Event Sourcing: Why Your Database Should Be an Append-Only Log

Most applications store the current state of things. A user's profile has a name, an email, and a phone number. When they change their email, you UPDATE the row. The old email is gone. You have no idea what it was, when it changed, or why. This is how 99% of software works, and it is fundamentally wrong for many use cases.

What Is Event Sourcing?

Event sourcing flips the model. Instead of storing the current state, you store every event that ever happened. A user signed up (with these details). They changed their email (from X to Y). They updated their phone number. They changed their email again. The current state is derived by replaying all events in order.

Think of it like a bank account. Banks do not store your balance and update it. They store every transaction: deposits, withdrawals, transfers, fees. Your balance is the sum of all transactions. If there is a dispute, they can trace exactly what happened, when, and why. That is event sourcing.

The Append-Only Log

At the core of event sourcing is an append-only log, sometimes called a Write-Ahead Log (WAL). Events are written sequentially and never modified or deleted. Each event is immutable — once written, it is there forever.

This has profound implications. There is no UPDATE or DELETE in the traditional sense. You can never accidentally lose data by overwriting it. The log is a complete, auditable history of everything that has ever happened in the system. If you need to know what the state was at any point in time, you replay events up to that moment.

How LuperIQ Uses Event Sourcing

LuperIQ, the CMS platform I am building, uses event sourcing as its primary storage mechanism. There is no PostgreSQL, no MySQL, no MongoDB. The entire system runs on an append-only WAL with blake3 checksums and Merkle chaining for tamper detection.

Every action in the CMS creates an event: page created, page updated, theme changed, user logged in, blog post published, product added. These events are stored in a binary-encoded WAL file using bincode serialization. The current state is projected from events into memory on startup, with periodic snapshots to speed up recovery.

Each event has an aggregate type (like 'Content' or 'ThemeStudio:Profile') and an aggregate ID. Events for the same aggregate are stored together, making it efficient to replay the history of a specific entity.

Advantages Over Traditional Databases

Complete audit trail: Every change is recorded with a timestamp. You can answer questions like 'What did the homepage look like on March 5th?' by replaying events to that point. In a traditional database, that information is gone the moment you update the row.

Debugging and support: When a customer reports an issue, you can replay their exact sequence of events. No guessing, no trying to reproduce. The events tell you exactly what happened.

Temporal queries: Want to know how many products were in the catalog on January 15th? Replay to January 15th. Want to compare the current theme to what it was last week? Replay to last week. This is trivial with event sourcing and nearly impossible with a traditional database.

Performance: Writing to an append-only log is extremely fast. There is no row locking, no index maintenance during writes, no write amplification from B-tree updates. Sequential writes are the fastest operation on both SSDs and spinning disks.

Simplicity: No ORM, no schema migrations, no JOIN queries. The data model is just events. Adding a new field to an event type is forward-compatible — old events still deserialize correctly with defaults for the new field.

The Challenges

Storage growth: An append-only log grows forever. In practice, this is manageable. Modern storage is cheap, and periodic snapshots let you compact the event history. LuperIQ uses binary encoding (bincode) which is extremely compact compared to JSON or relational storage.

Query complexity: Ad-hoc queries on event-sourced data require replaying events or maintaining projections (pre-computed views). If you need to run 'SELECT * FROM users WHERE city = 'Austin'', you cannot do that directly against the event log. You need a projection that indexes users by city.

Learning curve: Event sourcing requires a different mental model. Developers are used to thinking in terms of 'get the thing, change the thing, save the thing.' Event sourcing requires thinking in terms of 'what happened?' which takes practice.

Eventual consistency: If you use projections for reads and the event log for writes, there is a window where a read might not reflect the latest write. For many applications, this is acceptable. For others (like financial transactions), you need synchronous projection updates.

When to Use Event Sourcing

Event sourcing is a great fit when: you need a complete audit trail (financial systems, healthcare, legal compliance), you need temporal queries (what was the state at time X?), you want to decouple writes from reads (CQRS pattern), or your domain naturally expresses itself as events (order placed, payment received, item shipped).

Event sourcing is probably overkill when: you have simple CRUD requirements with no audit needs, your data model is mostly relational joins, you need complex ad-hoc queries across many entity types, or you are building a prototype and need to move fast.

Real-World Examples

Banking and finance have used event sourcing (under the name 'ledger') for centuries. Every transaction is an immutable record. The balance is derived. This is not a new idea.

Apache Kafka is essentially a distributed append-only log. Companies like LinkedIn, Uber, and Netflix use Kafka as the backbone of their event-driven architectures. If your data flows through Kafka, you are already using event sourcing concepts.

Git is event sourcing for code. Every commit is an immutable event. The current state of your repository is the projection of all commits. You can check out any point in history. You never lose data (unless you force-push, which is the event sourcing equivalent of deleting your WAL).

The Bottom Line

Event sourcing is not the right choice for every application, but it is the right choice for more applications than most developers realize. The combination of complete auditability, temporal queries, and write performance makes it a powerful pattern. If your application deals with money, compliance, content management, or any domain where knowing what happened is as important as knowing the current state, event sourcing deserves serious consideration.