This draws material from the following excellent article: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Based on Jay Kreps’ seminal article, the log is not just a file format for errors. It is the fundamental backbone of distributed systems.
The Theory of the Log in Distributed Systems
In distributed systems theory, the log is an append-only, totally-ordered sequence of records. While simple, it serves as the primary mechanism for managing the two hardest problems in the field: ordering and consistency.
1. The State Machine Replication (SMR) Principle
The academic core of the log is the State Machine Replication principle. It posits that:
- The Law: If two identical, deterministic processes start in the same state and receive the same inputs in the same order, they will produce the same output and end in the same state.
- The Log’s Role: The log acts as the "input sequencer." By forcing all incoming requests into a single, ordered log, you effectively "squeeze out" the non-determinism of a distributed network.
2. Physical vs. Logical Logging
The theory distinguishes between what you put in the log:
- Logical Logging: Recording the operations themselves (e.g., "Add 5 to account X"). This is compact but requires deterministic execution.
- Physical Logging: Recording the resulting state change (e.g., "Account X is now 105"). This is more robust against execution differences but requires more bandwidth.
3. Table-Log Duality
A profound theoretical takeaway is that tables and logs are duals.
- A Table represents a snapshot of data at a point in time (State).
- A Log represents the transformation over time (Events).
If you have a log, you can reconstruct the table at any point in history by "replaying" the events. If you have a table and record every change, you have a log.
4. The Log as a Consensus Abstraction
While researchers often focus on consensus algorithms like Paxos or Raft, the log is the practical application of those algorithms. Consensus is about agreeing on a single value; a distributed log is simply a sequence of consensus decisions—one for each "slot" in the log.
Modern Platforms Relying on the Log
Today’s most powerful data infrastructures are essentially "logs-as-a-service" or systems built entirely on log-centric principles.
1. Apache Kafka
Kafka is the direct realization of the ideas in the article. It treats the log as the primary architectural element rather than an implementation detail.
- How it uses the log: It provides a distributed, partitioned, and replicated commit log. Producers append events to the end of the log, and many different consumers (databases, search indexes, or microservices) read from the log at their own pace to build their own "view" of the world.
2. Apache Flink
Flink is a stream processing framework that treats data as an infinite log of events.
- How it uses the log: Flink uses the log for fault tolerance. It takes periodic "savepoints" of its state, but between those points, it relies on the replayability of the input log to recover and ensure "exactly-once" processing semantics.
3. CockroachDB / TiDB
These are "NewSQL" distributed databases that provide global scalability with ACID guarantees.
- How it uses the log: They use the Raft consensus algorithm to maintain a distributed log across different geographic nodes. Every write must be appended to this log and acknowledged by a majority of nodes before it is considered "committed," ensuring that the database remains consistent even if a data center goes offline.
4. Elasticsearch
While primarily a search engine, Elasticsearch's reliability depends on a log.
- How it uses the log: It uses a Translog (Transaction Log). Because writing to the heavy search index (Lucene) is expensive, Elasticsearch first writes the operation to an append-only log. This ensures that if a node crashes, it can recover any data that was acknowledged but not yet fully indexed.
5. Git (Version Control)
Though we think of it as a tool for code, Git is a pure implementation of a distributed log.
- How it uses the log: Every "commit" is a record in an immutable log. The "state" of your code at any branch is just the result of replaying the log of commits from the beginning. Merging and rebasing are simply operations that manipulate the order or structure of this log.