Change Data Capture (CDC)
Log-Based CDC Explained
Bane Tech Engineering · 6 min read
Log-based change data capture reads a database's transaction log — the record the database itself keeps of every committed change — and converts those entries into a stream of change events. Because it reads the log rather than querying tables, it captures every insert, update and delete with minimal impact on the source system.
Where the changes come from
Relational databases write each change to a log before applying it, for durability and recovery. Different engines name it differently: the write-ahead log (WAL) in PostgreSQL, the binary log in MySQL, the redo log in Oracle and the transaction log in SQL Server. A log-based CDC process reads these entries in commit order.
Comparison with other CDC methods
| Method | How it works | Trade-offs |
|---|---|---|
| Query-based | Periodically selects rows changed since a timestamp | Misses deletes and intermediate changes; adds query load |
| Trigger-based | Database triggers write changes to shadow tables | Captures everything but adds overhead to every write |
| Log-based | Reads the transaction log | Low overhead, complete and ordered; requires log access and configuration |
Advantages
- Captures all changes, including deletes, in commit order.
- No changes to application code or schema.
- Low latency between commit and event.
- Low additional load on the source database.
Operational considerations
- Log retention must exceed the longest expected consumer downtime.
- An initial snapshot is usually needed before streaming begins.
- Schema changes in the source must be detected and propagated.
- Required permissions and log settings differ per database engine.
- Replication lag should be monitored continuously.
For background on CDC in general, see What Is Change Data Capture (CDC)?.