Design Change data capture: learn it by breaking it, in Tailspin

Change data capture. The copies fall behind the source. Copy chosen tables from the shop's database to the warehouse, then keep the copy up to date. Capture every insert, update and delete, in order for each row. Search and the cache within 10 seconds of a change; the warehouse within minutes. Never slow the shop: capture may add at most 5% to its database.

Its design on a normal day: p99 22ms, 0% of requests failing, 72/s answered.

What Tailspin shows: Reading the database log costs the source almost nothing; reading its tables does not; Each reader of the change log falls behind on its own, without slowing the others; Partitions, not readers, cap how much is worked on at once; The log's retention as the safety net when a reader slows down.

Following Design change data capture, Engineering Atlas (the free part).