Writing

• AI agents for a data team, without handing them the keys to production

  Case study · 7 min · What I built in 2026 for a 12-engineer billing data team, what the agentic code reviewer actually caught, and which guardrails mattered.

• Idle CPUs, 60-second queries: a global lock in ClickHouse 23.8

  Case study · 6 min · How a dashboard widget that needed two numbers took a ClickHouse cluster down for 48 minutes, and why the fix was a cheaper query rather than more hardware.

• 665 GiB of dictionaries: when ClickHouse dictionaries stop scaling

  Case study · 3 min · Nine hashed dictionaries quietly grew to two thirds of a terabyte per node, and every reload tried to build a second copy. How we found that only five views used them, and replaced them with joins that read 17 granules out of 282,102.

• Airflow tasks stuck in "queued" forever after a twelve-second database blip

  Case study · 3 min · A short network flap between two datacenters left a task in "queued" for two and a half hours, and its DAG with it. The cause is a lost executor event: SELECT … FOR UPDATE SKIP LOCKED skips exactly the row it needs. Why restarting the scheduler is the wrong fix.

• Expired certificates broke production twice, and nothing turned red

  Case study · 3 min · Once as a load-balancer black hole that swallowed 23 of 24 connections for fifteen hours, once as a silent 30-hour replica lag behind a database that kept answering. Both times the metrics existed and the alerts did not.

• LowCardinality in ClickHouse: I measured the rule of thumb, and it was wrong both ways

  Note · 2 min · "LowCardinality helps between a thousand and a hundred thousand distinct values, and hurts otherwise." A code review quoted this rule, so I measured it on ClickHouse 23.8. LowCardinality wins at any small number of values, even two, and starts losing badly at about twenty thousand per part.

• Partitioning by modulo in ClickHouse: the DDL I wrote that made queries 13.7 times slower

  Note · 2 min · PARTITION BY supplier_id % 100 sounds like it spreads the load evenly. In practice it destroyed primary-key pruning: a lookup read 64% of the table instead of 3%. Here are the measurements, and why it happens.