Real-time Wikipedia Edits Pipeline
Wikipedia publishes a live feed of every edit made anywhere in the world — hundreds per second, all day, every day. This project catches that firehose, cleans it, and turns it into tables and dashboards you can actually trust. Running on one laptop. For $0 a month.
In plain terms: imagine trying to catch rain from a storm with a bucket, but you're not allowed to miss a single drop, and if you trip, you have to figure out which drops you already caught so you don't catch them twice. That's this project. The hard part was never reading the feed — it's staying correct when the wifi hiccups, the same edit shows up twice, or a job dies halfway through writing.
Problem
Wikimedia's recentchange feed is public, noisy, and never stops. That's only useful if:
- a reconnect doesn't leave a hole in your data,
- a duplicate doesn't inflate your counts,
- junk payloads don't sneak into your dashboard,
- and a killed job can restart without you babysitting it at 2 AM.
The goal was an honest freshness target of two minutes — not sub-second latency that nobody actually needs. And keep the raw events around, so if I change my mind about how aggregates work, I can rebuild them from scratch.
Architecture
The ingest service reconnects gracefully using Last-Event-ID and writes raw events to storage. Spark then checks every record, throws the malformed ones into a DLQ (dead letter queue — the intern's bin of the data world), and appends the good ones to bronze Iceberg tables. From there it builds gold tables for Grafana. Prometheus watches freshness; MinIO and the Iceberg REST catalog keep the whole thing portable — it's Kafka and S3, just wearing local clothes.
Key decisions
- ADR-0001: local-first Docker Compose. The whole system must run without a cloud bill. My wallet agreed with this decision immediately.
- ADR-0002: Redpanda over Kafka. Redpanda speaks Kafka's API but runs as one lightweight binary — my laptop thanks me, since Spark and Grafana are already fighting over RAM. And because it's Kafka-compatible, moving to real Kafka/MSK later needs zero code changes.
- ADR-0003: Spark Structured Streaming over Flink. Flink is amazing if you need sub-second everything. I needed two-minute freshness. Spark's micro-batches clear that easily and pair perfectly with Iceberg. Choosing Flink here would've been me buying a race car to drive to the grocery store.
- ADR-0004: Iceberg on MinIO. Iceberg gives atomic snapshots, schema evolution, and time travel — like git, but for tables. MinIO speaks the S3 API, so the cloud path is a config change, not a rewrite.
- ADR-0005: stage-by-stage delivery semantics. "Exactly-once" is a marketing term until you prove it per hop. Each boundary here has a stated contract, and the gold tables are effectively-once with the residual risk written down honestly.
- ADR-0006: two-minute event-time watermark. Events that show up late don't get silently dropped — they land in a visible late-arrivals table. Late data isn't a bug, it's a fact of life. You just need to count it.
Delivery semantics
| Hop | Delivery contract | Mechanism |
|---|---|---|
| Wikimedia SSE → ingest | At-least-once | Last-Event-ID resumes the feed after reconnects; overlap is possible, gaps are avoided within source retention. |
| Ingest → Redpanda | Idempotent producer, acks=all | Producer acknowledgements and idempotence prevent duplicates inside a producer session; a restart can still replay. |
| Redpanda → Spark → Iceberg | Effectively-once | Spark checkpoints offsets and Iceberg's native sink deduplicates a replayed (queryId, epochId) snapshot commit. |
| Bronze → silver/gold | Watermark-bounded deduplication | A two-minute event-time watermark deduplicates on Wikimedia meta_id, covering SSE overlap and producer-restart duplicates while keeping state bounded. |
Failure modes and recovery
| Failure | What happens | Recovery |
|---|---|---|
| Ingest process crash | The stream resumes from Last-Event-ID; an overlap may be re-emitted. | Restart automatically; downstream meta_id dedup removes the overlap. |
| Redpanda unavailable | Ingest retries with backoff and retains its source position. | Broker recovery lets the producer catch up without discarding source events. |
| Spark killed mid-batch | The checkpoint is not advanced and an incomplete Iceberg commit is not visible. | Restart Spark; it replays the batch from checkpoint and commits one atomic snapshot. |
| Upstream schema drift | The DQ gate rejects malformed required fields instead of merely warning. | Inspect the DLQ, evolve the Iceberg schema, then replay retained raw events. |
| MinIO or catalog outage | Spark fails the batch before checkpoint progress. | Restore the dependency and restart; Redpanda retention and checkpointing prevent loss. |
| Late event beyond watermark | Bronze still records it, but the settled window excludes it and increments a late-arrival counter. | Monitor the counter; rebuild from bronze if the watermark policy changes. |
Chaos demo: kill a streaming batch, then prove the outcome
Below: I literally kill Spark mid-batch (SIGKILL — no goodbye, no cleanup, very rude). Bronze grows from 70,233 to 74,015 rows across the kill; the duplicate check reports total = 74,015 and distinct = 74,015. Freshness recovers in under 120 seconds. The point isn't "processes never die" — of course they die. The point is that when they do, recovery produces a result you can check with your own eyes.
Verified numbers. Steady-state end-to-end freshness holds sub-minute against the 120s SLO with 3–17s batch durations on a 60s trigger (~4× headroom). An earlier 15s-trigger configuration measured 12.4s fresh — then degraded to ~8.5 minutes after three hours as streaming state grew; the fix (longer trigger + RocksDB state store) is documented in the repo and the blog post. Compaction reduced bronze from 1,334 → 72 files and 14.3MB → 4.9MB. In the recorded chaos demo, rows grew 11,937 → 18,032 across a SIGKILL with zero duplicates (20,418 = 20,418) and freshness recovered to 28s. A cold start is reproducible with
make up && make spark-up && make verify-spark.
Cost and portability
This whole thing costs $0/month by design — the most my wallet will ever approve of. Redpanda speaks Kafka's API and MinIO speaks the S3 API, so moving to AWS (MSK, S3, a managed catalog) is a configuration change, not an app rewrite. And to be clear: this runs on a laptop, and I present laptop numbers as laptop numbers — not pretend they prove production scale.
Try the data
The exported Parquet files run entirely in your browser with DuckDB-WASM — no server, no setup. Start with a preset or write your own SQL and go poking around.