December 7, 2025
Stream dreams vs Kafka memes
The end of the road for Kafka-delta-ingest
Scribd pulls the plug, Rust fans cheer, Kafka gets side-eye
TLDR: Scribd retired kafka‑delta‑ingest after building an even cheaper data pipeline, keeping delta‑rs alive as a test bed. Comments cheered the Rust-driven savings, slammed Kafka’s cost and weight, and debated alternatives like WarpStream—turning a shutdown into a victory lap and a cost-cutting slugfest
Scribd just sunset its five‑year workhorse, kafka‑delta‑ingest, after slashing ingestion costs 95% and then outdoing itself with an even cheaper “oxbow” setup. Translation: their data pipeline got leaner, meaner, and cheaper. The community reaction? A mix of nostalgia, victory laps for Rust, and a whole lot of Kafka skepticism. One early contributor, nevi‑me, popped in with feel‑good vibes (“seeing my name mentioned cheered me up”) and a blunt verdict: Kafka is too heavy for this job. Others nodded along to the post’s spicy thesis: if you’re only running Kafka to stuff data into storage, there are cheaper paths. Cue the drama.
The crowd split into camps: the “Rust saved us” crew calling the project a win for kickstarting delta‑rs, and the “ditch Kafka, keep the savings” crowd. rmnclmnt summed up the mood: even if the tool is retired, the push behind delta‑rs made it worth it. Then redwood tossed a grenade: “What about WarpStream?”—a cloud service promising lower Kafka costs—suggesting the debate isn’t over, just moving to the checkout aisle. Meanwhile, jokes flew about 2020 being “otherwise totally chill,” and folks memed the old process as a “sidecar” you only keep if Kafka isn’t the elephant in the room. The vibe: pour one out, celebrate the Rust wins, and keep your receipts on Kafka
Key Points
- •Scribd decommissioned kafka-delta-ingest after five years in production and removed it from its infrastructure.
- •kafka-delta-ingest inspired the creation of delta-rs and initially reduced streaming ingestion costs by 95% using a Rust-based approach.
- •A newer pipeline using the oxbow suite and medallion architecture reduced Delta Lake ingestion to under 10% of total platform costs, prompting the shutdown.
- •The tool’s reliance on Apache Kafka reduced its value at Scribd as other Kafka use cases declined; it remains useful where Kafka is already deployed.
- •Maintainers will keep kafka-delta-ingest updated for delta-rs testing but have no plans to expand the project further.