August 1, 2026
Lake house or data madhouse?
Indexing the Data Lake for Online Point Queries
Spotify wants lightning-fast lookups from giant storage, and commenters are not calm
TLDR: Spotify says it can answer user-specific questions quickly straight from massive cloud storage by adding a smart index instead of moving everything into a separate fast database. Commenters are split between “pretty clever” and “why not use a normal database,” with extra skepticism about the AI sales pitch.
Spotify just dropped a big idea: instead of copying huge amounts of user data into expensive fast databases, it wants to search directly inside its gigantic cloud storage and still answer one-person questions quickly. The pitch is simple enough for non-engineers: if someone asks, “What was I listening to last summer?”, Spotify wants that answer fast without dragging half the internet through a slow reporting system first. Their trick is adding an outside index — basically a super-organized table of contents — so they can jump straight to the right spot in massive files.
But the real action is in the comments, where admiration instantly collided with side-eye. One camp called it smart and practical, especially because it keeps the same files for analytics, machine learning, and online features instead of making extra copies. That got a respectful nod from readers who love anything that saves storage bills. Then came the panic-sweat faction: if Spotify has truly enormous amounts of data, building that index the first time sounds like a wallet-melting backfill nightmare. People could practically hear the finance team screaming.
And then the classic internet tech fight broke out: why not just use a normal database? One commenter basically asked why Spotify is doing architectural gymnastics instead of using Postgres or MySQL like a civilized company. Another wasn’t sold on the article’s trendy “AI agent” framing at all, treating it like marketing glitter sprinkled on a real infrastructure problem. In other words: cool idea, suspicious hype, and a deliciously nerdy “are we overcomplicating this?” showdown.
Key Points
- •Spotify says online services and AI agents need low-latency point queries over datasets too large to keep entirely in key-value stores.
- •The article argues that cloud object storage latency has improved, while distributed SQL engines such as Trino and BigQuery still add too much overhead for interactive single-key lookups.
- •Spotify presents Random Access Parquet (RAP) as an external-index approach that maps keys to Parquet file locations and uses precise ranged reads.
- •The article shows how partitioning by key and Bloom filters can reduce the number of candidate Parquet files dramatically before reading any file contents.
- •The remaining bottleneck described is the chain of dependent reads required to locate rows within Parquet files, including reading footers, row-group metadata, and page indexes.