July 30, 2026
Offset under fire
Paging Through a Parquet File in DuckDB: File_row_number or Offset?
DuckDB paging drama: the ‘easy’ method worked, but commenters still came for it
TLDR: DuckDB found a faster way to page through a huge file, but the comments were obsessed with a bigger warning: the usual skip-ahead method can give messy results if the order isn’t guaranteed. In other words, speed mattered, but reliability sparked the real debate.
A seemingly nerdy debate about how to slice up a giant data file turned into a classic internet pile-on: the author expected the obvious method to be a disaster, only to discover it wasn’t the real villain at all. The setup is simple enough for non-database people: if you have a file so huge you can’t send it all back in one go, you break it into pages. The surprise? In tests, using the file’s built-in row positions was still 2.53 times faster than the usual “skip ahead” approach across the full file, but the bigger lesson was that the file’s internal layout mattered even more than raw speed.
That’s where the comments got spicy. One of the strongest reactions was basically, “Hold on, you can’t just page through data with skip-and-limit unless you lock down the order.” Commenter matharmin came in with a stern, almost teacherly warning that this should be “common knowledge,” which is exactly the kind of line that makes a technical thread instantly smell like drama. It’s less a slap-fight and more that very specific internet energy where someone says, politely but firmly, “You forgot the rule everyone should know.”
Then came the helpful side-quest: RobinL dropped in to remind everyone there’s also a built-in row ID feature for DuckDB tables, which felt like the comments section equivalent of someone kicking open the door yelling, “Bonus tip!” The overall mood? Equal parts cautionary lecture, useful trivia, and quiet amazement that the supposedly bad method wasn’t actually catastrophically bad after all.
Key Points
- •The article addresses stateless pagination of a large Parquet file in DuckDB for API delivery under response-size constraints.
- •It compares `LIMIT/OFFSET` pagination with filtering on `file_row_number` from `read_parquet(..., file_row_number => true)`.
- •In tests on a 20-million-row file with 163 row groups, the row-range approach completed 2.53x faster across the full file and won in 37 of 37 runs.
- •The main reason for the speedup is that DuckDB can use Parquet footer metadata to skip entire row groups outside the requested row range without decompressing them.
- •The article says file layout is decisive: with a single large row group, the performance advantage drops sharply and both paging approaches become much slower.