Hand-rolled Parquet reader

Replacing a third-party library with 884 lines of C.

Setup

12 Parquet benchmarks: full scan, filtered scan, aggregation, joins, subqueries, and analytical queries on files ranging from 50K to 200K rows. Measured as wall-clock latency through the PostgreSQL wire protocol.

CREATE FOREIGN TABLE events
    OPTIONS (filename '/data/events.parquet');

SELECT event_type, COUNT(*), SUM(amount)
FROM events
GROUP BY event_type;

Problem

The original Parquet support used Carquet, a third-party C library built via CMake. It worked, but introduced a heavy build dependency: a full CMake clone-and-build step, a large dependency tree, and a schema traversal API that did not map cleanly to mskql’s flat columnar storage. The build was slow, the code was opaque, and bugs in the Thrift decoding layer were impossible to diagnose without understanding the library’s internals.

Cause

Parquet files use Apache Thrift compact protocol for metadata serialization. The format is simple—zigzag varints, field deltas, nested structs—but the field IDs must match the official Parquet Thrift IDL exactly. Any off-by-one in field mapping silently corrupts the metadata parse, producing wrong column counts, wrong offsets, or garbage compression codecs.

Fix

Replaced Carquet with pq_reader.c (884 lines): a minimal read-only Parquet decoder that handles the full read path from file open to materialized columns.

Thrift compact protocol — hand-written decoder for varints, zigzag encoding, field headers, nested structs, and skip-ahead for unknown fields. Field IDs match the official Parquet Thrift IDL: SchemaElement (10 fields), ColumnMetaData (11 fields), RowGroup (3 fields), PageHeader (8 fields).
Page decoding — supports PLAIN encoding for all types (BOOLEAN, INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY), RLE/bit-packed encoding for definition levels, and DELTA_BINARY_PACKED for integer columns.
Compression — zstd and zlib decompression via system libraries. Uncompressed pages pass through directly.
Dictionary pages — PLAIN_DICTIONARY encoding with dictionary page support for string columns.

The build dependency shrank from a full CMake project to two linker flags: -lzstd -lz.

Five bugs found in the first version

The initial implementation had 5 bugs in Thrift field ID mapping that prevented it from opening any Parquet file. These were invisible in testing because the test runner swallowed setup errors (psql returns exit 0 on SQL errors). After fixing the test harness to detect setup failures:

Stop byte handling — byte 0x00 (Thrift struct stop) was treated as field delta=0, consuming extra bytes and corrupting the stream.
SchemaElement field IDs — fields 5–10 were off by one (field 5 mapped to converted_type instead of num_children).
ColumnMetaData field IDs — fields 4–11 were all shifted by one position relative to the Thrift IDL.
RowGroup num_rows — field 2 was used for num_rows but the spec says field 3 (field 2 is total_byte_size).
List/set/map type IDs — used 13/14/15 but Thrift compact uses 9/10/11.

Result

Zero external C dependencies beyond zstd and zlib. The Parquet reader compiles in under a second. All 12 Parquet benchmarks pass with the same or better performance. The code is fully auditable—every field ID, every encoding, every decompression call is visible in a single 884-line file.