Tins · File formats
parquet-full
Every Parquet codec the spec defines: parquet-mojo, plus ZSTD, BROTLI and LZ4 through libzstd, libbrotli and liblz4. It lives in the full/ directory of the parquet repository and depends on parquet-mojo, so adding it gives you both, and the import is unchanged.
Install
plain pixi:
pixi init app -c https://conda.modular.com/max -c conda-forge && cd app
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml
pixi add "mojo-compiler==1.2.0"
# Without mojoshelf: the tin as a git dependency at its published commit.
# Transitive dependencies may be wrong: pixi takes the tin's own git deps as
# pinned in its pixi.toml, which may not match what other tins pin, and then
# fails with "duplicate records". mojoshelf picks revisions that agree.
pixi add parquet-full-mojo --git https://github.com/magmalake/parquet.mojo --rev c15f51aee06ccf73102810e7de143745abba6c90 --subdirectory fullNo mojoshelf needed. If pixi reports duplicate records, the mojoshelf tab picks revisions that agree.
Version
0.1.8 on mojoshelf
What it does
Works today
- Take it unless you control the files you read: ZSTD is what most Parquet in the wild uses, Iceberg included
- A reader without it that meets one of these codecs raises an error naming it
- A consumer that never needs ZSTD never builds the three C libraries
Correctness
Checked against A write and read round trip through each of the three codecs, and the ZSTD-compressed Iceberg fixtures
Also in File formats
- parquet read · write · every encoding · Arrow out
- avro object container files, both ways
- arrow-mlake Arrow layout · C Data Interface · IPC
- roaring 32/64-bit · deletion vectors
- Every tin