# Docs

> How to use parquet, a magmalake tin.

[magmalake](https://magmalake.dev/) [parquet](https://magmalake.dev/s/parquet)

## What it’s for

parquet reads and writes Apache Parquet natively: footer, page headers, levels and values are all decoded in Mojo, and a read comes back as Arrow arrays from [arrow-mlake](https://magmalake.dev/s/arrow-mlake). It is the data-file layer under [iceberg](https://magmalake.dev/s/iceberg), which uses its field-id projection, statistics, page index and bloom filters to plan and prune scans. This tin handles `UNCOMPRESSED`, `SNAPPY` and `GZIP` with no FFI. For `ZSTD`, `BROTLI` and `LZ4`, add [parquet-full](https://magmalake.dev/s/parquet-full) as well: it plugs into the same reader and writer.

## Install

A new project needs Modular’s channel, pixi’s build preview and the compiler before the tin:

mojoshelf:

```bash
pixi init app -c https://conda.modular.com/max -c conda-forge && cd app
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml
pixi add "mojo-compiler==1.2.0"
pixi shelf add parquet-mojo    # brings arrow-mlake, thrift, hashes, snappy, deflate, threads
```

`pixi shelf` comes with mojoshelf: [install it first](https://mojoshelf.org/getting-started).

plain pixi:

```bash
pixi init app -c https://conda.modular.com/max -c conda-forge && cd app
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml
pixi add "mojo-compiler==1.2.0"
# Without mojoshelf: the tin as a git dependency at its published commit.
# Transitive dependencies may be wrong: pixi takes the tin's own git deps as
# pinned in its pixi.toml, which may not match what other tins pin, and then
# fails with "duplicate records". mojoshelf picks revisions that agree.
pixi add parquet-mojo --git https://github.com/magmalake/parquet.mojo --rev c15f51aee06ccf73102810e7de143745abba6c90
```

No mojoshelf needed. If pixi reports duplicate records, the mojoshelf tab picks revisions that agree.

Most Parquet in the wild, Iceberg’s included, is ZSTD. Unless you control the files you read, add `parquet-full-mojo` instead; see [parquet-full](https://magmalake.dev/s/parquet-full).

## Quick start

Open a file, prune row groups by their statistics, read what’s left, and write it back out. `prune.parquet` is in the repository’s `tests/fixtures`:

```
from std.sys import argv

from parquet import (
    OP_GE,
    OP_LT,
    ParquetReader,
    ParquetWriter,
    Predicate,
    ScalarValue,
    WriterOptions,
)

def main() raises:
    var path = String(argv()[1]) if len(argv()) > 1 else "prune.parquet"
    var r = ParquetReader.open(path)
    print(r.num_rows(), "rows in", r.num_row_groups(), "row groups")

    # Keep only the row groups whose statistics allow 250 <= k < 260.
    r.select_columns(["k"])
    var preds: List[Predicate] = [
        Predicate("k", OP_GE, ScalarValue.of_int(250)),
        Predicate("k", OP_LT, ScalarValue.of_int(260)),
    ]
    print(r.prune_row_groups(preds), "row groups left")

    var t = r.read_table()
    var k = t.column_i64(0)                 # (values, validity)
    var n = 0
    for i in range(len(k[0])):
        if k[1][i] and k[0][i] >= 250 and k[0][i] < 260:
            n += 1
    print(t.num_rows, "rows read,", n, "match")

    # Write what was read back out (SNAPPY by default) and read that.
    var w = ParquetWriter(WriterOptions())
    for b in t.batches:
        w.write_batch(b.arena, b.roots)
    var again = ParquetReader(w^.finish())
    print("rewritten:", again.num_rows(), "rows,", again.num_row_groups(), "row group")
```

## API

Everything below is exported from `parquet`, along with the arrow-mlake types it returns (`RecordBatch`, `ArrayData`, `ArrayArena`, `ArrowType`, `export_c`, `import_c`).

**`ParquetReader[Codecs: CodecSet = DefaultCodecs]`**: one file, projected and read in batches. Pass `AllCodecs` from [parquet-full](https://magmalake.dev/s/parquet-full) as `Codecs` for the FFI codecs.

- `ParquetReader.open(path: StringSlice) raises -> Self`: reads the whole file into memory.
- `ParquetReader.open_projected(path: StringSlice, columns: List[String]) raises -> Self`: reads the footer, then only the column chunks those columns need.
- `ParquetReader(var data: List[UInt8])` and `ParquetReader.from_span(data: Span[UInt8, _])`: a file already in memory, owned or borrowed. A borrowed span must outlive the reader.
- `num_rows()`, `num_row_groups()`, `created_by()`, `key_value_metadata()`, `split_offsets()`: the file’s metadata; `schema` is its `ParquetSchema`.
- `statistics(self, rg: Int, leaf: Int) raises -> TypedStats`: a column chunk’s min, max, null and distinct counts, decoded to the column’s type.
- `select_columns(mut self, names: List[String]) raises`, `select_field_ids(mut self, ids: List[Int32]) raises`, `select_field_ids_deep(mut self, ids: List[Int32]) raises`: project by top-level name, by Parquet field id, or into nested fields by id.
- `select_row_groups(mut self, var groups: List[Int]) raises`: read only these row groups.
- `prune_row_groups(mut self, predicates: List[Predicate]) raises -> Int`: drops the row groups whose statistics prove no row can match all the predicates; returns how many are left.
- `prune_pages(mut self, predicates: List[Predicate]) raises -> Int`: the same with the page index, so pages outside the matching rows are never decompressed; returns the rows left. `pages_read()` counts the pages that were decoded.
- `has_next(mut self) -> Bool`, `read_batch(mut self) raises -> RecordBatch`, `rewind(mut self)`: stream up to `batch_size` rows (default 65536) at a time, one row group in memory.
- `read_table(mut self) raises -> Table`: every selected row group at once.
- Fields you can set: `batch_size`, `verify_crc` (default true), `num_workers` (default 1; 0 means one per core), `max_batch_value_bytes`.

**`Predicate(column: String, op: Int, value: ScalarValue)`**: `column op value`, with `op` one of `OP_EQ`, `OP_NE`, `OP_LT`, `OP_LE`, `OP_GT`, `OP_GE`. Build the value with `ScalarValue.of_int`, `of_uint`, `of_float`, `of_bool`, `of_string` or `of_bytes`.

**`Table`**: `batches: List[RecordBatch]` and `num_rows`, with `num_columns()`, `name(i)`, `type(i)` and the typed accessors `column_i64`, `column_f64`, `column_bool` and `column_str`, each returning `(values, validity)` across all batches.

**`ParquetWriter[Codecs: CodecSet = DefaultCodecs](var options: WriterOptions)`**: writes Arrow arrays as Parquet.

- `write_batch(mut self, arena: ArrayArena, roots: List[Int]) raises`: appends a batch, in the shape `RecordBatch` has.
- `add_metadata(mut self, key: StringSlice, value: StringSlice)`: a key/value pair for the footer.
- `finish(deinit self) raises -> List[UInt8]`: the file’s bytes.

**`WriterOptions()`**: `codec` (a thrift `CompressionCodec` value, default `SNAPPY`), `row_group_size`, `data_page_size`, `use_dictionary`, `write_statistics`, `write_page_index` and `created_by`.

Also exported: `build_schema` and `ParquetSchema` (with `field_by_name`, `field_by_id`, `leaf_by_path`), `decode_stats`, `CodecSet` and `DefaultCodecs`. Bloom filters are `read_bloom_filter(file, column_metadata) -> Optional[BloomFilter]` and `BloomFilter.might_contain(ScalarValue)`.

## Errors and limits

- Errors are `Error`s whose message starts with `parquet:` or the module, such as `parquet.page:` or `parquet.writer:`, and names what was wrong: `parquet: no column named 'x'`, `parquet: row group 12 does not exist`, `parquet.page: CRC32 mismatch on the page at ...`. Truncated or corrupt files raise; they don’t crash or read out of bounds.
- A page in a codec this reader’s `CodecSet` lacks raises `parquet: page codec ZSTD is not available — use parquet_full.AllCodecs ...` when the read reaches it. Project that column away and the rest of the file still reads.
- With `num_workers` above 1, a failing column chunk raises the same error as it would sequentially, in row-group then column order.
- One row with more than 2 GiB of `BYTE_ARRAY` data in a column raises: Arrow `large_binary` isn’t produced. Larger column chunks read, cut into smaller batches.
- Not supported: Parquet encryption, and pyarrow’s `ARROW:schema` metadata, so a `large_string` column reads as `string`.
- The writer writes v1 pages with `PLAIN` or `RLE_DICTIONARY` values, statistics, field ids and a page index. It doesn’t write v2 pages, the delta or byte-stream-split encodings, page CRCs, `INT96` or bloom filters.

## Tests

pyarrow is the oracle: 24 fixtures written by pyarrow and nine real Iceberg data files are read and compared with pyarrow’s dump of every value, at several batch sizes and worker counts. `pixi run test` runs that suite and the writer’s round trips; `pixi run verify-written` has pyarrow read back everything the writer writes, and `pixi run verify-c` imports the C Data Interface export into pyarrow. CI runs them on Linux and macOS, on nightly and Mojo 1.2.0, with the sibling tins checked out beside the repository. The codec tests are [parquet-full](https://magmalake.dev/s/parquet-full)’s.

The full reference is [the README on GitHub](https://github.com/magmalake/parquet.mojo#readme).

[A magmalake project](https://magmalake.dev/)

Apache 2.0 · © 2026 Marius Seritan