Tins · Tables & catalogs
iceberg
Native Apache Iceberg. Read a table's metadata.json, pick a snapshot, decode its manifests, plan a scan and read the rows — then create, append, delete, overwrite and expire. No JVM, no Python, no Rust in the path.
plain pixi:
pixi init app -c https://conda.modular.com/max -c conda-forge && cd app
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml
pixi add "mojo-compiler==1.2.0"
# Without mojoshelf: the tin as a git dependency at its published commit.
# Transitive dependencies may be wrong: pixi takes the tin's own git deps as
# pinned in its pixi.toml, which may not match what other tins pin, and then
# fails with "duplicate records". mojoshelf picks revisions that agree.
pixi add iceberg-mojo --git https://github.com/magmalake/iceberg.mojo --rev ec4f4e07605b687a2ea9e2fda5dfd4a01aa6ba97No mojoshelf needed. If pixi reports duplicate records, the mojoshelf tab picks revisions that agree.
0.8.6 on mojoshelf
What it does
Works today
- Format v1–v3 reads: position and equality deletes, v3 deletion vectors, schema evolution, nested types
- Writes: create, fast-append, delete (copy-on-write and merge-on-read), overwrite, dynamic partition overwrite, expire_snapshots
- Schema evolution as a write: add optional columns, rename, make optional, widen types and union_by_name, committed through every catalog
- REST, filesystem and SQL catalogs — the SQL one on SQLite or PostgreSQL, PyIceberg's schema; local, HTTP, S3, GCS and Azure IO
Not yet
- Compaction — rewrite_manifests
- Encryption
- Predicates inside list and map elements
The spec's v4 drafts — relative paths and content_stats — are tracked but not implemented. What is proposed but unbuilt lives in the issue tracker. The last spec there, a PostgreSQL tin, shipped as postgres.mojo and put the SQL catalog on a server.
Correctness
Row-sets cell-exact against PyIceberg 0.11.1 and DuckDB 1.5.5 across 48 of 48 filter cases, in both directions, including deletes, deletion vectors and schema evolution.
Tables magmalake writes are read back cell-exact by both — and PyIceberg can append to a table magmalake created.
Building it found bugs upstream
Holding three implementations against each other surfaces disagreements, and not all of them turned out to be my fault. I cut each one below down to a standalone script or unit test with no magmalake code in it, then either reported it upstream or — where someone had got there first — matched it to the report that already existed.
- iceberg-rust apache/iceberg-rust#3118 reported An In predicate whose literals straddle a file's bounds is never pruned: the lower and upper bounds are each tested against the whole literal set instead of narrowing it, as Java and PyIceberg do.
- PyIceberg apache/iceberg-python#3690 reproduced v3 manifests and manifest lists are read with the v2 projection, so first_row_id, referenced_data_file, content_offset and content_size_in_bytes are dropped on the way in.
- PyIceberg apache/iceberg-python#3620 reproduced The mirror image on the write path: the manifest writer builds its record schema at DEFAULT_READ_VERSION, so those same v3 fields would serialise as null.
- PyIceberg apache/iceberg-python#3833 reproduced A null list<struct> or map<K, struct> is rebuilt as an empty one — on the write path too, so the null is gone at rest and no reader can recover it.
Also in Tables & catalogs