Tins · File formats
arrow-mlake
The Apache Arrow memory layout, the C Data Interface and IPC streams in pure Mojo, with no FFI: its one dependency is memory-region-mojo. A stopgap until marrow, the community Arrow for Mojo, builds on Mojo 1.x.
Install
plain pixi:
pixi init app -c https://conda.modular.com/max -c conda-forge && cd app
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml
pixi add "mojo-compiler==1.2.0"
# Without mojoshelf: the tin as a git dependency at its published commit.
# Transitive dependencies may be wrong: pixi takes the tin's own git deps as
# pinned in its pixi.toml, which may not match what other tins pin, and then
# fails with "duplicate records". mojoshelf picks revisions that agree.
pixi add arrow-mlake-mojo --git https://github.com/magmalake/arrow-mlake.mojo --rev 178aae6e0da0dbe105f9cb02632a51cfd35b9ddaNo mojoshelf needed. If pixi reports duplicate records, the mojoshelf tab picks revisions that agree.
Version
0.2.1 on mojoshelf
What it does
Works today
- ArrayData and ArrayArena: flat, nested and extension types
- Export and import over the C Data Interface, release callbacks and all
- IPC stream writer and reader, with a bounds-checked FlatBuffers reader
- What parquet hands its batches back in
Correctness
Checked against pyarrow imports what it exports over C, and reads and writes its IPC streams both ways
Also in File formats
- parquet read · write · every encoding · Arrow out
- parquet-full parquet · plus ZSTD, BROTLI and LZ4
- avro object container files, both ways
- roaring 32/64-bit · deletion vectors
- Every tin