Docs

Using Orca

Orca turns a Mojo file of steps into a pipeline binary with a command line, and a scheduler runs those binaries on this machine, on Kubernetes or on EC2, recording every run in Postgres. This page installs it; the pages below cover the rest. The plan is in the repository's DESIGN.md.

The docs

• Writing a pipeline: Steps and pipelines, partitions by date or by name, schedules, Iceberg tables and datasets, sensors, Dista inside steps, and building the binaries.
• The command line: The commands you type, on a pipeline binary and on orca, the ones orca runs for you, and how the same binary acts locally or on AWS.
• Scheduling and running: The scheduler, runs on Kubernetes and EC2, placement and remote workers, fills, the watchdog and alerts, runs in Postgres, logs and machine metrics, lineage and rebuilds, the environment, and what happens when things fail.
• The UI: The pages and their JSON, the actions, logins, timelines and links to the code.
• Debugging failures: Where a failed run's errors and logs are, adding context to errors, symbolized crash dumps, reproducing a partition locally, and resuming it.
• Remote workers: Running some pipelines on a second machine: a worker token, a placement, a worker container that needs only outbound HTTPS, and the census as the example.
• Roadmap: How Orca will measure, admit and contain runs by memory and CPU, in the order it will be built, and the work it draws on.

Install it

Orca is the mojoshelf tin orca-mojo. With pixi and mojoshelf's pixi shelf installed, a new project adds it, and the tins it needs come with it. Plain pixi can add it from its repository instead:

pixi init -c https://conda.modular.com/max-nightly -c conda-forge lake && cd lake
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml   # tins build with pixi-build
pixi add "mojo-compiler==1.2.0"
pixi shelf add orca-mojo

pixi init -c https://conda.modular.com/max-nightly -c conda-forge lake && cd lake
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml   # tins build with pixi-build
pixi add "mojo-compiler==1.2.0"
# Without mojoshelf: the tin as a git dependency at its published commit.
# Transitive dependencies may be wrong: pixi takes the tin's own git deps as
# pinned in its pixi.toml, which may not match what other tins pin, and then
# fails with "duplicate records". mojoshelf picks revisions that agree.
pixi add orca-mojo --git https://github.com/magmalake/orca.mojo --rev 1db5de9aab671bf1b4de0ff1f9525535b9810b1a

A pipeline is one Mojo file whose main hands it to orca_main, compiled into a binary of its own. Build one and run it in memory to try it:

pixi run mojo build -I src pipelines/lake.mojo -o build/lake
./build/lake run --partitions 2024-01..2024-03

Writing a pipeline says what goes in the file, and how to build the orca command that schedules it.

The playground

To see the UI before running anything real, the repository's playground starts it on this machine with a throwaway Postgres, the example pipelines and their runs, two of them on a stand-in EC2:

git clone https://github.com/magmalake/orca.mojo && cd orca.mojo
git clone https://github.com/magmalake/dista.mojo ../dista.mojo
git clone https://github.com/magmalake/k8s.mojo ../k8s.mojo
pixi run playground          # then http://127.0.0.1:8270, log in as demo / demo-password

From source

Building from source works the same way: the repository compiles Dista and k8s.mojo from those checkouts beside it, pixi run test runs the tests, and pixi run orca builds the scheduler to build/orca. It targets Mojo 1.2.0, with a nightly environment alongside. Its state goes through postgres-mojo, which loads libpq; the tests start a throwaway Postgres server of their own. orca.tables, the Iceberg part, has its tests in the data environment (pixi run -e data test-tables).