Skip to content

Jobs

Compute code that runs as workloads submitted into a shared runtime — bounded or unbounded. This is the placement rule, not uptime: an unbounded Flink streaming job still lives here, because the long-lived process is owned by the platform cluster (platform/flink-cluster), not by the job code. Deployable always-on components (own port/files, deployed as containers) go in ../services/ instead. Engine groups share toolchain affinity; each job's data-flow ownership, where it matters, is recorded in docs/.

Engine Jobs Runs on
flink/ parse-node-outputs: HyperLiquid node output JSONL → typed rows → Parquet → Iceberg (hypercore.* raw tables). Dual-mode: --source files (bounded backfill) / Kafka (unbounded live/replay). platform Flink cluster (Java fat-jar)
dbt/ SQL modeling + data-quality tests over the warehouse (staging → intermediate → marts). dbt engine, triggered from Airflow
spark/ Iceberg maintenance (compaction, snapshot expiry — Airflow-scheduled), wallet-graph algorithms Spark cluster

Cluster definitions for Flink live in platform/flink-cluster; the Spark master wiring currently builds from spark/ (TODO: split cluster from job code the same way, platform/spark-cluster).