Declarative, event-sourced orchestration: YAML Playbooks drive APIs, databases, scripts and AI agents, and every step lands on an append-only event log.
These properties come from the runtime, not from code you write. Each card links to where it lives in the source.
State is rebuilt from an append-only log of events and never edited in place. What runs next is a pure function of that history, so a restarted server rebuilds a run from its events and carries on.
With plain scripts: state lives in memory or in rows that get overwritten, so a crash mid-step loses the work or repeats it.
orchestrate-core · Orchestrator::evaluateAdd a Python, Rhai or shell task to any step’s tool pipeline. It gets the playbook’s retry and backoff policy, its start and result land on the event log, large outputs are kept as durable references, and credentials arrive by keychain alias instead of living in the script.
With plain scripts: a script that dies halfway leaves no record of what ran, retries are hand-rolled, and credentials sit in env files or code.
orchestrate-core · PipelineTaskAn agent step uses the same closed set of tool kinds and the same five policy actions as any other step. It can pick among the routes the playbook declares, only the server starts a step, and every step it runs is recorded on the log. Agents reach MCP servers over HTTP.
With plain scripts: an agent that writes and runs its own script has no fixed set of actions, nothing outside it deciding what runs next, and no durable record of what it did.
server · playbook policy validationOn shutdown a worker stops claiming, finishes what it is running, then leaves. A command that was never acknowledged goes to another worker, and a redelivered event is recognised by its id instead of being appended twice. Worker pools autoscale on the command backlog.
With plain scripts: a queue and a cron job either lose the job that was running when a pod went away, or run it again and record it twice.
worker · drain on SIGTERMEvery change to the event log goes through a single fenced chokepoint, and the Rust compiler enforces that no other path exists. With leader election on, a writer that has lost its lease is refused. The log is EHDB, NoETL’s own event store: each append is synced to disk before it returns, and sealed parts are uploaded to object storage.
With plain scripts: naive leader election lets a paused old leader keep writing after it has lost the lease, and two writers corrupt the history.
ehdb · FencedEngine::writeDelivery is at-least-once: redelivered events are deduplicated by id within a bounded window, and steps that call external systems should be written to tolerate a retry.
Case study: NoETL as the orchestration harness for verifiable quantum circuit compilation, with an example playbook.
Eight parts, one event log. Select any node to see its role, ports and responsibilities, or tap the floor to send a ripple. Glowing packets show how commands, events and results move. Routes are simplified.
Also in the box: the CLI, noetl / ntl.
Pick a preset or edit the YAML. Press Run to play the whole lifecycle, from execute to quiescence. The simulator reads what you type, so your edits change the route it takes.
Playbook syntax and event names are simplified for this demo. simulate: is a demo-only hint that scripts a tool's outcome. The demo runs in your browser and contacts no cluster.
The server decides what runs next. Workers claim commands and run the pipeline inside a step. A worker's policy can steer its own tasks, but it can never create a new step-run.
Task kinds and which pool runs them are examples for this diagram.
Tap or hover a service to light up what it talks to and list each connection. NATS is optional and tool-facing only, so its island has a dashed outline.
Links to the GUI and to observability are indicative, and their ports are not shown.
Small results stay close. Large results move out, and workers pass along a ResultRef in their place. The vessels below sit on the same log scale as the slider.
The tiers are real. The cutoffs here (64 KB, 1 MB, 256 MB) are example values chosen for this demo.
The catalog is where a NoETL server keeps what you register. Below is an example catalog built into this page: each box is one entry, and each arrow is one entry naming another. It is not connected to a NoETL server, and nothing on it is live, registered, or executed.
Each catalog entry stores a document under a path, with a kind. Playbooks and agents can run. A subscription is activated and runs a playbook for each message. MCP descriptors, memory, and credentials never run; other entries name them.
noetl catalog register hello_world.yaml noetl catalog list Playbook noetl catalog get examples/hello_world
Register a changed file at the same path and the server stores the next version. Earlier versions stay. A run by path uses the latest; to run an older one, name it. Credentials are the exception: registering one again updates it in place.
noetl run catalog://workflows/fetch_and_store@1
An arrow starts at the entry that names another. A kind: playbook task runs another playbook by path, optionally pinned with version. A task names a credential with auth:. A subscription names its playbook in dispatch. An agent names an MCP server and calls it over HTTP.
- name: run_store kind: playbook path: workflows/fetch_and_store version: 2
Six kinds, from noetl.resource: the schema DDL seeds playbook and agent (executable), credential, mcp, and memory, and the Rust server seeds subscription (executable) at startup. A subscription is activated rather than run, and registering its path again adds a version; a running subscription keeps the version it was registered with until it is deactivated and registered again. Credentials registered with noetl register credential are stored by name in noetl.credential, so registering one again updates it rather than adding a version. The intro diagram also draws connectors, notebooks, models, datasets, and more. Those are not catalog kinds, so they have no boxes here.
CLI catalog commands · DSL reference · schema DDL · next version · subscription kind · subscription spec · register adds a version · subscription register · subscription lifecycle · credential update