Moorhold docs

Desired state and reconciliation

On this page

Moorhold keeps a record of what should exist, the desired state, and a reconciler that keeps making the tools match it. This page explains how that works in the latest release (2026.11.4), and what it means for you.

Everything Moorhold manages is a resource in its Postgres store: the Stack, its Components, the roles each group gets, each source. Each resource says what should exist, and the reconciler works out the steps. A source says "this web service, loaded into this schema, read by these tools", and the reconciler makes the Trino catalog, the Superset database and the rest.

You change the desired state by changing resources: in the portal, with sudo moorhold apply on the server, or through a proposal that an approver accepts. The resource kinds reference lists every kind.

The reconciler compares the desired state with what each tool holds, and changes the tools until they match. For each resource, it goes through the adapters it needs in order, and for each one:

  1. Observes what the tool has now.
  2. Plans what to change, from what it observed and what the resource asks for. Planning changes nothing, which is how moorhold plan -f and a proposal's plan can show a change before it happens.
  3. Applies the plan.
  4. Records the outcome in the resource's status: its conditions, and an event.

Adapters run in dependency order. Trino has to have a source's catalog before Superset can point at it, so if one adapter fails, the ones after it wait.

The reconciler is level-triggered: each pass looks at where things are now, whatever happened before. The same pass creates what's missing, repairs what's drifted, and does nothing when everything matches. Running it twice is safe.

It runs a pass when a resource changes, when a tool's health changes, when someone asks it to (Stack page or moorhold reconcile), and on a regular resync, which is how it notices drift no one told it about. After a failure it tries again, waiting longer each time.

Drift is when a tool no longer matches the desired state. Moorhold sorts it into three kinds:

Kind What happened What Moorhold does
Repairable Something Moorhold made is missing or was changed. Puts it back on the next pass, and records a DriftRepaired event.
A collision Something Moorhold would create already exists, made by someone else. Leaves it alone and sets Drifted=True. The moorhold.adopt annotation tells it to take the object over.
An unhealthy tool A tool the resource needs isn't healthy. Sets Degraded=True and skips that tool and the ones after it until it recovers.

A tool that's asleep because nobody has used it isn't a fault. The resource shows Ready=False with the reason Asleep, and it catches up when the tool wakes.

Only one reconciler may change the tools at once. It holds a lease, an advisory lock in Postgres. On an installed server the reconciler runs all the time and holds it.

When you run moorhold apply -f or moorhold reconcile and the running reconciler holds the lease, the command stores your change and hands it over. It then waits for that reconciler to finish, up to --wait seconds. Your change still happens if you stop waiting.

Removing a resource is reconciled too. The reconciler runs the adapters in reverse order, then observes again to confirm each tool no longer has what it made. Only then does it forget the resource.

Delete Keeps
Soft (the default) What people built on the resource: catalogued datasets in DataHub, and removed DAGs' run history in Airflow.
Hard (--hard-catalog) Nothing Moorhold made. It removes only its own objects.

A delete waits while other objects depend on what it would remove, such as a dashboard on a dataset, and the resource shows BlockedByDependents=True. moorhold delete --force removes it anyway.

  • Edit resources. A change made by hand in a tool to something Moorhold manages is repaired on the next pass.
  • What you build yourself in a tool, such as a dashboard, isn't Moorhold's, and Moorhold leaves it.
  • Read a resource's conditions to see where it is: on the Stack page, the Sources page, or with moorhold status.