Release state
Replaced #
This was the Latest release until 2026.10.14 replaced it. Before it became Latest, we updated our own Moorhold server to it from 2026.10.12, and it passed a restore drill there. Its files stay on get.moorhold.dev and still work. We haven't tested an update from this release to 2026.11.4. If your server runs it, email hello@moorhold.dev and we'll help you move it.
We wrote this release's notes, including their Still to be tested line, before we ran it on our own server.
This release brings a paid plan that adds automatic updates and off-site backups to your servers. With it, your server can install updates by itself in a maintenance window you choose, and keep its backups in an S3 bucket of your own, off the server. If you lose a server whose backups are in a bucket, you can bring it back onto a new one, with or without the plan. Updating by hand, with the update notice and its security fixes, stays free, as do backups and restores on the server. Edited 9 November 2026: the plan was withdrawn before it went on sale, so the price and the line about buying it have been removed. An admin can now set the mail server's password on the Mail page, without the server's command line, and "Send a test email" works. A data engineer can start a web service's first load from the source's page, with no Airflow permission of their own. sudo moorhold users reset-password now makes the person choose a new password when they next sign in.
Before you update #
- What changes
- Automatic updates in a maintenance window and off-site backups to a bucket of your own were to come with a paid plan, since withdrawn. A server whose backups are in a bucket can be brought back onto a new one. An admin can set the mail password on the Mail page, and a data engineer can start a web service's first load from its page. Resetting a password now makes the person choose a new one when they sign in.
- Downtime
- Moorhold is unavailable from when the update stops the running release until the new one has started. It hasn't been measured for this release.
- Backup
- Yes, on a server with backups on, which is the default: before it changes your Stack or the running release, the update takes a rollback point, a backup of every database on this server. If the new release doesn't start, it goes back to that point by itself. If you've turned backups off, the update takes no rollback point.
- Manual steps
- None for most servers. On 2026.10.12, if DataHub or Superset is asleep, open it just before you run sudo moorhold upgrade or ask for the update on the Updates page, or the update refuses to start: see Updating. If a script of yours runs sudo moorhold users reset-password and the person should keep the password it sets, add --no-temporary. If a data source names its secret in a way this release no longer takes, the Sources page shows it as Failed with where to move the secret, except for a web service in your Stack file whose reference isn't a file directly in /etc/moorhold/secrets named in capital letters, digits and underscores: the update refuses to start until you rename or move that one, see Updating. From 2026.10.10, also see 2026.10.11's notes: if you added a data source in the portal or with `moorhold apply -f`, run sudo moorhold apply once after the update, and if you changed the sample data's source file, see 2026.10.11's Updating.
- Still to be tested
- This release hasn't been through its test servers for the parts that need the paid plan, a bucket or a later release to update to: automatic updates installing in a window, off-site backups to a real bucket, bringing a server back onto a new one, and an update that stops when its rollback point fails. A data engineer starting a first load wasn't seen on the test server, and the Updates page offering 2026.10.13 is checked once it's Latest.
Updating #
On a server running 2026.10.12 or 2026.10.10, update from the server's command line with:
sudo moorhold upgrade 2026.10.13
It checks the release and the server before it changes anything, and takes the same rollback point as the installer. Once this release is the latest one, an admin can also press "Check for updates" on the Updates page in the portal and ask for the update there.
On 2026.10.12, wake any tool that's asleep before you update this way. sudo moorhold upgrade and the Updates page both run the checks of the release you're updating from, which count a sleeping tool's stopped services as failed, so the update refuses to start and changes nothing. Open each sleeping tool in your browser just before you update, and start the update straight away. The installer below runs this release's checks, so it updates a server while a tool sleeps.
On a server running 2026.10.12 or 2026.10.10, you can instead download this release's installer and run it on the server as you did to install, but without --hostname:
curl -fsSLo install.sh https://get.moorhold.dev/2026.10.13/install.sh sudo bash install.sh
The installer sees the release that's running and updates it in place, with the same checks and the same rollback point. On a server that already runs this release, it says there's nothing to update and changes nothing.
A server on 2026.10.11, which was published but never the latest release, updates the same ways. Read 2026.10.12's notes as well: they fix a security problem in 2026.10.11's mail settings, and say what to do if the Mail page then shows "Moorhold isn't using the saved mail settings."
If you're updating from 2026.10.10, 2026.10.11's Updating says what else that update changes, the sample data among them. If you added a data source in the portal, or with moorhold apply -f, and its secret is a file directly in /etc/moorhold/secrets, such as file:///etc/moorhold/secrets/SUPPLIER_PASSWORD, run this once the update has finished, so the source gets its secret. A source that names its secret any other way shows as Failed instead: see "A web service's secret" below.
sudo moorhold apply
What else the update changes:
- Resetting a password.
sudo moorhold users reset-passwordnow gives a temporary password: the person chooses a new one at their next sign-in. If a script of yours runs it and the person should keep the password it sets, add--no-temporary.--temporaryis still accepted. - A web service's secret. A web service's password or key must now be in a file directly in
/etc/moorhold/secrets, with a plain name of letters, digits and underscores, such asfile:///etc/moorhold/secrets/SUPPLIER_PASSWORD. A reference to a subfolder, a path with.., a path inside Moorhold's containers, or any reference that isn'tfile://is refused, and so is a file named like one of Moorhold's own, which includes any name ending_OIDC_SECRET. This holds for the web services in your Stack file on the server as well as those added in the portal. A web service added in the portal that names its secret any other way shows as Failed on the Sources page after the update, with where to put the secret, and so does one in your Stack file whose file is named like one of Moorhold's own. Move it to a file there, change the reference, and runsudo moorhold apply. A web service in your Stack file must also use a name in capital letters, digits and underscores: if its reference isn't a file directly in/etc/moorhold/secretswith a name like that, the update stops before it changes anything and names the reference it refused. A lower-case name such asfile:///etc/moorhold/secrets/supplier_passwordstops it too. Rename the file or move the secret there, change the reference in the Stack file, and run the update again. - A database source's secret. A database source's
file://reference can now only be the warehouse reader's password,file:///run/secrets/airflow-warehouse-reader-password, with the warehouse database on this server and the reader's own user name. On a server installed with--profile core, a database source can't use afile://reference at all. Avault://reference must sit underkv/sources/in OpenBao, such asvault://kv/sources/orders#password. A database source that names anything else shows as Failed after the update, with a message saying where to move the secret. Database sources you already have are still listed, and you can still edit or delete them (see Fixed for adding one). - OpenBao's access for the reconciler. This applies only if your Stack runs OpenBao, which the installer's Stack doesn't. A store set up by this release lets the reconciler read only source secrets under
kv/sources/and the database logins it leases. A store set up by an earlier release keeps its wider access until itsopenbao-datavolume is recreated, and Moorhold has no command that does that. Recreating the volume empties OpenBao, so any source secret you keep there has to be put back afterwards with OpenBao's ownbaocommand. Until then, the reference check above keeps a source tokv/sources/. - Two new secrets. The update makes two secrets the server didn't have: an id for this server, which every backup now carries, and a mark for the bucket path it backs up to. Backups taken before the update still count as this server's.
- Who may start a first load. The update adds
firstLoadGroups: [data-engineers, shipboard-admins]underspec.agentsin your Stack, just afterapproverGroups, with a comment above it. It leaves the Stack alone if you've changed the groups that may propose or approve changes there, or if it namesfirstLoadGroupsalready. Without that line, only the groups that approve changes may start a first load. See New. - An update from the portal that ends with a warning. If you ask for this update on the Updates page, and 2026.10.13 starts but the restore drill after it fails or
moorhold.servicecan't be written, the release you're updating from records the update as failed. The Updates page then says "The update failed" with exit 66, 67 or 68, although 2026.10.13 is running. Runsudo moorhold statuson the server to see what needs doing.
New #
- The paid plan. Moorhold is free for one server. A paid plan adds automatic updates and off-site backups. Each server asks account.moorhold.dev for its plan only when it's already in touch about updates, from the portal or the command line, or when you turn off-site backups on. It sends its access key and its Moorhold version, nothing else, and the key is never logged. To see the plan, or ask again now, run:
sudo moorhold plan sudo moorhold plan check
moorhold plan -f <dir>still shows what applying a Stack would change, as it did before. A server that can't reach account.moorhold.dev keeps what it has switched on. If the plan ends, what you'd switched on keeps working for 30 days, counted from the end date or from when the server learns that the plan has ended, whichever is later. Then automatic updates pause and new backups are made on the server. The backups already in your bucket can still be restored, free. - Automatic updates. With the paid plan, an admin can choose "Install updates automatically" on the Updates page and choose whether all releases or only security fixes install by themselves, in a maintenance window in the server's time zone. Typing
automaticconfirms it. Updates then install inside the window, with the same checks and rollback point as an update you start yourself, and go back by themselves if the new release doesn't start. A normal release waits a day after it's found, and a security fix goes in the next window. A release that needs approval or has manual steps always waits for an admin. An update doesn't start while a pipeline in Airflow is running or queued: it waits and looks again, and if a pipeline is still running at the window's last chance, it tries in the next window. Backups must be on, with one in the last 48 hours and a passing restore drill. A failed update pauses automatic updates until an admin turns them back on. Any admin can narrow, pause or turn them off at once. Without the plan, the Updates page says the setting needs a plan. To keep them off on one server whatever the portal says, runsudo systemctl mask moorhold-update.timer. - Off-site backups. With the paid plan, your backups can go to an S3 bucket of your own, so losing the server doesn't lose them. Make a bucket and an access key for it, then run:
sudo moorhold backup target s3 --endpoint https://<endpoint> --region <region> --bucket <bucket>
It asks for the key id and the secret without showing them, or reads them from files only root can read, and never takes them on the command line. It checks the bucket, takes the first backup there with a restore drill, and puts everything back if that backup fails. If the backup works and only the restore drill fails, the bucket stays the target:
sudo moorhold backup statussays what failed. Each server gets its own path,moorhold/<hostname>unless you give--path. Your earlier backups stay where they were, and no copy is kept on the server while the bucket is the target. Moorhold also keeps a copy of the server's secrets with the backups, encrypted as they are, so you can bring the server back onto a new one. Keep a copy of the backup repository key away from the server:sudo moorhold backup keyprints it, and no backup can be restored without it. The bucket's keys on the server can delete the backups too, so turn on object lock or versioning for the bucket if they must survive someone taking over the server. Don't add an expiry rule to the bucket: Moorhold removes old backups itself. If the bucket doesn't answer, that night's backup fails, and an update stops before it changes your Stack or the running release, because it can't take its rollback point there. To bring backups back onto the server, which is always free, runsudo moorhold backup target volume. - Bring a server back onto a new one. If you lose a server whose backups were in a bucket, install Moorhold on a new server, then run
sudo moorhold backup restore-serverwith the bucket's details. It asks for the bucket's keys and the old server's repository key, brings back every database and the old server's secrets, starts the stack, and takes a backup in the bucket with a restore drill. It refuses a server that holds data of its own unless you add--replace-own-data. It's free, with or without the plan. Without the plan, once the new server learns its plan, it makes its backups on the server, and the ones in the bucket can still be restored. The new server takes the bucket path over, so the old one, if it comes back, refuses to back up there and says to give it a new--path. - Set the mail password on the Mail page. On the Mail page, once the mail settings have a user name, an admin can now type the mail server's password under "Set the mail server's password", without the server's command line. It asks you to sign in again if you last signed in more than five minutes ago. The server keeps the password where
sudo moorhold mail passwordkeeps it, and the portal never shows it. Setting it restarts the portal and the reconciler, so the portal is unavailable while they restart. Members see whether a password is set, never the password. The local admin you sign in as without Keycloak can't set it, and still usessudo moorhold mail password. On a server that can't take the password from the portal safely, the page gives that command instead, as before. If the server doesn't pick the password up within 15 minutes, it deletes it unread and the page asks you to type it again. - Start a web service's first load. On a web service's page in Sources, people in the groups the Stack names in
spec.agents.firstLoadGroups(data engineers and admins, on a server installed or updated with the installer's groups) see "Start first load". Moorhold starts the source's load in Airflow with its own login, so they need no Airflow permission. It's offered until a load has succeeded, so a failed first load can be started again. Starting it switches on the source's schedule in Airflow too, and the page says so.
Fixed #
- Sending a test email from the Mail page. "Send a test email" now sends one. Before, it opened a "We can't find that page" page.
sudo moorhold mail passwordnow ends by pointing you to it. The last two releases' notes listed this as still known. - Saving after leaving a page open. A portal form sent after the page was left open for more than a few minutes no longer shows "Not a portal session: token expired." Moorhold renews your session and carries on, or, when it can't, sends you to sign in and back to the page. The Mail page's settings keep what you typed across that sign-in, but a password you typed doesn't, and on other pages you make the change again. The last two releases' notes listed this as still known.
- An update while a tool sleeps. The checks an update runs first now count a tool that's asleep while idle as asleep, so from this release on an update starts while a tool sleeps, however it was asked for. A tool counts as asleep only when every one of its services was stopped the way sleeping stops it. The update to this release is checked by the release you're updating from, unless you use the installer: see Updating. The last two releases' notes listed this as still known.
- A short password for a web service. On a server reachable from the internet, a web service's password or key, which its supplier chose, no longer stops
sudo moorhold startwhen it's shorter than 20 characters. Moorhold refuses it only when it's empty. Moorhold's own secrets are still checked in full. - Postgres on an installed server. The first-run card and Add source now offer only a web service on an installed server, because a database source added there can't connect. An agent can't propose one through the portal's MCP server, and a proposal to add one can't be approved, in the portal or from the command line. Database sources the server already has are still listed, and can be edited or deleted.
- Where a source's secret goes. The Sources form now checks a secret reference before anything is stored, and says on the field where that kind of source's secret has to be. A web service added in the portal shows "Needs a host step before it can load:
sudo moorhold apply" until someone runs that on the server. The Sources list now shows why a source is Failed or Degraded. - The forgotten password page. For the only admin, "Forgot your password?" now says they'll be asked to choose a new password when they sign in after
sudo moorhold users reset-password. - A source's page with the keyboard. On a source's page, the table showing the source in each tool can now be scrolled with the keyboard, and screen readers announce it as "Source status in each tool".
- Signing out of DataHub. When you sign out of DataHub, the page that asks "Do you want to sign out?" now has the heading "Sign out".
Verified #
- Unit and golden tests: pass.
- Images: every image passed the licence check, and each image is the one the manifest names. No tool changes version, and every image from upstream keeps its 2026.10.12 digest.
- On the test servers: the update from 2026.10.12 on the long-running test server ended on 2026.10.13, took its rollback point, kept the server's settings and data, and passed the restore drill afterwards, with every service healthy. On fresh 2026.10.12 and 2026.10.10 installs, the update by
sudo moorhold upgradeand by the installer each checked the release and the server before changing anything, ended on 2026.10.13 and passed the restore drill. Running the installer again said there was nothing to update and changed nothing. Both new secrets are on the long-running test server. The signing drill passed. - In the portal: a web service added in the portal shows the host step until
sudo moorhold apply, and one with its secret in a file directly in/etc/moorhold/secretsis Ready in every tool.reset-passwordwithout flags asks for a new password at sign-in, and with--no-temporaryit doesn't. An analyst sees no first-load button. Without the plan, automatic updates can't be chosen. An admin set the mail password on the Mail page, and "Send a test email" sent one. A Mail page left open past its session's life kept the draft through a fresh sign-in, with no error. On an installed server, Add source offers a web service only, even when asked for a database. - In the code and its tests: off-site backups and
backup target, bringing a server back onto a new one and--replace-own-data, automatic updates in a window and what makes them wait, the plan commands and what they send, and the checks that count a sleeping tool as asleep. - Owed: the parts that need the paid plan, a bucket or a later release to update to: automatic updates installing in a window, off-site backups to a real bucket, bringing a server back onto a new one, and an update that stops when its rollback point fails. A data engineer starting a first load wasn't seen on the test server; see Still known. The Updates page offering 2026.10.13 once it's Latest is checked after that.
Images #
Every one of Moorhold's own images is rebuilt from this release's source.
Still known #
- Signing back in to DataHub returns to its home page. After you sign out of DataHub and sign in again from the signed-out page, you land on DataHub's home page, not the page you left.
- Signing back in after Superset's sign-out. After you sign out of Superset and sign in again from the signed-out page, you land on the portal's Proposals page. Open Superset again from the launchpad.
- More memory for a sleeping DataHub. If a request for more memory for DataHub's services comes due while DataHub sleeps, the resize can fail and put the old size back. If it does, open DataHub to wake it and ask again.
- The launchpad and a sleeping tool. The launchpad can't tell that a tool is asleep, so a sleeping tool's tile can show it as not answering. Opening it wakes it.
- DataHub on a shared computer. If the last person to use the browser didn't sign out, DataHub can still open as them, with their access, until its session runs out, even after you sign in to Moorhold as yourself. Sign out before someone else uses the browser.
- Users and Groups in DataHub. If your role can't manage users, DataHub's Users and Groups page shows "Failed to load users!". That's DataHub's message when your role isn't allowed to see the list of users.
- Working only in Airflow or Superset. Time spent in Airflow or Superset alone doesn't keep your Moorhold session going, so if you use only one of them until the session times out, you see the sign-in page.
- API tokens made in Airflow. They keep working after you sign out, until they expire or are deleted in Airflow.
- An agent working alone. An agent's calls to DataHub's MCP server don't count as use, so DataHub can go to sleep while an agent is working with it. The agent's next call wakes it.
- System tables for agents. Agents that search through DataHub's own MCP server still see the system tables, because DataHub applies its default view only on its own pages.
- A reboot after a rollback. 2026.10.7 and earlier releases can't stop Moorhold at shutdown. So after a rollback to one of them, a reboot shows services failing and restarting until Moorhold has started, as before. They recover by themselves.
- Airflow's Triggerer. Airflow shows its Triggerer as unhealthy because Moorhold doesn't run one. Nothing in Moorhold uses deferrable tasks, and Moorhold's own health check ignores it.
- First loads after an update, on some servers. The update adds
firstLoadGroupsonly where the Stack's group lines are exactly as the installer wrote them. If they were ever written back with spaces inside the brackets, as in[ data-engineers, shipboard-admins ], the update leaves them alone, so only admins can start a first load. To let data engineers start one, addfirstLoadGroups: [data-engineers, shipboard-admins]underspec.agentsin your Stack, then runsudo moorhold apply. - An error printed during the update from 2026.10.10. The update from 2026.10.10 prints an error ending "Unknown option '--kind'", then "Couldn't read the store's sources". The update still finishes. A web service added in the portal gets its secret once you run
sudo moorhold apply, as Updating says. - x86_64 (amd64) servers only.
Upgrading #
On a server running 2026.10.7 or later, update from the server's command line with sudo moorhold upgrade <version>, giving the release you're moving to. On a server running 2026.10.6 or later, you can instead download the new release's installer and run it on the server as you did to install, but without --hostname.
This release's notes give its steps under Updating, and say which releases it updates from. If they don't cover the release your server runs, email hello@moorhold.dev and we'll help. More about upgrading
Recovery steps #
There's no automatic rollback yet, so there are no recovery steps to follow on your own. If something has gone wrong with your Moorhold, email hello@moorhold.dev with the version you're on and what you saw, and we'll help. Please don't send passwords or keys.
Download and verify #
The signed release is on get.moorhold.dev. Check the signature on release.lock first, then each file's SHA-256.
| File | SHA-256 |
|---|---|
| moorhold-2026.10.13.tar.gz | 60aa2d6ddf495535b4f443d3fbaa8cb43f25968dcf083205e58732dd30a890e2 |
| manifest.yaml | ff4ff75a0baeccfcc1c3be33e5684feebe8555f30f30ff33a6a8ecce69ec2aa3 |
| release.lock | 5f1513e01669ef0b2c917e08cddf9bd45e108cec2777b9387732e4e688257827 |
| install.sh | 202f6f508fc9e6d2704c7207b0219026f088e7a26426f3a53df87c1056e1a9a9 |
| cloud-init-ionos.yaml | afb21c18356f4d7ebbfd4ba900c22eb69ed853cc6173673b102a3e60a5a2d39f |
| cloud-init-hetzner.yaml | 6293ce59784e97d7bb07db4e3c624a3169383ca24dffa672b36f860e25e78aa5 |
| release.lock.minisig | Signature, key 7F6D12F3F6217F9C |
| SHA256SUMS | The checksums above |
minisign -Vm release.lock -p moorhold-release.pub sha256sum -c SHA256SUMS
Public key: get.moorhold.dev/moorhold-release.pub