Moorhold docs

You can put one database, or the lake, back as it was in a backup. Moorhold restores it into a new place first, then swaps it in, and keeps the database it replaced until you drop it. To look at old data without replacing anything, restore it beside the original instead. If you've lost a server whose backups were in a bucket of your own, you can bring it back onto a new server (see Bring a server back onto a new one).

To go back to the release and the data you had before an upgrade, see Roll back an upgrade.

  • You need root on the server, on a terminal. The restore asks before it changes anything. To run it without a terminal, add --yes.

  • The apps stop while it runs. Restoring a database stops every service except Postgres and the backup services, and starts them again at the end, whatever the outcome.

  • Anything written since the backup is lost from the restored database. It stays in the kept copy until you drop it.

  • Find the snapshot you want. Each backup is a snapshot with an id:

    Terminal window
    sudo moorhold backup --list

    You can also use latest for the most recent one.

  1. See the plan first:

    Terminal window
    sudo moorhold backup restore <snapshot> --database <database> --dry-run

    It prints "Plan (nothing has changed):" and the steps, then "Dry run: nothing has changed.", and changes nothing.

  2. Run the restore:

    Terminal window
    sudo moorhold backup restore <snapshot> --database <database>

    It shows the plan again and asks. Type y to go ahead.

The restore then:

  1. Stops the apps. If any service doesn't stop, it starts them all again and changes nothing.
  2. Restores the snapshot into a new database, <database>_r<time>.
  3. Swaps them in one step. The restored copy becomes <database>, and the database you had is kept as <database>_old<time>.
  4. Starts the apps again.

It ends with "Swapped. The previous <database> is kept as <database>_old<time>. Once you've checked the stack, drop it with: sudo moorhold backup drop-kept <database>_old<time>".

From 2026.10.10, the restore checks that the snapshot is a backup of the database you named before it stops anything. If it isn't, it says "snapshot <id> is a backup of <other>, not <database>. Nothing has changed." and tells you how to list the right snapshots. From 2026.10.13, it also refuses a snapshot that another server took, with "snapshot <id> is a backup of another server, not this one. Nothing has changed."

If new backups are being made on the server and earlier ones are still in the bucket, the ones in the bucket can still be restored. Add --from-bucket to list and restore them:

Terminal window
sudo moorhold backup --list --from-bucket
sudo moorhold backup restore <snapshot> --database <database> --from-bucket

Restore a database beside the original

Section titled "Restore a database beside the original"

To look at the data in a backup without replacing anything, restore it into a new database:

Terminal window
sudo moorhold restore <database> --into <new-database> --snapshot <snapshot>

Leave out --snapshot to use the latest one. It never writes over a database that exists.

Terminal window
sudo moorhold backup restore <snapshot> --lake --dry-run
sudo moorhold backup restore <snapshot> --lake

This puts SeaweedFS's data back from a lake backup (see Backups). It stops the apps and SeaweedFS, restores beside the live data, then swaps it in. The data you had is kept in a folder named .moorhold-before-restore-<time> on the volume.

Once you've checked that the apps and your data look right, drop the copy the restore kept:

Terminal window
sudo moorhold backup drop-kept --dry-run
sudo moorhold backup drop-kept <database>_old<time>

Until you do, it takes space and the daily backups include it. See Backups.

Bring a server back onto a new one

Section titled "Bring a server back onto a new one"

From 2026.10.13, if you lose a server whose backups were in a bucket of your own, you can bring its databases and secrets back onto a new server. This is free.

Before you start:

  • The old server's backups must be in a bucket, and it must have run 2026.10.13 or later with them there. From 2026.10.13, a server with its backups in a bucket keeps a copy of its secrets with them, and a recovery needs that copy. Without it, the old databases can't be read.
  • Have the old server's backup repository key, which you kept when you ran sudo moorhold backup key on it, and an access key for the bucket.
  • Know the old server's path in the bucket. Unless it was given --path, it's moorhold/<old-hostname>.
  • Install Moorhold on the new server with backups on, which is the default, and the same profile as the old one. Don't finish the setup in the portal first: the command refuses a server whose admin password has been set, or which has sources of its own (see If something goes wrong).
  1. See the plan first:

    Terminal window
    sudo moorhold backup restore-server --endpoint https://<endpoint> --region <region> --bucket <bucket> --path moorhold/<old-hostname> --dry-run

    It prints "Plan (nothing has changed):" and the steps, and changes nothing.

  2. Run it:

    Terminal window
    sudo moorhold backup restore-server --endpoint https://<endpoint> --region <region> --bucket <bucket> --path moorhold/<old-hostname>

    It asks for the bucket's access key id, its secret access key and the old server's repository key, without showing what you type. To give them from files instead, add --access-key-id-file <file>, --secret-access-key-file <file> and --repository-key-file <file>. Each file must be readable only by root (mode 0600). The command never takes them on the command line.

  3. It opens the old server's backups with that key and lists the newest backup of each database, of the roles and of the secrets. Read the list, then confirm when it asks "Replace this server's databases and secrets with the old server's? This stops the apps."

The command then:

  1. Stops the apps, restores each of the old server's databases beside this server's, then swaps them in and gives the Postgres roles the old server's passwords. This server's databases are kept as <database>_old<time>.
  2. Keeps the old server's secrets and repository key in /etc/moorhold/secrets. This server's own are kept in /etc/moorhold/secrets/.before-recovery-<time>.
  3. Points the backups at the bucket, starts the stack and takes the old server's bucket path over.
  4. Takes a backup in the bucket, then runs a restore drill.

From 2026.10.16, the command brings back only the old server's live databases. Copies that an earlier rollback or restore kept there stay in the backups, and the list says so with "Left in the backups, not brought back:" and their names. The same line gives the command to bring one back under a new name.

Nothing in the bucket is deleted. If a restore fails, this server stays as it was.

When it's done, the new server backs up to that path as the old server, with its id. It says "The old server must never back up to this bucket path again". If the old server comes back, it refuses to back up there and says to give it a new --path. Turn it off, or point its backups elsewhere, before it starts again.

The new server now uses the old server's repository key, so the copy you already keep still opens every backup. The backups already in the bucket can still be restored.

If the old server had backed up its lake, the command's last lines give the sudo moorhold backup restore <snapshot> --lake command that puts it back. The lake isn't restored by itself. Once you've checked the apps, drop the databases this server had before (see Drop the kept copy).

  • The restore's last line starts with "Swapped."
  • sudo moorhold status says all the services are healthy, and lists the kept copy.
  • The data in the apps is as it was at the time of the snapshot.
  • After bringing a server back, the command says "Recovered:" with the number of databases, that a backup was taken and that the restore drill passed. sudo moorhold backup status names the bucket.
  • "confirming needs a terminal: add --yes to go ahead": run it on a terminal, or add --yes.
  • "These services didn't stop": nothing has changed, and the services are started again. Try again, or see Check your server's status.
  • "The restore failed": <database> is unchanged. From 2026.10.10, if the restore had made an empty <database>_r<time>, it drops it and says so. Otherwise it says the copy is left and gives the command that drops it.
  • "the backup service on this server is older than this moorhold command": run the sudo moorhold start --profile <profile> command the message gives, with full or core, whichever your server runs. Then run the restore again. Nothing has changed.
  • A lake restore stops halfway. The stack is left stopped, and the message says where the live, moved-aside and restored data are. Don't start SeaweedFS. The message says which folders to move the data back from, by hand.
  • "This server isn't as the install left it": the server you're recovering onto has its admin password set, or sources of its own. Nothing has changed. To recover onto it anyway, add --replace-own-data. Its databases are then kept as <database>_old<time>, and its secrets in /etc/moorhold/secrets/.before-recovery-<time>.
  • "Opening the old server's backups failed": the lines above it say why, for example a wrong repository key or a bucket that doesn't answer. Nothing has changed.
  • "the repository holds no copy of the old server's secrets": the old server didn't keep a copy of its secrets in the bucket, which starts with 2026.10.13. The server can't be brought back from these backups. Nothing has changed.
  • "the repository holds no backup of a database": check that the bucket and --path are the old server's. Nothing has changed.
  • "This path holds backups from more than one server": the message lists the server ids it found, each with its newest backup. Run the command again with --server-id <id>.
  • "Restoring the old server's databases failed": nothing live has changed, and this server is as it was.