Retention, failure alerts and testing a restore

Setting up a backup is half the job. The other half is deciding how long to keep what you have taken, knowing the same day when a backup fails, and checking now and then that a restore really works. A backup that nobody monitors fails quietly for weeks. A backup that was never restored may turn out to be empty on the day you need it.

This article covers the three settings that answer those questions: retention, notifications and a regular restore test.

Before you begin#

• At least one backup configuration on a server. See Schedule database backups or Back up site files. File backups use exactly the same fields.
• Permission to manage backups to edit a configuration and permission to restore backups for the test.
• A decision about your risk. How much data can you afford to lose (this sets how often you back up), and how far back might you need to go (this sets how long you keep)? A shop with daily orders and a corruption that is noticed after three weeks needs more than seven daily copies.

1. Set the retention#

Retention is a property of each backup configuration. Open Backups, open the menu at the end of a configuration's row and choose Edit.

1. Keep the last. The number of finished backups to keep, from 1 to 365. The form starts at 7. SrvYard always keeps at least this many of the newest finished runs, even when they are older than the age limit below, and failed runs do not count towards it.
2. And anything newer than (optional). An age in days, from 1 to 3650. With it set, a backup is kept if it is among the newest N or younger than this many days, whichever keeps more. Leave it empty and only the count applies. Older archives are deleted from the bucket every night.

An example: Keep the last 14 with And anything newer than 30 days. A server that backs up daily keeps the 14 newest backups, and since 30 days of daily backups are younger than 30 days, it keeps about 30. A server that backs up weekly keeps its 14 newest, which go back about three months.

How the clean-up works:

• It runs once a night, at 04:40 UTC. A backup that has just become too old therefore stays until the next night.
• For every backup that falls out of the rules, SrvYard deletes its archives from your bucket and marks the run Expired. An expired run stays in the list as history (for about 90 days) but cannot be restored or downloaded.
• If the bucket cannot be reached, the run is left as it is and the next night tries again. Nothing is lost.
• Deleting a configuration stops its clean-up: archives of a configuration that no longer exists are never removed automatically.
• Failed runs, which have no archive, are forgotten after 30 days.

2. Tell the right people#

Under the retention fields the same panel has the notification settings.

3. Email when a backup fails (optional). Up to five addresses, separated by commas. The note under the field says what always happens: Owners and admins are always told in the app and by email. The addresses you add here are for people who are not owners or admins, such as the person who runs the database.
4. Also tell me when a backup succeeds. After every good run, a short message goes to the addresses you listed, with the number of databases and the size. It is useful for the first weeks of a new configuration and noisy afterwards. Owners and admins are not mailed for successes.

A failure message contains the server, the configuration, the error text and a link to the Backups tab. When a run fails, the list shows the same error in its row.

1. The error of the run. Here mysqldump: Got error: 2013: Lost connection to MySQL server during query when… tells you that the database server dropped the connection during the dump. Use the Status filter above the table (Finished, Failed, Running, Waiting, Expired) to see only failed runs.

A run that is Running for more than two hours belongs to a worker that died. SrvYard marks it Failed during its half-hourly check, so a hung backup eventually produces an alert too.

3. Send backup events to your team's channel#

For a team channel (Slack, Discord, Telegram or email) use a notification channel under Settings > Notifications.

1. Backup failed. The one every channel should have.
2. Backup finished. Useful in a quiet channel for a daily "all is well".

The group Backups and snapshots contains five events: Backup finished, Backup failed, Backup restored, Snapshot created and Snapshot failed. Switch All events off to choose them one by one, and use Severity (Warnings and errors or Errors only) and Limit to to keep a channel focused on some servers or tags. Press Test on the channel to send a test message.

4. Test a restore#

Do this once now and again every few months, and after you change anything about the backup setup (provider, bucket, schedule, server).

1. Into a new database. Open Backups, choose the newest finished run, Restore… and pick the tab Into a new database.
2. Name of the new database. The suggested …_restored is fine.
3. Restore. Wait until it finishes, then check the result: open the new database with phpMyAdmin or through an SSH tunnel (see Remote access, phpMyAdmin and SSH tunnels), look at the number of tables, count the rows of an important table and check that the newest record is from the day of the backup.

Then clean up: delete the test database on the Databases tab. For files, restore Into another directory and look at the folder. Write down how long it took: that number is your real recovery time, and it is the one to compare with what your business expects.

A short checklist for a healthy setup:

• Each production database has a configuration with an Email when a backup fails address that someone reads.
• The newest backup is younger than your schedule says it should be.
• Keep the last covers the longest time you might need to go back.
• You have restored a backup into a new database in the last three months.
• The storage provider shows Verified.

Troubleshooting#

• I did not get a failure email. Owners and admins always receive it, so check the spam folder and who the owners are under Settings > Members. For other people, add their address to Email when a backup fails.
• Backups older than I expected are gone. Keep the last and the age limit decide. Increase them; archives already deleted do not come back.
• The bucket keeps growing although I keep only 7. Archives of deleted configurations are not cleaned, and a bucket that could not be reached skips the night's clean-up. Delete old archives in the bucket, and check the storage provider.
• A run is marked Expired but I need it. Its archives have been deleted. Restore from a newer run, or from a copy of your own, if you made one.
• The restore test fails. Read the log of the restore in the operation log. Common causes are a database engine of another family, a full disk or a name that already exists. Fix it now, while it is only a test.
• A scheduled backup did not run at all. The configuration may be paused or the server not Ready. A paused configuration shows a Paused badge.

Related#

• Schedule database backups: where the retention fields live.
• Restore a database or files: every restore option in detail.
• Add an S3-compatible storage provider: keep the provider verified.
• Provider snapshots: the whole-server safety net, with its own retention.