Get told when a scheduled task stops running
Some failures are silent. A nightly invoice run that stops, a backup script that crashes, a Laravel scheduler that never starts: nothing is slow and nothing is down, the work just does not happen. A heartbeat catches this by turning the question round. Instead of SrvYard checking your task, your task checks in with SrvYard when it finishes. If a check-in does not arrive on time, you are notified.
This article explains how to create a heartbeat, how to ping it from any script, the quicker way of letting a scheduled job do it for you, and how the three states work. It also says plainly what SrvYard does not do: it does not poll your public site from outside on a timer.
Before you begin
- A task that runs on a schedule. A cron entry, a scheduled job from the server's Scheduler tab, or any script that runs regularly and can make a web request. See Scheduled commands on a server.
- Permission to manage scheduled jobs on the server, or to manage heartbeats on the site. Members who can only view see the list and the state, not the buttons.
- A way to be told. Missed heartbeats raise a Heartbeat missed notification for the owners and admins of the organisation, by email and in the app, as set in their own notification settings. See Notification channels to send them to Slack or Discord as well.
1. Open the Heartbeats tab
Heartbeats belong to a server, and each one can also be tied to a site. Open the server and choose Heartbeats.
- Add heartbeat. Creates a new heartbeat. The next step walks through it.
- A heartbeat row. The name, then a line that reads, for example, "Every 5 minutes, notify after 5 minutes", followed by the site it belongs to and, when it was created with a scheduled job, the words "scheduled job".
- Status. Waiting for the first ping, Healthy or Missed (explained in step 5). Last ping next to it shows when the last check-in arrived.
- Ping URL. The address your task calls. Copy puts it on the clipboard. The three-dot menu at the end of the row has Edit and Delete.
The ping URL is the only credential. Anyone who has the address can mark the heartbeat as healthy. Keep it in the script on your server, not in a public repository, and do not paste it into chats.
2. Create a heartbeat by hand
Press Add heartbeat. A panel opens with the title New heartbeat.
- Name. What you will recognise in a notification, such as
Nightly invoices. - Frequency. How often the task runs: every minute, 5, 10, 15 or 30 minutes, every hour, 6 or 12 hours, every day or every week.
- Notify me after. How long to wait after a check-in was due before you are told: 1, 5, 10, 15 or 30 minutes, 1 or 3 hours, or 1 day. The panel suggests five minutes. A task that sometimes runs long deserves a longer wait.
- Create heartbeat. Saves it. The new heartbeat shows Waiting for the first ping until your task calls it for the first time.
SrvYard works out the deadline as the time of the last check-in plus the frequency plus the "notify me after" time. For a task that runs every day with a one-hour wait, the deadline is 25 hours after the last ping.
3. Ping it from your task
Add one line at the end of the script, after the work has succeeded. The address answers both GET and POST, and replies with an empty 204 when the ping was accepted.
/usr/bin/php /home/srvyard/shop.example/current/artisan invoices:run && curl -fsS -m 10 --retry 3 -o /dev/null "<ping URL>"
The && matters. The ping is only sent when the task finished without an error, so a crashing task misses its check-in and you hear about it. Replace <ping URL> with the address you copied.
An unknown or mistyped ping URL answers 404. If your script uses curl -f you will see the failure; without it the typo is silent. Run the line once by hand and check that the heartbeat leaves Waiting for the first ping.
4. Let a scheduled job ping for you
If the task is a scheduled job on the server, you can skip steps 2 and 3. In the Scheduler tab, open New job (or edit an existing one) and turn on Monitor with heartbeats.
- Monitor with heartbeats. "The job pings a heartbeat URL after every successful run. You are notified when a ping is missed." SrvYard creates the heartbeat for you, works out the frequency from the job's schedule and adds the ping to the job's command, so it is only sent after a run that exited without an error.
- Notify me after. The same choice as before. After you save, the heartbeat appears on the Heartbeats tab marked "scheduled job".
A heartbeat made this way follows its job. Change the schedule and the expected frequency changes with it. Delete the job and the heartbeat goes too. You cannot delete it separately while the job exists: turn the switch off instead. A job that runs only at reboot has no rhythm to watch, so it gets no heartbeat.
The Laravel scheduler is the best candidate. A heartbeat on the schedule:run job that runs every minute tells you within minutes when the whole scheduler has stopped, which would otherwise silently stop every job it starts.
5. Read the states
A check runs every minute over all heartbeats.
- Waiting for the first ping. Created, but no check-in has arrived yet. The deadline counts from the moment of creation, so a heartbeat whose task never calls it is reported as missed once the frequency and wait have passed.
- Healthy. The last check-in arrived in time.
- Missed. The deadline passed with no check-in. SrvYard records "Heartbeat ... was not received" in the activity feed, and sends the notification once. It does not repeat it every minute.
When a missed heartbeat finally checks in, it returns to Healthy, the activity feed notes that it checked in again, and a recovery notification is sent. Pings are limited to 120 a minute per heartbeat, far above anything a scheduled task needs.
6. See the heartbeats of one site
A heartbeat that belongs to a site also appears under the site's Observe tab, so whoever looks after the site does not need to know which server it is on.
- Heartbeats. The sub-tab under Observe. It has its own Add heartbeat button, and what you create here is tied to the site.
- A heartbeat row. The same columns as on the server. See Site activity and Observe for the other Observe pages.
What SrvYard does not do
SrvYard does not request your public site on a timer to see whether it is up. The only site check it runs by itself is the health check after each deployment, described in Site activity and Observe. If you need an outside view of availability, use an external uptime service. A heartbeat only proves that something on your server ran.
Troubleshooting
- The heartbeat stays on "Waiting for the first ping". Your task has not called the URL yet, or the call fails. Run the curl line by hand on the server and look for errors. Check that the server can reach SrvYard's address and that the token in the URL is complete.
- A ping returns 404. The token is wrong or the heartbeat was deleted. Copy the address again from the Heartbeats tab.
- Every heartbeat on a server is missed at once. Check the server itself: a stopped scheduler, a full disk, a firewall that blocks outgoing requests or a server that is down all produce this. See Server metrics and charts.
- I get a notification for a task that ran. The task took longer than the frequency plus the wait. Raise Notify me after with Edit.
- I cannot delete a heartbeat. It belongs to a scheduled job. Turn Monitor with heartbeats off in the job instead.
- Nobody received the notification. Missed heartbeats go to owners and admins. Check their notification settings for Heartbeat missed, and whether the server is muted for them.
Related
- Scheduled commands on a server: create the jobs that heartbeats watch.
- Site activity and Observe: heartbeats, the health check and the activity of one site.
- Notification channels: send missed-heartbeat alerts to a team channel.
- Alert thresholds and silencing: alerts for CPU, memory and disk.