Send requests, jobs and exceptions to TraceBeat or Nightwatch

Server charts tell you that a machine is busy. They do not tell you which request is slow, which queued job keeps failing or which exception your visitors are hitting. For that you need application monitoring: a small agent next to your Laravel application collects requests, jobs, commands, exceptions and logs and sends them to a service where you can search them.

SrvYard connects a site to two such services, TraceBeat and Nightwatch. It writes the settings into the site's .env, keeps the agent running as a background process of the site, shows whether events are arriving, and marks deployments so the charts can show them. This article covers connecting once for the organisation, connecting a site, the options under Advanced, and the card that shows its health.

Before you begin#

• A Laravel site on PHP. The Application monitoring card only appears for Laravel-based PHP sites. Other site types do not show it.
• A site that has been deployed once and has an .env. The Connect monitoring button stays disabled until the site is deployed and SrvYard holds a copy of its .env. The card says "Deploy the site first" or "No copy of the .env yet" when that is the reason.
• The laravel/nightwatch package in your application. Both services use it. SrvYard never changes your repository, so you add it yourself: run composer require laravel/nightwatch, commit and deploy. The card shows the command and tells you when the package is missing.
• An account at TraceBeat or Nightwatch. With TraceBeat you can also let SrvYard create the application for you, using an API key (step 1). With Nightwatch you paste a token from its dashboard.
• Permission to manage integrations on the site. Viewing the card needs only view access.

1. Connect TraceBeat once for the organisation (optional)#

If you use TraceBeat, give SrvYard an API key once and it can create the application and the environment of every site by itself. Open Organization settings > Integrations.

1. Settings, API. The link to the page in TraceBeat where you create the key.
2. Scopes. The key needs apps:read, apps:write, envs:read and envs:write. SrvYard uses them to create and change applications and environments, to check that the agent is connected and to rotate tokens. The key must not be limited to some environments.
3. API key. Paste the key. SrvYard asks TraceBeat to confirm the key and its scopes before saving it. The key is stored encrypted and is never shown again.
4. Connect TraceBeat. Saves the key.

You can skip this step. Without a key, you create the environment in TraceBeat yourself and paste its token on the site (step 3). Nightwatch has no API, so it always works that way.

2. Open the site's integrations#

Open the site, choose Settings, then Integrations in the left-hand list. The first card is Application monitoring.

1. Integrations. The settings section that holds the card (below it sits the card about the connection to your Git host).
2. Connect monitoring. Opens the panel for the next step. The text above it explains what connecting does: it adds a block to the .env and runs the Nightwatch agent as a background process of the site.

3. Connect the site#

Press Connect monitoring. A panel slides in from the right.

1. Service. TraceBeat or Nightwatch.
2. Set it up for me. TraceBeat only. SrvYard creates the application and environment in TraceBeat and receives the environment token itself. You can add an environment to an application that already exists instead of creating a new one. This choice needs the organisation key from step 1: without it the panel says "The TraceBeat API key is not connected" and links to the integrations page.
3. Use an environment token. You created the environment yourself and paste its token. This is the only way for Nightwatch.
4. Token. Copy it from the service: in TraceBeat from your application, Environments, Setup; in Nightwatch from your application's settings. SrvYard writes it to the site's .env and keeps a copy encrypted.
5. Advanced. Sampling and deployment options, see step 4.

Press Connect. SrvYard then:

• adds a block to the .env, between marker comments so that repeating it changes nothing: the token, for TraceBeat also NIGHTWATCH_BASE_URL, and the address the agent listens on (port 2407 on the local machine by default);
• creates a background process named apm for the site that runs php artisan nightwatch:agent, so it starts again after a reboot;
• shows the card in the Connecting state until both are in place.

4. Choose sampling and deployment options#

Unfold Advanced in the panel (you can come back later with Settings on the card).

1. Sampling. Three numbers between 0 and 1: Requests, Commands and Exceptions. They become NIGHTWATCH_REQUEST_SAMPLE_RATE, NIGHTWATCH_COMMAND_SAMPLE_RATE and NIGHTWATCH_EXCEPTION_SAMPLE_RATE. The default 1.0 keeps everything and 0.1 keeps one in ten. Lower the rates on a very busy site to keep the volume and the cost down. Keep exceptions at 1.0 unless you have a reason not to.
2. Send deployment events. After every deployment SrvYard tells the service, so its charts can mark the moment a release went live. This is how you see that a slowdown started with a release.
3. Restart the agent after a deployment. The agent keeps the old release in memory until it restarts. This switch is on by default and should stay on.

5. Read the card#

Once connected, the card replaces the button with the details and a status badge at the top.

• Connecting. SrvYard is still writing the .env or starting the agent.
• Package missing. Add laravel/nightwatch to the application and deploy.
• Agent not running. Supervisor reports the process as stopped or failed. The message says what it reports. Look at the process on the site's Processes tab.
• Waiting for first event. The agent runs but nothing has arrived. Open the site in a browser to produce a request.
• Connected, last event … Events are arriving. This is the state you want.
• Not receiving. The agent runs but the service has received nothing for a while. Check that the site is being used and that the server can reach the service.
• Paused in TraceBeat. The environment is paused on the TraceBeat side, so it ignores what the agent sends.

The rows below list the Service, Set up (with a link to the application and environment in TraceBeat), Events, Package and its version, Agent and its command, Sampling, Deployment events, the last deployment event, and when it was last checked. Check again asks again immediately; the card also refreshes itself every few seconds while something is changing.

With TraceBeat and "Set it up for me" the buttons are Settings, Rotate token and Disconnect. Rotate token asks TraceBeat for a new token, which stops the old one at once; SrvYard writes the new one to the .env and restarts the agent, so events pause for a few seconds. When you connected by pasting a token the button is Replace token. Disconnect removes the block from the .env, deletes the agent process and forgets the token. The package stays in your application, and nothing is deleted at the service.

Troubleshooting#

• The Connect monitoring button is greyed out. The site has not been deployed yet, or SrvYard has no copy of its .env. Deploy first, then reload the page.
• I do not see the Application monitoring card. It is only shown for Laravel-based PHP sites.
• "This site is already connected." Disconnect it first, then connect again with the new settings.
• Stuck on "Package missing". The deployed code does not contain laravel/nightwatch. Check that composer.json and composer.lock are committed, and that the deploy script runs composer install.
• Connected, but I see no exceptions. The exception sampling rate may be below 1.0, or the exception is handled and never reported. Check the sampling row on the card.
• After a deployment the data stops. Turn on Restart the agent after a deployment, so the agent loads the new release.
• "SrvYard could not ask TraceBeat." SrvYard could not reach TraceBeat to read the state. The agent may still work. Use Check again in a minute.

Related#

• Site activity and Observe: logs, heartbeats and the health check, which need no extra package.
• Environment variables: see the block SrvYard adds to the .env.
• Queue workers and the Laravel scheduler: the other background processes of a Laravel site.
• Write and test a deploy script: make sure composer install runs on every deployment.