13. Laravel Health. Configuration and Usage in Docker Healthcheck.

Video version
(Leave your feedback on YouTube)

TL; DR

The Laravel Health package is not a substitute for a full-fledged monitoring system but can complement it. Use external monitoring systems for notifications and to run the check command.

For my project, I created a separate module for health checks and moved the methods to an Action. If your project doesn't use modules or Actions, I recommend doing so.

Always close the web-view in production and staging. Ideally, provide access only from a corporate IP address!

Workflow

An important point - the documentation suggests scheduling the check in the project. This is not the best approach because if your application "dies", no one will run the scheduled check, and no notification will be sent.

Also, sending notifications about a broken application from a broken application is not okay. Therefore, the workflow is as follows:

  1. Install or purchase any external monitoring service. For example, uptime kuma. Ideally, on a different server than the APP being monitored.
  2. The service pings the /health-check path and sends notifications to the desired channel.
  3. /health-check will return a 503 status when at least one of the checks fails. Be careful with the setup - skipped statuses are also considered errors. If this is not acceptable, write your own SimpleHealthCheckController implementation.
  4. Upon notification, developers go to /health if it is available, if not - to the server, and so on.

Authorization

Don't forget about authorization. A corporate IP is still the best option.

Configuration for authorization in the Service Provider:

config/health.php
[
  'path' => env('HEALTH_PATH', 'health'),
  'check_path' => env('HEALTH_CHECK_PATH', 'health-check'),

  'middleware' => [
      'web',
      VeryBasicAuth::class
  ]
]
app/Providers/HealthCheckServiceProvider.php
class HealthCheckServiceProvider extends ServiceProvider
{
    public function boot(): void
    {
        $this->registerRoutes();
    }
    
    protected function registerRoutes(): void
    {
        if ($this->app instanceof CachesRoutes && $this->app->routesAreCached()) {
            return;
        }
    
        Route::get(config('health.path'), HealthCheckResultsController::class)
            ->middleware(config('health.middleware', 'web'));

        Route::get(config('health.check_path'), CheckSystemHealth::class)
            ->middleware(config('health.middleware', 'web'));

        Route::get('/up', fn() => '');
    }
}

Here:

  • HEALTH_PATH - the path to the Laravel health web interface. Must be closed to users.
  • HEALTH_CHECK_PATH - the path to a simple "Everything is okay" check. Also recommended to be closed.
  • /up - a simple check to see if the APP is up. I suggest not closing this.

Micro-optimization

Saving the history of health checks makes sense in production. Understanding what went wrong at night when everyone was asleep is important.

Therefore, I implemented my version of SimpleHealthCheckController.php:

Modules/HealthCheck/Actions/CheckSystemHealth.php
<?php

namespace Modules\HealthCheck\Actions;

use Illuminate\Http\Response;
use Lorisleiva\Actions\Concerns\AsController;
use Spatie\Health\Enums\Status as HealthCheckStatus;
use Spatie\Health\Models\HealthCheckResultHistoryItem;
use Symfony\Component\HttpKernel\Exception\ServiceUnavailableHttpException;
use Throwable;

class CheckSystemHealth
{
    use AsController;

    /**
     * @throws Throwable
     */
    public function handle(): Response
    {
        $unhealthy = HealthCheckResultHistoryItem::query()
            ->whereNotIn(
                'health_check_result_history_items.status',
                [
                    HealthCheckStatus::ok(),
                    HealthCheckStatus::skipped(),
                ]
            )
            ->where(
                'health_check_result_history_items.batch',
                fn ($query) => $query
                    ->select('health_check_result_history_items.batch')
                    ->from('health_check_result_history_items')
                    ->latest()
                    ->limit(1)
            )
            ->exists();

        throw_if($unhealthy, new ServiceUnavailableHttpException('Application not healthy'));

        return response()->noContent();
    }
}

Laravel Health and Docker

For Docker healthcheck, the package does not offer ready-made solutions. Therefore, I implemented my own command:

Modules/HealthCheck/Actions/CheckSystemHealth.php
<?php

namespace Modules\HealthCheck\Actions;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\Artisan;
use Illuminate\Support\Str;
use Lorisleiva\Actions\Concerns\AsCommand;
use Spatie\Health\Commands\RunHealthChecksCommand;
use Spatie\Health\Enums\Status as HealthCheckStatus;
use Spatie\Health\Models\HealthCheckResultHistoryItem;

class CheckPropertyHealth
{
    use AsCommand;

    public $commandSignature = 'health:check-property {name} {--fresh}';
    public $commandDescription = 'Check single health property by name';

    public function handle(
        string $name
    ): ?string {
        return HealthCheckResultHistoryItem::query()
            ->whereRaw("LOWER(health_check_result_history_items.check_name) = '{$name}'")
            ->orderByDesc('health_check_result_history_items.created_at')
            ->value('health_check_result_history_items.status');
    }

    public function asCommand(Command $command)
    {
        $name = Str::lower($command->argument('name'));

        if ($command->option('fresh')) {
            Artisan::call(RunHealthChecksCommand::class);
        }

        $status = $this->handle($name);

        if (!$status) {
            $command->warn("No health check results found for {$name}");
            return $command::INVALID;
        }

        if ($status !== HealthCheckStatus::ok()->value) {
            $command->error($status);
            return $command::FAILURE;
        }

        $command->info($status);
        return $command::SUCCESS;
    }
}

And integrated it into Docker healthcheck:

docker-compose.yml
app:
 # ...
 healthcheck:
   test: php artisan health:check-property octane --fresh
   interval: 60s
   timeout: 5s
   retries: 3

horizon:
 # ...
 depends_on:
  - app
 healthcheck:
   test: php artisan health:check-property horizon
   start_period: 5s
   interval: 60s
   timeout: 5s
   retries: 3

schedule:
 # ...
 depends_on:
   - app
 healthcheck:
   test: php artisan health:check-property schedule
   start_period: 5s
   interval: 60s
   timeout: 10s
   retries: 3

Here, --fresh runs the health check command. Calling it as a schedule is not an option because this is a self-check.

Accepted Checks


Set of Checks

Health::checks
OctaneCheck::new()
  • I use it in the healthcheck of the app service. Be careful if you run it from the schedule service - Octane is not running there.
Health::checks
HorizonCheck::new(),
  • Checks if the Horizon status is Active. TOP for the Horizon service healthcheck.
Health::checks
ScheduleCheck::new()->useCacheStore(config('cache.default')),
  • Here it is important to understand the principle of operation. You must add a check command. It writes a timestamp to the cache every minute. If the time in the cache is more than 2 minutes behind, the command did not execute, and the schedule service is not working. Of course, for the healthcheck of the schedule container.
registerSchedule some where
$schedule->command(ScheduleCheckHeartbeatCommand::class)->everyMinute();
Health::checks
DatabaseConnectionCountCheck::new()
    ->failWhenMoreConnectionsThan(20),
  • Here you need to understand. For Laravel Octane - one connection per worker and several system processes: autovacuum, background writer, and so on. This is Octane's feature - one constant connection. If you are not using Octane, each request = +1 connection (maybe more). So, in the case of Octane, 20 connections sound okay. For others, it depends on the system load.
Health::checks
DatabaseSizeCheck::new()
             ->failWhenSizeAboveGb(errorThresholdGb: 25.0),
  • This check is useful if you use cloud services and fear that you will receive a large bill for maintenance tomorrow. And in general, it is just nice to see how your project "grows".
Health::checks
QueueSizeCheck::new()
             ->queue('default', 100),

The number of processes in the queue is a very useful metric. It will indicate abnormal activity or cases where things are going wrong and processes are accumulating.

Health::checks
UsedDiskSpaceCheck::new()->unless($isLocal),
  • Very useful for those who store files or keep a database on the server with the code (not recommended). But even otherwise, it will help react promptly if memory gets clogged, for example, by Docker images or excessive log files. No need to run locally.
Health::checks
RedisCheck::new()->unless($isLocal),
  • Makes sense, will remind if something is wrong with Redis. Not necessary to enable locally.
Health::checks
CpuLoadCheck::new()
             ->unless($isLocal)
             ->failWhenLoadIsHigherInTheLastMinute(8)
             ->failWhenLoadIsHigherInTheLast5Minutes(8.0)
             ->failWhenLoadIsHigherInTheLast15Minutes(8.5),
  • Here you need to know about Unix CPU load numbers. In short, it's the processor load over the last minute, 5 minutes, and 15 minutes. 1 represents one core. This metric is useful for timely problem detection, like stuck processes, etc. Set limits based on your server's metrics! No need to enable locally.
Health::checks
DebugModeCheck::new()
    ->unless($isLocal),
  • Very useful on servers. At a minimum, it will alert you during the first deployment because DevOps usually doesn't pay attention to your DEBUG_MODE. And at maximum, if you did have to enable it manually, it will remind you until you disable it. Also, DEBUG_MODE exposes a lot of critical information. DO NOT ENABLE IT ON SERVERS!
Health::checks
RedisMemoryUsageCheck::new()
                ->unless($isLocal)
                ->failWhenAboveMb(3000),
  • Helps identify any stuck cache entries. And of course, if you use cloud solutions, it will immediately hint if something is wrong and money is about to fly away.
Health::checks
SecurityAdvisoriesCheck::new()
    ->unless($isLocal),
  • I talked about security checks as part of the PHP Insigts package. And this check is a pleasant surprise. Absolutely useful if the project is already released and not updated every day. It allows you to react to vulnerabilities earlier than unscrupulous competitors.
Health::checks
SslCertificationExpiredCheck::new()
    ->unless($isLocal)
    ->url($appUrl)->warnWhenSslCertificationExpiringDay(7)
    ->failWhenSslCertificationExpiringDay(3),
SslCertificationValidCheck::new()
    ->unless($isLocal)
    ->url($appUrl),
  • Very useful to know and react promptly to expired certificates. This check is often present in external monitors, so you can use them. And if the project has many domains, specify each one.
Health::checks
OptimizedAppCheck::new()
    ->unless($isLocal),
  • Essential for prod and stage environments. Developers often forget about optimization. DevOps doesn't care at all. And of course, there are cases when a developer clears the cache directly on the server and forgets to revert it.
Health::checks
CacheCheck::new()
   ->unless($isLocal)
  • Checks if everything is okay with the cache. Theoretically overlaps with the Redis check. But still detects configuration and directory access issues if any.

Other checks

  • BackupsCheck - if you store a database or files on the server with the project. I don't recommend it. Use S3 for files, Cloud for databases. If you still use such backups, of course, check them.
  • DatabaseCheck - useful if you have more than one database. Or don't use the database as a storage for checks. In my case, the check will fall anyway if the database dies.
  • EnvironmentCheck - sounds okay, but I absolutely don't understand how to use it in practice. Check only if it's prod?
  • FlareErrorOccurrenceCountCheck - specific if you use Flare as a bug tracker.
  • MeiliSearch - if you use MeiliSearch specifically for full-text search. For others, PingCheck is enough.
  • PingCheck - useful for microservices, external HTTP services, and so on. I'll use it when needed.
  • QueueCheck - makes sense to hang on queues with an especially important tag or when not using HorizonCheck. Or several queue services.
  • EnvVars - an interesting check, it will help catch accidental deletion of some configuration. But I don't see the point in checking every minute (maybe I'm wrong). Theoretically, only during deployment will it be okay.

Telescope setup

На жаль багато з health функціоналу не вдалося ігнорувати в Laravel Telescope. Але деякі речі можна

config/telescope.php
Watchers\CommandWatcher::class => [
        'enabled' => env('TELESCOPE_COMMAND_WATCHER', true),
        'ignore' => [
            'health:check-property',
            'health:check',
            'health:schedule-check-heartbeat'
        ],
    ],