Introducing Laravel Horizon - a Dashboard for your Queues

At Laracon 2017, today, Taylor introduced the latest package in the Laravel world. Like Cashier, Scout, and Passport before it, Horizon is an open-source package that you can bring into your apps but isn't distributed with the core.

Update: After I wrote this post, Taylor also released an intro post on Medium covering a lot of these topics from the official angle.

What is Horizon?

Horizon is a package for configuring and understanding your queues. It provides you control, insight, and analytics into the number of queues and queue workers you have, your failed jobs, and your job throughput. Horizon makes it easy to configure your queues and see how they're doing.

Code-based configuration

Using code-based configuration, just like you're used to with any other Laravel apps and components, you can tell Horizon how many supervisors to run and for each define which connection they'll use, which queues they should operate on, which mechanism to use for balancing the work, and the maximum number of processes they can spin up.

Horizon makes it easy to define all of these settings uniquely for each environment.

<?php

// horizon config file
[
// ...,
'environments' => 
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default'],
            'balance' => 'simple',
            'processes' => 10,
            'tries' => 3
        ]
    ],

    'local' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default', 'notifications'],
            'balance' => 'simple',
            'processes' => 20,
            'tries' => 3,
            'min-processes' => 5, // optional config
        ]
    ]


// ...

'waits' => ['redis:default' => 5] // If I read this syntax correctly, sets how long to wait before consider queue "backed up"
];

Commands

There are a few commands you can pass to Horizon. Here are those Taylor shared:

php artisan horizon:pause # pause but not stop worker
php artisan horizon:continue # resume after pause
php artisan horizon:terminate # gracefully stop during deploy process
php artisan horizon:snapshot # take a metrics snapshot; cron as often as you want

High-level analytics

Horizon provides you with a few key metrics on your entire queue:

• Jobs per minute
• Jobs in the past hour
• Failed jobs in the past hour
• Queue worker status
• Total processes
• Max wait time for your jobs
• Max run time for your jobs
• Max throughput for your jobs

It'll also show you a list of the supervisors you have running, how many processes they're supervising, which queues they're operating on, and whether they're a balancing supervisor.

Job- and queue- specific analytics

Horizon provides throughput-over-time and runtime-over-time graphs for each of your individual queued jobs.

Note: This also works for anything else that's queued. Event listeners, notifications, queued mail, etc.

Tags and Monitoring

Horizon makes it simple to "tag" your queued jobs and to monitor for given tags, giving you even better insight into certain classes of jobs or certain users.

To tag a job, add it to the tags() method on the job:

class MyJob
{
    // ...
    public function tags()
    {
        return ['videos', 'video:' . $this->video->id];
    }
}

Then later you can go to the tag monitoring and choose to pull just specific tags; e.g. customer emails having trouble with Invoice 14; you can monitor Invoice:14 tag and just watch it for a bit to see what's happening, failing, etc.

If you don't provide a tags method, Laravel will auto-tag jobs by the Eloquent model Ids; if you have a "Video" eloquent model attached to your job with the ID of 4, your job will automatically get the tag App\Video:4 applied to it.

Recent jobs

Horizon keeps track of the most recent queued jobs and shows all of the important information you might want to know about each: which queue they ran on, which tags they had, when they were queued, how long they took to run, and whether or not they succeeded.

Failed jobs

For each failing job, Horizon tracks the stack trace and all of the relevant data, making it easy to understand the reason for the failure. You can also choose to retry a failed job after resolving whatever caused it to fail.

If you retry a job, you can see all of the additional tries after the first to see how they did (or didn't) change.

Even if you're not monitoring a tag, you can search the "Failed Jobs" list for any tags—allowing you to debug after the fact.

Failed jobs are retained for seven days (configurable) and the ability to search their tags is retained for 48 hours. All of this metadaa is stored directly in Redis.

Queue balancing strategies

You may have noticed that one of the options in the configuration is "balance", which is set to "simple" in each of the given examples. Queue balancing describes strategies to handle how Horizon splits resources between two queues.

• simple splits the processes between the two, regardless of the workload
• auto auto-balances your queue workers based on the number of remaining jobs and average wait time on each queue

Notifications

Horizon can send SMS or Slack messages to notify the app owner if the wait is getting long on a queue.

// AppServiceProvider
Horizon::routeSlackNotificationsTo('slack endpoint');
Horizon::routeSmsNotificationsTo('phone number');

"Long" is defined via a "waits" configuration setting.

Authentication

On first installation, dashboard is local-only.

You can also choose who has access to it:

// AppServiceProvider
// Choose who can see the dashboard
Horizon::auth(function ($request) {
    return true;
});

Conclusion

I'm looking forward to diving into Horizon as soon as I can get my hands on it, and I'll be sure to write up anything I haven't covered here. But as someone who runs quite a few projects that rely on queues, I'm very much looking forward to adding Horizon to everything.