Laravel Forge - Adding a Queue Worker with Beanstalkd

Laravel has a suite of features offering the ability to push data and/or functionality onto queues, and to assign "workers" to pull items off the queue one-by-one and operate on them.

There a plenty of use cases for this, but one common one is that it saves your users from waiting while the server peforms a complex operation like processing an uploaded image. With queues, the user's interaction pushes the "process image" task (along with any details about the particular image, the user's id, etc.) onto a queue. Then it releases the user back to whatever they were doing. Meanwhile, the queue worker is silently popping one more task off the top of the queue, acting on it, deleting it, and moving on to the next.

There are many different drivers for Laravel; the default is "sync", which just runs the code as if there were no queue. You can use queues on external services, like Iron.io and Amazon SQS. Or you can run your own queue locally, which is what Laravel Forge provides with beanstalkd. In addition, Forge makes it simple to start up a queue worker to run down any of your queues, whether locally on beanstalkd or remotely on Iron.io, and act on them.

Warning: No detectEnvironment closures

I've written about this before, but using a Closure (often with environment variables) to detect your environment instead of an array of hostnames is a little bit of a second-class citizen in Laravel and even more so in Forge. By this, I mean this won't work in Forge:

$env = $app->detectEnvironment(function() {
   return getenv('APP_ENV') ?: 'production';
});

Well, this is even more true with Queue Workers--so much that, unless someone shows me what I'm missing, I'm ready to say it is impossible (or at least wildly impractical) to use Laravel beanstalkd queues together with a detectEnvironment Closure. Not only does the detectEnvironment Closure not have the environment variables available to it, but when you're running commands from a queue, it ignores detectEnvironment entirely if you're using a Closure.

I could only get my push queues to work (including correctly detecting environment) if A) I switched detectEnvironment to use a hostnames array or B) was satisfied with the environment always being "production" (which is fallback response if you use a Closure for your detectEnvironment).

I would guess this is more of a bug or an oversight than an intentional design decision. Or, it's me just doing something wrong. I hope to dig through the source soon and try to wrap my brain around the Artisan bootstrap to understand it better. But for now: If you use a closure to detect your environment, and the environment name for your Forge server isn't "production," you'll have to hold off on this for now. (Am I wrong? Please let me know!)

Writing your code

Assuming we're OK with either A) detecting environment using hostname or B) defaulting to the environment name 'production' for all queues, we're ready to go.

First, you'll want to write a Job Handler, which is really any class with a 'fire' method (and you can even customize which method name gets called). At the end of the fire method, make sure to delete that job so it's removed from the qeueue. You can learn a lot more about Laravel queues at the docs.

namespace Company\Twitter;

class ProfilePuller
{
    public function fire($job, $data)
    {
        // do something with $data['twitter_handle']

        $job->delete();
    }
}

Now that you have a job handler, you'll want to push a job up to your queue, referencing that class and passing some data, somewhere in your code--in a controller, for example:

Queue::push('Company/Twitter/ProfilePuller', [
    'twitter_handle' => 'stauffermatt'
]);

At this point, if you run your code that triggers this queue it's going to work perfectly. Wait, is it that easy to get your queues set up?

Not quite. The Laravel default queue driver is 'sync', which means "run this code synchronously as if we weren't using queues at all." When the controller hits that Queue::push line, it runs the code in your job handler just like it was inline code. But we want it to run asynchronously.

Updating your config

The next step is telling your app to use your beanstalkd queue instead of the 'sync' queue.

Find the config files for your present environment. For me it was app/config/forge/queue.php (create this file and structure it like the default queue.php if it doesn't exist).

The queue config file has a parameter that's named 'default', which is set to 'sync'. If you've ever edited your app's database settings, this format will be very familiar. Change 'sync' to 'beanstalkd' and your queue pushes will now hit your Forge Beanstalkd queue.

return array(
    'default' => 'beanstalkd'
);

Requiring Pheanstalk

There's a composer package that Laravel requires in order to interact with beanstalkd: pda/pheanstalk. Add this to your composer.json and install it.

$ composer require pda/pheanstalk

Starting the worker

Push your code up to your Forge server.

Log into Forge, click through the interface to your Site, click the Queue Workers tab, and click Start Worker with all the defaults still entered. These defaults will start a worker that uses the beanstalkd driver, the "default" queue, and some default timeouts and tolerances. You now have a worker up and running, hitting your default queue on your beanstalkd server, managed and kept running by Supervisor.

Watching the worker do its thing

That's it! Go trigger your Queue::push code from earlier. That'll push the queue task up onto beanstalkd, the worker will pull it down and act on it, and then delete it, and the queue will be clean again. You're good to go, and now your users can breeze around your app, unaware of the raw processing power being thrown at their tasks. There are also plenty of other use cases for queues, but other people (and the docs) have already covered that well.

If you are having any trouble, or want to see evidence of the queue working, go check your logs--which, of course, by this point, are logging to Papertrail, right? Nice and easy.