Managing Laravel Work Queues With Beanstalk and Supervisor on Ubuntu 16.04

Published on: Fri, May 4, 2018 at 11:10 am EST
Linux Guides PHP Programming System Admin Ubuntu

Beanstalk is a fast and simple work queue. It allows you to run time-consuming tasks asynchronously, such as sending emails, connecting to external APIs or processing images. By doing so, you will reduce your web app latency. Laravel provides out-of-the-box support for beanstalkd.

In this tutorial we will install beanstalkd, setup a demo Laravel application and manage the queue workers through Supervisor. The demo application will get a list of available OSes from the Vultr API and randomly choose one.

It is assumed that you already have Supervisor installed on your system. If you do not, you should read the following tutorial on how to setup Supervisor: Installing and Configuring Supervisor on Ubuntu 16.04

It is also assumed that you have SSH access to your Vultr instance.

Installing Beanstalk

The first step to take is to install beanstalkd.

sudo apt-get update
sudo apt-get install beanstalkd

Start the service.

sudo systemctl start beanstalkd

You may also enable the service to start upon system initialization.

sudo systemctl enable beanstalkd

Check the service status by issuing the following command.

sudo systemctl status beanstalkd

By default, beanstalkd listens on the port 11300. Beanstalk uses a simple text-based protocol described in it's github repository. You can test it's protocol by running telnet.

telnet localhost 11300

Write the following and hit ENTER.

list-tubes

You should see a list of tubes available on the server:

OK 14
---
- default

To close the connection simply type quit and then press ENTER.

Tubes in Beanstalk represent work queues. Beanstalk is composed basically by producers, consumers, jobs and tubes. Producers put jobs into a tube to be consumed (processed) by any number of consumers. Note that both producers and consumers are simply clients of the Beanstalk server and are totally independent of each other. In practical terms this means that by using Beanstalk you may produce your jobs in your PHP application and have it processed in a NodeJS app for example. Luckily, Laravel abstracts all of this and provides us with a very simple API to dispatch and handle jobs, as we will see next.

Sample project

To build our sample project, we first need to install its dependencies. We are going to install PHP and Composer.

sudo apt-get install php php-mbstring php-xml php-common php-zip composer

Now, create the project based on Laravel 5.5.

composer create-project --prefer-dist laravel/laravel vultr "5.5.*"

Next cd into the recently created folder. We will be working in this directory from now on. This folder may be referenced later as PROJECT_ROOT.

cd vultr/

To use Beanstalk in Laravel we need to install one more dependency, the PHP Beanstalk client.

composer require pda/pheanstalk ~3.0

We also need to create a Job representation. In Laravel, this is a class inside the app/Jobs folder. Laravel has a console command to help us create a Job class. Let's create our sample Job.

php artisan make:job FindFavoriteOS

Update the app/Jobs/FindFavoriteOS.php file to the following.

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\Log;

class FindFavoriteOS implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $rawData = file_get_contents('https://api.vultr.com/v1/os/list');
        $list = json_decode($rawData, true);
        shuffle($list);
        $key = array_rand($list);
        $favorite = $list[$key];
        Log::info('My Favorite OS is: ' . $favorite['name']);
    }
}

The handle method is what will be effectively executed when the job is consumed. Here, we fetch data from the Vultr API, randomly pick an OS and write the chosen OS to the log file. The log file is located at storage/logs/laravel.log.

We have already installed Beanstalk and defined the Job to be executed by it. Now it is time to tell Laravel to use Beanstalk as the default work queue. Copy the default .env file provided by the Laravel installation.

cp .env.example .env

Now open the .env file and update the line where the queue driver is specified.

QUEUE_DRIVER=beanstalkd

Finally, generate an application key.

php artisan key:generate

We are now ready to dispatch jobs to the Beanstalk work queue.

Dispatching and running jobs

Dispatching a job in Laravel is quite simple. Update the routes/web.php file.

<?php

Route::get('/', function () {
    for ($i = 0; $i < 50; $i++) {
        \App\Jobs\FindFavoriteOS::dispatch();
    }

    return '50 Jobs dispatched!';
});

Despite not being recommended to run the built-in provided server in production, we will be using it here for the sake of brevity.

php artisan serve --host 0.0.0.0 --port 8000

Now in a web browser, navigate to http://[vultr-instance-ip]:8000. You will see the following message.

50 Jobs dispatched!

Open a new SSH connection to your server and inside of our project root, execute the following.

php artisan queue:work --once

This is the expected output:

[2018-02-14 00:03:52] Processing: App\Jobs\FindFavoriteOS
[2018-02-14 00:03:53] Processed:  App\Jobs\FindFavoriteOS

Confirm that the log was generated.

cat storage/logs/laravel.log

Configuring Supervisor

To avoid the need to manually process the queue, we will use supervisord. Create the following program configuration in /etc/supervisor/conf.d/vultr.conf.

[program:vultr]
process_name=%(program_name)s_%(process_num)02d
command=php [PROJECT_ROOT]/artisan queue:work
autostart=true
autorestart=true
numprocs=8
redirect_stderr=true
stdout_logfile=/var/log/worker.log

Notice that the right path to put the Supervisor configuration file will depend on your setup. Also, remember to replace [PROJECT_ROOT] with the full path to the PROJECT_ROOT on your system.

Here we are configuring Supervisor to automatically start processing the queue and, in case of the script breaking, to restart it. Also note that we are not instantiating a single worker, but 8 processes. You are free to instantiate however many processes you find necessary depending on your application.

To allow Supervisor to manage our worker, force it to reread its configuration.

sudo supervisorctl reread
sudo supervisorctl update

If the Supervisor service has not been started, you many need to start it.

sudo systemctl start supervisord

Now let's see if the jobs are being processed.

tail -f storage/logs/laravel.log

On a web browser, navigate to http://[vultr-instance-ip]:8000. You will see the logs being generated on your console.

Conclusion

We have successfully configured a Laravel application to use Beanstalk as a work queue. Also, we demonstrated how to use Supervisor to manage workers.

Want to contribute ?

You could earn up to $300 by adding new articles