Category: Tools

When it comes to web applications and softwares, speed is crucial for a satisfactory user experience. However, certain tasks require more time to complete, such as sending automated emails, importing large data from CSV or excel files or generating complex reports from extensive data sets. These kinds of tasks cannot be performed synchronously. The well-designed web applications utilize such asynchronous processing in the background, which can be performed using a capability provided by Laravel Queues.

Prerequisites:

This article aims to guide you through the process of executing jobs asynchronously using queues and workers in Laravel. Prior to diving in, please ensure that you have PHP, Composer and MySQL installed on your system, and that you are familiar with bootstrapping and initiating a new Laravel project.

The Concept:

Before delving into the code, I would like to describe the concept of asynchronous processing with a real-life analogy. Imagine yourself as the owner of a local grocery shop that accepts orders over the phone and delivers the requested items to customers’ doorsteps.

Now, assume that you receive a phone call from a customer and take a new order. You write this order on the piece of paper. You then proceed to the storeroom, locate the individual products, pack them into a box, and arrange for delivery. In this scenario, you cannot accept any new orders until you have dispatched the current one. This is called synchronous approach. It blocks your ability to handle other tasks until you complete the currently running process at hand.

However, instead of sticking to a synchronous model, you have the option to hire an additional worker for the storeroom. By doing so, you can accept a new order, jot it down on a piece of paper (referred to as a job), and place it in a queue. One of the storeroom workers will pick a job from the queue, assemble the order, and arrange for delivery. Afterwards, the worker will return to the queue, select the next job, and commence its processing. This is called an asynchronous approach. It  allows for a more efficient workflow. While employing extra workers may require additional resources, it enables you to handle more orders without impeding your overall productivity. This, in turn, leads to improved performance and heightened customer satisfaction.

The Basic Implementation:

Now, let’s see how to implement this asynchronous approach of a job, a worker, and a queue in Laravel. To start, create a new Laravel project somewhere on your computer. Open the project in your favourite IDE and update the code for the routes/web.php file as follows:

Route::get('/', function () {

    dispatch(function() {
        sleep(5);
        logger('job done!');
    });

    return view('welcome');
});

The dispatch() helper function in Laravel sends a new job to a queue. Here, a job is regular PHP code in the form of a closure or class. You can also use job classes instead of closures in above code.

To simulate a long-running task, I have added an explicit delay of five seconds using the sleep function in the job closure. Then, I used the logger() helper function to print the line “job done!” in the project’s laravel.log file located at storage/logs/laravel.log.

Now, start the project by running the following command:

php artisan serve

And open http://127.0.0.1:8000 in your browser. You will see the browser loading for five seconds, and then the welcome page will show up. If you check the log file at location storage/logs/laravel.log, you will see the following log entry:

[2021-08-23 14:37:50] local.DEBUG: job done!

This means that the job ran successfully after the five-second delay, but still it is not running asynchronously. It is still blocking the I/O operations as the browser was in a loading state for five seconds. It is because of the QUEUE_CONNECTION variable in the project’s .env file. This variable indicates the connection to be used in the queue service in Laravel. By default, it has been set to sync, which means that Laravel will process all the jobs synchronously.

Asynchronous Approach:

For asynchronous job-processing, you will have to use a different connection. Laravel has provided a pre-configured list of connections inside the config/queue.php file as follows:

'connections' => [

    'sync' => [
        'driver' => 'sync',
    ],

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 90,
        'after_commit' => false,
    ],

    'beanstalkd' => [
        'driver' => 'beanstalkd',
        'host' => 'localhost',
        'queue' => 'default',
        'retry_after' => 90,
        'block_for' => 0,
        'after_commit' => false,
    ],

    'sqs' => [
        'driver' => 'sqs',
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
        'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
        'queue' => env('SQS_QUEUE', 'default'),
        'suffix' => env('SQS_SUFFIX'),
        'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
        'after_commit' => false,
    ],

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => env('REDIS_QUEUE', 'default'),
        'retry_after' => 90,
        'block_for' => null,
        'after_commit' => false,
    ],
],

You can use any connection except sync,which is used for synchronous job-processing. For this example, we will use the database connection. To update the connection, open the project’s .env file and change the value of QUEUE_CONNECTION to database and save the file.

But, to work with database connection, we have to create a table to maintain our asynchronous jobs. To create the migration for this table, run the following command:

php artisan queue:table

It will generate the migration script for creating the jobs table. Open the newly generated migration script, it will have the following content:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    /**
     * Run the migrations.
     */
    public function up(): void
    {
        Schema::create('jobs', function (Blueprint $table) {
            $table->bigIncrements('id');
            $table->string('queue')->index();
            $table->longText('payload');
            $table->unsignedTinyInteger('attempts');
            $table->unsignedInteger('reserved_at')->nullable();
            $table->unsignedInteger('available_at');
            $table->unsignedInteger('created_at');
        });
    }

    /**
     * Reverse the migrations.
     */
    public function down(): void
    {
        Schema::dropIfExists('jobs');
    }
};

Now, run the migration command as follows to create the jobs table in the database:

php artisan migrate

This table will hold all the queued jobs until a worker processes them. Before starting the laravel, clear the project’s log file located at storage/logs/laravel.log and run the following command:

php artisan serve

And open http://127.0.0.1:8000 in your browser. This time, the page will render almost immediately. After waiting for five seconds, if you check the storage/logs/laravel.log file, you’ll find it empty. Because, the above setup will only add the job into the jobs table. But, execution of this job is still pending.

If you look at the jobs table in your database, you’ll see that the framework has pushed a new job to the queue. To execute this job, we have to run the Laravel queue worker process.

Laravel Queue Worker:

A queue worker is a regular process that runs in the background and polls the queue for unprocessed jobs. To start a new worker, execute the following command inside your project directory:

php artisan queue:work

The worker will start and begin processing the unprocessed job immediately. Now, check the storage/logs/laravel.log file. It will display the following logs: 

[2021-08-23 15:30:34][1] Processing: Closure (web.php:18)
[2021-08-23 15:30:39][1] Processed:  Closure (web.php:18)

Note: To process jobs automatically in the future, you’ll have to keep the worker process running.

And that concludes this article. I have covered the basic concepts of how synchronous and asynchronous approach are working and how you can use Laravel Queues to run asynchronous jobs. There are numerous other aspects of queue management that you should familiarize yourself with, such as job uniqueness, handling race conditions, and implementing throttling. It is advisable to begin incorporating queues into your applications and learn through hands-on experience. Additionally, make sure to thoroughly explore the official documentation on Laravel Queues. If you have any questions or concerns about any of the concepts presented in this article, please feel free to reach out to me on LinkedIn, or GitHub. Until the next article, stay safe and continue your learning journey.