Laravel Queues: An Asynchronous Processing

This article aims to guide you through the process of executing jobs asynchronously using queues and workers in Laravel.

When it comes to web applications and softwares, speed is crucial for a satisfactory user experience. However, certain tasks require more time to complete, such as sending automated emails, importing large data from CSV or excel files or generating complex reports from extensive data sets. These kinds of tasks cannot be performed synchronously. The well-designed web applications utilize such asynchronous processing in the background, which can be performed using a capability provided by Laravel Queues.

Prerequisites:

This article aims to guide you through the process of executing jobs asynchronously using queues and workers in Laravel. Prior to diving in, please ensure that you have PHP, Composer and MySQL installed on your system, and that you are familiar with bootstrapping and initiating a new Laravel project.

The Concept:

Before delving into the code, I would like to describe the concept of asynchronous processing with a real-life analogy. Imagine yourself as the owner of a local grocery shop that accepts orders over the phone and delivers the requested items to customers’ doorsteps.

Now, assume that you receive a phone call from a customer and take a new order. You write this order on the piece of paper. You then proceed to the storeroom, locate the individual products, pack them into a box, and arrange for delivery. In this scenario, you cannot accept any new orders until you have dispatched the current one. This is called synchronous approach. It blocks your ability to handle other tasks until you complete the currently running process at hand.

However, instead of sticking to a synchronous model, you have the option to hire an additional worker for the storeroom. By doing so, you can accept a new order, jot it down on a piece of paper (referred to as a job), and place it in a queue. One of the storeroom workers will pick a job from the queue, assemble the order, and arrange for delivery. Afterwards, the worker will return to the queue, select the next job, and commence its processing. This is called an asynchronous approach. It  allows for a more efficient workflow. While employing extra workers may require additional resources, it enables you to handle more orders without impeding your overall productivity. This, in turn, leads to improved performance and heightened customer satisfaction.

The Basic Implementation:

Now, let’s see how to implement this asynchronous approach of a job, a worker, and a queue in Laravel. To start, create a new Laravel project somewhere on your computer. Open the project in your favourite IDE and update the code for the routes/web.php file as follows:

Route::get('/', function () {

    dispatch(function() {
        sleep(5);
        logger('job done!');
    });

    return view('welcome');
});

The dispatch() helper function in Laravel sends a new job to a queue. Here, a job is regular PHP code in the form of a closure or class. You can also use job classes instead of closures in above code.

To simulate a long-running task, I have added an explicit delay of five seconds using the sleep function in the job closure. Then, I used the logger() helper function to print the line “job done!” in the project’s laravel.log file located at storage/logs/laravel.log.

Now, start the project by running the following command:

php artisan serve

And open http://127.0.0.1:8000 in your browser. You will see the browser loading for five seconds, and then the welcome page will show up. If you check the log file at location storage/logs/laravel.log, you will see the following log entry:

[2021-08-23 14:37:50] local.DEBUG: job done!

This means that the job ran successfully after the five-second delay, but still it is not running asynchronously. It is still blocking the I/O operations as the browser was in a loading state for five seconds. It is because of the QUEUE_CONNECTION variable in the project’s .env file. This variable indicates the connection to be used in the queue service in Laravel. By default, it has been set to sync, which means that Laravel will process all the jobs synchronously.

Asynchronous Approach:

For asynchronous job-processing, you will have to use a different connection. Laravel has provided a pre-configured list of connections inside the config/queue.php file as follows:

'connections' => [

    'sync' => [
        'driver' => 'sync',
    ],

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 90,
        'after_commit' => false,
    ],

    'beanstalkd' => [
        'driver' => 'beanstalkd',
        'host' => 'localhost',
        'queue' => 'default',
        'retry_after' => 90,
        'block_for' => 0,
        'after_commit' => false,
    ],

    'sqs' => [
        'driver' => 'sqs',
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
        'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
        'queue' => env('SQS_QUEUE', 'default'),
        'suffix' => env('SQS_SUFFIX'),
        'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
        'after_commit' => false,
    ],

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => env('REDIS_QUEUE', 'default'),
        'retry_after' => 90,
        'block_for' => null,
        'after_commit' => false,
    ],
],

You can use any connection except sync,which is used for synchronous job-processing. For this example, we will use the database connection. To update the connection, open the project’s .env file and change the value of QUEUE_CONNECTION to database and save the file.

But, to work with database connection, we have to create a table to maintain our asynchronous jobs. To create the migration for this table, run the following command:

php artisan queue:table

It will generate the migration script for creating the jobs table. Open the newly generated migration script, it will have the following content:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    /**
     * Run the migrations.
     */
    public function up(): void
    {
        Schema::create('jobs', function (Blueprint $table) {
            $table->bigIncrements('id');
            $table->string('queue')->index();
            $table->longText('payload');
            $table->unsignedTinyInteger('attempts');
            $table->unsignedInteger('reserved_at')->nullable();
            $table->unsignedInteger('available_at');
            $table->unsignedInteger('created_at');
        });
    }

    /**
     * Reverse the migrations.
     */
    public function down(): void
    {
        Schema::dropIfExists('jobs');
    }
};

Now, run the migration command as follows to create the jobs table in the database:

php artisan migrate

This table will hold all the queued jobs until a worker processes them. Before starting the laravel, clear the project’s log file located at storage/logs/laravel.log and run the following command:

php artisan serve

And open http://127.0.0.1:8000 in your browser. This time, the page will render almost immediately. After waiting for five seconds, if you check the storage/logs/laravel.log file, you’ll find it empty. Because, the above setup will only add the job into the jobs table. But, execution of this job is still pending.

If you look at the jobs table in your database, you’ll see that the framework has pushed a new job to the queue. To execute this job, we have to run the Laravel queue worker process.

Laravel Queue Worker:

A queue worker is a regular process that runs in the background and polls the queue for unprocessed jobs. To start a new worker, execute the following command inside your project directory:

php artisan queue:work

The worker will start and begin processing the unprocessed job immediately. Now, check the storage/logs/laravel.log file. It will display the following logs: 

[2021-08-23 15:30:34][1] Processing: Closure (web.php:18)
[2021-08-23 15:30:39][1] Processed:  Closure (web.php:18)

Note: To process jobs automatically in the future, you’ll have to keep the worker process running.

And that concludes this article. I have covered the basic concepts of how synchronous and asynchronous approach are working and how you can use Laravel Queues to run asynchronous jobs. There are numerous other aspects of queue management that you should familiarize yourself with, such as job uniqueness, handling race conditions, and implementing throttling. It is advisable to begin incorporating queues into your applications and learn through hands-on experience. Additionally, make sure to thoroughly explore the official documentation on Laravel Queues. If you have any questions or concerns about any of the concepts presented in this article, please feel free to reach out to me on LinkedIn, or GitHub. Until the next article, stay safe and continue your learning journey.

Laravel order by relation column with example

Some examples that demonstrate the usage of Laravel’s orderBy method with relationship columns, which can be applied in Laravel versions 5, 6, 7, 8, 9, and 10.

In this post, we focus on examples to use Laravel order by relation column. There are many different ways to achieve it.

Below are some examples that demonstrate the usage of Laravel‘s orderBy method with relationship columns. These examples can be applied in Laravel versions 5, 6, 7, 8, 9, and 10.

Example 1: Ordering by a BelongsTo Relationship Column

Consider the scenario where you have a "User" model that belongs to a "Role" model. You can use the orderBy method to sort the users based on the role name in ascending order.

$users = User::with(['role' => function ($q) {
    $q->orderBy('name');
}])->get();

You can use the same code for descending order by adding 'desc' argument to orderBy method as follows,

$users = User::with(['role' => function ($q) {
    $q->orderBy('name', 'desc');
}])->get();

For above example to work properly, you have to define belongsTo relationship of "Role" model inside the "User" model as follows,

public function role(): BelongsTo
{
    return $this->belongsTo(Role::class, 'role_id', 'id');
}

Example 2: Using inner join with relation table

For above scenario, where “User” model that belongs to “Role” model, you can use join method to perform user sorting based on role name as follows,

$users = User::select('*')
                 ->join('roles', 'users.role_id', '=', 'roles.id')
                 ->orderBy('roles.name', 'asc');

As above example, you need to replace the second argument of orderBy method to 'desc' for sort records in descending order. There is no need of any relationship required for this query.

Example 3: Using sortBy() and sortByDesc() methods

You can use sortBy() and sortByDesc() methods to order the records for the same scenario as follows,

$users = User::get()->sortBy(function($query){
    return $query->role->name;
})->all();

In above example, it first get the users collection from the database and then sort them by the provided relation column. For these methods, you have to define belongsTo relationship in "User" model.

You can use sortByDesc() method same as above.

Example 4: Using subquery and whereColumn method

You can also use subqueries to sort records. For the above scenario, you can use subqueries as follows,

$users = User::select('*')
    ->orderBy(Role::select('name')
        ->whereColumn('roles.id', 'users.role_id')
    );

In above example, we have we used subquery from roles table using whereColumn method to get the name of the role of each row of users table. After getting the role name, we used orderBy method to achieve the sorted records.

For sorting records in descending order, you can use the above example with orderByDesc method.

You can use this example, without defining belongsTo relationship in model.

These are some useful examples to order records based on relationship model in Laravel.

Using Laravel to upload a file

Uploading a file in any programming is a challenge. So, we focus on uploading a file and some validations to use with file upload using Laravel.

Uploading a file in any programming is a challenge. In this post, we focus on uploading a file and some validations to use with file upload using Laravel.

To upload a file using Laravel, you can follow these steps:

Create a new form in your Laravel view with an input field for the file:

<form method="POST" action="{{ route('file.upload') }}" enctype="multipart/form-data">
    @csrf

    <input type="file" name="file">

    <button type="submit">Upload</button>
</form>

In above code, we have added file.upload route as an action of the form. So, we need to define this route in routes/web.php file. This route should point to the controller method that will handle the file upload.

The following code will define a new route in your routes/web.php file:

Route::post('/file/upload', [App\Http\Controllers\FileController::class, 'upload'])->name('file.upload');

Above code has defined a route, which points to the upload method of the FileController. So, create a new controller FileController and method upload inside that to handle the file upload as follows:

class FileController
{
    public function upload(Request $request)
    {
        // Validate the uploaded file
        $request->validate([
            'file' => 'required|file|max:1024', // limit file size to 1 MB
        ]);

        // Store the uploaded file in the storage/app/public directory
        $path = $request->file('file')->store('public');

        // Generate a URL for the uploaded file
        $url = Storage::url($path);

        // Redirect back with a success message
        return back()->with('success', 'File uploaded successfully: ' . $url);
    }
}

In the upload() method, we first validate that the uploaded file meets our requirements. So, we have added some validations for our file. These validations are as follows:

required:

The field under this validation must be present in the input data and must not empty. A field is “empty” if it meets one of the following criteria:

  • The value is null.
  • The value is an empty string.
  • The value is an empty array or empty Countable object.
  • The value is an uploaded file with no path.

file:

The field under this validation must be a successfully uploaded file.

max:1024:

The field under this validation must be less than or equal to a 1024 bytes. Here, 1024 is value of file size. You can change it according to your requirements.

We then use the store() method on the uploaded file to store it in the storage/app/public directory. This directory is publicly accessible, so we can generate a URL for the file using the url() method on the Storage facade. Finally, we redirect back to the form with a success message that includes the URL of the uploaded file.

You can now test the file upload functionality by navigating to the form and selecting a file to upload. If the file meets the validation requirements, it will be uploaded and a success message will be displayed. You can then access the uploaded file at the generated URL.

Finding bugs using PHPStan as a Static Analyzer

With PHP being an interpreted language it has a downside when it comes to finding bugs in your code. It will not show you errors in your software until you actually run it. PHPStan tries to solve this problem by doing static analysis on your code. It was recently created by Ondrej Mirtes.

Running PHPStan will tell you about bugs in your codebase almost instantly (yes, it’s very fast). At the time of writing this article, PHPStan currently checks your code on:

  • The existence of classes and interfaces in an instance of, catch type hints, other language constructs, and even annotations. PHP does not do this and just stays silent instead.
  • Existence of variables while respecting scopes of branches and loops.
  • Existence and visibility of called methods and functions.
  • Existence and visibility of accessed properties and constants.
  • Correct types assigned to properties.
  • The correct number and types of parameters are passed to constructors, methods, and functions.
  • Correct types returned from methods and functions.
  • The correct number of parameters passed to sprintf/printf calls is based on format strings.
  • Useless casts like (string) ‘foo’.
  • Unused constructor parameters – they can either be deleted or the author forgot to use them in the class code.
  • That only objects are passed to the clone keyword.

As you can see, it contains a lot of useful checks which will warn you of potential bugs before you even run your code.

Installing PHPStan

Installing PHPStan is as easy as including it in your project through composer:

$ composer require --dev phpstan/phpstan

We can now run PHPStan from the base directory of our project:

$ vendor/bin/phpstan analyze -l 4 src

A breakdown of this command:

  • vendor/bin/phpstan is the executable
  • analyze tells PHPStan to analyze all files in the given directories
  • -l 4 means that we want to analyse on the most strict level
  • src is the directory we want to analyse

Try running this in your own project and see what kind of potential errors are living in your codebase.

Integrating PHPStan into CI

It’s super easy to use PHPStan in Continuous Integration. For most of my personal projects, I use TravisCI. Since we’ve included PHPStan as a dev-dependency in our composer.json file we just have to add the PHPStan executable to the scripts that the CI-software needs to run.

For TravisCI, this means just changing the default script in a .travis.yml like this:

language: php
php:
  - '8.0'
install: composer install

# Simply add these lines
script:
    - vendor/bin/phpunit
    - vendor/bin/phpstan analyse src tests --level=4

The default script that TravisCI runs for PHP projects is simply phpunit. Now we’ve added PHPStan to it. If PHPStan finds any errors within your project, the build will fail.

What are ORM Frameworks?

ORM is a short form of Object Relational Mapping, which means as ORM framework is written specifically in OOP (object-oriented programming) language (like PHP, C#, Java, etc…) and it is like a wrapper around a relational database (like MySQL, PostgreSQL, Oracle, etc…). So, ORM is basically mapping objects to relational tables.

What does an ORM framework do?

The ORM framework generates objects (as in OOP) that virtually map the tables in a database. So, any programmer could use these objects to interact with the database without writing an optimized SQL code.

For example:

We have 2 tables in a database:

  • Products
  • Orders

The ORM framework would create 2 objects corresponding to the above tables (like products_object and orders_object) with little configuration, which will handle all the database interactions. So, if you want to add a new product to the products table, you would have to use the products_object and save() method like below,

product = new products_object("Refrigerator","Electronics");
product.save();

You can see, how much easier an ORM framework can make things. No need to write any SQL syntax. And the application code would be very clean.

Some other advantages of using ORM frameworks

1. Syncing between OOP language and the relational database data types is always creating a problem. Sometimes variable data types have to be converted properly to insert into the database. A good ORM framework will take care of these conversions.

2. Using an ORM will create a consistent code base for your application since no SQL statements are written in the code. This makes it easier to write and debug any application, especially if more programmers are using same code base.

3. ORM frameworks will shield your application from SQL injection attacks since the framework will be filtering the data before any operation in the database.

4. Database Abstraction; Switching databases for the application is easier as, ORM will take care of writing all the SQL code, data type conversions etc …

When to use an ORM framework?

An ORM framework becomes more useful as the size and complexity of the project increases. An ORM framework may be overkilling an application on a simple database with 5 tables and 5-6 queries to be used for the application.

Consider the use of ORM when:

  • 3 or more programmers are working on an application.
  • Application database consists of 10+ tables.
  • The application is using 10+ queries.

About 80-90% of application queries can be handled by the ORM generated objects. It is inevitable that at some point straight SQL query is required, which can’t be handled by ORM generated objects.

In fact, ORM frameworks often have their own *QL query language that looks a lot like SQL. Doctrine, a popular PHP based ORM framework has DQL (Doctrine Query Language) and the very popular Hibernate (used in the Java and .Net world) has HQL. Going even further, Hibernate allows writing straight SQL if need be.

ORM Frameworks for PHP programmers

  • CakePHP, ORM, and framework for PHP 5
  • CodeIgniter, a framework that includes an ActiveRecord implementation
  • Doctrine, open source ORM for PHP 5.3.X
  • FuelPHP, ORM, and framework for PHP 5.3. Based on the ActiveRecord pattern.
  • Laravel, a framework that contains an ORM called “Eloquent” an ActiveRecord implementation.
  • Maghead, a database framework designed for PHP7 includes ORM, Sharding, DBAL, SQL Builder tools etc.
  • Propel, ORM and query-toolkit for PHP 5, inspired by Apache Torque
  • Qcodo, ORM, and framework for PHP 5
  • QCubed, A community-driven fork of Qcodo
  • Redbean, ORM layer for PHP 5, creates and maintains tables on the fly
  • Yii, ORM, and framework for PHP 5. Based on the ActiveRecord pattern.
  • Zend Framework, a framework that includes a table data gateway and row data gateway implementations. ZendDb