On the "Oh Dear!" service's blog there's a recent post covering the use of queues in a Laravel application and how to size and scale them to most effectively use the resources you have.

Laravel offers a convenient way to create asynchronous background tasks using its queues. We utilize those heavily at Oh Dear! for all our monitoring jobs and in this post we'll share some of our lessons learned and what we consider to be best practices.

The post starts out with an introduction to queues and their handling in Laravel applications, pushing jobs into workers to be executed. While a simple single-worker queue can handle a decent amount of work, problems can arise as the work grows larger. He mentions splitting up the work as a potential solution and defines the differences between "fast" and "slow" jobs to act as a dividing line for the queues. He also makes the suggestion of single-purpose queues based on types and, finally, mixing in the idea of a better user experience through immediate feedback.

On the Laravel News site today there's a tutorial posted showing you how to combine Docker and Laravel's Scheduler/Queue and make them still run as they would on a virtual server.

In Laravel, one of the tricky changes when switching from a virtual server to Docker is figuring out how to run a scheduler and a queue worker. I see this question come up quite a bit when PHP developers are trying to figure out how to use Laravel with Docker.

Should you run them on the host server? Should you run via cron in a Docker container?

There are a couple of ways I recommend running the scheduler command and Laravel queues in Docker, and we’re going to cover the basics of running both with a complete (albeit simple) Docker setup you can use to experiment.

Their approach uses a single multi-purpose Docker image rather than splitting the functionality up and making it more complex (Laravel subscribes to the monolithic approach anyway). The post then gets into the setup of this environment using Docker and docker-compose to configure several services: application (app), a Redis container and a MySQL container. The contents of the docker-compose and Dockerfile configurations are included as well as the VirtualHost configuration for the main site. Next it shows the use of the CMD directive to run a bash script when the build is brought up. This is what kicks off the scheduler/queue handling. The post finishes up with a few other changes needed to the Docker configuration and the creation of the "scheduler" service.

Sergey Zhuk has posted the second part of his "fast web scraping" series that makes use of the ReactPHP package to perform the requests. In part one he laid some of the groundwork for the scraper and made a few requests. In this second part he improves on this basic script and how to throttle the requests so as to not overload the end server.

t is very convenient to have a single HTTP client which can be used to send as many HTTP requests as you want concurrently. But at the same time, a bad scraper which performs hundreds of concurrent requests per second can impact the performance of the site being scraped. Since the scrapers don’t drive any human traffic on the site and just affect the performance, some sites don’t like them and try to block their access. The easiest way to prevent being blocked is to crawl nicely with auto throttling the scraping speed (limiting the number of concurrent requests). The faster you scrap, the worse it is for everybody. The scraper should look like a human and perform requests accordingly. A good solution for throttling requests is a simple queue.

He shows how to integrate the clue/mq-react package into the current scraper to interface with a RabbitMQ instance and handle the reading of and writing to the queue. He includes the code needed to update the ReactPHP client. The mq-react package makes the update simple with the HTTP client reading from the queue instance rather than the array of URLs. One the queue is integrated, he then shows how to create a "parser" that can read in the HTML and extract only the wanted data using the DomCrawler component.

The Yappa.be blog has a tutorial posted sharing the method they used to implement queued and scheduled execution of Symfony commands. This is made possible by the Enqueue package.

At Yappa, we have always used Johannes' JMSJobQueueBundle to run and schedule Symfony console commands for background jobs.

However, we've stumbled upon a much more elegant solution called Enqueue. [...] It's packed with features, supports major brokers such as RabbitMQ, Kafka, Amazon SQS, Google PubSub, Redis etc. and has a bundle ready to be used with Symfony.

[...] One downside is that the Enqueue Symfony bundle doesn't provide an out of the box solution to queue Symfony console commands and there's no 100% straight forward way to implement this. In this post I'll cover the basics in setting up the Enqueue Symfony bundle so we can easily queue Symfony console commands!

The tutorial then walks you through the installation of the Symfony bundle, adding it to the list of installed bundles and configuring it with basic parameters and queue connection details. Next they've included the code to create the processor used when a command is pushed to the queue. To handle the other side (execution of the command when the queue is handled) they create a "QueuedCommand" value object and a command handler class. With this structure in place they show the addition of test commands to the queue and the result when the queue is consumed.

On the Tuleap blog there's an article posted showing how they made the switch from RabbitMQ to Redis in their PHP application.

There are various places in Tuleap were message queues are needed but the primary one is to run jobs in background. Historically it was for all system related stuff (creating users, git/svn repositories, etc) that required special unix permissions that we were not eager to grant to a web app.

[...] More recently we add a need to share events across servers, for distributed Tuleap setup for instance. Our good old system was no longer able to deal with that because we needed a real queue management that works across servers. We choose RabbitMQ because we were looking for a queue system, PHP support quite decent and tutorials were good. However, we had to setup a quite complicated queue system.

The post outlines the exchange to exchange binding system they created and includes the code use to make the RabbitMQ-based system work. Then they introduce Redis and how, when they learned about RPOPLPUSH, decided to make the switch. The code for this replacement is also included.

The TutsPlus.com site has a new tutorial posted for the Laravel users showing how to defer tasks to queues using functionality already included in the framework.

In this article, we're going to explore the Queue API in the Laravel web framework. It allows you to defer resource-intensive tasks during script execution to enhance the overall end user experience. After introducing the basic terminology, I'll demonstrate it by implementing a real-world example.

The article starts by talking about some of the advantages of using queues to help with reducing page load times and improving the overall performance/quality of your site's user experience. It then starts in on the Laravel-specific pieces of the queuing puzzle including the queue drivers, connection types and the queues themselves. Next they move on to the code, showing how to create a simple first job that will manipulate the images a user uploads. The post includes the code to create the "jobs" table, build the "Images" model and the "ProcessImageThumbnails" job that will build the smaller thumbnail images from larger ones. Finally they create the controller and views for the upload handling, show how to delegate it to the job for the image processing and run the queue processor do to the actual work.

On the Laravel News site today they have a post announcing a new member of the Laravel family that was announced at the current Laracon conference: Laravel Horizon.

The moment everyone in the Laravel community has been waiting for has finally arrived! Laravel Horizon is software to “supercharge your queues with a beautiful dashboard and code-driven configuration.”

[...] Laravel Horizon is designed to make it easy monitor your queues from a web interface and receive notifications when things go wrong.

They list out some of the key features including auto-balancing, code-driven configuration, queue monitoring and a notification system. They briefly describe each of these features and what they see as the "most awesome part" - that the entire thing is open source and 100% free.

On the Delicious Brains site they've posted a tutorial showing how to create a command line daemon that will emulate the Amazon Web Services SQSD handling. The SQSD is a worker daemon service that Amazon offers as a part of its Elastic Beanstalk support.

Sometimes when you’re building a project there are parts of the architecture that exist on production that don’t exist on your development machine. Those missing parts (like proprietary software that’s specific to your hosting provider) can sometimes mean unwelcome surprises when you deploy to production.

Recently as part of my work on Mergebot, I decided to address this. My local machine was missing the AWS Elastic Beanstalk Worker Environment SQS daemon (known as SQSD). AWS isn’t open source so there’s, unfortunately, no official way to replicate it. So I decided to build a small PHP command line (CLI) app to attempt to replicate its functionality. In this article, I’m going to cover some of the aspects of creating a command line app in PHP and explain how I implemented them for my replica SQSD CLI.

He starts off with a brief overview of the Laravel queue worker and how it compares to the SQSD functionality. He then starts in on the code to create the daemon (outside of a framework) and adding in the while loop to keep it running as a daemon making use of the SQSD Worker class as a base. The post ends with some instructions on packaging up the command line tool using the phar functionality already included in the PHP language.

The Hackernoon.com site has a tutorial posted from Cloudways showing you how to use the queue handling for email verification in a Laravel 5.4 application. The tutorial walks you through the updates and additions you'll need to make to the user signup process to verify their email.

Today, I will demonstrate how you can use Queue in Laravel 5.4 for email verification by using integrated auth RegisterController. In addition, I will also show you how to add the email sending process to a queue to make your app processes faster.

In this article, I am going to use a database for the queue jobs. However, you can use Redis or any other drivers, as defined in the official docs.

The updates assume you've created the authentication/authorization system with Laravel's make:auth command. Once you've run that and the code is generated you can then make the changes:

• updating the users table to store the email token
• adding a table for the queue records
• migrating the tables
• updating the .env file with queue and mail information

They then go through the functionality to add to the app including the EmailVerification class, the email template and a SendVerficationEmail queue job. The last updates are to the auth registration process to push the sending of the verification email to the background and send another email when the email is confirmed.

The Delicious Brains site has a new tutorial posted introducing WP Image Processing Queue, a tool that allows for on-the-fly image processing in your WordPress application via background processing.

I think the best solution is to get background processing into WordPress core so that all themes/plugins can share a single queue and ensure we don’t impact server performance. And so started my crusade.

At PressNomics, I had a great chat with Mike Schroder. He presented a very good path to core: find a feature that WordPress core needs and that needs background processing. In other words, piggyback! This is exactly how the image optimization stuff made it into core last year: by piggybacking off of responsive images. For background processing, he proposed coming up with an alternative to on-the-fly image processing (OTFIP). Whoa, turns out OTFIP is a problem we regularly deal with for WP Offload S3 as well. This could be a “two birds – one stone” kind of thing. Stars were aligning.

He talks more about some of the current discussions and efforts around processing the images like this (with OTFIP, On The Fly Image Processing). He covers some of the libraries that are currently out there for this processing and how, ultimately, the image processing queue came out to replace them as a result of some work at WordCamp US Contributor Day. He gives an example of the code needed to resize the images and the resulting markup. The post ends with the work he's planning on getting this queuing into the WordPress core and encourages plugin authors to use the OTFIP functionality rather than an external library.