On the Delicious Brains site they've posted a tutorial showing how to create a command line daemon that will emulate the Amazon Web Services SQSD handling. The SQSD is a worker daemon service that Amazon offers as a part of its Elastic Beanstalk support.

Sometimes when you’re building a project there are parts of the architecture that exist on production that don’t exist on your development machine. Those missing parts (like proprietary software that’s specific to your hosting provider) can sometimes mean unwelcome surprises when you deploy to production.

Recently as part of my work on Mergebot, I decided to address this. My local machine was missing the AWS Elastic Beanstalk Worker Environment SQS daemon (known as SQSD). AWS isn’t open source so there’s, unfortunately, no official way to replicate it. So I decided to build a small PHP command line (CLI) app to attempt to replicate its functionality. In this article, I’m going to cover some of the aspects of creating a command line app in PHP and explain how I implemented them for my replica SQSD CLI.

He starts off with a brief overview of the Laravel queue worker and how it compares to the SQSD functionality. He then starts in on the code to create the daemon (outside of a framework) and adding in the while loop to keep it running as a daemon making use of the SQSD Worker class as a base. The post ends with some instructions on packaging up the command line tool using the phar functionality already included in the PHP language.

Continuing his series about new functionality in Laravel v5.3 Matt Stauffer has posted this quick article covering updates to the queue worker functionality.

Queues are one of those tools in Laravel that everyone knows is there, but very few people understand deeply. It's understandable--Laravel is often the first place folks have run into queues, and to be honest, they're not simple.

Thankfully, very little has changed on a user-facing front with regard to how queues work in Laravel 5.3.

One of the main updates is that the "listen" command is now "work" and the action then runs as a daemon by default instead of requiring the command to be long-running. He talks about the difference in this shift and how something like Supervisor can now be used to manage the daemon (including some documentation specific to Laravel). He finishes the post looking at what has changed "under the hood" and the benefits the changes bring.

The SitePoint PHP blog has posted a new tutorial from author Daniel Sipos showing you how you can use custom annotations in your Symfony-based application. You can also do annotation parsing outside of Symfony but that requires other external libraries to accomplish.

In this article, we are going to look at how we can create and use our own custom annotations in a Symfony 3 application. You know annotations right? They are the docblock metadata/configuration we see above classes, methods and properties. You’ve most likely seen them used to declare Controller routes (@Route()) and Doctrine ORM mappings (@ORM()), or even to control access to various classes and methods in packages like Rauth. But have you ever wondered how can you use them yourself?

[...] In this article we are going to build a small reusable bundle called WorkerBundle. [...] We’re going to develop a small concept that allows the definition of various Worker types which “operate” at various speeds and which can then be used by anyone in the application. The actual worker operations are outside the scope of this post, since we are focusing on setting up the system to manage them (and discover them via annotations).

He then gets into the code, creating the WorkerInterface the workers will implement and a sample worker class with an annotation describing it. Next up he creates the WorkerManager to create and get the current set of workers. Then comes the discovery process and the creation of a simple class that looks through files and finds those with the @Worker annotation and makes them available as a worker instance. Finally he "wires it all together" in the services configuration and shows an example of a basic worker instance and using it by calling its work method.

In a post to the Madewithlove blog Maxime Fabre takes a look at threading in PHP using the pthreads support that can be included into your PHP installation.

As far as I can remember, PHP has always had a terrible reputation at handling very heavy (or asynchronous) tasks. [...] But PHP can do threading, and more importantly it's a lot easier than you probably think.

[...] In this article I'm going to dive into the pthreads extension (short for POSIX Threads). It has been around for a while (since 2012) but I feel like too many people forget it exists or assume it is going to be painful to use – mostly because the official documentation is rather slim about it.

They start by getting the pthreads support installed locally (it assumes you use OS X and the "brew" package manager but it can be installed manually too). The article starts off by defining some basic nomenclature from the pthreads world and gives a diagram of how it all fits together. From there it gets into some examples, showing a simple thread class to fetch Google results and how to fire off multiple instances at the same time. They then extend this even further and look at the concept of "workers" and using them to manage individual jobs. It then moves up the next level and looks at "pools" of workers and processing multiple workers at the same time.

There's also a section dealing with one "gotcha" that can happen with class inheritance between parent and child threads. They show how to work around this with a custom Worker class that performs the autoloading for you and is executed at the start of a Pool. Finally they cover the messaging between the child threads and, as a bonus, how threading could be used in a command bus setup.

Lorna Mitchell has posted a new tutorial to her site today walking you through using Beanstalkd with PHP for a simple queuing setup in your application. Beanstalkd is "a simple, fast work queue. Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously."

I have an API backend and a web frontend on this project (there may be apps later. It's a startup, there could be anything later). Both front and back ends are PHP Slim Framework applications, and there's a sort of JSON-RPC going on in between the two. The job queue will handle a few things we don't want to do in real time on the application, such as: updating counts of things like comments, [...] cleaning up, [...] other periodic things like updating incoming data/content feeds or talking to some of the 3rd party APIs we use like Mailchimp and Bit.ly.

She starts with a look at how to add jobs to the queue (she assumes that you've already set up the Beanstalkd instance at this point). She uses the Pheanstalk library for the job handling and includes a sample call to configure the connection and create an instance to make the connection. The sample job contains an array of data including an "action" and "data" for it to use when processing. She also includes an example of a basic PHP-based Beanstalkd worker that will go through currently pending jobs and execute them based on the action/data combination. In the sample worker script, she defines the action as a method in the class to be executed directly on the worker instance. She finishes off the post with a few "things to remember" about working with workers and long-running PHP scripts.

Wan Qi Chen has returned with the next three parts in his "Background jobs with PHP and resque" series (first parts here) with parts four, five and six helping you implement the concepts from the first three parts.

• Background jobs with php and resque: part 4, managing worker
• Background jobs with php and resque: part 5, creating jobs
• Background jobs with php and resque: part 6, integration into CakePHP

That sixth part of the series uses the CakeRisque plugin to make the integration simpler. Save that, you could integrate this setup pretty easily into whatever framework you choose.

In this latest post to his blog Justin Carmony shares some of his experience using Redis and Solo to asynchronously run queries and return data without the user having to wait.

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don’t want to make the user wait after pressing a button. This is where "Workers" can come in. They are independent scripts that run along side of your application, performing tasks, or "jobs."

Solo is a very basic Perl script that ensures only one process of a type is running at once. Using this and a PHP library called predis, he shows how to set up workers and add items to your processing queue. The workers themselves run on a cron job and connect to the queue server to see what they need to do. He also throws in some "bells and whistles" - extras that can enhance your worker system: queue monitoring, version numbering and killing workers based on a hash value.

His code examples are posted on his github account and a screencast is included in the post to show the system in action.

Matthew Wells has a new post that looks at combining Kohana and Gearman to create a system to handle large data processing without slowing down the rest of the application.

A commonly identified bottleneck arises when dealing with large, 'expensive' data. This is commonly seen when an application posts a large volume of well structured data to the API (that some process must be carried out upon), before some form of structured receipt is then returned as a request response. [...] Analysing such a request tends to show high PHP CPU usage with lower database consumption. [...] The structured nature of data exchanged via an API means that we can, relatively simply and reliably, divide the submitted data and process it simultaneously with the help of a great tool called Gearman.

He walks you through the entire process including his initial thoughts on what the system should be and how it should behave when the requests are made. He shares the code he used to implement the system - a simple worker that processes part of the request and returns the results. The command-line calls to run the worker manually for testing are also included.

Gonzalo Ayuso had previously shown how to dynamically add watermarks to your images with the help of a mod_rewrite rule. One of the side effects of this is that, with a high load on your server, things can get bogged down quite quickly. His alternative? Add the watermarks to your images with Gearman workers.

In this second solution I will use a gearman worker to generate the watermarks. The benefits of gearman is the possibility of use a pool of workers. We can add/remove workers if our application scales. Those workers can be placed even at different hosts, and we can swap easily from one configuration to another.

He includes some sample scripts to illustrate the process - a Gearman client that'll call the watermarking process and a worker that takes in the image and transforms it with GD to add a new layer for the watermark.

Lorna Mitchell has a new post today about using a popular backend processing tool, Gearman, from inside of PHP. Her example gives a full overview of how to add a new job and write the worker for it to make things happen.

Basically, this application generates some PDFs from a variety of data sources, makes images, and emails it. Since the whole data processing, image handling, PDF generation process is fairly heavy, I'm putting the requests to generate these onto a gearman queue and having some workers process the jobs.

You'll need to have the Gearman server and extension installed (sample instructions here for Ubuntu) to use the sample code she gives using the Gearman_Client and GearmanWorker objects to create the pieces of the puzzle. You can find out more about the Gearman PECL extension here.