• Background jobs with php and resque: part 4, managing worker
• Background jobs with php and resque: part 5, creating jobs
• Background jobs with php and resque: part 6, integration into CakePHP

That sixth part of the series uses the CakeRisque plugin to make the integration simpler. Save that, you could integrate this setup pretty easily into whatever framework you choose.

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don't want to make the user wait after pressing a button. This is where "Workers" can come in. They are independent scripts that run along side of your application, performing tasks, or "jobs."

Solo is a very basic Perl script that ensures only one process of a type is running at once. Using this and a PHP library called predis, he shows how to set up workers and add items to your processing queue. The workers themselves run on a cron job and connect to the queue server to see what they need to do. He also throws in some "bells and whistles" - extras that can enhance your worker system: queue monitoring, version numbering and killing workers based on a hash value.

His code examples are posted on his github account and a screencast is included in the post to show the system in action.

A commonly identified bottleneck arises when dealing with large, 'expensive' data. This is commonly seen when an application posts a large volume of well structured data to the API (that some process must be carried out upon), before some form of structured receipt is then returned as a request response. [...] Analysing such a request tends to show high PHP CPU usage with lower database consumption. [...] The structured nature of data exchanged via an API means that we can, relatively simply and reliably, divide the submitted data and process it simultaneously with the help of a great tool called Gearman.

He walks you through the entire process including his initial thoughts on what the system should be and how it should behave when the requests are made. He shares the code he used to implement the system - a simple worker that processes part of the request and returns the results. The command-line calls to run the worker manually for testing are also included.

In this second solution I will use a gearman worker to generate the watermarks. The benefits of gearman is the possibility of use a pool of workers. We can add/remove workers if our application scales. Those workers can be placed even at different hosts, and we can swap easily from one configuration to another.

He includes some sample scripts to illustrate the process - a Gearman client that'll call the watermarking process and a worker that takes in the image and transforms it with GD to add a new layer for the watermark.

Basically, this application generates some PDFs from a variety of data sources, makes images, and emails it. Since the whole data processing, image handling, PDF generation process is fairly heavy, I'm putting the requests to generate these onto a gearman queue and having some workers process the jobs.

You'll need to have the Gearman server and extension installed (sample instructions here for Ubuntu) to use the sample code she gives using the Gearman_Client and GearmanWorker objects to create the pieces of the puzzle. You can find out more about the Gearman PECL extension here.

With most web applications, there comes a time where there is a need to perform either CPU or I/O intensive work based on user actions. Whether processing uploaded files, or performing system-wide database updates, developers are increasingly turning to Gearman as a simple way to hand off the heavy lifting to another server to be performed asynchronously.

He's created a basic module that can be downloaded and dropped right into your Magento install to make it work with your Gearman queues. He includes an installation guide and a bit of code on how to make a sample job to run an event and the class sitting on the other side for the worker to use in handling it.

We're looking at migrating to Git for our primary development on the framework. There are a lot of use cases we need to accommodate [...] Git allows you to define "hooks", scripts that are run at various points of the git commit lifecycle. One hook that can run on the server is called "post-receive". [...] And now we get to Gearman. I realized I could have my post-receive script queue a background task to Gearman. Since this is almost an instantaneous operation, it meant that my hook was completed before Gearman started a worker; if it wasn't, I could always do a sleep() within my worker to ensure it was.

He uses the gearman PECL extension to interface with the Gearman backend service and shows how to push off his post-recieve and rss processes to be handled separately from the current process. He includes the PHP code for both of these processes as well as the actual worker process.

Sometimes things just get too heavy for a straight forward approach. Memory usage might be too high or interaction might be delayed. In this case it might make sense to queue the task up for later execution.

The technique uses beanstalkd as a messaging queue to handle the requests based on the user's request via the interface the beanstalkd library provides. Then, to keep the queue running in the background and available, he uses daemotools to run a worker process. You can download the complete code for a working example.

Well, first, what is an MPM? It stands for Multi-Processing Module. [...] The most commonly used threaded MPM is the Worker MPM. In this MPM, you have several processes that run multiple threads within it. This is the one I will be talking about.

He goes on to describe the "huge memory savings" that using the module got them - using the worker process to increase the child capacity of the Apache instance. He recommends keeping it simple when compiling a server to work with the worker functionality and mentions how it can also help with serving static pages right along side the dynamic ones (without a second server!).

Check out the post for a few more tips.