Distributing Work with Gearman and Perl

I’ve been playing with Gearman over the past few weeks and I’m impressed by how little friction there is between the idea of distributing work across machines and actually making it happen. The Perl module, Gearman::Client and Gearman::Worker, is straightforward enough that you can have a client and a worker talking to each other through a job server in under an hour.

The basic model is simple. You have a client that submits jobs, a worker that performs them, and a job server sitting in the middle dispatching work to whichever worker is free. The client and worker can be written in different languages, run on different machines, and you don’t need to worry about the networking yourself — Gearman handles all of that through TCP sockets to the job server.

Here’s a minimal client in Perl:

use Gearman::Client;

my $client = Gearman::Client->new;
$client->job_servers('127.0.0.1:4730');
my $result = $client->do_task('reverse', 'Hello World');
print "$$result\n";

And the matching worker:

use Gearman::Worker;

my $worker = Gearman::Worker->new;
$worker->job_servers('127.0.0.1:4730');
$worker->register_function('reverse', sub {
    my $job = shift;
    return scalar reverse $job->workload;
});
$worker->work while 1;

The function name — ‘reverse’ in this case — is entirely arbitrary. Gearman doesn’t care what you call your functions or what data you pass to them. It just routes packets between clients and workers based on function names.

The thing that makes this genuinely useful rather than just an academic exercise is the ability to offload expensive operations away from your web servers. If your application needs to resize images, generate PDFs, or run database queries that take seconds instead of milliseconds, you can ship that work to a pool of worker machines and keep your web tier responsive. The job server handles load balancing automatically — it only sends new jobs to idle workers, so adding more worker processes or machines is as simple as starting them up and pointing them at the job server.

You can also run jobs in the background without waiting for results, which is the more common pattern in practice. Instead of do_task, you call dispatch_background and get back a handle you can use later to check on the job’s status.

I did hit a snag with the Perl module, though. If you register a job server without specifying the port — just the hostname — you’ll get a rather unhelpful error about syswrite on an undefined value deep inside Gearman::Taskset. The fix is to include the port explicitly:

$client->job_servers('127.0.0.1:4730');

Rather than:

$client->job_servers('127.0.0.1');

It’s a minor gotcha but worth noting if you’re following examples that omit the port.

For redundancy you can run multiple job servers and configure clients and workers to try each one in turn. If one goes down, the others pick up the load without any manual intervention. Two or three job servers is probably plenty for most setups.

I’m not sure yet where Gearman fits into my own projects, but the combination of language independence, minimal setup, and built-in load balancing makes it worth keeping in mind for anything that needs to scale beyond a single machine.