Gearman is a distributed queue with several language bindings.

While Gearman has a nice Python implementation (python-gearman) of the client and worker, I chose to use the libgearman bindings (python-libgearman) directly since they are already packaged for Debian (as python-gearman.libgearman).

Unfortunately, these bindings are not very well documented, so here's the sample application I wished I had seen before I started.

Using the command-line tools

Before diving into the Python bindings, you should make sure that you can get a quick application working on the command line (using the gearman-tools package).

Here's a very simple worker which returns verbatim the input it receives:

gearman -w -f myfunction cat

and here is the matching client:

gearman -f myfunction 'test'

You can have have a look at the status of the queues in the server by connecting to gearmand via telnet (port 4730) and issuing the status command.

Using the Python libgearman bindings

Once your gearman setup is working (debugging is easier with the command-line tools), you can roll the gearman connection code into your application.

Here's a simple Python worker which returns what it receives:

#!/usr/bin/python  

from gearman import libgearman  

def work(job):  
   workload = job.get_workload()  
   return workload  

gm_worker = libgearman.Worker()  
gm_worker.add_server('localhost')  
gm_worker.add_function('myfunction', work)  

while True:  
   gm_worker.work()

and a matching client:

#!/usr/bin/python  

from gearman import libgearman  

gm_client = libgearman.Client()  
gm_client.add_server('localhost')  

result = gm_client.do('myfunction', 'test')  
print result

This should behave in exactly the same way as the command-line examples above.

Returning job errors

If you want to expose to the client errors in the processing done by the worker, modify the worker like this:

#!/usr/bin/python  
  
from gearman import libgearman  
  
def work(job):  
   workload = job.get_workload()  
   if workload == 'fail':  
       job.send_fail()  
   return workload  
  
gm_worker = libgearman.Worker()  
gm_worker.add_server('localhost')  
gm_worker.add_function('myfunction', work)  
  
while True:  
   gm_worker.work()

and the client this way:

#!/usr/bin/python  
  
from gearman import libgearman  
  
gm_client = libgearman.Client()  
gm_client.add_server('localhost')  
  
result = gm_client.do('myfunction', 'fail')  
print result

License

The above source code is released under the following terms:

To the extent possible under law, Francois Marier has waived all copyright and related or neighboring rights to this sample libgearman Python application. This work is published from: New Zealand.