Eric Isselée, 123RF

Boost performance of your Python programs

Snakebyte

Article from ADMIN 02/2010

By Ankur Kumar Sharma

Overcome the Global Interpreter Lock in Python and take advantage of run-time parallelism on multicore machines.

The modern way to boost run-time performance of Python programs is to leverage the run-time parallelism provided by modern multicore machines along with the SMP-enabled operating systems. Thus, if you could design your Python applications so that their independent parts run in parallel on these multicore machines at run time, then performance of the applications would be improved a lot. In other words, if your existing Python applications are multithreaded or you write your new code in a way that exploits concurrency, things run in parallel on the multicore machines.

Unfortunately, the current implementation of standard Python known as CPython is unable to provide run-time parallelism for the Python threads on multicore machines because of the Global Interpreter Lock (GIL). I'll explore ways to conquer this GIL issue in CPython.

The objective of this article is to get you started with parallel computing on multicore/multiprocessor and computing cluster hardware using Python, but you will need some understanding of parallel computing [1]. I used the Ubuntu 9.10 x86_64 desktop distribution along with Python 2.7 to test the code in this article and to generate the screen shots.

Breaking GIL and Extracting Performance

GIL limits the number of the threads that can execute concurrently in the CPython interpreter to one. The lock also impairs the ability of CPython to take advantage of parallel execution in modern multicore machines. To overcome this limitation on Python programs, one uses subprocesses in place of the threads in the Python code. Although this subprocess model does have some creation and interprocess communication overhead, it's free of the GIL issue and adequate most of the time when dealing with parallelism in Python programs.

If this picture seems complicated to you,

...

Use Express-Checkout link below to read the full article (PDF).