High-performance Python – compiled code and C interface

Step Lively


Cython takes Python code, converts it to C, and compiles it for you, but what if you have existing C code that you want to use in Python like a Python module? This is where ctypes [14] can help.

The ctypes foreign function library provides C-compatible data types and lets you call functions in dynamic link libraries (DLLs) or shared libraries from within Python. In essence, it "wraps" these libraries so they can be called from Python. You can find ctypes with virtually any Python distribution.

To use ctypes, you typically start with your C/C++ code and build the shareable object as usual. However, be sure to use the position-independent code (PIC) flag and the shared flag (you'll be building a library). For example, with gcc, you use the -fPIC and -shared options:

$ gcc -fPIC -shared -o libsource.so source.c

It is up to you to compile the C code and create a library using any method you like, as long as you use the -fPIC and -shared options.

Ctypes Example – sum

In the previous example, most of the work is done in the summation, so now I'll rewrite that routine in C to get better performance. According to an online tutorial [15], an example in C for computing the sum and building it into a library is shown in Listing 5.

Listing 5

C Summation

01 int sum_function(int num_numbers, int *numbers) {
02    int i;
03    int sum = 0;
04    for (i = 0; i < num_numbers; i++) {
05       sum += numbers[i];
06    }
07    return sum;
08 }

The function is named sum_function and the code sum.c. This code can be compiled into a shared object (library) with gcc:

$ gcc -fPIC -shared -o libsum.so sum.c

The compiler creates the shared object libsum.so, a library.

To use the library in Python, a few specific ctypes functions and variables are needed within Python. Because it can make the Python code a bit complex, I write a "wrapper function" for the library in Python (Listing 6). Notice that the specific function sum_function is defined.

Listing 6

Wrapper Function

01 import ctypes
03 _sum = ctypes.CDLL('libsum.so')
04 _sum.sum_function.argtypes = (ctypes.c_int, ctypes.POINTER(ctypes.c_int))
06 def sum_function(numbers):
07    global _sum
08    num_numbers = len(numbers)
09    array_type = ctypes.c_int * num_numbers
10    result = _sum.sum_function(ctypes.c_int(num_numbers), array_type(*numbers))
11    return int(result)

If you have a library with more than one function, you will have to create the interface for each function in this file. Now to test the C function in Python (Listing 7). The eagle has landed! It works! However, you might notice that it is slower than even the pure Python code. Most likely the arithmetic intensity is not great enough to show an improvement. Do not let this deter you. It's always worth trying ctypes if you want or need more performance.

Listing 7

Python C Test

import sum
import numpy
x = numpy.arange(10000000)
%time sum.sum_function(x)
CPU times: user 2.15 s, sys: 68.4 ms, total: 2.22 s
Wall time: 2.22 s


Python is amazingly popular right now, with thousands of modules that can be used to extend its capability. However, in the high-performance world, Python is not known for being fast. A number of people have written tools and extensions to make Python faster.

In this article, I presented Numba, which compiles Python code with a JIT compiler invoked with a decorator. A simple summation example was found to be much faster than the original Python code.

Cython, also discussed briefly, is a tool for "translating" Python code into C code. A simple Python "makefile" translates and compiles the code for you. It's actually pretty simple to use and allows you to create a large Python module with lots of functions.

Finally, I presented ctypes, which uses an opposite approach by taking existing C code that is compiled into a library, coupled with some ctypes functions and variables, to create a module that can be used in Python. The ctypes library has a great deal of flexibility, so you can incorporate code written in C into Python.

The Author

Jeff Layton has been in the HPC business for almost 25 years (starting when he was four years old). He can be found lounging around at a nearby Frys enjoying the coffee and waiting for sales.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs

Support Our Work

ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.