High-Performance Python – Compiled Code and Fortran Interface

Fortran functions called from Python bring complex computations to a scriptabllanguage.

Soon after C became the first language to be callable from Python, people wanted the same for the millions of lines of proven Fortran code in scientific applications. In the past, combining C and Fortran was dependent on the compiler. The advent of Fortran 2003 brought a standard, compiler-independent C/Fortran interface.

The approach for combining C and Fortran for Python is to write a C wrapper for the Fortran functions and subroutines that you want to use in Python. Then, you build the Python function around the C wrapper and use it as a Python module. Calling the C function calls the Fortran routine. Although a little indirect, the concept is straightforward.

In the first example, I integrate Cython and Fortran.

Fortran/C and Cython

The Fortran 90 website, which lists best practices, discusses how elements of the Fortran 2003 standard can integrate C and Fortran. The iso_c_binding module of Fortran 2003-compliant compilers matches Fortran and C types with named constants used as KIND-type parameters in Fortran.

Without writing any C code, Cython can then be used to interface with the Fortran code. The Cython code is written in Fortran with the use of C calling conventions. The Fortran code interface with C uses the iso_c_binding module to link Cython directly against the Fortran library. Granted, this method doesn’t really combine C and Fortran; rather, it uses the standardized Fortran/C interface. However, it really is integrating Fortran and C with Python.

If you are interested, the Fortran 90 website has an example of interfacing Fortran with Python via Cython that is as simple as creating a C interface in Fortran. Another good example can be found on Pierre Augier’s website.

Fortran/C and ctypes

In the previous article on high-performance Python, I presented ctypes as a way to integrate “foreign” function libraries (i.e., libraries outside of Python) into Python. You can use ctypes to integrate Fortran code with Python, as well.

The following examples show the multiplication of two integers and the addition of two integers. Save the Fortran integer multiplication code function in file mult.f90,

integer function multiply(a, b)
    integer, intent(in) :: a, b
    multiply = a * b

and save the Fortran integer addition code in file add.f90:

integer function addtwo(a, b)
    integer, intent(in) :: a, b
    addtwo = a + b
end function addtwo

You should compile each function individually into shared objects:

$ gfortran -shared -fPIC -g -o mult.so mult.f90
$ gfortran -shared -fPIC -g -o add.so add.f90

For the sake of development, I compile the code with the debug option (-g), but that is purely a personal preference.

If you want to explore the shareable objects, you can use the Linux nm tool:

$ nm -ao mult.so | grep multiply
mult.so:00000000000005cc T multiply_

Note that the function name in the shareable object has a trailing underscore.

Now you can use the ctypes module to check whether the objects work with code saved in a file named testfunc.py (Listing 1). Notice that the variables a and b are typed by cytpes and the Fortran functions are called with a trailing underscore. Also notice that variables are passed to the function by reference,which is different from C code, which passes by value.

Listing 1: Using ctypes

from ctypes import byref, cdll, c_int
 
mult = cdll.LoadLibrary('./mult.so')
add = cdll.LoadLibrary('./add.so')
a = c_int(2)
b = c_int(4)
print mult.multiply_(byref(a), byref(b))
print add.addtwo_(byref(a), byref(b))

The output from the Python code is:

$ python3 testfunc.py
8
6

If you like, you can write a simple Python wrapper function for the shareable objects, so it's easier to understand in the main Python code. A wrapper allows you to avoid the trailing underscore in the main Python code and the use of byref functions to make it look more Pythonic. The previous article on high-performance Python has an example of how to write wrapper functions.

f2py

Early in the development and growth of Python, the desire to integrate Fortran into Python led people to develop the necessary tools. To make the process more accessible, Pearu Peterson decided to create a tool for make porting easier. The tool, f2py, was first released in 1999 and was known as f2py.py. Eventually, the tool was incorporated into NumPy (numpy.f2py) and is still there today.

F2py is probably the most popular tool for interfacing Fortran with Python and has been used in a large number of projects. It creates an extension module that is imported into Python by the import Python command. The module has automatically generated wrapper functions that create the interface between Fortran and Python.

The f2py users guide introduction shows three methods of use. Instead of presenting all three methods, I will focus on the “quick and smart way.”

The first step is to create a signature file for the Fortran (or C) code (f2py also accommodates C code) that describes the wrapper for the Fortran or C functions. F2py refers to these as “signatures” of the functions. Usually f2py can create a signature file for you just by scanning the source. Once the signatures are created, you compile the code to make it ready for Python.

f2py example

The simple Fortran code in Listing 2 illustrates how to call a subroutine with input data:

Listing 2: secret.f90

subroutine hellofortran(n)
integer n
 
write(*,*)  "Hello from Fortran! The secret number is: ", n
 
return
end

The next step is to create the signature of the code for f2py:

$ python3 -m numpy.f2py secret.f90 -m secretcode -h secret.pyf

For this command, f2py takes the Fortran90 code and creates the signature file secret.pyf. The Python module is named secretcode (the -m option). The resulting signature file is shown in Listing 3.

Listing 3: secret.pyf

$ more secret.pyf
!    -*- f90 -*-
! Note: the context of this file is case sensitive.
 
python module secretcode ! in 
    interface  ! in :secretcode
        subroutine hellofortran(n) ! in :secretcode:secret.f90
            integer :: n
        end subroutine hellofortran
    end interface 
end python module secretcode
 
! This file was auto-generated with f2py (version:2).
! See http://cens.ioc.ee/projects/f2py2e/

The final step is to compile the code with f2py and create the Python module:

$ python3 -m numpy.f2py -c secret.pyf secret.f90

The output from the command (not shown) provides some good information about what it's doing and is useful in understanding what f2py does.

F2py creates a shared object (.so) suitable for Python to import:

$ ls -s
total 120
112 secretcode.cpython-37m-x86_64-linux-gnu.so    4 secret.f90    4 secret.pyf

To test the code, run python3 (Listing 4). Remember, the Python module is secretcode.

Listing 4: Testing the Code

$ python3
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import secretcode
>>> secretcode.hellofortran(5)
 Hello from Fortran! The secret number is:            5

The first command (see the >>> prompts) imports the secretcode Python module. The shared object has the function hellofortran, which you use in the second command, secretcode.hellowfortran(). I used the argument 5 in the test.

OpenMP and f2py

If you can compile Fortran routines to use in Python, shouldn't you be able to compile OpenMP Fortran code for Python? After all, Python only uses a single core. If you either take existing Fortran routines or write new Fortran routines that use OpenMP to take advantage of all of the CPU cores, you would make much better use of computational resources.

Previously, I wrote an article that introduced the concept of using multicore processing with Fortran and OpenMP. The article has a simple example of finding the minimum of an array. F2py was used on the command line with several options for the compiler to build OpenMP code:

$ f2py --f90flags=-fopenmp -L/usr/lib/gcc/x86_64-redhat-linux/4.8.2/ -lgomp -c -m test test.f90

In this case, the GFortran compiler option -fopenmp was used to compile the OpenMP. It also added the appropriate gomp library with the full path. This test.f90 code also used the ompmin function and created the test module. Read the article to understand how you can use OpenMP with your Fortran code in Python.

gfort2py

During my research for this article, I ran into the interesting gfort2py project, which uses GFortran, the GNU Fortran compiler, and Fortran modules (.mod files) to translate the Fortran code's ABI (application binary interface) to Python-compatible types with Python's ctypes library.

In principle, gfort2py can accommodate anything the compiler can compile (i.e., valid Fortran), as long as it is in a Fortran module. Gfort2py is mostly written in Python and requires no changes to your Fortran code, as long as it is a module; however, you can only use GFortran.

The requirements for gfort2py is a GFortran version greater than 5.3.1. It works with both Python 2.7 and Python 3. Although it is set up to be installed by pip, because I’m using Anaconda, I will install it the old-fashioned way and build it on the target system. The instructions for installation on the gfort2py website are easy to follow, and it's simple to install. Just be sure to start with the Python version you intend to use later.

The Fortran test code is very simple (Listing 5) and almost the same as the previous code. It just writes out a variable that is passed in. Notice that the code is in a module, as required.

Listing 5test1.f90

module test1
contains 
subroutine hello(n)
    integer :: n
    write(*,*) "hello world. The secret number is ",n
    return 
end
end module test1

The next step is to compile the code and create the library (Listing 6). The resulting Python library is very easy to use: You use a function from the gfort2py Python module, and it handles the heavy lifting, so to speak (Listing 7).

Listing 6: Compiling Fortran Code

$ gfortran -fPIC -shared -c test1.f90
$ gfortran -fPIC -shared -o libtest1.so test1.f90
$ ls -s
total 28
8 libtest1.so  4 test1.f90  4 test1.fpy  4 test1.mod  4 test1.o  4 test1.py

Listing 7: Using the Python Library

import gfort2py as gf
 
SHARED_LIB_NAME = "./libtest1.so"
MOD_FILE_NAME="test1.mod"
 
x = gf.fFort(SHARED_LIB_NAME, MOD_FILE_NAME)
 
y = x.hello(6)

First, variables are defined pointing to the library and the compiled .mod file. Next, an object is defined that contains the functions in the Fortran module file (x). Finally, a function from the code is used. Running the Python code shows the output:

$ python3 test1.py
 hello world. The secret number is            6

Going beyond this simple code would make the article too long, but it seems like almost anything you can put into a Fortran module would work – perhaps even OpenMP or an other extension. Gfort2py seems to be under active development, so check back in from time to time to see what new features have been added.

Summary

Fortran code is accessible by Python a number of ways, ranging from C wrappers for Fortran functions to dedicated tools that all are pretty simple to use. F2py is the most popular tool, although I have had trouble using it with non-GFortran compilers.

In the next article, I present a few of the many Python libraries that allow you to use GPUs. I think you will be pleasantly surprised at the state of GPU computing in Python.