High Performance Python

Strengths of Python

• Flexible and powerful in-built type system

  • Arbitrary data size (ints, strings, lists etc.)

  • Objects, modules, inheritance etc.

• Fast to develop, can try scripts in REPL

• Extensibility and wide range of libraries available

Downsides to Python

• Type conversion based on context

• Overhead as variables ‘change type’

• Magic behind the scenes to support abstraction (arbitrary sized integers, arbitrary length strings…)

• This costs space and time…

Trade-offs

• The flexibility comes at a cost (performance)

• Python source is translated to python bytecode (.py -> .pyc), pyc updated on first run

• The bytecode is interpreted at runtime

Speeding things up

We will need to trade some of the flexibility to get more performance

• Introduce fixed types so managing memory is leaner and more efficient

• Get closer to the machine architecture

  • either use other program components written in a compiled language

  • Or, ‘compile’ our python in some way

Working with other Processes is Useful when

1. using python to ‘script’ other things, e.g. automation of routine system tasks;

2. running toolkits written in other languages (such as C)

Pros of Process Level Integration

• verbose API designed for humans

• simple to use

• cross platform

• very versatile

Cons of Process Level Integration

• start-up cost of process

• difficult to maintain state outside python script

• low level of granularity

• difficult to efficiently pass large amounts of data efficiently

Optimise for the common case of Python and C

• Actively try to create programs that utilise multiple programming languages

• Allow each language to excel at what it does best

• Write user facing scripts in Python

• Write the high-performance modules in a language that is more computationally efficient

• Create Foreign Function Interfaces between them…

Foreign Function Interfaces

• A function that executes code written in a different programming language

• Provides finer grained integration of library code into applications than process-based approaches

• C is the most common language to interface with, as it complements Python well

• Typically implemented through dynamically linking C libraries into Python interpreter (Java and C# can do this too)

CPython Extensions

• CPython extension modules are just shared object C libraries (.so files, or DLLs)

• Placed in a well know search path, defined by sys.path…

• The Python VM exposes a set of C library functions and types (classes/structs) that allow:

  • Python modules and functions to be created

  • Marshalling of types between C/Python

  • Initialization callbacks

Example of CPython Extension

• C code that makes up the functionality of the module

• Wrappers methods for each underlying C function. Note standardized naming convention and parameter list

• Initialization callback function, that registers a table of exposed functions to the Python VM

• Explicit functions to manage type conversion between C and Python

Using CPython Extensions

• Compile as a dynamic library

• Include Python VM library

• Place resultant library in the Python search path

• Use like any other Python module

Limitations of CPython

• Powerful, but quite Complex to write

• Developer needs to handle memory management (ref counts!)

• Exception handling requirements need to be obeyed

• Only works with CPython…

3 Higher Level Approaches to CPython

• Using the ctypes library

• The C Foreign Function Interface or CFFI

• The Cython language