mirror of
https://github.com/OPM/ResInsight.git
synced 2026-09-03 20:53:13 -05:00
430 lines
16 KiB
Plaintext
430 lines
16 KiB
Plaintext
Developer documentation
|
|
-----------------------
|
|
|
|
1 Summary
|
|
2 About ctypes
|
|
2.1 Loading the shared library
|
|
2.2 Type mappings
|
|
2.2.1 Fundamental types
|
|
2.2.2 User defined types and classes
|
|
3 The C code
|
|
4 Common structure of the wrapping
|
|
4.1 Loading the shared library
|
|
4.2 Prototyping
|
|
4.3 Python classes
|
|
5 Garbage collection and 'data_owner'
|
|
6 Installation in Statoil
|
|
|
|
|
|
|
|
1 Summary
|
|
|
|
The ert Python classes are based on wrapping (some) of the C libraries
|
|
which constitute the ERT application. The wrapping is based on the
|
|
python ctypes module which does the wrapping dynamically. The python
|
|
classes are quite thin, most of the actual code is in C, however a
|
|
user of the ert Python should NOT need to know that there is some C
|
|
code deep down.
|
|
|
|
|
|
2 About ctypes
|
|
|
|
ctypes is a standard Python module for dynamic wrapping of C code. The
|
|
C code should compiled into a shared library. The ctypes library works
|
|
by loading the shared library with dlopen(). In addition there is a
|
|
quite good system for transparently mapping between Python types and C
|
|
types. More extensive documentation of ctypes is available at:
|
|
http://www.python.org/org/library/ctypes.html
|
|
|
|
2.1 Loading the shared library
|
|
|
|
In the ert Python wrapping loading the shared library is handled by
|
|
the ert.util.clib function load(). When the shared library has been
|
|
loaded, all the symbols of the shared library are available as
|
|
attributes of the ctypes load handle. At the lowest level the loading
|
|
of shared libraries is through the ctypes.CDLL() call like:
|
|
|
|
lib_handle = ctypes.CDLL( "libname" )
|
|
|
|
however in the ert py library this is wrapped by the function load()
|
|
in the ert.util.clib module:
|
|
|
|
import ert.util.clib as clib
|
|
lib_handle = clib.load("name1" , "name2" , "name3")
|
|
|
|
The clib.load() function will only load one library, but it can take
|
|
several names to that library and will try loading until one of them
|
|
succeeds. This is quite useful because several of the standard
|
|
libraries are identified with different names on the different RedHat
|
|
releases, for instance zlib is loaded like this:
|
|
|
|
zlib_handle = clib.load("libz", "libz.so.1")
|
|
|
|
When the library has been loaded, all functions of the shared library
|
|
will be available as python function attributes of the library
|
|
handle. I.e. if loading the standard library like:
|
|
|
|
clib_handle = clib.load( "libc" )
|
|
|
|
The attribute clib_handle.getenv will be a pointer to a Python
|
|
function object which wraps the standard library getenv()
|
|
function. Before this is really usable we have to "tell" the function
|
|
object which input arguments and return to expect, this is done with
|
|
the restype attribute and the argtypes attribute of the function
|
|
object, more details of the prototype process can be found in section
|
|
4.2
|
|
|
|
|
|
2.2 Type mappings
|
|
|
|
The ctypes library automagically handles conversion between common
|
|
C-types and Python types, so for the example above we would be able to
|
|
write:
|
|
|
|
PATH = clib_handle.getenv( "PATH" )
|
|
|
|
And the conversion between Python strings and NULL based char* in C is
|
|
handled transparently. In the ert wrapping setting the type of return
|
|
values and input values is handled by the function prototype() in
|
|
ert.cwrap.cwrap.
|
|
|
|
The type mappings necessary to get prototype() function to work is
|
|
maintained in the CWrapper class in ert.cwrap.cwrap.py. During
|
|
initialization of the ert Python code there are many calls
|
|
associating the name of a C type you whish to use in the prototype
|
|
process (as a string) and the corresponding Python type:
|
|
|
|
CWrapper.registerType( "int" , ctypes.c_int )
|
|
CWrapper.registerType( "double" , ctypes.c_double )
|
|
CWrapper.registerType( "char*" , ctypes.c_char_p )
|
|
CWrapper.registerType( "int*" , ctypes.POINTER( ctypes.c_int ))
|
|
|
|
|
|
|
|
|
|
2.2.1 Fundamental types
|
|
|
|
All the fundamental C types are known to the ctypes library, and
|
|
transparently converted <-> the correct Python type.
|
|
|
|
C type ctypes Python type
|
|
--------------------------------------------------------
|
|
int c_int int
|
|
float c_float float
|
|
double c_double double
|
|
.....
|
|
char * (NULL terminated) c_char_p string
|
|
--------------------------------------------------------
|
|
|
|
All these types can be created by calling them with an optional
|
|
initializer of the correct type and value:
|
|
|
|
cint = ctypes.c_int( 42 )
|
|
|
|
With the function ctypes.POINTER() you can create pointers to the
|
|
fundamental types - but this is approaching dangerous territory!
|
|
|
|
|
|
2.2.2 User defined types and classes
|
|
|
|
Let us say you have a C function which expects a pointer to an
|
|
abstract data type as input:
|
|
|
|
void abstract_method( abstract_type * at );
|
|
|
|
And that you have implemented the Python class AT which wraps a
|
|
pointer to a at instance in a "c_ptr" attribute. If the AT class
|
|
contains a method:
|
|
|
|
def from_param( self ):
|
|
return ctypes.c_void_p( self.c_ptr )
|
|
|
|
The AT class can then be used as a Python type in the argtypes
|
|
attributes. To make it available in the prototyping as well you must
|
|
register the type:
|
|
|
|
CWrapper.registerType( "at" , AT )
|
|
|
|
Observe that the AT class should NOT be used as restype, i.e. for the
|
|
return value:
|
|
|
|
/-- Warning: ----------------------------------------------\
|
|
| The mapping of types generally works quite well, however |
|
|
| observe that the return type of C functions which return |
|
|
| an instance pointer is NOT the current class, but |
|
|
| c_void_p, that is because this return value is |
|
|
| internalized in the c_ptr attribute of the object. |
|
|
\----------------------------------------------------------/
|
|
|
|
|
|
3 The C code
|
|
|
|
The C libraries are to a large extent developed with abstract types
|
|
like:
|
|
|
|
typedef struct {
|
|
.....
|
|
.....
|
|
} abs_type;
|
|
|
|
and "methods" like:
|
|
|
|
abs_type * abs_type_alloc( ) { }
|
|
void abs_type_free( abs_type * at ) {}
|
|
int abs_type_get_an_int( const abs_type * at) {}
|
|
void abs_type_set_an_int( abs_type * at , int value) {}
|
|
|
|
it has therefore been relatively easy to map this onto Python classes
|
|
and give a pythonic feel to the whole thing. As a ground rule each C
|
|
file implements one struct; this struct is wrapped in a Python module
|
|
with the same name and a Python class with CamelCaps naming:
|
|
|
|
C Python
|
|
ecl_kw.c implements ecl_kw_type ecl_kw.py implements EclKW
|
|
|
|
|
|
|
|
|
|
4 Common structure of the wrapping
|
|
|
|
The wrapping of the libraries (currently only libecl is anything close
|
|
to complete) follow roughly the same structure:
|
|
|
|
1. Load the shared library
|
|
|
|
2. For each C "class" you wish to wrap {
|
|
|
|
2.1. Prototype the C functions you wish to use by creating Python
|
|
function objects with correct restype and argtypes
|
|
attributes for the functions you are interested in, using
|
|
the CWrapper.prototype() function.
|
|
|
|
2.2. Create the Python class - based on the prototype functions
|
|
from 2.1 above.
|
|
|
|
}
|
|
|
|
The following excerpt from ecl_kw.y illustrates this:
|
|
|
|
------------------------------------------------------------
|
|
# Load the shared library
|
|
import libecl <--- Step 1
|
|
|
|
Class EclKW: <-------------·
|
|
# Pure Python code to implement the EclKW class, |
|
|
# based on the functions prototyped below. | Step 3
|
|
.... <-------------·
|
|
|
|
# Create a wrapper instance which wraps the libecl library. <--------·
|
|
cwrapper = CWrapper( libecl.lib ) |
|
|
|
|
|
# Register the type mapping "ecl_kw" <-> EclKW | Step 2
|
|
cwrapper.registerType( "ecl_kw" , EclKW ) |
|
|
|
|
|
# Prototype the functions needed to implement the Python class |
|
|
cfunc.alloc_new = cwrapper.prototype("long ecl_kw_alloc( char* , int , int )")
|
|
cfunc.free = cwrapper.prototype("void ecl_kw_free( ecl_kw )") |
|
|
.... <--------·
|
|
------------------------------------------------------------
|
|
|
|
These three steps are described in more detail in 4.1 - 4.3 below.
|
|
|
|
|
|
4.1 Loading the shared library
|
|
|
|
For loading the shared library 'libecl.so' there is a module
|
|
'libecl.py' which calls the actual loading function, load() in
|
|
ert.cwrap.clib.py. Subsequently the library instance will be a 'lib'
|
|
attribute of the the libecl module, so that subsequent functions which
|
|
need access to the function objects pointing to C functions, like
|
|
e.g. the prototype function in the CWrapper class, will use this
|
|
attribute.
|
|
|
|
If you take a detailed look at some of the "import libxxx" statements
|
|
they might seem superfluous, in the sense that the active scope does
|
|
not reference the imported library, however in many cases the "import
|
|
libxxxx" statements are for side-effects only.
|
|
|
|
|
|
4.2 Prototyping
|
|
|
|
As mentioned in section 2.1 we must "tell" the function pointers the
|
|
type of the return value and the input arguments. This is done by
|
|
setting the restype and argtypes attributes. For the getenv() function
|
|
which takes one (char *) input and returns a (char *) pointer this
|
|
would be:
|
|
|
|
clib_handle.getenv.restype = ctypes.c_char_p
|
|
clib_handle.getenv.argtypes = [ ctypes.c_char_p ]
|
|
|
|
In the ert python code this is achieved with the helper class CWrapper
|
|
implemented in the module ert.cwrap.cwrap:
|
|
|
|
from ert.cwrap.cwrap import CWrapper
|
|
|
|
lib = clib.load( "libc" )
|
|
wrapper = CWRapper( lib )
|
|
getenv = wrapper.prototype("char* getenv( char* )")
|
|
|
|
The prototype() method of the CWrapper class is essentially one
|
|
massive regular expression which parses the "char* getenv( char* )"
|
|
string and assigns the correct restype and argtypes attributes.
|
|
|
|
|
|
All the calls to the prototype() method are typically assembled in the
|
|
bottom of the file implementing a class defintion. E.g. the bottom of
|
|
the ecl_kw.py file looks like this:
|
|
|
|
# 1: Load the shared library
|
|
cwrapper = CWrapper( libecl.lib )
|
|
|
|
# 2: Register type mapping ecl_kw <--> EclKW
|
|
cwrapper.registerType( "ecl_kw" , EclKW )
|
|
|
|
# 3: Prototype the C functions we will use from Python
|
|
cfunc = CWrapperNameSpace("ecl_kw")
|
|
cfunc.get_size = cwrapper.prototype("int ecl_kw_get_size( ecl_kw )")
|
|
cfunc.get_type = cwrapper.prototype("int ecl_kw_get_type( ecl_kw )")
|
|
cfunc.iget_char_ptr = cwrapper.prototype("char* ecl_kw_iget_char_ptr( ecl_kw , int )")
|
|
cfunc.iset_char_ptr = cwrapper.prototype("void ecl_kw_iset_char_ptr( ecl_kw , int , char*)")
|
|
cfunc.iget_bool = cwrapper.prototype("bool ecl_kw_iget_bool( ecl_kw , int)")
|
|
cfunc.iset_bool = cwrapper.prototype("bool ecl_kw_iset_bool( ecl_kw , int, bool)")
|
|
cfunc.alloc_new = cwrapper.prototype("c_void_p ecl_kw_alloc( char* , int , int )")
|
|
....
|
|
....
|
|
|
|
This makes the function cfunc.get_size, cfunc.get_type,
|
|
cfunc.iget_char_ptr, ... available as first class Python functions,
|
|
and the Python classes can be implemented based on these
|
|
functions. Observe that the return value from "ecl_kw_alloc()" is
|
|
c_void_p' and not ecl_kw - that is because this return value is
|
|
internalized in the c_ptr of the object (this is not particularly
|
|
elegant, and could probably be improved upon..??).
|
|
|
|
The prototyped functions is quite low-level, and they should __NOT__
|
|
be used outside the file scope where they are defined.
|
|
|
|
|
|
4.3 Python classes
|
|
|
|
All the main Python classes wrap a C based structure, and contain
|
|
roughly the same structure:
|
|
|
|
* The underlying C structure is 'held' with the attribute c_ptr,
|
|
c_ptr is just a c_void_p instance which holds the pointer value of
|
|
the underlying C structure. The name c_ptr is just convention, and
|
|
could be anything. The type of c_ptr is 'c_void_p'.
|
|
|
|
Mapping between a class instance and the C pointer held by this
|
|
instance is handled with the 'from_param' function:
|
|
|
|
def from_param( self ):
|
|
return self.c_ptr
|
|
|
|
The name 'from_param' is goverened by ctypes; the from_param
|
|
method is called automatically by the ctypes runtime.
|
|
|
|
* The __init__() function, or alternatively a classmethod
|
|
constructor calls the C based constructor, like
|
|
e.g. ecl_grid_alloc() and internalize the return value in the
|
|
attribute 'c_ptr'; alternatively the c_ptr might come as a shared
|
|
reference from another function.
|
|
|
|
* The __del__ function is called when the Python object is garbage
|
|
collected, then the corresponding C free function should be
|
|
called, e.g. ecl_grid_free().
|
|
|
|
|
|
|
|
|
|
|
|
5 Garbage collection and 'data_owner'
|
|
|
|
The Python language has garbage collection, and as a user of the ert
|
|
python code you should just assume that the garbage collection works
|
|
for the ert py types. However, if you want to wrap new types (or maybe
|
|
fix a bug ...) you should understand how the the ert py classes
|
|
interact with garbage collection.
|
|
|
|
When the Python interpreter decides that an object will be collected,
|
|
the __del__() method of that object is called, and any non-standard
|
|
cleanup code can be called here. In the case of ert py, this is where
|
|
the C based destructor can be called. The important point is that it
|
|
differs from case to case whether the C based constructor should
|
|
indeed be called.
|
|
|
|
The EclKW class illustrates this quite well. Broadly speaking you can
|
|
instantiate an EclKW instance in two ways:
|
|
|
|
* Through the EclwKW.new() function
|
|
* As a reference to an existing ecl_kw through an EclFile instance.
|
|
|
|
In the first case the c_ptr of the EclKW instance will point to fresh
|
|
storage dedicated to this EclKW, and when the EclKW instance goes out
|
|
of scope the memory should be freed with a call to the C function
|
|
ecl_kw_free().
|
|
|
|
EclKW (Python) ecl_kw (C)
|
|
c_ptr
|
|
| ---------
|
|
\--------------->| PORO |
|
|
| REAL |
|
|
| 10000 |
|
|
|-------|
|
|
| 0.15 |
|
|
| 0.19 |
|
|
| ... |
|
|
---------
|
|
|
|
|
|
In the second case an EclFile instance has been created, and this
|
|
object points to ecl_file C struct which is essentially a container
|
|
containing many ecl_kw instances. We can then query the EclFile
|
|
instance to get reference to the various ecl_kw keywords, i.e. for a
|
|
small restart file:
|
|
|
|
|
|
EclFile (Python) ecl_file (C) EclKW (Python)
|
|
c_ptr c_ptr
|
|
| ------------ |
|
|
\---------------->| PRESSURE | |
|
|
| SGAS <---+------------------/
|
|
| SWAT |
|
|
| .... |
|
|
------------
|
|
|
|
The situaton above (which is quite typical) could for instance come
|
|
about from:
|
|
|
|
restart_file = ecl.EclFile( "ECLIPSE.X0057" )
|
|
sgas_kw = restart_file.iget_named_kw( "SGAS" , 0 )
|
|
|
|
Now, when the sgas_kw object goes out of scope it is important that
|
|
the SGAS ecl_kw in the ecl_file file container is _NOT_ deleted, that
|
|
keyword is owned by the ecl_file container and the whole thing will
|
|
crash and burn if the ecl_kw is destroyed bypassing the container. The
|
|
sgas keyword will be destroyed when the restart_file object goes out
|
|
scope at a later stage.
|
|
|
|
To facilitate this difference in behaviour when an EclKW goes out
|
|
scope the object contains a field 'data_owner', and the __del__()
|
|
method looks like this:
|
|
|
|
def __del__( self ):
|
|
if self.data_owner:
|
|
cfunc.free( self )
|
|
|
|
I.e. the C based free() function is only called if the object is the
|
|
owner of the underlying C structure. This technique is implemented in
|
|
many of the objects. In the current code the data_owner field is set
|
|
alongside with the c_ptr field, and not modified during the lifetime
|
|
of the object (i.e. the implementation does not support 'hostile
|
|
takeover' or 'orphaning' of objects).
|
|
|
|
|
|
|
|
6 Installation in Statoil
|
|
|
|
In Statoil the ert Python libraries are installed in the /project/res
|
|
hierarchy.
|