User Tools

Site Tools


Sidebar

Table Of Contents

endurox-python:v8.0.x:guides:module_developer_guide

Enduro/X Python Internal/External Developer Guide

Enduro/X Python Internal/External Developer Guide


1. Intro

This document is for external and internal purposes of the Enduro/X Python module developer. The document covers details about dependency installation on the target platform, how to build the module, and how to prepare distribution binaries.

Enduro/X Python module uses Pybind11 C++ binding library and Enduro/X invocation code is written in C++. The minimum language level is C++ 11.

Build scripts also rely on "python3" binary (or symlink) which shall be in the PATH. This name is used for detecting the Python version, and this name is used in test scripts.

Module documentation is built by Sphinx. The documentation by itself is embedded in the module, meaning that Python interactive shell can provide help.

This document does not cover the Enduro/X Python API description. For this purpose see:

2. Preparing system

To build Enduro/X Python module, the following dependencies:

  • C++11 (or later) Compiler.
  • CMake, at least version 3.1 or later
  • pkg-config

Optionally, if user guides are built, AsciiDoc must be installed.

For installing C/C++ Compiler and CMake, see Enduro/X Core building instructions. Such as:

For installing pkg-config, see endurox-go-book instructions.

3. General Enduro/X Python architecture details

This section lists details about Enduro/X Python module internals.

3.1. Module structure

Enduro/X Python mode is based on native code binding. Because Python loads shared libraries with RTLD_LOCAL flag, the consequent shared libraries loaded by Enduro/X code (XA libs) do not see the original libraries loaded by the Python. To overcome this problem actual binding code is found in "endurox.endurox" sub-module. The sub-module is automatically loaded with RTLD_GLOBAL flag at the "endurox" module init.

3.2. Build system

The module uses CMake build scripts to perform C++ code builds. This allows the module to utilize Enduro/X build scripts and perform advanced product configuration. CMake configuration and building are activated from setup.py script.

The generic CMake configuration may be started in the project root directory i.e. "$ cmake .". This will generate Makefiles in the project directory and that shall be sufficient for IDEs such as Visual Studio Code to perform full C++ code analysis and builds.

When testing such code (edited from IDE), install it with the setuptools: "$ python3 setup.py install --user".

When switching to a fully Python-controlled build, clean-up the project with: "$ python3 setup.py clean", as Python-controlled builds use different directories for build outputs.

4. Installing Python 3

This section describes tasks to be performed on the given operating system& for getting the required Python 3 version. Additional dependencies are being installed such as pip, wheel and sphinx.

4.1. CentOS/RHEL/Oracle Linux 7.x

To install the Python3 for RHEL system, use the following command:

$ su - root

# yum install python3 python3-devel python3-pip python3-distutils

4.2. CentOS/RHEL/Oracle Linux 8.x

To install the Python3 for RHEL system, use the following command:

$ su - root

# yum install python3 python3-devel python3-pip python3-distutils platform-python-devel

4.3. CentOS/RHEL/Oracle/Rocky Linux 9.x

To install the Python3 for RHEL system, use the following command:

$ su - root

# yum install python3 python3-devel python3-pip platform-python-devel

4.4. Ubuntu systems / Debian / Raspberry systems

$ sudo apt-get install python3 python3-pip python3-dev

4.5. SLES 15:

$ sudo zypper install python3-devel python3-pip

4.6. FreeBSD systems

$ sudo pkg install python38 py38-pip


$ sudo ln -s /usr/local/bin/python3 /usr/local/bin/python3.8

4.7. Installing Python 3 from sources

$ wget https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz

$ tar -xzf Python-3.6.8.tgz

$ cd Python-3.6.8

$ ./configure --prefix=/usr --with-ensurepip=install

$ make

$ sudo make install

$ sudo ln -s /usr/bin/python3.6 /usr/bin/python3

4.8. MacOS

As of MacOS Monterey, OS ships with the Python3 pre-installed.

Building endurox-python on MacOS

To build Enduro/X Python module on the MacOS Monterey or later version, follow environment variables must be set, before running the build:

export SDKROOT="`xcrun --show-sdk-path`"
export CMAKE_MODULE_PATH=/usr/local/share/endurox/cmake
export CFLAGS="-I/usr/local/include"
export CXXFLAGS="-I/usr/local/include"
export LDFLAGS="-L /usr/local/lib"
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
export PATH=$PATH:/usr/local/bin

CFLAGS flags above are mainly used by viewc(5) tool which generates VIEW buffer headers for integration testing.

4.9. Oracle Solaris

Python 3.6+ is not available from CSW, the only option is to build Python 3.6 from the sources. CPython can be built on Solaris OS by the:

  • Oracle Solaris Studio compiler
  • Or GCC compiler.

To perform the build, please see the previous section.

However when building Enduro/X Python module, only GCC compiler is suitable, as Solaris Studio does not support cxx_user_literals syntax.

Before building Python on Solaris, the following environment must be prepared:

$ export CFLAGS="-m64 -D_XOPEN_SOURCE_EXTENDED -I /usr/include/ncurses"

$ export LDFLAGS="-m64"

...

Building endurox-python on Solaris

  • g++ compiler shall be installed:
$ su - root
# /opt/csw/bin/pkgutil -y -i gcc5g++
# /opt/csw/bin/pkgutil -y -i pkgconfig

The following environment variables must be set before running setuptools or pip3:

$ export CXX=g++
$ export CC=gcc
$ export LDFLAGS="-m64 -L/usr/lib64"
$ export CFLAGS="-m64"
$ export CXXFLAGS="-m64"
$ export PKG_CONFIG_PATH=/usr/lib64/pkgconfig
$ export LD_LIBRARY_PATH=/usr/lib64

Additionally, when performing the tests, must ensure that platform dependent library path from (e.g.) /usr/lib/64 comes first in LD_LIBRARY_PATH and only then /opt/csw/lib/64 (if used). For example

$ export LD_LIBRARY_PATH=/usr/lib64:/usr/lib/64:/opt/csw/lib/64

otherwise expect that endurox-python module exceptions will not work and will generate such errors as:

...
terminate called after throwing an instance of 'atmi_exception'

4.10. IBM AIX

IBM AIX v7.3 ships with Python3 which is built with xlC compiler suite. For AIX v7.2, Linux toolbox version of Python3 is available. In the case of Linux toolbox, Python3 3 is built with GCC.

As Enduro/X Python uses C++ Exceptions, the stdc++ library must match with the compiler which is used for building Python3, otherwise exceptions will not work and core dumps will be generated.

So it is recommended that the compiler is matched with the Python compiler.

  • In the case of AIX 7.3, build Enduro/X Python with xlC
  • In the case of AIX 7.2 toolbox version, build Enduro/X Python with GCC

To install the Linux toolbox version of Python3 on AIX use the following commands:

$ su - root

# yum install python3 python3-devel

Building endurox-python on IBM AIX (xlC)

To build Python module, C++11 standard is required to be supported by the compiler. If using IBM xlC compiler suite, then "xlclang++" shall be used as a C++ compiler.

For installing xlC compiler, please follow the instructions from the compiler vendor.

As AIX shared libraries require that all dependencies for external symbols are met, the CMake version shall be at least 3.12

export OBJECT_MODE=64
export CC=xlc
export CXX=xlclang++
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig

Building endurox-python on IBM AIX (GCC)

export CC=gcc
export CXX=g++
export CFLAGS=-maix64
export CXXFLAGS=-maix64
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig

After this configuration standard package build/install will work.

4.11. Installing Python 3 libraries

Once Python is installed on the target operating system, install the sphinx and wheel packages. Installation would be done with pip package manager. Usually, the name of the command is pip or pip3, depending on the operating system.

$ sudo pip3 install sphinx wheel

In case if pip3 loader is not found, try with:

$ python3 -m pip ...

5. Building the module

This section describes the Enduro/X Python build/install procedures, as well as how to perform the testing.

5.1. Getting the source

$ git clone --recursive https://github.com/endurox-dev/endurox-python

Another option is to download the official sources release at https://www.mavimax.com/downloads

$ tar -xzf endurox-8.0.1.tar.gz

$ cd endurox-8.0.1

In the document, it assumed that sources are located in endurox-python directory.

5.2. Building and installing module with setuptools

$ cd endurox-python

$ python3 ./setup.py build

$ python3 ./setup.py install --user

5.3. Building and installing module with pip3

$ cd endurox-python

$ pip3 install .  --user

In case Enduro/X Core is not installed system-wide, a custom Enduro/X Core installation path may be specified in the following environment variables (before running the pip or setup.py installation instructions).

This example specified a custom installation path to /usr/local:

$ export CMAKE_MODULE_PATH=/usr/local/share/endurox/cmake
$ export CXXFLAGS=-I/usr/local/include
$ export LDFLAGS=-L/usr/local/lib
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
$ export PATH=$PATH:/usr/local/bin

-- if previously did attempted the build, clean must be invoked to remove
-- cmake caches:

$ ./setup.py clean

5.4. Cleaning project (for re-building)

$ cd endurox-python

$ python3 ./setup.py clean

5.5. Building documentation

Documentation shall be built after the packages are installed for the user or system wide.

$ cd endurox-python/doc

$ make html

HTML pages are unloaded in "_build/html" directory. Optionally user guides and mapages can be built (in case asciidoc is installed).

$ cd endurox-python/doc/guides

$ make

$ cd ../manpage

$ make

5.6. Preparing packages

Binary distribution in WHL format can be prepared by:

$  python3 ./setup.py bdist_wheel

Source distribution

$  python3 ./setup.py sdist

Distribution files are located in endurox-python/dist directory.

5.7. Testing

Enduro/X Python module contains unit/integration tests. Tests are executed by the following command:

$ cd endurox-python

$ python3 ./setup.py test

Oracle DB tests are not executed as part of this run, due to fact that Oracle DB must be configured prior the test.

5.8. Testing with Oracle DB

If Oracle DB tests are required, system must be configured. Oracle DB must be installed, and Python package cx_oracle shall be installed:

$ pip3 install cx_oracle --user

Database configuration (users, environment variables) are configured as part of the Enduro/X Building Guide, Enduro/X basic Environment configuration for HOME directory. Once the Oracle environment is configured, a test database table must be created. that could be done in the following way (assuming that ~/ndrx_home is properly set):

$ source ~/ndrx_home

$ cd endurox-python/tests/test005_oraclexa/runtime/conf

$ ./sqlplus.run

SQL> @tables.sql

Table created.

finally, Oracle DB test cases can be run:

$ cd endurox-python/tests/test005_oraclexa

$ ./run.sh

6. Troubleshooting

6.1. Different Python versions used for Python code and C++ code build

In case if after the build/install steps, Enduro/X package does fail with error: ModuleNotFoundError: No module named endurox.endurox, for example:

$ python3

>>> import endurox as e
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/user1/Library/Python/3.11/lib/python/site-packages/endurox-8.0.4-py3.11-macosx-13-arm64.egg/endurox/__init__.py", line 13, in <module>
    from .endurox import *
ModuleNotFoundError: No module named 'endurox.endurox'

Verify that C++ shared library is built with the proper Python version. That can be checked by inspecting the module install location:

$ /Users/user1/Library/Python/3.11/lib/python/site-packages/endurox-8.0.4-py3.11-macosx-13-arm64.egg/endurox/

total 2472
-rw-r--r--  1 user1  staff     1194 Dec 10 01:37 __init__.py
drwxr-xr-x  4 user1  staff      128 Dec 10 01:37 __pycache__
-rwxr-xr-x  1 user1  staff  1229024 Dec 10 01:37 endurox.cpython-312-darwin.so
-rw-r--r--  1 user1  staff    25861 Dec 10 01:37 ubfdict.py

Compare the Python version in the shared library name: endurox.cpython-312-darwin.so which corresponds to version 3.12 and module install path ndurox-8.0.4-py3.11-macosx-13-arm64.egg, which suggests that Python code is built and installed with 3.11 version. To work correctly, these versions must match the compiler version from which the module is expected to work.

The issue can be fixed by:

  • Either masking or uninstalling one of the Python versions, which are not expected to be used;
  • Update CMakeLists.txt of the Enduro/X for Python package, with replacing section for Python executable detection:
if(${CMAKE_VERSION} VERSION_EQUAL "3.12.0" OR ${CMAKE_VERSION} VERSION_GREATER "3.12.0")
    # Find the python...
    find_package (Python3 COMPONENTS Development Interpreter)
else()
    set(Python3_EXECUTABLE "python3")
endif()

with hard-coding the binary name, for example:

#if(${CMAKE_VERSION} VERSION_EQUAL "3.12.0" OR ${CMAKE_VERSION} VERSION_GREATER "3.12.0")
    # Find the python...
#    find_package (Python3 COMPONENTS Development Interpreter)
#else()
    set(Python3_EXECUTABLE "python3.11")
#endif()

However this normally shall not be done, and it is expected that CMake would properly resolve the Python version (having the CMake above the 3.12). Also, note that at the build/install time setup.py, does pass the current Python binary name to the CMake.

After any of the above fixes, perform the clean and build/install steps again.

6.2. On MacOS pip3 install of source or binary package might fail

If performing "pip3 install . --user" of the source or binary wheel package and error such as is generated:

$ pip3 install . --user
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
...

Behavior can be overridden (installation process fixed) by:

$ mkdir -p ~/.config/pip/

$ cat << EOF > ~/.config/pip/pip.conf
[global]
break-system-packages = true
EOF

After the pip.conf edit, retry the installation process with the pip3.

However, another option is to follow the installation process of the Enduro/X for Python3 by using source package and "python3 ./setup.py install --user" command, which with default configuration does succeed (no above error generated).

7. Conclusions

This instruction showed how to perform common tasks, starting from module build, and installation and completing it with the module testing. The given information shall be enough to start to dig into module development details. Also, the document can help to prepare module for systems for which the official build is not available or Python 3 is not available.