Table of Contents
This document is for external and internal purposes of the Enduro/X Python module developer. The document covers details about dependency installation on the target platform, how to build the module, and how to prepare distribution binaries.
Enduro/X Python module uses Pybind11 C++ binding library and Enduro/X invocation code is written in C++. The minimum language level is C++ 11.
Build scripts also rely on "python3" binary (or symlink) which shall be in the PATH. This name is used for detecting the Python version, and this name is used in test scripts.
Module documentation is built by Sphinx. The documentation by itself is embedded in the module, meaning that Python interactive shell can provide help.
This document does not cover the Enduro/X Python API description. For this purpose see:
To build Enduro/X Python module, the following dependencies:
Optionally, if user guides are built, AsciiDoc must be installed.
For installing C/C++ Compiler and CMake, see Enduro/X Core building instructions. Such as:
For installing pkg-config, see endurox-go-book instructions.
This section lists details about Enduro/X Python module internals.
Enduro/X Python mode is based on native code binding. Because Python loads shared libraries with RTLD_LOCAL flag, the consequent shared libraries loaded by Enduro/X code (XA libs) do not see the original libraries loaded by the Python. To overcome this problem actual binding code is found in "endurox.endurox" sub-module. The sub-module is automatically loaded with RTLD_GLOBAL flag at the "endurox" module init.
The module uses CMake build scripts to perform C++ code builds. This allows the module to utilize Enduro/X build scripts and perform advanced product configuration. CMake configuration and building are activated from setup.py script.
The generic CMake configuration may be started in the project root directory i.e. "$ cmake .". This will generate Makefiles in the project directory and that shall be sufficient for IDEs such as Visual Studio Code to perform full C++ code analysis and builds.
When testing such code (edited from IDE), install it with the setuptools: "$ python3 setup.py install --user".
When switching to a fully Python-controlled build, clean-up the project with: "$ python3 setup.py clean", as Python-controlled builds use different directories for build outputs.
This section describes tasks to be performed on the given operating system& for getting the required Python 3 version. Additional dependencies are being installed such as pip, wheel and sphinx.
To install the Python3 for RHEL system, use the following command:
$ su - root # yum install python3 python3-devel python3-pip python3-distutils
To install the Python3 for RHEL system, use the following command:
$ su - root # yum install python3 python3-devel python3-pip python3-distutils platform-python-devel
To install the Python3 for RHEL system, use the following command:
$ su - root # yum install python3 python3-devel python3-pip platform-python-devel
$ sudo apt-get install python3 python3-pip python3-dev
$ sudo pkg install python38 py38-pip $ sudo ln -s /usr/local/bin/python3 /usr/local/bin/python3.8
$ wget https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz $ tar -xzf Python-3.6.8.tgz $ cd Python-3.6.8 $ ./configure --prefix=/usr --with-ensurepip=install $ make $ sudo make install $ sudo ln -s /usr/bin/python3.6 /usr/bin/python3
As of MacOS Monterey, OS ships with the Python3 pre-installed.
To build Enduro/X Python module on the MacOS Monterey or later version, follow environment variables must be set, before running the build:
export SDKROOT="`xcrun --show-sdk-path`" export CMAKE_MODULE_PATH=/usr/local/share/endurox/cmake export CFLAGS="-I/usr/local/include" export CXXFLAGS="-I/usr/local/include" export LDFLAGS="-L /usr/local/lib" export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig export PATH=$PATH:/usr/local/bin
CFLAGS flags above are mainly used by viewc(5) tool which generates VIEW buffer headers for integration testing.
Python 3.6+ is not available from CSW, the only option is to build Python 3.6 from the sources. CPython can be built on Solaris OS by the:
To perform the build, please see the previous section.
However when building Enduro/X Python module, only GCC compiler is suitable, as Solaris Studio does not support cxx_user_literals syntax.
Before building Python on Solaris, the following environment must be prepared:
$ export CFLAGS="-m64 -D_XOPEN_SOURCE_EXTENDED -I /usr/include/ncurses" $ export LDFLAGS="-m64" ...
$ su - root # /opt/csw/bin/pkgutil -y -i gcc5g++ # /opt/csw/bin/pkgutil -y -i pkgconfig
The following environment variables must be set before running setuptools or pip3:
$ export CXX=g++ $ export CC=gcc $ export LDFLAGS="-m64 -L/usr/lib64" $ export CFLAGS="-m64" $ export CXXFLAGS="-m64" $ export PKG_CONFIG_PATH=/usr/lib64/pkgconfig $ export LD_LIBRARY_PATH=/usr/lib64
Additionally, when performing the tests, must ensure that platform dependent library path from (e.g.) /usr/lib/64 comes first in LD_LIBRARY_PATH and only then /opt/csw/lib/64 (if used). For example
$ export LD_LIBRARY_PATH=/usr/lib64:/usr/lib/64:/opt/csw/lib/64
otherwise expect that endurox-python module exceptions will not work and will generate such errors as:
... terminate called after throwing an instance of 'atmi_exception'
IBM AIX v7.3 ships with Python3 which is built with xlC compiler suite. For AIX v7.2, Linux toolbox version of Python3 is available. In the case of Linux toolbox, Python3 3 is built with GCC.
As Enduro/X Python uses C++ Exceptions, the stdc++ library must match with the compiler which is used for building Python3, otherwise exceptions will not work and core dumps will be generated.
So it is recommended that the compiler is matched with the Python compiler.
To install the Linux toolbox version of Python3 on AIX use the following commands:
$ su - root # yum install python3 python3-devel
To build Python module, C++11 standard is required to be supported by the compiler. If using IBM xlC compiler suite, then "xlclang++" shall be used as a C++ compiler.
For installing xlC compiler, please follow the instructions from the compiler vendor.
As AIX shared libraries require that all dependencies for external symbols are met, the CMake version shall be at least 3.12
export OBJECT_MODE=64 export CC=xlc export CXX=xlclang++ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig
Once Python is installed on the target operating system, install the sphinx and wheel packages. Installation would be done with pip package manager. Usually, the name of the command is pip or pip3, depending on the operating system.
$ sudo pip3 install sphinx wheel
In case if pip3 loader is not found, try with:
$ python3 -m pip ...
This section describes the Enduro/X Python build/install procedures, as well as how to perform the testing.
$ git clone --recursive https://github.com/endurox-dev/endurox-python
Another option is to download the official sources release at https://www.mavimax.com/downloads
$ tar -xzf endurox-8.0.1.tar.gz $ cd endurox-8.0.1
In the document, it assumed that sources are located in endurox-python directory.
$ cd endurox-python $ python3 ./setup.py build $ python3 ./setup.py install --user
$ cd endurox-python $ pip3 install . --user
In case Enduro/X Core is not installed system-wide, a custom Enduro/X Core installation path may be specified in the following environment variables (before running the pip or setup.py installation instructions).
This example specified a custom installation path to /usr/local:
$ export CMAKE_MODULE_PATH=/usr/local/share/endurox/cmake $ export CXXFLAGS=-I/usr/local/include $ export LDFLAGS=-L/usr/local/lib $ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig $ export PATH=$PATH:/usr/local/bin -- if previously did attempted the build, clean must be invoked to remove -- cmake caches: $ ./setup.py clean
Documentation shall be built after the packages are installed for the user or system wide.
$ cd endurox-python/doc $ make html
HTML pages are unloaded in "_build/html" directory. Optionally user guides and mapages can be built (in case asciidoc is installed).
$ cd endurox-python/doc/guides $ make $ cd ../manpage $ make
Binary distribution in WHL format can be prepared by:
$ python3 ./setup.py bdist_wheel
Source distribution
$ python3 ./setup.py sdist
Distribution files are located in endurox-python/dist directory.
Enduro/X Python module contains unit/integration tests. Tests are executed by the following command:
$ cd endurox-python $ python3 ./setup.py test
Oracle DB tests are not executed as part of this run, due to fact that Oracle DB must be configured prior the test.
If Oracle DB tests are required, system must be configured. Oracle DB must be installed, and Python package cx_oracle shall be installed:
$ pip3 install cx_oracle --user
Database configuration (users, environment variables) are configured as part of the Enduro/X Building Guide, Enduro/X basic Environment configuration for HOME directory. Once the Oracle environment is configured, a test database table must be created. that could be done in the following way (assuming that ~/ndrx_home is properly set):
$ source ~/ndrx_home $ cd endurox-python/tests/test005_oraclexa/runtime/conf $ ./sqlplus.run SQL> @tables.sql Table created.
finally, Oracle DB test cases can be run:
$ cd endurox-python/tests/test005_oraclexa $ ./run.sh
In case if after the build/install steps, Enduro/X package does fail with error: ModuleNotFoundError: No module named endurox.endurox, for example:
$ python3 >>> import endurox as e Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/user1/Library/Python/3.11/lib/python/site-packages/endurox-8.0.4-py3.11-macosx-13-arm64.egg/endurox/__init__.py", line 13, in <module> from .endurox import * ModuleNotFoundError: No module named 'endurox.endurox'
Verify that C++ shared library is built with the proper Python version. That can be checked by inspecting the module install location:
$ /Users/user1/Library/Python/3.11/lib/python/site-packages/endurox-8.0.4-py3.11-macosx-13-arm64.egg/endurox/ total 2472 -rw-r--r-- 1 user1 staff 1194 Dec 10 01:37 __init__.py drwxr-xr-x 4 user1 staff 128 Dec 10 01:37 __pycache__ -rwxr-xr-x 1 user1 staff 1229024 Dec 10 01:37 endurox.cpython-312-darwin.so -rw-r--r-- 1 user1 staff 25861 Dec 10 01:37 ubfdict.py
Compare the Python version in the shared library name: endurox.cpython-312-darwin.so which corresponds to version 3.12 and module install path ndurox-8.0.4-py3.11-macosx-13-arm64.egg, which suggests that Python code is built and installed with 3.11 version. To work correctly, these versions must match the compiler version from which the module is expected to work.
The issue can be fixed by:
if(${CMAKE_VERSION} VERSION_EQUAL "3.12.0" OR ${CMAKE_VERSION} VERSION_GREATER "3.12.0") # Find the python... find_package (Python3 COMPONENTS Development Interpreter) else() set(Python3_EXECUTABLE "python3") endif()
with hard-coding the binary name, for example:
#if(${CMAKE_VERSION} VERSION_EQUAL "3.12.0" OR ${CMAKE_VERSION} VERSION_GREATER "3.12.0") # Find the python... # find_package (Python3 COMPONENTS Development Interpreter) #else() set(Python3_EXECUTABLE "python3.11") #endif()
However this normally shall not be done, and it is expected that CMake would properly resolve the Python version (having the CMake above the 3.12). Also, note that at the build/install time setup.py, does pass the current Python binary name to the CMake.
After any of the above fixes, perform the clean and build/install steps again.
If performing "pip3 install . --user" of the source or binary wheel package and error such as is generated:
$ pip3 install . --user error: externally-managed-environment × This environment is externally managed ╰─> To install Python packages system-wide, try brew install xyz, where xyz is the package you are trying to install. ...
Behavior can be overridden (installation process fixed) by:
$ mkdir -p ~/.config/pip/ $ cat << EOF > ~/.config/pip/pip.conf [global] break-system-packages = true EOF
After the pip.conf edit, retry the installation process with the pip3.
However, another option is to follow the installation process of the Enduro/X for Python3 by using source package and "python3 ./setup.py install --user" command, which with default configuration does succeed (no above error generated).
This instruction showed how to perform common tasks, starting from module build, and installation and completing it with the module testing. The given information shall be enough to start to dig into module development details. Also, the document can help to prepare module for systems for which the official build is not available or Python 3 is not available.