exsinglesv — Singleton process group lock provider. XATMI server.
exsinglesv is a special XATMI server used for providing lock services for singleton process groups. For each singleton group, one copy of the server is required to be configured.
The lock server supports file system-based fnctl() locking. When process acquires such lock, it reports the status to the shared memory, so that ndrxd(8) and cpmsrv(8) can do the booting or failover sequences. At the lock time, secondary "heartbeat" lock file is updated, with the maximum value of counters seen in the file and incrementing that by some value larger than 1 (e.g. set by NDRX_SGLOCKINC ex_env(5) parameter). This ensures that if for some error reason another node continues to do the heartbeats, it would see that it has lost the lock.
The lock server periodically checks the lock health, by doing the following actions:
In case if any checks fail, a singleton process group in shared memory (if was locked) and lock files are unlocked, and the process exits with failure.
Lock provider distinguishes two locking modes:
The status for the process groups can be checked by running the following xadmin(8) command:
$ xadmin psg
exsinglesv has two user exits which may be set in the configuration file, parameters: exec_on_bootlocked and exec_on_locked. The first program is executed in case if lock was acquired during the normal startup, and the second (exec_on_locked) is executed in case if failover has happened (lock files which were locked by other node, were unlocked) or if exsinglesv did crash, was restarted and acquired the lock. If programs specified by these flags return non 0 status, the exsinglesv would unlock any resources it locked and exits with failure.
exsingleckl provides cluster consistency check service used by tmsrv(8), in case if in ndrxconfig.xml(5) tag <procgroups>/<procgroup> attribute sg_nodes_verify is set to Y. For load balancing service can be started in multiple copies. Separate process instances are required for each of the singleton groups.
Advertised service name is "@SGLOC-<node_id>-<grpno>". When request is received by the service, it check the if local node process group is locked, and if so it tries to access network services for other nodes, to verify group statuses (they all must not be locked). If network service is not available for the node, checks are done in read-only mode against heartbeat file (lockfile_2). The current group must have the highest counter number for the whole cluster. If that is not met, the group is unlocked and service reports group not locked.
exsingleckr provides remote check services for the locks. Requests are received from exsinglesv and exsingleckl servers via tpbridge(8). Remote check service read local shared memory singleton process group table and provides group status back to the caller. Service name "@SGREM-<node_id>-<grpno>" is advertised. Several instances of the server can be started. This service is optional, however for performance reasons, it is recommended to configure exsingleckr, regardless of the sg_nodes_verify attribute value, as exsinglesv during the periodic checks, always attempts to perform network call firstly.
Binaries exsinglesv, exsingleckl and exsingleckr uses the same configuration section. For these XATMI servers, tag <procgrp_lp> must be set to group name to which they provide locking services. However sections may be separated by CCTAGs.
For more detailed setup of the singleton groups, functions and limitations, please see more Enduro/X Administration Manual, High availability features section.
Binary is available for Enduro/X version 8.0.10.
Process requires common configuration (ini-files) to be set up for the Enduor/X instance, where exsinglesv is booted.
Process reads settings from "[@exsinglesv[/<NDRX_CCTAG>]]" section.
If doing manual xadmin(8) command based exsinglesv start (or restart/sreload) on booted application, the return from command might delay, as depending on current ndrxd(8) sanity cycle, the singleton process group might pass for startup and only then xadmin will return results to command line back. However this is subject to change for future releases, where after the first boot all start/stop operations on the exsinglesv might be considered as failover recovery. Currently manual start stops, are assumed and locked as fresh boot operations (i.e. doing immediate lock of the group).
This section demonstrates simple configuration for one group. Note that such configuration shall match an all involved cluster nodes which serves the given singleton group.
ndrxconfig.xml demonstrates configuration for the group named "GRPV":
<?xml version="1.0" ?> <endurox> <procgroups> <procgroup grpno="5" name="GRPV" singleton="Y" sg_nodes="1,4" sg_nodes_verify="Y"/> </procgroups> <servers> <!-- lock provider for group 5 --> <server name="exsinglesv"> <!-- only one lock provider for the group! --> <min>1</min> <max>1</max> <srvid>10</srvid> <sysopt>-e ${NDRX_ULOG}/exsinglesv.log -r</sysopt> <procgrp_lp>GRPV</procgrp_lp> <cctag>GRPVCCT</cctag> </server> <!-- support servers, local --> <server name="exsingleckl"> <min>10</min> <max>10</max> <srvid>15</srvid> <sysopt>-e ${NDRX_ULOG}/exsingleckl.log -r</sysopt> <procgrp_lp>GRPV</procgrp_lp> <cctag>GRPVCCT</cctag> </server> <!-- support servers, remote --> <server name="exsingleckr"> <min>10</min> <max>10</max> <srvid>30</srvid> <sysopt>-e ${NDRX_ULOG}/exsingleckr.log -r</sysopt> <procgrp_lp>GRPV</procgrp_lp> <cctag>GRPVCCT</cctag> </server> <!-- banksv1 is configured as singleton in the cluster --> <server name="banksv1"> <min>1</min> <max>1</max> <srvid>120</srvid> <sysopt>-e ${NDRX_ULOG}/banksv1.log -r</sysopt> <procgrp>GRPV</procgrp> </server> ... <!-- for demo purposes, we show configuration for client daemon processes too --> <server name="cpmsrv"> <min>1</min> <max>1</max> <srvid>9999</srvid> <sysopt>-e ${NDRX_ULOG}/cpmsrv.log -r -- -k3 -i1</sysopt> </server> </servers> <clients> <!-- bankcl is also singleton in the cluster --> <client cmdline="bankcl" procgrp="GRPV"> <exec tag="BANK1" subsect="1" autostart="Y" log="${NDRX_ULOG}/bankcl-1.log"/> </client> ... </clients> </endurox>
app.ini
... [@exsinglesv/GRPVCCT] lockfile_1=/path/to/shared/file/system/GRPV_lock_1 lockfile_2=/path/to/shared/file/system/GRPV_lock_2 ...