Servers using XA failing to start because of missing ndrx_get_xa_switch

Added by Pablo over 7 years ago

Trying to start services that are using Oracle XA. They fail to start with following:

NDRX:5:31151:000:20160908:163502195:i/init.c:0194:Max servers set to 10000
NDRX:5:31151:000:20160908:163502195:i/init.c:0210:Max services set to 10000
NDRX:5:31151:000:20160908:163502195:i/init.c:0226:Random key set to: [fWreqXvs]
NDRX:5:31151:000:20160908:163502195:i/init.c:0242:Posix queue msg_max set to: [100]
NDRX:5:31151:000:20160908:163502195:i/init.c:0258:Posix queue msgsize_max set to: [10000]
NDRX:5:31151:000:20160908:163502195:i/init.c:0276:Posix queue prefix set to: [/ctdrun]
NDRX:5:31151:000:20160908:163502195:i/init.c:0294:Posix queue queue path set to: [/dev/mqueue]
NDRX:5:31151:000:20160908:163502195:i/init.c:0310:SystemV SEM IPC Key set to: [41000]
NDRX:5:31151:000:20160908:163502195:i/init.c:0317:Using comms timeout: 120
NDRX:5:31151:000:20160908:163502195:i/init.c:0334:Cluster node id=1
NDRX:5:31151:000:20160908:163502195:i/init.c:0352:NDRX_LDBAL set to 0
NDRX:5:31151:000:20160908:163502195:i/init.c:0362:[NDRX_CLUSTERISED] says: We run in cluster mode
NDRX:5:31151:000:20160908:163502195:i/init.c:0399:[NDRX_XA_CLOSE_STR]: XA Close String defaulted to: []
NDRX:5:31151:000:20160908:163502195:i/init.c:0428:[NDRX_XA_LAZY_INIT]: Lazy XA Init: TRUE
NDRX:5:31151:000:20160908:163502195:i/init.c:0495:[NDRX_NRSEMS]: Number of services shared memory semaphores set to: 30 (used only for poll() mode) (default: 30)
NDRX:5:31151:000:20160908:163502195:i/init.c:0516:[NDRX_MAXSVCSRVS]: Max number of local servers per service set to: 30 (used only for poll() mode) (default: 30)
NDRX:5:31151:000:20160908:163502195:i/init.c:0520:env loaded ok
NDRX:5:31151:000:20160908:163502195:rvmain.c:0322:Using comms timeout: 120
NDRX:3:31151:000:20160908:163502195:i/atmi.c:0474:Trying to tpfree NULL buffer!
NDRX:5:31151:000:20160908:163502195:ed_buf.c:0203:_tpalloc: type=FML32 len=1024
UBF :5:31151:000:20160908:163502195:bf/ubf.c:0096:Binit p_ub=0x229ff30 len=1076
UBF :5:31151:000:20160908:163502195:bf/ubf.c:1233:Balloc: Returning 0x229ff30!
NDRX:5:31151:000:20160908:163502195:i/init.c:0144:env already loaded...
NDRX:5:31151:000:20160908:163502195:i/init.c:0801:Got prefix [/ctdrun]
NDRX:5:31151:000:20160908:163502195:i/init.c:0115:Got free context id=0
NDRX:5:31151:000:20160908:163502195:i/init.c:0124:Returning context id=0
NDRX:5:31151:000:20160908:163502195:i/init.c:0830:my_id=[srv,hcrdstatsv,8182,31151,1]
NDRX:5:31151:000:20160908:163502195:i/init.c:0849:Client queue [/ctdrun,clt,reply,hcrdstatsv,31151,0] opened.
NDRX:5:31151:000:20160908:163502195:i/init.c:0701:NDRXD queue: [/ctdrun,sys,bg,ndrxd]
NDRX:5:31151:000:20160908:163502195:mi/sem.c:0083:Using service semaphore key: [266240]
NDRX:5:31151:000:20160908:163502195:mi/sem.c:0098:ndrx_sem_attach enter
NDRX:5:31151:000:20160908:163502195:mi/sem.c:0133:sem: [229377] attached
NDRX:5:31151:000:20160908:163502195:mi/sem.c:0137:ndrx_sem_attach return 0
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0105:G_srvinfo.size = 12440000 (1244 * 10000)
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0109:G_svcinfo.size = 7440000 (372 * 20000)
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0113:G_brinfo.size = 7440000 (4 * 32)
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0334:ndrx_shm_attach enter
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0370:Shm: [/ctdrun,shm,svcinfo] attach
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0381:ndrx_shm_attach return 0
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0334:ndrx_shm_attach enter
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0370:Shm: [/ctdrun,shm,brinfo] attach
NDRX:5:31151:000:20160908:163502195:mi/shm.c:0381:ndrx_shm_attach return 0
NDRX:2:31151:000:20160908:163502195:tmi/xa.c:0166:Failed to get symbol `ndrx_get_xa_switch': 
NDRX:3:31151:000:20160908:163502195:perror.c:0267:_TPset_error_fmt: 12 (TPESYSTEM) [Failed to get symbol `ndrx_get_xa_switch': ]
NDRX:2:31151:000:20160908:163502195:rvmain.c:0399:tpsvrinit() fail
NDRX:3:31151:000:20160908:163502195:i/ulog.c:0104:Failed to open [{MOSTDIR}/tmp/ULOG.20160908]
NDRX:2:31151:000:20160908:163502195:tmi/xa.c:0166:Failed to get symbol `ndrx_get_xa_switch': 
31151:20160908:16350219:??????:statsv.c:  53:tpopen() fail: 12:[12:TPESYSTEM (last error 12: Failed to get symbol `ndrx_get_xa_switch': )]
31151:20160908:16350219:hcrdstatsv  :tpsvrinit() fail
31151:20160908:16350219:??????:statsv.c:  73:tpclose() fail: 12:[12:TPESYSTEM (last error 12: Failed to get symbol `ndrx_get_xa_switch': )]
Server exit: -1, id: 8182
NDRX:3:31151:000:20160908:163502195:perror.c:0267:_TPset_error_fmt: 12 (TPESYSTEM) [Failed to get symbol `ndrx_get_xa_switch': ]
NDRX:2:31151:000:20160908:163502195:mi/shm.c:0150:cannot close shm [/ctdrun,shm,srvinfo] as fd is -1
NDRX:5:31151:000:20160908:163502195:v/init.c:0389:epollfd = 0
Error: 12:TPESYSTEM (last error 12: Failed to get symbol `ndrx_get_xa_switch': )

Here are my library PATHes:
LD_LIBRARY_PATH=/lib:/usr/lib/pa20_64:/usr/local/lib:/ndrtools/most/lib/64bit:/ndrtools/oracle/v12c1/lib:/ndrtools/enduro/dist/lib:/most/runtime/project/ctd/lib:/most/runtime/project/ctd/lib/CUST:/most/runtime/project/ctd/lib/BASE:/ndrtools/oracle/v12c1/rdbms/lib
LD_LIBRARY_PATH_64=/lib:/usr/lib/pa20_64:/usr/local/lib:/ndrtools/most/lib/64bit:/ndrtools/oracle/v12c1/lib:/ndrtools/enduro/dist/lib:/most/runtime/project/ctd/lib:/most/runtime/project/ctd/lib/CUST:/most/runtime/project/ctd/lib/BASE:/ndrtools/oracle/v12c1/rdbms/lib
LIBPATH=/opt/freeware/lib:/lib:/usr/lib/pa20_64:/usr/local/lib:/ndrtools/most/lib/64bit:/ndrtools/oracle/v12c1/rdbms/lib:/ndrtools/enduro/dist/lib:/most/runtime/project/ctd/lib:/most/runtime/project/ctd/lib/CUST:/most/runtime/project/ctd/lib/BASE
SHLIB_PATH=/lib:/usr/lib/pa20_64:/usr/local/lib:/ndrtools/most/lib/64bit:/ndrtools/oracle/v12c1/lib:/ndrtools/enduro/dist/lib:/most/runtime/project/ctd/lib:/most/runtime/project/ctd/lib/CUST:/most/runtime/project/ctd/lib/BASE


Replies (9)

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Pablo over 7 years ago

Forgot to mention - Enduro/X is installed in /ndrtools/enduro/dist/lib

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Madars over 7 years ago

Can you please given me output of command:

$ xadmin pe | grep NDRX_

will print the enviroment of Enduro/X which is actually loaded.

And can you please give me endurox.ini (file of NDRX_CCONFIG) [@global] section.

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Pablo over 7 years ago

[ctd@hce1 conf]$ xadmin pe | grep NDRX_
Enduro/X 3.2.2, build Sep  5 2016 12:44:31, using epoll for LINUX (64 bits)

Enduro/X Middleware Platform for Distributed Transaction Processing
Copyright (C) 2015, 2016 ATR Baltic, SIA. All Rights Reserved.

This software is released under one of the following licenses:
GPLv2 (or later) or ATR Baltic's license for commercial use.

ndrxd PID (from PID file): 3858
NDRX_HOME=/ndrtools/enduro/dist
NDRX_CCONFIG=/most/runtime/project/ctd/conf/ndrxconf.ini
NDRX_APPHOME=/most/runtime/project/ctd
NDRX_CLUSTERISED=1
NDRX_CMDWAIT=1
NDRX_CONFIG=/most/runtime/project/ctd/conf/ndrxconfig.xml
NDRX_DMNLEV=5
NDRX_DMNLOG=/most/runtime/project/ctd/tmp/ndrxd.log
NDRX_DPID=/most/runtime/project/ctd/ndrxd.pid
NDRX_DQMAX=3000
NDRX_IPCKEY=41000
NDRX_LDBAL=0
NDRX_LEV=5
NDRX_LOG={MOSTDIR}/tmp/ndrx.log
NDRX_MSGMAX=100
NDRX_MSGSIZEMAX=10000
NDRX_NODEID=1
NDRX_QPATH=/dev/mqueue
NDRX_QPREFIX=/ctdrun
NDRX_RNDK=fWreqXvs
NDRX_SHMPATH=/dev/shm
NDRX_SRVMAX=10000
NDRX_SVCMAX=20000
NDRX_TOUT=120
NDRX_UBFMAXFLDS=16000
NDRX_ULOG={MOSTDIR}/tmp
NDRX_XA_LAZY_INIT=1
[ctd@hce1 conf]$ 

and
[ctd@hce1 conf]$ cat ndrxconf.ini 
[@global]
NDRX_APPHOME=${MOSTDIR}
NDRX_CLUSTERISED=1
NDRX_CMDWAIT=1
NDRX_CONFIG=${MOSTDIR}/conf/ndrxconfig.xml
NDRX_DMNLEV=5
NDRX_DMNLOG=${MOSTDIR}/tmp/ndrxd.log
NDRX_DPID=${MOSTDIR}/ndrxd.pid
NDRX_DQMAX=3000
NDRX_HOME=/ndrtools/enduro/dist
NDRX_IPCKEY=41000
NDRX_LDBAL=0
NDRX_LEV=5
NDRX_LOG={MOSTDIR}/tmp/ndrx.log
NDRX_MSGMAX=100
NDRX_MSGSIZEMAX=10000
NDRX_NODEID=1
NDRX_QPATH=/dev/mqueue
NDRX_QPREFIX=/ctdrun
NDRX_RNDK=fWreqXvs
NDRX_SHMPATH=/dev/shm
NDRX_SRVMAX=10000
NDRX_SVCMAX=20000
NDRX_TOUT=120
NDRX_UBFMAXFLDS=16000
NDRX_ULOG={MOSTDIR}/tmp
NDRX_XA_LAZY_INIT=1

[@debug]
*=ndrx=5 ubf=1 lines=1 bufsz=1 file=${MOSTDIR}/tmp/enduro.log
xadmin= ndrx=5 file=${MOSTDIR}/tmp/xadmin.log
ndrxd=ndrx=1 ubf=5 lines=1 bufsz=1000 file=${MOSTDIR}/tmp/ndrxd.log
hcrdstatsv=ndrx=5 ubf=5 lines=5 bufsz=1000 file=${MOSTDIR}/tmp/HCRDSTATSV

[ctd@hce1 conf]$ 

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Madars over 7 years ago

I see few problems here:

1. The XA settings are missing:

       NDRX_XA_RES_ID=XA_RESOURCE_ID
           XA API two phase commit resource id. Must be set if using distributed transactions.

       NDRX_XA_OPEN_STR=XA_OPEN_STRING
           This is open string for two phase commit XA driver. Must be set if using distributed transactions.

       NDRX_XA_CLOSE_STR=XA_CLOSE_STRING
           This is close string for two phase commit XA driver. This is optional, NDRX_XA_OPEN_STR will be used if not set.

       NDRX_XA_DRIVERLIB=ENDUROX_XA_DRIVER_LIB
           This is full path to Enduro/X shared library which loads the XA api.

       NDRX_XA_RMLIB=RESOURCE_MANAGER_SPECIFIC_DRIVER
           This is full path to resource manager’s specific driver (shared library). Must be set if using distributed transactions.

Sample for Oracle static XA driver:

# XA SECTION
export ORACLE_SID=ROCKY
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export NDRX_XA_RES_ID=1
export NDRX_XA_OPEN_STR="ORACLE_XA+SqlNet=ROCKY+ACC=P/endurotest/endurotest1+SesTM=180+LogDir=/tmp/xa+nolocal=f" 
export NDRX_XA_CLOSE_STR=$NDRX_XA_OPEN_STR
export NDRX_XA_DRIVERLIB=/usr/lib/libndrxxaoras.so
export NDRX_XA_RMLIB=/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
export NDRX_XA_LAZY_INIT=1

or for common-config:

[@global]
...
# XA SECTION
ORACLE_SID=ROCKY
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
NDRX_XA_RES_ID=1
NDRX_XA_OPEN_STR="ORACLE_XA+SqlNet=ROCKY+ACC=P/endurotest/endurotest1+SesTM=180+LogDir=/tmp/xa+nolocal=f" 
NDRX_XA_CLOSE_STR=$NDRX_XA_OPEN_STR
NDRX_XA_DRIVERLIB=/usr/lib/libndrxxaoras.so
NDRX_XA_RMLIB=/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
NDRX_XA_LAZY_INIT=1

2. Not related, but "{MOSTDIR}/tmp" I guess you mean "${MOSTDIR}/tmp"

3. I would recommend you to upgrade to "Enduro/X 3.3.1", it have fixes for common config & debugging (just do the git pull & rebuild).

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Pablo over 7 years ago

About the XA stuff - right, I missed this.
But when I add it to config ini file like this:

NDRX_XA_RES_ID=1
NDRX_XA_OPEN_STR=ORACLE_XA+SqlNet=SID+ACC=P/USER/password+SesTM=180+LogDir=/tmp/xa+nolocal=f 
NDRX_XA_CLOSE_STR=ORACLE_XA+SqlNet=SID+ACC=P/USER/password+SesTM=180+LogDir=/tmp/xa+nolocal=f
NDRX_XA_DRIVERLIB=/ndrtools/enduro/dist/lib/libndrxxaoras.so

config becomes wrong and giving me:
[ctd@hce1 conf]$ xa stop -y
Failed to initialize!
[ctd@hce1 conf]$ 

How can I get more detailed output?

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Pablo over 7 years ago

I tried to set NDRX_XA_OPEN_STR value with quotes, but it doesn't make any difference.

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Madars over 7 years ago

Logs must be in xadmin= ndrx=5 file=${MOSTDIR}/tmp/xadmin.log

check the xadmin.log

RE: Servers using XA failing to start because of missing ndrx_get_xa_switch - Added by Pablo over 7 years ago

It says:

NDRX:2: 1151:000:20160908:170416556:i/init.c:0444:Invalid XA configuration, missing NDRX_XA_OPEN_STR or NDRX_XA_CLOSE_STR or NDRX_XA_DRIVERLIB or NDRX_XA_RMLIB keys...

NDRX_XA_RMLIB is what is missing.
Added it (pointing to /ndrtools/oracle/v12c1/lib/libclntsh.so.12.1) and it worked!

Thank you!!

(1-9/9)