Sidebar
endurox:v8.0.x:manuals:tmqueue.8
Name
tmqueue — Transactional Message Queue server.
DESCRIPTION
This is a special ATMI server that is used for transactional persistent message queue operations. tmqueue is backend server used by tpenqueue and tpdequeue() calls. Queue server works in pair with tmsrv(8) instance, where both tmqueue and tmsrv are configured in XA environment with NDRX_XA_RMLIB=libndrxxaqdisk.so (actual disk driver) and NDRX_XA_DRIVERLIB=libndrxxaqdisks.so (static registration) or NDRX_XA_DRIVERLIB=libndrxxaqdiskd.so (dynamic registration) driver loaders. Related tmsrv instance shall be started first, before tmqueue process.
Actual queue configuration is described either in configuration file q.conf(5) which may be present as a separate file or as part of the common-configuration ini section.
Current storage backends uses file system for persistence. For more details on back-end configuration see libndrxxaqdisks(8).
The tpenqueue(3) and tpdequeue(3) can be invoked either as part of the global transaction or not. If global transaction is not used, then tmqueue will open the transactions internally.
The queue server internally uses thread pools for handling the work. Multiple thread pools are used to avoid the deadlocking where internal processing might do the XA driver calls which in the end sends notifications to the same server back.
Three thread-pools are used:
1) The first one is for accepting the ATMI tpcall(3) requests.
2) The second thread pool is used by forward services (which basically are senders for automatic queues).
3) The third thread pool is used by notifications received from transaction manager ™. TM notifies the queue server for XA operation progress (prepare, abort, commit).
Every instance of tmqueue will advertise following list of services:
- @TMQ-<Cluster (or Virtual) Node ID>-<Server ID>
- @QSP<Queue space>
For example for Queue Space MYSPACE, Cluster Node ID 6, Enduro/X Server ID 100 services will look like:
- @TMQ-6-100
- @QSPMYSPACE
The automatic forwarder pool, ever configured time, will scan the automatic queues for non-locked messages. Once such message is found, the message is submitted for worker thread. The worker thread will do the synchronous call to target server (srvnm from q.conf), wait for answer and either update tries counter or remove the message if succeed. If message is submitted with TPQREPLYQ then on success, the response message from invoked service is submitted to reply queue. If message destination service fails for number of attempts, then message is forwarded to TPQFAILUREQ, in this case original TPQCTL is preserved.
Tmqueue server is optimized for waking up the forwarder dispatch thread if new processable message is enqueued in automatic queue. If thread which enqueued new message detects that forwarder thread sleeps, then immediate forwarder thread wakeup is performed. This approach does not synchronize fully with dispatcher thread (for performance reason), thus it might be still possible that scan time sleep is done, even processable message was enqueued.
During the startup, tmqueue will read messages from the XA backend driver. All messages are read. All active messages found in persistent storage are rolled back at startup. Only prepared and committed commands/messages are read and stored in message lists. Any prepared commands automatically puts the lock marking on corresponding messages.
In case if tmqueue was shutdown with incomplete transactions:
- Active transactions are rolled back at startup. Any request from tmsrv regarding these transactions will return XAER_NOTA error.
- Tmsrv may rollback any prepared transaction (if needed).
- If for some reason tmsrv lost the track of any prepared transaction, tmrecovercl(8), tmrecoversv(8) may be used to rollback such prepared transactions.
Tmqueue server for any transaction keeps internal accounting with active transaction expiry setting. If transaction is not completed in given time frame, the tmqueue will automatically rollback the transaction.
For more usage of the persistent queues, configuration, command line and codding samples, see atmitests/test028_tmq folder.
OPTIONS
- [-q QUEUE_CONFIG]
- This is either full or relative path to the queue definition file (see q.conf(5) for more info on syntax. When changes are done in this file, it is possible to reload the configuration during the runtime, with xadmin mqrc command. Note that queues are not removed, but changed or added. If flag is not set, then data is searched in [@queue] ini section. And uses <cctag> for sub-section search, if multiple Q-spaces are defined. Firstly it attempts to load form this flag-specified configuration file, afterwards attempt is made to load from CConfig if available.
- [-s SCAN_TIME]
- This time in seconds used by main forwarder thread to scan for any automatic messages to be submitted for processing.
- [-p SERVICE_THREAD_POOL_SIZE]
- This is thread pool size of used for tpenqueue(), tpdequeue() serving. Also this thread pool is utilized by xadmin mqX commands. Default is 10.
- [-u NOTIFY_THREAD_POOL_SIZE]
- This is number of threads which are processing callbacks from XA driver (for operation completion notifications). Default value is 10.
- [-f FORWARD_THREAD_POOL_SIZE]
- This is number of worker threads for forwarder. Default value is 10.
- [-t XA_TRANSACTION_TIMEOUT]
- Maximum number of seconds to wait for XA transaction to be completed. This used internally when global transactions are not used. Default is 30.
- [-T XA_SESSION_TIMEOUT]
- Session timeout for in-active transactions in seconds - if transaction is not completed ( no tpcommit(3) or tpabort(3) performed), or new tpenqueue(3) or tpdequeue(3) is not performed in given time frame, the transaction internally is rolled back. When new tpenqueue(3) or tpdequeue(3) is performed in the global transaction, the time-out counter is reset. After internal rollback any further enqueue/dequeue operations in global transaction will return TPETRAN error. Commit of such global transaction would be terminated with TPEABORT. Value 0 disables transaction sessions checks. It is strongly discouraged to use 0, as this in case of system error may lock resources until the tmqueue server process is restarted. Default value is 180.
- [-N]
- If flag is set, the enqueue operations will not trigger immediate forwarder thread wakeup.
- [-n ENDUROX_NODE_ID]
- Indicates the virtual Enduro/X cluster node ID. If the parameter is not set, then given parameter matches the local application domain node ID set in NDRX_NODEID environment variable. Parameter normally is set to some common cluster node id number for singleton process group operations. So that in case of group failover, the tmqueue server from shared storage would read, recognize and process failed (other) node’s persisted messages. Additionally, it is required that <srvid> for the <server> tag in ndrxconfig.xml(5) used by tmqueue matches the failed nodes <srvid> value. At event of group failover to other Enduor/X node, if Node ID or Server ID does not match, existing stored messages would be ignored.
- [-X NR_OF_SECONDS]
- Number of seconds for periodic verification of the committed message directory. Functionality is enabled in case if value is greater than 0. Granularity of the flag is set by -s parameter. Processing of committed directory scans are done by notification thread (from the thread pool, setting -u). Verification process ensures that all messages on the disk are loaded into memory, which is normal operation of the tmqueue. If any unregistered message is found on the disk, that triggers tmqueue to restart (load all messages from the disk and continue operations. Note that restart would abort any active tmqueue transactions) and for restart period of time (including time needed for queue space message reading and loading from the disk to memory), the queue space would not be available. Normally setting is not required, however with singleton failover groups if in the cluster STONITH device is not used with the shared file system and if shared file system does not become read-only or un-available at the moment of the failure, for certain node failures and node removal from the cluster scenarios (such as node suspend, timeout, other node takes the lock, and then failed node is resumed), for small period of time duplicate runs of the tmqueue processes might be possible. During this duplicate run there is small chance that message is moved to committed directory by the lock lost tmqueue instance and that new activated instance would not see that message. To protect against such cases, flag -X is recommended, however the value shall not trigger checks very often, as directory listing operations on large shared file systems might be slow. The value such as 60 (once per two minutes) or greater would be recommended, however this depends on system criticality and requirements of the message availability, as if such chance of unregistered committed message happens, the tmqueue during the operations would not be able to dequeue or forwards the message.
LIMITATIONS
When commit is performed for several enqueued messages within a global transaction, each message is unblocked (made available in queue space for dequeue or automatic forward) separately, message by message. This means that during the commit execution, even though commit is not completed yet, part of the messages will appear as committed.
Flag NOJOIN is not supported for XA libndrxxaqdisks and libndrxxaqdiskd drivers.
Only 1 tmqueue server is allowed per queue space. Thus it is mandatory that <min> and <max> tags are set to 1. Otherwise incorrect queue logic may be expected.