tcpgatesv — Enduro/X user programmable TCP gateway (client/server roles, XATMI server).
Use programmable tcp gateway. The server gives for programmer abstraction over the tcp sockets. This module gives programmer abstraction over of the socket to the next level - XATMI services. By using tpc gateway for programmer to send network message, it is sufficient for dong the XATMI service invocation with specific UBF buffer, containing the message to be sent to network. The network address now is abstracted to XATMI service name. All other details technical details of socket operations are covered by tcpgatesv.
The getaway is able to handle multiple simultaneous connections. The connections are identified by connection id, which consists of two parts - simple connection id (started from Nr. 1) and by compiled connection id, where part of the number contains the actual timestamp when connection is established.
Getaway can operate with persistent connections, so that tcpgatesv opens the number of connections, and callers (from both ends XATMI and network) can do the invocations synchronously or asynchronously. In async mode it is still possible to simulate synchronous service invocations, that is done in case if corr_svc parameter is set. This parameter indicates which service which to invoke for sending incoming message for parsing. The correlation service can parse the message, and respond with parsed correlation id (in EX_NETCORR). Then by this given correlation ID, the gateway server will lookup any XATMI callers in progress, if found, then they are replied with buffer coming from corr_svc. If correlation id is not parsed or no waiters in progress, then incoming_svc is called with given incoming/corr_svc response buffer.
Geteway can operate in one request-new connection mode from both ends, it can accept the request from network (in this case we must be passive - wait for conn). And it can create new connection when doing the XATMI request to network. After receiving response from network or getting timeout, the connection is closed.
Module allows the system to track the status of the connections. When gateway boots, it sends disconnected status to target connection tracking server (status_svc service in ini config). When connection is established either in active or passive mode, the status of connection is sent to status service.
It is possible to configure gateway to send keep-a-live messages. The keep a live message is zero length. Thus can be safely ignored by network target device. The same as tcpgatesv does.
Tcpgateway uses different message framing algorithms, to avoid partially read messages from network, which can occur if message size is larger than message transfer unit (MTU). Framing mode is controlled by framing parameter. Following options are supported:
Note that if even bytes are used for connection length indication, it is possible, to swap them in half by using framing_half_swap parameter. This is suitable for connecting to VISA payment network, which uses little endian 4 bytes, which are swapped in big endian order by two halves.
The administrator can configure tcpgatesv for different roles:
The tcpgateway can operate in different logical modes (type param):
The next sections will show samples for different framing methods, including settings of framing and framing_half_swap. The test message that will be demonstrated in bellow framing modes are: AHELLO WORLD\\10\\11\\12\\13TEST\\ff, the total number of bytes: 21.
framing = aa
0000 32 31 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 10 11 21HELLO WORLD... 0010 12 13 54 45 53 54 ff ..TEST.
framing = AAAA
0000 30 30 32 35 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 0025HELLO WORLD. 0010 10 11 12 13 54 45 53 54 ff ....TEST.
framing = llll
0000 00 15 00 00 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 ....HELLO WORLD. 0010 10 11 12 13 54 45 53 54 ff ....TEST.
framing = LLLLLL
0000 00 00 00 00 00 1b 48 45 4c 4c 4f 20 57 4f 52 4c ......HELLO WORL 0010 44 00 10 11 12 13 54 45 53 54 ff D.....TEST.
framing = bbbb
0000 15 00 00 00 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 ....HELLO WORLD. 0010 10 11 12 13 54 45 53 54 ff ....TEST.
framing = BBBBBBBB
0000 00 00 00 00 1d 00 00 00 48 45 4c 4c 4f 20 57 4f ........HELLO WO 0010 52 4c 44 00 10 11 12 13 54 45 53 54 ff RLD.....TEST.
framing = pp
0000 00 21 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 10 11 .!HELLO WORLD... 0010 12 13 54 45 53 54 ff ..TEST.
Mode: framing = PPPP, framing_half_swap=0
'framing' = PPPP
0000 00 00 00 25 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 ...%HELLO WORLD. 0010 10 11 12 13 54 45 53 54 ff ....TEST.
Mode: 'framing' = d, framing_half_swap=N/A
framing = d
Using default message delimiter 0x03 (delim_stop).
0000 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 10 11 12 13 HELLO WORLD..... 0010 54 45 53 54 ff 03 TEST..
framing = D
Using default message marker delim_start=0x02 and default message end marker delim_stop=0x03.
0000 02 48 45 4c 4c 4f 20 57 4f 52 4c 44 00 10 11 12 .HELLO WORLD.... 0010 13 54 45 53 54 ff 03 .TEST..
TCP Gateway is programmed by using UBF buffers. Buffers contains specific fields including CARRAY (BLOB) message that needs to be delivered or is received from network.
In case of sending data to network standard tpcall(3) or tpacall(3) are used. The target service of invocation is configured in gateway parameter, that is advertised by tpcgatewsv.
When message is received from network, with incoming data, the corr_svc will be invoked if configured. Finally message is delivered to incoming_svc, the invocation by tcpgatesv will be done synchronously or asynchronously depending on configuration parameters and message specification.
Sending message to network - request
To send message to network in use following UBF buffer (tpcall(3)):
EX_NETDATA - The BLOB/CARRAY data to delivery to target connection
EX_NETCONNID - connection id either compiled or simple. The compiled connection id can be used when generating response back to network. The connection id is composed of 64bit integer, where first 24 bits are connection id, and oldest 39bits are set to UTC epoch milliseconds since start of 1970. The compiled id can be used for doing reply to exact connection.
EX_NETCORR - Optional Correlator string, used for synchronous connections.
Response from gateway service
EX_NERROR_CODE - Error code, can be one of followings:
atmi.NEMANDATORY (6) - Mandatory field is missing (EX_NETDATA)
atmi.NETOUT (8) - timeout waiting on reply
atmi.NENOCONN (9) - Connection not found by EX_NETCONNID or no connection established.
atmi.NELIMIT (10) - Connection count limit reached
EX_NERROR_MSG - Corresponding error message.
Network endpoints identification
When connection is established, and when Enduro/X sends incoming data buffer to XATMI target or correlation service, the UBF buffer includes following meta-data:
The identification data is available for established connections. When reporting connection statuses, and connection is down (disconnected), then fields are optional. In case if included, then data is from last established connection.
Sync service req/reply - example
Request/reply example (from client perspective - in this example server process does change the data bytes doing +1 over the data starting from position 5):
$ ud < test.ud SENT pkt(1) is : EX_NETCONNID 1 EX_NETCORR AELL EX_NETDATA AELLO WORLD\00\10\11\12\13TEST\ff RTN pkt(1) is : EX_NERROR_CODE 0 EX_NETCONNID 6481138401960525826 EX_NERROR_MSG SUCCEED EX_NETGATEWAY TCP_P_ASYNC_P EX_NETCORR AELL EX_NETDATA AELLP!XPSME\01\11\12\13\14UFTU\00 EX_NETOURPORT 53972 EX_NETTHEIRPORT 29999 EX_NETCONMODE A EX_NETOURIP 127.0.0.1 EX_NETTHEIRIP 127.0.0.1
Incoming request at correlation service (other end reads network and sends data to (corr_svc), at the destination with no reply waiter, it will just invoke the incoming service (see after this dump).
EX_NETCONNID 6481138401943748609 EX_NETGATEWAY TCP_P_ASYNC_A EX_NETDATA AELLO WORLD\00\10\11\12\13TEST\ff EX_NETOURPORT 29999 EX_NETTHEIRPORT 53972 EX_NETCONMODE A EX_NETOURIP 127.0.0.1 EX_NETTHEIRIP 127.0.0.1
Incoming request at server (incoming_svc):
EX_NETCONNID 6481138401943748609 EX_NETGATEWAY TCP_P_ASYNC_A EX_NETCORR AELL EX_NETDATA AELLO WORLD\00\10\11\12\13TEST\ff EX_NETOURPORT 29999 EX_NETTHEIRPORT 53972 EX_NETCONMODE A EX_NETOURIP 127.0.0.1 EX_NETTHEIRIP 127.0.0.1
Note that when message is received back from other host, it is sent for correlation service so that we can match the response. For this particular case the invocation did look like:
N:NDRX:5:26407:7f21c98357c0:000:20170131:010926331:_tplog.c:0099:CORSVC: Incoming request: EX_NETCONNID 6481138401960525826 EX_NETGATEWAY TCP_P_ASYNC_P EX_NETDATA AELLP!XPSME\01\11\12\13\14UFTU\00 EX_NETOURPORT 29999 EX_NETTHEIRPORT 53972 EX_NETCONMODE A EX_NETOURIP 127.0.0.1 EX_NETTHEIRIP 127.0.0.1 t:USER:4:26407:7f21c98357c0:000:20170131:010926331:estsv.go:0081:Extracted correlator: [AELL] N:NDRX:5:26407:7f21c98357c0:000:20170131:010926331:_tplog.c:0099:Reply buffer afrer correl EX_NETCONNID 6481138401960525826 EX_NETGATEWAY TCP_P_ASYNC_P EX_NETCORR AELL EX_NETDATA AELLP!XPSME\01\11\12\13\14UFTU\00 EX_NETOURPORT 29999 EX_NETTHEIRPORT 53972 EX_NETCONMODE A EX_NETOURIP 127.0.0.1 EX_NETTHEIRIP 127.0.0.1
Example connection status buffer
The connection 2 is disconnected.
EX_NETCONNID 2 EX_NETGATEWAY TCP_P_SYNC_A EX_NETFLAGS D
Note that connection related fields: EX_NETOURPORT/EX_NETTHEIRPORT/ EX_NETCONMODE/EX_NETOURIP/EX_NETTHEIRIP are present always when connection is established. In case if connection is closed, then these fields are optional and may not be present.
The configuration is written in CCONFIG ini file. The section for tcp gateway is [@tcpgate/CCTAG]. The CCTAG is optional. Following parameters are available for tcp gateway:
l+ - little endian byte order, does not include length of it self
L+ - little endian byte order, include length of it self
b+ - big endian byte order, does not include length of it self
B+ - big endian byte order, include length of it self
a+ - ASCII text byte order, does not include length by it self
A+ - ASCII text byte order, does include length by it self
p+ - packed BCD, does not include length by it self
P+ - packed BCD, does include length by it self
d - Use message stop indicator (set by delim_start)
D - Use message start & stop indicators (set by delim_stop)
0 - Persistent connection mode, asynchronous messages, including sync with correlation. Supported connection type active (A) and passive (P). In active mode gateway will try to establish the max number of connections. In passive mode gateway waits for max number of incoming connections.
When XATMI client invokes the gateway service, the service waits for outgoing (workers_out) XATMI context object. If object is acquired, the message is submitted to free network connection thread for further processing. If the connection id is specified by EX_NETCONNID, then connection is searched, if not found reject is generated, if found the message is enqueued. At this point response is generated and send back to caller either success (message sent to network thread) or error.
When message is received from network, and correlator service corr_svc returns EX_NETCORR field, then reply waiter (XATMI request object waiting for reply) is located, if found, then reply is passed back to caller. If reply is not found or EX_NETCORR does not exists in UBF buffer, then incoming message is passed to incoming_svc. The invocation is done with tpacall(3), TPNOREPLY mode. Meaning that no answer is waited back from target server back to tcpgatesv.
In correlated connections, the time-out waiting on network is determined by req_reply_timeout parameter in seconds.
1 - Persistent, sync by connection, Enduro/X sends to Network. No matter of the role from active or passive (both are supported in this mode). The connection will be opened as in req_reply mode 0 (above). But the difference is that each invocation will be done in synchronous way, meaning that for each connection only one request can be be sent at the same time. When the response is received from network, the waiter is looked up by connection id. If waiter is found then answer is delivered to waiter with tpreturn(3). If waiter is not found, then target service incoming_svc is called in asynchronous way with out waiting a reply. This can be suitable for cases to detect any late response messages. The service name can be set to dummy one. If service invocation generates error, it will be logged in logfile and connection will continue to serve.
In correlated connections, the time-out waiting on network is determined by req_reply_timeout parameter in seconds.
2 - Persistent, sync by connection, Network sends to Enduro/X. The role of connection type active or passive does not matter here. The connection establishment will be done according to req_reply mode 0 and 1. In this mode, connection receives request it waits for free workers_in XATMI object. Once incoming object is got, the service incoming_svc is invoked with tpcall(3). If response is received and EX_NETDATA is present, the answer is sent to network back. If service call did succeed, but EX_NETDATA is not present, connection is restarted. If service invocation did not succeed, the call is ignored. The timeout for service invocation is standard XATMI timeout flag (NDRX_TOUT environment or [@global] section parameter).
3 - Non-persistent, sync each request - new connection, Enduro/X sends to Net. In this mode for each of the requests, new connection is created. Once response is received, connection is closed. For this mode, type must be A - active, in order to establish a connection.
The time-out waiting on network is determined by req_reply_timeout parameter in seconds.
4 - In this mode Enduro/X receives connection from network and invokes target service incoming_svc. The invocation is done with tpcall(3). If call does not succeed, the error is ignored. If call succeeds but EX_NETDATA is not present connection is closed. If call did succeed and EX_NETDATA is present, the response message is prepared and sent back to network and then connection is closed.
The incoming_svc service invocation timeout is governed by NDRX_TOUT parameter.
In this mode the gateway must be configured in passive mode (waiting for connection), i.e. type=P.
[@tcpgate] ... debug=ndrx=5 ubf=0 tp=5 file=/tmp/tcpgatesv.log
Meaning that Enduro/X internal ATMI level logging (ndrx setting) is set to 5 - debug, and user logging tp (tcpgatesv binary) logging also is set to 5 - debug. Output file will be set to /tmp/tcpgatesv.log. UBF logging is set to none.
To see the usage different usage settings, see tests/02_tcpgatesv/runtime/conf/tcpgate.ini'.
Typical configuration would look like:
[@tcpgate] gateway=TESTSVC incoming_svc=INCSVC type=P framing=ll periodic_zero_msg=60 ip=0.0.0.0 port=9999 max_connections=10
Report bugs to support@mavimax.com
In case if running software for MacOS, it might be required to set:
# defaults write NSGlobalDomain NSAppSleepDisabled -bool YES
Otherwise tcpgatesv binary might receive Go panics. Otherwise in high processing intensity, system warns user in dmesg with:
process tpcgatesv[43069] caught causing excessive wakeups. Observed wakeups rate...
And in this result, seems like Go binary is interrupted by OS, which causes Go binary to corrupt its internal scheduler.
Also problems is found with MacOS only when background processes (with out TTY) are running. If user logins in shell and executes the tcpgatesv tests, then problems does not appear.