Name
restousv — Enduro/X HTTP REST outgoing request server
DESCRIPTION
This is XATMI server which provides external REST service mappings for XATMI
sub-system. The restoutsv can be started with in multiple copies in order to
gain better system performance and have higher stability (so that administrator
can reload the configuration, e.g. adding some new service mapping). The data
to HTTP REST services are send in POST format.
This is XATMI REST gateway server can send requests in HTTP/HTTPS format. Different
content types (say XATMI buffer types are supported). The output format which is
sent to network is mapped from input XATMI buffer type. All Enduro/X buffers are
supported:
-
UBF/FML buffer is converted to JSON message and send as mime type application/json.
The buffer format it self is one level key:value json with possible array elements
(UBF occurrences). The buffer is converted by tpubftojson(3) C function. The error
handling is made via two fields defined in Exfields UBF field definition file.
The field names are EX_IF_ECODE for XATMI error code and EX_IF_EMSG for error
message. The other option is to use HTTP standard error codes.
-
JSON with this XATMI buffer the message with mime type application/json is
directly sent to network. The error fields can be provided in configuration. Other
option is to use HTTP standard error codes.
-
STRING with this XATMI buffer the string message is directly printed to HTTP
request stream. the mime type is set to text/plain. The errors are matched in
response message by regular expressions or HTTP error code.
-
CARRAY with this buffer type the received message is sent to HTTP server in
binary format as mime type application/octet-stream. The errors are match in
response message by regular expressions or HTTP error code.
-
VIEW are converted to two level JSON. The first level includes one object. The
object name matches view name. The for conversion from VIEW to JSON functions like
'tpviewtojson(3)* and tpjsontview(3) are used. If error view based error handling
mechanism is selected (errors="json2view"), then views shall contain one "short"
type field for error code and another C string field (char array) for error message.
The restincl and restoutsv can be combined so that it would make a HTTP bridge
between the systems.
restousv supports service monitoring and setting the services to be depend on
echo server. For echo server there could be set max failures to remove services
from shared memory (service board) and min echo OK messages, to advertise
services back to shared memory.
Internally the restoutsv is multi-threaded XATMI server, where main thread consumes
the service requests and then it they are dispatched to XATMI client sessions
associated go routine (workers). The number is configured by workers configuration
parameter (default 5). When all workers are busy the main thread will wait for free
worker. During this time if any other restoustsv is started, it they can handle
the traffic.
The restoustsv does not keep the http connections open between multiple requests,
this is linked with fact that with go version 1.7, the open connections was leaking
thus this functionality is disabled.
The error handling supports for following error types:
-
Service error
-
No entry (no service)
-
Timeout
Whenever format error is occurred (the response cannot be parsed), the XATMI error
type 13 (TPETIME) is generated.
The typical restousv configuration looks like:
[@restout]
# Enduro/X debug string
debug = tp=5 ndrx=5 iflags=detailed file=
gencore=0
workers=5
scan_time=7
defaults={"urlbase":"https://example.com"
,"errors":"http"
,"noreqfilersp": true
,"sslinsecure": true
}
# this will post to: https://example.com/svc1
service XATMISERVICE1={
"url":"/svc1"
,"errors":"json2ubf"
,"timeout":5
}
# this will post to: https://example.com/svc2
service XATMISERVICE2={
"url":"/svc2"
,"errors":"json2ubf"
,"timeout":5
}
Services with dependency on echo example
This section list the example where JSON echo service is defined and once there
is enough succeed messages, the dependent services are advertised to service board.
[@restout]
# Enduro/X debug string
debug = tp=5 ndrx=5 iflags=detailed file=
gencore=0
workers=5
scan_time=7
defaults={"urlbase":"https://example.com"
,"errors":"http"
,"noreqfilersp": true
,"sslinsecure": true
}
# This echo host
service ECHO_JSON={
"url":"/echo_json"
,"errors":"json"
,"echo":true
,"echo_time":1
,"echo_max_fail":2
,"echo_min_ok":3
,"echo_conv":"json"
,"echo_data":"{\"StringField\":\"Hello Echo\"}"
}
service DEP_JSON={
"url":"/dep_jsonhte"
,"errors":"http"
,"timeout":5
,"depends_on":"ECHO_JSON"
}
From above example it could be seen that, the https://example.com/echo_json will
be requested every 1 second (echo_time) and if the succeed count is 3 (echo_min_ok)
the service will advertise any dependent services, in this case DEP_JSON. In
case if the service fails 2x times (echo_max_fail), the DEP_JSON is unadvertised.
The data which is send to service are {"StringField":"Hello Echo"} (echo_data),
the buffer type is json.
The advertise and unadvertised operations by echo threads are actually scheduled
for XATMI servers main thread and executed with interval of the 7 seconds (scan_time).
Error handling type: http - return error codes in HTTP protocol
With this error handling method, the error codes are directly used from HTTP protocol.
and mapped to XATMI return codes. The error code can be mapped from XATMI subsystem
to HTTP codes manually by using errors_fmt_http_map parameter in service or
default parameter block. The default mapping which is set if
errors_fmt_http_map is not present, is following:
-
http.StatusOK (200) = atmi.TPMINVAL(0)
-
http.StatusGatewayTimeout (504) = atmi.TPETIME(13)
-
http.StatusNotFound (404) = atmi.TPENOENT(6)
-
Anything else (\*) = atmi.TPESVCFAIL(11)
This method is suitable for all buffer formats. Also in case if using other error
handling method and the HTTP error code is not http.StatusOK(200), then default
http error mapping is used or the one set by errors_fmt_http_map.
Error handling type: json2ubf - UBF request buffer error handling method
This error handling mechanism is suitable for XATMI UBF buffer type.
The error message by server must be loaded into top level JSON field EX_IF_ECODE
and error message is loaded into EX_IF_EMSG field. This is suitable in case if
using restincl on the other Enduor/X server to bridge the servers using HTTP/Rest
method and sending data over UBF buffers. The response fields
Error handling type: json - response code embedded JSON response message
This is suitable for json buffer type. It is expected that at root level of
json message there are two fields: One for XATMI error code and another for
error message. The default value for configuration parameter errfmt_json_code
is set to error_message and the default value for parameter errfmt_json_msg
is set to errfmt_json_msg.
For example:
{"error_code":13,"error_message":"13:TPETIME (last error 13: ndrx_mq_receive failed: Connection timed out)"}
Error handling type: text - Free format text error code and message
This is free format text field provided back by server. The restoutsv can parse
the response and if parse is ok (got 2x arguments) then it is treated as response
and error code and value is extracted. If error format is not matched, then it
is assumed that there is no error and data is loaded back into original request
buffer (either STRING or CARRAY). The regular expression for parsing the error
is defined by errfmt_text parameter and default is set to "^([0-9]+):(.\*)$",
the first group must match the number and second part is the string containing
the error.
For example if response will be:
13:TPETIME (last error 13: ndrx_mq_receive failed: Connection timed out)
Then error code is extracted as 13 and error
Error handling type: json2view - VIEW request buffer error handling method
This is suitable for VIEW buffer type. When these errors are used, two
specific parameters must be defined: errfmt_view_code - view field name where
XATMI error code is stored and errfmt_view_msg - view field where the error
message is stored.
CONFIGURATION
-
workers = NUMBER_OF_XATMI_SESSIONS
-
Number XATMI sessions. These sessions are use for serving the outgoing calls -
i.e. doing the calls to HTTP server and responding back to XATMI caller.
If the number is less than outgoing calls, the calls will be suspended while
there will be no XATMI session free. Once it is XATMI session is found free,
then call will be served (i.e. called HTTP counterpart). So meanwhile this parameter
means max number of concurrent outgoing HTTP calls.
The default value for parameter is 5.
-
gencore = GENERATE_CORE_FILE
-
If set to 1, then in case of segmentation fault, the core dump will be generated
instead of Golang default signal handler which just prints some info in stderr.
The default value is 0 which means use default Golang panic handling mechanisms.
-
scan_time = SCAN_TIME_SECONDS
-
This is restoutsv wide configuration parameter which is used in case if any
echo services are define. This is time when main service thread is interrupted
for doing any advertise or un-advertise as a result of echo threads have scheduled
the changes in service board.
The default is 1 (second) and used only if there is at-least one echo service.
-
defaults = SERVICE_CONFIGURATION_JSON
-
This is JSON string (can be multi-line) with defaults setting for the services. It
is basically a service descriptor which is used as base configuration for services.
Once the service is being setup, firstly it uses this defaults config block and
then overrides it by additional flags in service definition block. The details
within the JSON are described in bellow section SERVICE CONFIGURATION.
-
service <SERVICE_NAME> = SERVICE_CONFIGURATION_JSON
-
This is the same configuration as for default, but describes the service routes.
The REST-OUT process can have as many as needed the service mapping routes. Then
<SERVICE_NAME>is XATMI service name to be advertised as outgoing for given
configuration.
SERVICE CONFIGURATION
-
urlbase = URL_BASE
-
This is first part of the URL to be used when compiling the final url to make
requests to. urlbase is used in case if url parameter starts with leading slash
symbol (/), then urlbase is contacted with url to get the final destination.
If url starts with any other symbol (like "http…"), then it is assumed
that URL is full and not partial. Schemes supported are: HTTP and HTTPS.
-
url = URL
-
Full or partial HTTP/HTTPS url to do the postings to. If the parameter starts with
leading / symbol, then urlbase from given definition or from defaults are used
as the start of the request address.
-
sslinsecure = SSL_INSECURE
-
If set to true the work with self-signed certificates on HTTPS server side are
accepted. The default is false, meaning that requests to self signed hosts will
be rejected with error.
-
timeout = REQUEST_TIMEOUT_SECONDS
-
Number of seconds in which HTTP/HTTPS server must respond. If the request goes out
of the scope of the given seconds, then time-out error TPETIME is returned to caller
process.
-
errors = ERROR_HANDLING
-
The parameter can be set to following values http, json, json2ubf and text.
See the working modes of each of the modes in above text.
The default value for this parameter is json2ubf.
-
errfmt_json_code = JSON_ERRORS_CODE
-
In case if errors handling mechanism is set json, then this field indicates
the top level JSON field which is expected in response. The codes used here are
XATMI error codes.
The default value is error_code.
-
errfmt_json_msg = JSON_ERRORS_MESSAGE
-
In case if error handling mechanism is selected in parameter errors to json,
then this field is used to indicate top level JSON field to store the response
message.
The default value is errfmt_json_msg.
-
errfmt_json_onsucc = EXPECT_JSON_ERROR_FIELDS_ON_SUCCEESS
-
If set to true in case of errors driving by json errors, then system will
expect to have the errfmt_json_code and errfmt_json_msg fields to be present
in response even the service invocation did succeed.
-
errfmt_view_code = VIEW_ERRORS_CODE
-
In case if errors handling mechanism is set json2view, then this field indicates
view buffer field in which error code should be installed according to JSON-To-VIEW
buffer formatting rules. See tpjsontoview(3) and tpviewtojson(3) for message
format.
-
errfmt_view_msg = VIEW_ERRORS_MESSAGE
-
In case if error handling mechanism in parameter errors is set to to json2view,
then this field is used to indicates VIEW field into which responses message is stored.
The message it self comes in the JSON message. The JSON message must be formatted
according to the JSON-To-VIEW rules, see tpjsontoview(3) and tpviewtojson(3).
-
errors_fmt_http_map = HTTP_ERROR_CODES_MAPPING
-
Error mapping between HTTP return codes and XATMI error codes used for returning
the value to XATMI service client.
The syntax for the string is following:
"errors_fmt_http_map":"<HTTP_STATUS_CODE_1>:<ATMI_ERROR_CODE_1>,...,
<HTTP_STATUS_CODE_N>:<ATMI_ERROR_CODE_N>,*:<ATMI_ERROR_CODE_FOR_ANY>"
for example (actually a default setting of the parameter):
"errors_fmt_http_map":"200:0,504:13,404:6,*:11"
Means that HTTP return code 200 is translated to Succeed, error code 504 is
translated to TPETIME (timeout) and 404 is translated on TPENOENT (no service)
and any other error (\*) is translated to 11 which is service failure.
-
parseonerror = PARSE_ON_ERROR
-
If set to true and the error response is received from target service, then
then service response buffer is populated with corresponding data received. Thus
the XATMI service will return the error and the return buffer will be changed too.
If set to false in case of error, from XATMI service only error will be returned
and service request buffer is not changed.
If error handling is different than http for given rest service, but http error
is received, then regardless of this parameter, the XATMI service call buffer is
not changed.
The default value for this parameter is false.
-
echo = ENABLE_ECHO
-
Make particular service a background echo daemon. This means that given service
definition will make periodical remote HTTP/HTTPS REST service invocations. The
service will count the succeeds or fails, and will schecdule advertise or unadvertise
any services from service board, for which depends_on is set to given echo
service name.
The default is false.
-
echo_time = ECHO_INTERVAL_TIME
-
Echo interval time. This is the time interval which is used for sending out
echo messages. Paramter is used only when echo is set to true.
The default is 5 (seconds).
-
echo_max_fail = ECHO_MAX_FAIL
-
Max number of failure transmissions to echo URL. After the ECHO_MAX_FAIL is
reached, all services which are depending on this echo server will mark for
unadvertise. The unadvertise will happen at period of the scan_time.
The default value is 2.
-
echo_min_ok = ECHO_MIN_OK
-
Minimum number of succeed echo messages after which dependent services will be
scheduled for advertise by main thread. Main thread is executed at interval
scan_time. Note that if service is marked
The default is 3.
-
echo_conv = ECHO_DATA_FORMAT
-
Data buffer format which is used for sending the data to remote HTTP service.
Possible values are json2ubf, json, text and raw. This parameter sets
how echo_data parameter is parsed to prepare the outgoing XATMI buffer that
is converted to target HTTP service data format.
The default is json2ubf.
-
echo_data = ECHO_DATA
-
Echo data is used for data preparation for sending to target service for doing
diagnostics of the HTTP service availability. The value of this JSON configuration
parameter depends on value configured echo_conv value.
Formats of data are following according to echo_conv:
json2ubf: The format is according to tpjsontoubf(3) function. One level json
with UBF fields inside. Arrays are supported, which are loaded into UBF buffer
occurrences. For example:
# This echo host
service ECHO_JSON2UBF={
"url":"/echo_json2ubf"
,"errors":"json2ubf"
,"echo":true
,"echo_time":1
,"echo_max_fail":2
,"echo_min_ok":3
,"echo_conv":"json2ubf"
,"echo_data":"{\"T_STRING_FLD\":\"Some echo data...\"}"
}
json: This is arbitrary JSON string. The double quotes shall be prefixed with
back slash symbol (\). For example:
service ECHO_JSON={
"url":"/echo_json"
,"errors":"json"
,"echo":true
,"echo_time":1
,"echo_max_fail":2
,"echo_min_ok":3
,"echo_conv":"json"
,"echo_data":"{\"StringField\":\"Hello Echo\"}"
}
text: This is arbitrary text buffer. Any double quotes shall be prefixed with
back slash symbol (\). Example:
# This echo host
service ECHO_STRING={
"url":"/echo_string"
,"errors":"text"
,"echo":true
,"echo_time":1
,"echo_max_fail":2
,"echo_min_ok":3
,"echo_conv":"text"
,"echo_data":"This is echo string!"
}
raw: This is raw data encoded in base64. For example:
service ECHO_RAW={
"url":"/echo_raw"
,"errors":"text"
,"echo":true
,"echo_time":1
,"echo_max_fail":2
,"echo_min_ok":3
,"echo_conv":"raw"
,"echo_data":"AQIDBAUGBwgJEBESExQV"
}
json2view: the data is encoded as described in tpviewtojson(3) C call. For example:
service ECHO_JSON2VIEW={
"url":"/echo_json2view"
,"errors":"json2view"
,"echo":true
,"echo_time":1
,"echo_max_fail":2
,"echo_min_ok":3
,"echo_conv":"json2view"
,"errfmt_view_msg":"rspmessage"
,"errfmt_view_code":"rspcode"
,"echo_data":"{\"REQUEST1\":{\"tshort1\": 5,\"tlong1\": 77777,\"tstring1\": [\"\",\"INCOMING TEST\"]}}"
}
-
depends_on = DEPENDS_ON
-
This parameter is used by normal services (non echo), to mark that the defined
service is depending on echo service. Thus if echo service name is specified
in this parameter, then given service will be advertised if echo_min_ok is
reached by echo. And unadvertised when echo_max_fail is reached. When restoutsv
is booting, any services which depends on echo are not automatically advertised,
instead they wait for echo service schedule.
The default value is empty, thus service is advertised automatically and does
not depend on echo service.