Feature #286

Feature #357: Enduro/X 7.0 master task

async logger and latent command framwork (LCF)

Added by Madars over 3 years ago. Updated 8 months ago.

Status:ClosedStart date:09/10/2017
Priority:HighDue date:
Assignee:-% Done:

100%

Category:-
Target version:-

Description

We shall enable new flag for debug string, which is async. This means that at init we spawn new thread which will be busy with writting data to log files. While the "clients" to logger, will just put the strings in linked lists and send for thread to write to files.

Also at periodical intervals, async logger shall check some shared memory block where there will be rotating list of commands which shall be executed by all LCFs. The command shall be command_id, and 3x arguments (something like 32 byte chars). custom commands shall be configurable to receive callback. The xadmin shall support custom command registration.

built in commands:
- change debug levels of Enduro/X logger
- re-parse cache data (requires cache thread safety on objects)

This will allow to change configuration of the binaries "on the fly" with out restart.

Binaries shall keep some pointers to last executed command, and loop over to next. And what the list by timestamps (monotonic should be fine). When command is processed, thread should remember that, and also it should update the counter in command block - how much processes are executed that command. So that admin have some overview of status.


Subtasks

Feature #213: Out of the box logrotate featureClosed

History

#1 Updated by Madars over 3 years ago

it shall be possible to send command to exe name or server id.
Also there shall be some header info, to point where commands start/stops?

#2 Updated by Madars over 3 years ago

  • Subject changed from async logger and latenet command framwork (LCF) to async logger and latent command framwork (LCF)

#3 Updated by Madars over 3 years ago

scheduled for 5.4 a.k.a 6.0

#4 Updated by Madars over 3 years ago

  • Parent task set to #328

#5 Updated by Madars about 3 years ago

Also it would be nice to have dump encryption facility. Something like public key loaded on disk, then in memory we generate some AES key, and then each dump consists of AES key under PUB key and dump encrypted under AES.

Also need a utility for log file decryption. The encoded dump segments must be marked specifically.

#6 Updated by Madars almost 3 years ago

  • Parent task changed from #328 to #357

#7 Updated by Madars over 2 years ago

lcf shall reload servicedb too (one that will be used for JOLT).

#8 Updated by Madars over 2 years ago

Spinlock RW:

var M_readers = 0;
pthread_spinlock_t M_lock;

ndrx_rwspin_rdlock()
{
    pthread_spin_lock(&M_lock);

    M_readers++;

    pthread_spin_unlock(&M_lock);
}

ndrx_rwspin_rdunlock()
{
    pthread_spin_lock(&M_lock);

    M_readers--;

    pthread_spin_unlock(&M_lock);
}

ndrx_rwspin_wrlock();
{
    for(;;)
    {
        pthread_spin_lock(&M_lock);
        if (M_readers==0)
        {
            //If got 0 readers, then keep the master spinlock locked
            break;
        }
        pthread_spin_unlock(&M_lock);
    }
}

ndrx_rwspin_wrunlock()
{
    pthread_spin_unlock(&M_lock);
}

#9 Updated by Madars over 1 year ago

We shall make remove of swait
Add asyng logging, or sync logging shall perform rw locks on the debug descriptors

#10 Updated by Madars over 1 year ago

For log rotate seems like we need:

- Hash of file descriptor / pointer open. has sub-hash of loggers (bellow), has file name. + count of facilities. have a flag, that this pointer is used by process level loggers.
- Each file descriptor in hash, has own hash with list of logger facilities. Key by facility.

If we choose to rotate the files (re-open).

- To close some log file: find descr in hash, decrement the count, if count == 0 file shall be closed. The sub-hash entry of facility shall be removed. If involved process level files, then needs to perform full debug locking.
- To rotate all files: loop over the hash of descriptors, if has proc level logger, put global logging lock (with counters of loggers, + on exit with needs conditional variable on 0 loggers), close the pointer, open it again, add to new hash list. Loop over the sub-hash, update file pointers. Add sub-hash to new hash element. Delete from origianl top-level hash. Once all rebuilt, swap the pointers -> assign new hash table to global descriptor table.

- we could use LCF with time-stamps, to indicate to processes to which time interval command applies -> i.e. for what process age the command shall be executed. Currently we could execute the LCFs at the debug handler. Only needs to think how it would affect the logging counters, probably needs to do LCF before the logging entry/exit counters.

+ counters needs to be:

- spin-locked
- where internally there would be additional flag which would mean to switch to mutex. So that we doing log file rotate/re-open, other threads would wait on mutex.

#11 Updated by Madars over 1 year ago

log rotate process would be:

- external script renames all log files
- calls $ xadmin rotatelog
- external program may archive + delete (or do other actions as required). Even if process ptr still shows the old file, it would be re-open as soon as log will attempt to write by process
- thus processes would slowly re-open the logs, as they continue to work.

#12 Updated by Madars over 1 year ago

Also might want to add lockfp=y|f
where - y lock at file pointer level
f - full locking (the debug output is fully synchronized.

#13 Updated by Madars over 1 year ago

Also for log-rotate we might want to send notification to admin queues, so that servers process the LCF commands.

#14 Updated by Madars over 1 year ago

In case if ndrxdebug.conf is specified file= (i.e. stderr), then we could extract the log file name from sysopts -e param from servers. And log= from client sections. The log file name could be set in environment variable. Like NDRX_PROCLOG=... The debug init could pick up this name instead of using stderr. In that case we could do the logrotate on outside specified file name.

#18 Updated by Madars 9 months ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

#19 Updated by Madars 8 months ago

  • Status changed from Resolved to Closed

Release notes:

- Implemented LCF command framework. Published two commands:
- xadmin lcf logrotate - allow to reopen the log handlers
- Added xadmin lcf command for printing current status of the LCF commands.
- and xadmin lcf lochg - to change the log levels for processes with out restart.
- Added APIs for custom lcf commands: ndrx_lcf_xadmin_add() and ndrx_lcf_func_add()

See xadmin(8) and ex_env(5) man pages for more details.

with this release additional shared memory and semaphore is open.

Also available in: Atom PDF