Multiprocess handler

Defines several log handlers to operate in a multi-process application.

All log handler defined here are propagating log messages generated in a sub-process to a central log process. From the central log process it’s easier to handle all log messages to a target log handler, like a file or syslog.

It depends on the way you created the sub-process to choose the right log handler. Here’s a short overview:

Sub-Process Created By Compatible Log Handler
os.fork() starlog.ZmqHandler
multiprocessing.Process() starlog.MultiprocessHandler, flexlog.ZmqHandler
multiprocessing.Process() in “spawn” mode starlog.ZmqHandler
class starlog.MultiprocessHandler(queue=None, manager_queue=True, logger='starlog.logsink')

The MultiprocessHandler uses multiprocessing.Queue to send log records between processes.

All subprocesses sends messages to the queue in the emit method. A background thread of the main process retrieves these messages and delegates them to the logger (default: starlog.sink) in the main process.

The MultiprocessHandler is expected to be set up by the main process.

Parameters:
  • queue – A multiprocessing capable queue. If not set, then a new multiprocessing queue is created.
  • manager_queue (bool) – If queue is None and this argument is set to True, then a new queue is created by calling multiprocessing.managers.SyncManager.Queue().
  • logger (str) – name of the logger where log records are send to in the main process.
close()

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

class starlog.ZmqHandler(address='tcp://127.0.0.1:5557', logger='starlog.logsink')

The ZmqHandler creates a zmq connection between the main process and its children processes. The main process establishes a zmq.PULL socket. Child processes sets up a zmq.PUSH socket.

All subprocesses sends messages to the zmq connection in the emit method. A background thread of the main process retrieves these messages and delegates them to the logger (default: starlog.sink) in the main process.

The ZmqHandler is expected to be set up by the main process.

Parameters:
  • address (str) – the address of the connection which is passed to zmq.Socket.connect() / zmq.Socket.bind(). Examples: tcp://127.0.0.1:12345, tcp://127.0.0.1, ipc:///tmp/log.sock. Omitting the port in a tcp:// address will bind the socket to a random port.
  • logger (str) – name of the logger where log records are send to in the main process.
close()

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.