Multiprocess handler¶
Defines several log handlers to operate in a multi-process application.
All log handler defined here are propagating log messages generated in a sub-process to a central log process. From the central log process it’s easier to handle all log messages to a target log handler, like a file or syslog.
It depends on the way you created the sub-process to choose the right log handler. Here’s a short overview:
Sub-Process Created By | Compatible Log Handler |
---|---|
os.fork() | starlog.ZmqHandler |
multiprocessing.Process() | starlog.MultiprocessHandler ,
flexlog.ZmqHandler |
multiprocessing.Process() in “spawn” mode | starlog.ZmqHandler |
-
class
starlog.
MultiprocessHandler
(queue=None, manager_queue=True, logger='starlog.logsink')¶ The MultiprocessHandler uses
multiprocessing.Queue
to send log records between processes.All subprocesses sends messages to the queue in the emit method. A background thread of the main process retrieves these messages and delegates them to the
logger
(default:starlog.sink
) in the main process.The MultiprocessHandler is expected to be set up by the main process.
Parameters: - queue – A multiprocessing capable queue. If not set, then a new multiprocessing queue is created.
- manager_queue (bool) – If queue is None and this argument is set
to True, then a new queue is created by calling
multiprocessing.managers.SyncManager.Queue()
. - logger (str) – name of the logger where log records are send to in the main process.
-
close
()¶ Tidy up any resources used by the handler.
This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.
-
class
starlog.
ZmqHandler
(address='tcp://127.0.0.1:5557', logger='starlog.logsink')¶ The ZmqHandler creates a zmq connection between the main process and its children processes. The main process establishes a
zmq.PULL
socket. Child processes sets up azmq.PUSH
socket.All subprocesses sends messages to the zmq connection in the emit method. A background thread of the main process retrieves these messages and delegates them to the
logger
(default:starlog.sink
) in the main process.The ZmqHandler is expected to be set up by the main process.
Parameters: - address (str) – the address of the connection which is passed to
zmq.Socket.connect()
/zmq.Socket.bind()
. Examples:tcp://127.0.0.1:12345
,tcp://127.0.0.1
,ipc:///tmp/log.sock
. Omitting the port in atcp://
address will bind the socket to a random port. - logger (str) – name of the logger where log records are send to in the main process.
-
close
()¶ Tidy up any resources used by the handler.
This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.
- address (str) – the address of the connection which is passed to