Allow using QueueHandler and QueueListener in logging.dictConfig

[spacemanspiff2007]

Feature or enhancement

Allow using instances in the dict passed to dictConfig

Pitch

Currently the logging configuration does not allow passing already created instances of e.g. logging.Handler.
However this would be very useful for an application that uses the dict to configure the logger but wants to use a QueueHandler that implements custom logic.
Since the logging.Handlers and logging.Filters need to be attached to the QueueListener for logging to files it's almost impossible to set it up properly.
If it would be possible to pass an instance the setup could be simplified a lot.

Previous discussion

[vsajip]

You can already do this with minimal effort, as documented here:

In order to provide complete flexibility for user-defined object instantiation, the user needs to provide a ‘factory’ - a callable which is called with a configuration dictionary and which returns the instantiated object. This is signalled by an absolute import path to the factory being made available under the special key '()'.

So, just create a callable which sets things up as you want, and reference it in the configuration as shown in the example given in the posted link.

[spacemanspiff2007]

So, just create a callable which sets things up as you want, and reference it in the configuration as shown in the example given in the posted link.

You are right - I must have been confused when I created the PR. Sorry about that.

I want to use a queue to act as a buffer so I can write out the logs in a different thread. The log messages are created in bursts and the normal logging significantly slows down the application.

flowchart LR
    Logger --> QueueHandler
    QueueHandler -->|Buffer| QueueListener
    QueueListener --> FileHandler

Loading

But for this to work I need to be able to attach the logging handlers to the QueueListener which is currently not possible because
the handlers are created from the config file.

Imho there are two ways to solve this:

1. Create all logging objects from the configuration. Then have another function on DictConfigurator which glues everything together. That way one could inherit from DictConfigurator and override the function. That way the Handlers could be attached to the user objects.
   That would be the most generic implementation and cover all use cases but a bigger code change.
2. Add the possibility to define target handlers for every handler - just like with the MemoryHandler. However it would be nice if one could attach multiple handlers to one handler.

What do you think?

[vsajip]

What do you think?

It's hard to say with the information you've provided. One way of doing it would be to define your configuration dict as follows (shown as YAML for brevity):

...
handlers:
  ...
  queue_handler:
    (): my.package.factory
    handlers:
      file:  # dict with configuration for file handler
      console: # dict with configuration for console handler
      ... and any other handlers you need, likewise

Then, your factory will be called with a handlers kwarg which will be a dict containing all of the handler configuration data (using any schema you like). In the factory, you then:

• Construct a queue
• Construct all the handlers you need
• Create a QueueListener configured with the queue and the handlers
• Create a QueueHandler configured with the queue
• Return that QueueHandler instance from the callable.

That assumes you need all the handler configuration in a configuration file. If not, you could make things simpler by just hardcoding it in the factory. Of course, you'll need to adapt this scheme to your specific needs.

This approach requires no changes to logging but should give you what you need. Of course you need to do a bit more work to instantiate the handlers etc. yourself, but that shouldn't be too onerous. This type of application is what the factory approach was meant for.

[spacemanspiff2007]

That assumes you need all the handler configuration in a configuration file. If not, you could make things simpler by just hardcoding it in the factory. Of course, you'll need to adapt this scheme to your specific needs.

This approach requires no changes to logging but should give you what you need. Of course you need to do a bit more work to instantiate the handlers etc. yourself, but that shouldn't be too onerous. This type of application is what the factory approach was meant for.

If I had full control over the logging configuration it really would be that easy.
But it's an application where the users supply the logging configuration through the dict which then is loaded through dictConfig.
That would mean in my factory I would have to duplicate the complete code from logging.config.

Adding the possibility to chain handlers would effectively solve this issue because it would allow to inject the queue.

Example:

The user adds this configuration

handlers:
  ...
  user_file_handler:
    class: logging.handlers.TimedRotatingFileHandler
    ....

Which could be changed into

handlers:
  ...
  user_file_handler:
    (): QueueHandler
    target: queue_listener
    ...
  queue_listener:
    class: logging.handlers.TimedRotatingFileHandler
    ....

That would be a small change that would allow for the queue to work.
If you are unsure about the target key one could use handlers since this the the key from the loggers and less likely to be used.

Please reopen the issue, I am not the only one who is having issues with the QueueHandler.
It is included in logging.handlers so I really think there should be a supported way to use them with dictConfig.

[vsajip]

OK, but let me think a bit about the best way of solving this in a general way. Leave the PR closed, though, because I wouldn't be taking that approach.

[spacemanspiff2007]

OK, but let me think a bit about the best way of solving this in a general way. Leave the PR closed, though, because I wouldn't be taking that approach.

No problem - better a well thought solution that takes longer than a quick and dirty fix.
Thank you for your help!

[vsajip]

@spacemanspiff2007 I've got a first cut of a possible fix in #93269 - take a look and see if it will meet the requirements (from the documentation part - I'm not asking you to review the code and test in detail, though you're of course free to do so.

[spacemanspiff2007]

Thank you for the quick implementation. I'll take a deeper look at it in the mid of next week.

From what I understand your idea is to configure the listener and handler in one entry in the config dict.
I think it would be helpful to add the listener key in the example, e.g. with my.module.queuelistener_factory.

So from what I understand from the docs this will work:

listener_instance = MyQueueListener()

cfg = {
  'qhand': 'logging.handlers.QueueHandler',
  'queue': lambda: listener_instance.queue,
  'listener': listener_instance.set_handler_by_names
}

setConfig(cfg)

---

It's nice that it's possible to get handlers by name with getHandlerByName.
Would it make sense to provide this kind of mechanism for other logging objects e.g. Formatters and Filters?
Could we also add an option (or another function) to return all handlers?
After startup I want to iterate over all rotating file handlers and based on some internal criteria do a rotate on some of them.