When the worker dies, the concurrency limit is not reset

[OlegChuev]

I have a job that can run for an extended period — let’s call it InfiniteSleepJob:

class InfiniteSleepJob < ApplicationJob
  # limit only one simultaneous execution of operation per key
  limits_concurrency to: 1, key: ->(key, *_args) { key },  group: "my_group", duration: 1.day

  def perform(*args)
    # do smt
  end
end

When I add a few jobs to the queue and the worker unexpectedly dies during execution (e.g., due to an HA event or if I forcibly kill the Docker container with docker kill container), the concurrency limit gets stuck, even after the job is pruned. Block is released only when it expires

Here’s an example of a job that was pruned, but the concurrency limit wasn’t removed:

Here’s my queue.yml:

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval:  120
  workers:
    - queues: "default"
      threads: 3
      processes: 1
      polling_interval: 1

I’d expect the queue to unlock as soon as the job fails, allowing the blocked jobs to proceed as expected.

Could you kindly clarify whether this is the expected behavior? Am I missing something?

[rosa]

Hey @OlegChuev, yeah, it's expected behaviour if the worker dies unceremoniously (eg. a kill -9 signal). It wouldn't have a chance to clean up after itself, which includes the concurrency limit. The duration parameter determines how long the system will wait to unblock jobs that didn't get successfully unblocked for any reason, so if your long running job starts at, say 17:00, another one gets enqueued and blocked at 18:00, and the first one dies at 20:00, it won't be until the following day at 17:00 (1.day duration) that the blocked job will be considered to be unlocked.

[rosa]

Although I think I could improve this case, when releasing claimed jobs whose worker got killed... 🤔 I'll work on this next week.

[nerisa]

Hello @rosa, I have tried to fix this issue in my PR #547