-
Notifications
You must be signed in to change notification settings - Fork 173
Add support for recurring tasks (cron style jobs) #155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
4ea040f
Pass default options for each dispatcher
rosa f0c1134
Extract concurrency maintenance work to its own class and allow optin…
rosa 16966b6
Add Fugit to parse cron-style schedules
rosa e554e7e
Refactor and simplify process callbacks setup
rosa d453743
Remove initial jitter for dispatcher
rosa c383d15
Use Zeitwerk as autoloader instead of explicit requires
rosa d871c27
Move #preserve_finished_jobs? to SolidQueue module for consistency
rosa 5e82dbf
Move ConcurrencyClerk under Dispatcher namespace
rosa f5e63a1
Refactor a bit the concurrency maintenance and stub recurring tasks
rosa 676383f
Implement basic recurring task parsing and loading in a schedule
rosa efadb87
Rename RecurringTask#id to RecurringTask#key
rosa e7dd9d3
Create recurring_executions table
rosa 8726b39
Fix configuration of recurring tasks when parsed via Configuration
rosa 3e4c0f0
Track recurring executions to prevent enqueuing the same task more th…
rosa b2f74a3
Log thread errors in development, just like in test
rosa b64178a
Support passing kwargs as last element of the arguments array
rosa ab5e504
Remove or change some logging from info to debug level
rosa a28ccda
Store full recurring task configuration in process metadata
rosa 6d510a5
Destroy recurring execution when deleting a job
rosa 1043984
Add a clear_in_batches method to delete recurring executions
rosa e104a5f
Fix indentation in Process::Runnable
rosa 8d1e27c
Refactor a bit the Poller vs. Runnable modules
rosa 38db619
Group all development dependencies together in solid_queue.gemspec
rosa 4ea0e2a
Add documentation about recurring tasks in README
rosa File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# frozen_string_literal: true | ||
|
||
module SolidQueue | ||
class Job | ||
module Recurrable | ||
extend ActiveSupport::Concern | ||
|
||
included do | ||
has_one :recurring_execution, dependent: :destroy | ||
end | ||
end | ||
end | ||
end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
# frozen_string_literal: true | ||
|
||
module SolidQueue | ||
class RecurringExecution < Execution | ||
scope :clearable, -> { where.missing(:job) } | ||
|
||
class << self | ||
def record(task_key, run_at, &block) | ||
transaction do | ||
if job_id = block.call | ||
create!(job_id: job_id, task_key: task_key, run_at: run_at) | ||
end | ||
end | ||
rescue ActiveRecord::RecordNotUnique | ||
SolidQueue.logger.info("[SolidQueue] Skipped recurring task #{task_key} at #{run_at} — already dispatched") | ||
end | ||
|
||
def clear_in_batches(batch_size: 500) | ||
loop do | ||
records_deleted = clearable.limit(batch_size).delete_all | ||
break if records_deleted == 0 | ||
end | ||
end | ||
end | ||
end | ||
end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
class CreateRecurringExecutions < ActiveRecord::Migration[7.1] | ||
def change | ||
create_table :solid_queue_recurring_executions do |t| | ||
t.references :job, index: { unique: true }, null: false | ||
t.string :task_key, null: false | ||
t.datetime :run_at, null: false | ||
t.datetime :created_at, null: false | ||
|
||
t.index [ :task_key, :run_at ], unique: true | ||
end | ||
|
||
add_foreign_key :solid_queue_recurring_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade | ||
end | ||
end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,24 +3,16 @@ | |
require "solid_queue/version" | ||
require "solid_queue/engine" | ||
|
||
require "active_job/queue_adapters/solid_queue_adapter" | ||
require "active_job/concurrency_controls" | ||
|
||
require "solid_queue/app_executor" | ||
require "solid_queue/processes/supervised" | ||
require "solid_queue/processes/registrable" | ||
require "solid_queue/processes/interruptible" | ||
require "solid_queue/processes/pidfile" | ||
require "solid_queue/processes/procline" | ||
require "solid_queue/processes/poller" | ||
require "solid_queue/processes/base" | ||
require "solid_queue/processes/runnable" | ||
require "solid_queue/processes/signals" | ||
require "solid_queue/configuration" | ||
require "solid_queue/pool" | ||
require "solid_queue/worker" | ||
require "solid_queue/dispatcher" | ||
require "solid_queue/supervisor" | ||
require "active_job" | ||
require "active_job/queue_adapters" | ||
|
||
require "zeitwerk" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ❤️ |
||
|
||
loader = Zeitwerk::Loader.for_gem(warn_on_extra_files: false) | ||
loader.ignore("#{__dir__}/solid_queue/tasks.rb") | ||
loader.ignore("#{__dir__}/generators") | ||
loader.ignore("#{__dir__}/puma") | ||
loader.setup | ||
|
||
module SolidQueue | ||
mattr_accessor :logger, default: ActiveSupport::Logger.new($stdout) | ||
|
@@ -42,11 +34,17 @@ module SolidQueue | |
mattr_accessor :clear_finished_jobs_after, default: 1.day | ||
mattr_accessor :default_concurrency_control_period, default: 3.minutes | ||
|
||
def self.supervisor? | ||
supervisor | ||
end | ||
class << self | ||
def supervisor? | ||
supervisor | ||
end | ||
|
||
def silence_polling? | ||
silence_polling | ||
end | ||
|
||
def self.silence_polling? | ||
silence_polling | ||
def preserve_finished_jobs? | ||
preserve_finished_jobs | ||
end | ||
end | ||
end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be at the end?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, @brunoprietog thanks for spotting this! It should totally be at the end 😆