Skip to content

[8.x] Rename some queue configurations #32728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
May 11, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Illuminate/Contracts/Queue/Job.php
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ public function timeout();
*
* @return int|null
*/
public function timeoutAt();
public function retryUntil();

/**
* Get the name of the queued job class.
Expand Down
6 changes: 3 additions & 3 deletions src/Illuminate/Events/CallQueuedListener.php
Original file line number Diff line number Diff line change
Expand Up @@ -40,18 +40,18 @@ class CallQueuedListener implements ShouldQueue
public $tries;

/**
* The number of seconds to wait before retrying the job.
* The number of seconds before a released job will be available.
*
* @var int
*/
public $retryAfter;
public $backoff;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others have already commented about the nomenclature of backoff here.

I just want to add that backoff has a certain connotation of gradually backing off, so that each time it fails the delay is increased 🤷‍♂️

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello 👋🏼

I know no one asked for my opinion here, but what triggered me writing this comment is that, now that I was faced with the delay/backoff rename in a project and researched how this came about, I was under the same impression and thought "this doesn't feel right". As you said: "gradually backing off" was also my expectation

But, IMHO, turns out we're mostly just so used to the (indeed very common place) e.g. "exponential backoff" that it's easy to assume (even myself) the "backoff" equals "exponential backoff".

But in fact there are just different types of backoff one can do:

  • linear
  • constant
  • exponential

My 2c FWIW 😄


/**
* The timestamp indicating when the job should timeout.
*
* @var int
*/
public $timeoutAt;
public $retryUntil;

/**
* The number of seconds the job can run before timing out.
Expand Down
6 changes: 3 additions & 3 deletions src/Illuminate/Events/Dispatcher.php
Original file line number Diff line number Diff line change
Expand Up @@ -524,10 +524,10 @@ protected function propagateListenerOptions($listener, $job)
{
return tap($job, function ($job) use ($listener) {
$job->tries = $listener->tries ?? null;
$job->retryAfter = method_exists($listener, 'retryAfter')
? $listener->retryAfter() : ($listener->retryAfter ?? null);
$job->backoff = method_exists($listener, 'backoff')
? $listener->backoff() : ($listener->backoff ?? null);
$job->timeout = $listener->timeout ?? null;
$job->timeoutAt = method_exists($listener, 'retryUntil')
$job->retryUntil = method_exists($listener, 'retryUntil')
? $listener->retryUntil() : null;
});
}
Expand Down
8 changes: 4 additions & 4 deletions src/Illuminate/Mail/SendQueuedMailable.php
Original file line number Diff line number Diff line change
Expand Up @@ -76,17 +76,17 @@ public function failed($e)
}

/**
* Get the retry delay for the mailable object.
* Get number of seconds before a released mailable will be available.
*
* @return mixed
*/
public function retryAfter()
public function backoff()
{
if (! method_exists($this->mailable, 'retryAfter') && ! isset($this->mailable->retryAfter)) {
if (! method_exists($this->mailable, 'backoff') && ! isset($this->mailable->backoff)) {
return;
}

return $this->mailable->retryAfter ?? $this->mailable->retryAfter();
return $this->mailable->backoff ?? $this->mailable->backoff();
}

/**
Expand Down
12 changes: 6 additions & 6 deletions src/Illuminate/Notifications/SendQueuedNotifications.php
Original file line number Diff line number Diff line change
Expand Up @@ -118,17 +118,17 @@ public function failed($e)
}

/**
* Get the retry delay for the notification.
* Get number of seconds before a released notification will be available.
*
* @return mixed
*/
public function retryAfter()
public function backoff()
{
if (! method_exists($this->notification, 'retryAfter') && ! isset($this->notification->retryAfter)) {
if (! method_exists($this->notification, 'backoff') && ! isset($this->notification->backoff)) {
return;
}

return $this->notification->retryAfter ?? $this->notification->retryAfter();
return $this->notification->backoff ?? $this->notification->backoff();
}

/**
Expand All @@ -138,11 +138,11 @@ public function retryAfter()
*/
public function retryUntil()
{
if (! method_exists($this->notification, 'retryUntil') && ! isset($this->notification->timeoutAt)) {
if (! method_exists($this->notification, 'retryUntil') && ! isset($this->notification->retryUntil)) {
return;
}

return $this->notification->timeoutAt ?? $this->notification->retryUntil();
return $this->notification->retryUntil ?? $this->notification->retryUntil();
}

/**
Expand Down
9 changes: 7 additions & 2 deletions src/Illuminate/Queue/Console/ListenCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ class ListenCommand extends Command
*/
protected $signature = 'queue:listen
{connection? : The name of connection}
{--delay=0 : The number of seconds to delay failed jobs}
{--delay=0 : The number of seconds to delay failed jobs (Deprecated)}
{--backoff=0 : The number of seconds before a released job will be available}
{--force : Force the worker to run even in maintenance mode}
{--memory=128 : The memory limit in megabytes}
{--queue= : The queue to listen on}
Expand Down Expand Up @@ -91,8 +92,12 @@ protected function getQueue($connection)
*/
protected function gatherOptions()
{
$backoff = $this->hasOption('backoff')
? $this->option('backoff')
: $this->option('delay');

return new ListenerOptions(
$this->option('env'), $this->option('delay'),
$this->option('env'), $backoff,
$this->option('memory'), $this->option('timeout'),
$this->option('sleep'), $this->option('tries'),
$this->option('force')
Expand Down
9 changes: 7 additions & 2 deletions src/Illuminate/Queue/Console/WorkCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ class WorkCommand extends Command
{--daemon : Run the worker in daemon mode (Deprecated)}
{--once : Only process the next job on the queue}
{--stop-when-empty : Stop when the queue is empty}
{--delay=0 : The number of seconds to delay failed jobs}
{--delay=0 : The number of seconds to delay failed jobs (Deprecated)}
{--backoff=0 : The number of seconds before a released job will be available}
{--force : Force the worker to run even in maintenance mode}
{--memory=128 : The memory limit in megabytes}
{--sleep=3 : Number of seconds to sleep when no job is available}
Expand Down Expand Up @@ -120,8 +121,12 @@ protected function runWorker($connection, $queue)
*/
protected function gatherWorkerOptions()
{
$backoff = $this->hasOption('backoff')
? $this->option('backoff')
: $this->option('delay');

return new WorkerOptions(
$this->option('delay'), $this->option('memory'),
$backoff, $this->option('memory'),
$this->option('timeout'), $this->option('sleep'),
$this->option('tries'), $this->option('force'),
$this->option('stop-when-empty')
Expand Down
10 changes: 5 additions & 5 deletions src/Illuminate/Queue/Jobs/Job.php
Original file line number Diff line number Diff line change
Expand Up @@ -266,13 +266,13 @@ public function maxExceptions()
}

/**
* Get the number of seconds to delay a failed job before retrying it.
* Get the number of seconds before a released job will be available.
*
* @return int|null
*/
public function delaySeconds()
public function backoff()
{
return $this->payload()['delay'] ?? null;
return $this->payload()['backoff'] ?? $this->payload()['delay'] ?? null;
}

/**
Expand All @@ -290,9 +290,9 @@ public function timeout()
*
* @return int|null
*/
public function timeoutAt()
public function retryUntil()
{
return $this->payload()['timeoutAt'] ?? null;
return $this->payload()['retryUntil'] ?? $this->payload()['timeoutAt'] ?? null;
}

/**
Expand Down
2 changes: 1 addition & 1 deletion src/Illuminate/Queue/Listener.php
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ protected function createCommand($connection, $queue, ListenerOptions $options)
$connection,
'--once',
"--queue={$queue}",
"--delay={$options->delay}",
"--backoff={$options->backoff}",
"--memory={$options->memory}",
"--sleep={$options->sleep}",
"--tries={$options->maxTries}",
Expand Down
6 changes: 3 additions & 3 deletions src/Illuminate/Queue/ListenerOptions.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,18 @@ class ListenerOptions extends WorkerOptions
* Create a new listener options instance.
*
* @param string|null $environment
* @param int $delay
* @param int $backoff
* @param int $memory
* @param int $timeout
* @param int $sleep
* @param int $maxTries
* @param bool $force
* @return void
*/
public function __construct($environment = null, $delay = 0, $memory = 128, $timeout = 60, $sleep = 3, $maxTries = 1, $force = false)
public function __construct($environment = null, $backoff = 0, $memory = 128, $timeout = 60, $sleep = 3, $maxTries = 1, $force = false)
{
$this->environment = $environment;

parent::__construct($delay, $memory, $timeout, $sleep, $maxTries, $force);
parent::__construct($backoff, $memory, $timeout, $sleep, $maxTries, $force);
}
}
22 changes: 11 additions & 11 deletions src/Illuminate/Queue/Queue.php
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,9 @@ protected function createObjectPayload($job, $queue)
'job' => 'Illuminate\Queue\CallQueuedHandler@call',
'maxTries' => $job->tries ?? null,
'maxExceptions' => $job->maxExceptions ?? null,
'delay' => $this->getJobRetryDelay($job),
'backoff' => $this->getJobBackoff($job),
'timeout' => $job->timeout ?? null,
'timeoutAt' => $this->getJobExpiration($job),
'retryUntil' => $this->getJobExpiration($job),
'data' => [
'commandName' => $job,
'command' => $job,
Expand Down Expand Up @@ -162,21 +162,21 @@ protected function getDisplayName($job)
}

/**
* Get the retry delay for an object-based queue handler.
* Get the backoff for an object-based queue handler.
*
* @param mixed $job
* @return mixed
*/
public function getJobRetryDelay($job)
public function getJobBackoff($job)
{
if (! method_exists($job, 'retryAfter') && ! isset($job->retryAfter)) {
if (! method_exists($job, 'backoff') && ! isset($job->backoff)) {
return;
}

$delay = $job->retryAfter ?? $job->retryAfter();
$backoff = $job->backoff ?? $job->backoff();

return $delay instanceof DateTimeInterface
? $this->secondsUntil($delay) : $delay;
return $backoff instanceof DateTimeInterface
? $this->secondsUntil($backoff) : $backoff;
}

/**
Expand All @@ -187,11 +187,11 @@ public function getJobRetryDelay($job)
*/
public function getJobExpiration($job)
{
if (! method_exists($job, 'retryUntil') && ! isset($job->timeoutAt)) {
if (! method_exists($job, 'retryUntil') && ! isset($job->retryUntil)) {
return;
}

$expiration = $job->timeoutAt ?? $job->retryUntil();
$expiration = $job->retryUntil ?? $job->retryUntil();

return $expiration instanceof DateTimeInterface
? $expiration->getTimestamp() : $expiration;
Expand All @@ -213,7 +213,7 @@ protected function createStringPayload($job, $queue, $data)
'job' => $job,
'maxTries' => null,
'maxExceptions' => null,
'delay' => null,
'backoff' => null,
'timeout' => null,
'data' => $data,
]);
Expand Down
14 changes: 7 additions & 7 deletions src/Illuminate/Queue/Worker.php
Original file line number Diff line number Diff line change
Expand Up @@ -397,9 +397,9 @@ protected function handleJobException($connectionName, $job, WorkerOptions $opti
// another listener (or this same one). We will re-throw this exception after.
if (! $job->isDeleted() && ! $job->isReleased() && ! $job->hasFailed()) {
$job->release(
method_exists($job, 'delaySeconds') && ! is_null($job->delaySeconds())
? $job->delaySeconds()
: $options->delay
method_exists($job, 'backoff') && ! is_null($job->backoff())
? $job->backoff()
: $options->backoff
);
}
}
Expand All @@ -421,13 +421,13 @@ protected function markJobAsFailedIfAlreadyExceedsMaxAttempts($connectionName, $
{
$maxTries = ! is_null($job->maxTries()) ? $job->maxTries() : $maxTries;

$timeoutAt = $job->timeoutAt();
$retryUntil = $job->retryUntil();

if ($timeoutAt && Carbon::now()->getTimestamp() <= $timeoutAt) {
if ($retryUntil && Carbon::now()->getTimestamp() <= $retryUntil) {
return;
}

if (! $timeoutAt && ($maxTries === 0 || $job->attempts() <= $maxTries)) {
if (! $retryUntil && ($maxTries === 0 || $job->attempts() <= $maxTries)) {
return;
}

Expand All @@ -449,7 +449,7 @@ protected function markJobAsFailedIfWillExceedMaxAttempts($connectionName, $job,
{
$maxTries = ! is_null($job->maxTries()) ? $job->maxTries() : $maxTries;

if ($job->timeoutAt() && $job->timeoutAt() <= Carbon::now()->getTimestamp()) {
if ($job->retryUntil() && $job->retryUntil() <= Carbon::now()->getTimestamp()) {
$this->failJob($job, $e);
}

Expand Down
8 changes: 4 additions & 4 deletions src/Illuminate/Queue/WorkerOptions.php
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ class WorkerOptions
*
* @var int
*/
public $delay;
public $backoff;

/**
* The maximum amount of RAM the worker may consume.
Expand Down Expand Up @@ -56,7 +56,7 @@ class WorkerOptions
/**
* Create a new worker options instance.
*
* @param int $delay
* @param int $backoff
* @param int $memory
* @param int $timeout
* @param int $sleep
Expand All @@ -65,9 +65,9 @@ class WorkerOptions
* @param bool $stopWhenEmpty
* @return void
*/
public function __construct($delay = 0, $memory = 128, $timeout = 60, $sleep = 3, $maxTries = 1, $force = false, $stopWhenEmpty = false)
public function __construct($backoff = 0, $memory = 128, $timeout = 60, $sleep = 3, $maxTries = 1, $force = false, $stopWhenEmpty = false)
{
$this->delay = $delay;
$this->backoff = $backoff;
$this->sleep = $sleep;
$this->force = $force;
$this->memory = $memory;
Expand Down
4 changes: 2 additions & 2 deletions tests/Queue/QueueBeanstalkdQueueTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ public function testPushProperlyPushesJobOntoBeanstalkd()
$pheanstalk = $queue->getPheanstalk();
$pheanstalk->shouldReceive('useTube')->once()->with('stack')->andReturn($pheanstalk);
$pheanstalk->shouldReceive('useTube')->once()->with('default')->andReturn($pheanstalk);
$pheanstalk->shouldReceive('put')->twice()->with(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'delay' => null, 'timeout' => null, 'data' => ['data']]), 1024, 0, 60);
$pheanstalk->shouldReceive('put')->twice()->with(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'backoff' => null, 'timeout' => null, 'data' => ['data']]), 1024, 0, 60);

$queue->push('foo', ['data'], 'stack');
$queue->push('foo', ['data']);
Expand All @@ -50,7 +50,7 @@ public function testDelayedPushProperlyPushesJobOntoBeanstalkd()
$pheanstalk = $queue->getPheanstalk();
$pheanstalk->shouldReceive('useTube')->once()->with('stack')->andReturn($pheanstalk);
$pheanstalk->shouldReceive('useTube')->once()->with('default')->andReturn($pheanstalk);
$pheanstalk->shouldReceive('put')->twice()->with(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'delay' => null, 'timeout' => null, 'data' => ['data']]), Pheanstalk::DEFAULT_PRIORITY, 5, Pheanstalk::DEFAULT_TTR);
$pheanstalk->shouldReceive('put')->twice()->with(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'backoff' => null, 'timeout' => null, 'data' => ['data']]), Pheanstalk::DEFAULT_PRIORITY, 5, Pheanstalk::DEFAULT_TTR);

$queue->later(5, 'foo', ['data'], 'stack');
$queue->later(5, 'foo', ['data']);
Expand Down
8 changes: 4 additions & 4 deletions tests/Queue/QueueDatabaseQueueUnitTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ public function testPushProperlyPushesJobOntoDatabase()
$database->shouldReceive('table')->with('table')->andReturn($query = m::mock(stdClass::class));
$query->shouldReceive('insertGetId')->once()->andReturnUsing(function ($array) use ($uuid) {
$this->assertSame('default', $array['queue']);
$this->assertEquals(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'delay' => null, 'timeout' => null, 'data' => ['data']]), $array['payload']);
$this->assertEquals(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'backoff' => null, 'timeout' => null, 'data' => ['data']]), $array['payload']);
$this->assertEquals(0, $array['attempts']);
$this->assertNull($array['reserved_at']);
$this->assertIsInt($array['available_at']);
Expand Down Expand Up @@ -59,7 +59,7 @@ public function testDelayedPushProperlyPushesJobOntoDatabase()
$database->shouldReceive('table')->with('table')->andReturn($query = m::mock(stdClass::class));
$query->shouldReceive('insertGetId')->once()->andReturnUsing(function ($array) use ($uuid) {
$this->assertSame('default', $array['queue']);
$this->assertEquals(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'delay' => null, 'timeout' => null, 'data' => ['data']]), $array['payload']);
$this->assertEquals(json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'backoff' => null, 'timeout' => null, 'data' => ['data']]), $array['payload']);
$this->assertEquals(0, $array['attempts']);
$this->assertNull($array['reserved_at']);
$this->assertIsInt($array['available_at']);
Expand Down Expand Up @@ -119,14 +119,14 @@ public function testBulkBatchPushesOntoDatabase()
$query->shouldReceive('insert')->once()->andReturnUsing(function ($records) use ($uuid) {
$this->assertEquals([[
'queue' => 'queue',
'payload' => json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'delay' => null, 'timeout' => null, 'data' => ['data']]),
'payload' => json_encode(['uuid' => $uuid, 'displayName' => 'foo', 'job' => 'foo', 'maxTries' => null, 'maxExceptions' => null, 'backoff' => null, 'timeout' => null, 'data' => ['data']]),
'attempts' => 0,
'reserved_at' => null,
'available_at' => 'available',
'created_at' => 'created',
], [
'queue' => 'queue',
'payload' => json_encode(['uuid' => $uuid, 'displayName' => 'bar', 'job' => 'bar', 'maxTries' => null, 'maxExceptions' => null, 'delay' => null, 'timeout' => null, 'data' => ['data']]),
'payload' => json_encode(['uuid' => $uuid, 'displayName' => 'bar', 'job' => 'bar', 'maxTries' => null, 'maxExceptions' => null, 'backoff' => null, 'timeout' => null, 'data' => ['data']]),
'attempts' => 0,
'reserved_at' => null,
'available_at' => 'available',
Expand Down
Loading