Skip to content

4.x dev #135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 44 commits into from
Mar 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
cb4bdc0
Move $tries and retryUntil() to Laravel
marickvantuil Feb 12, 2024
8dceed7
Update README.md
marickvantuil Feb 12, 2024
1f05570
Install Laravel Pint
marickvantuil Feb 12, 2024
ff8f7b2
Add strict_types
marickvantuil Feb 12, 2024
94ae3b4
Pint
marickvantuil Feb 14, 2024
678457b
Use larastan/larastan
marickvantuil Feb 14, 2024
6e1547f
Remove Vue dashboard
marickvantuil Feb 14, 2024
bd12aab
Remove dashboard remaining files
marickvantuil Feb 18, 2024
bcf92bc
Replace deprecated client
marickvantuil Feb 18, 2024
a99a0e9
Update api requests with new client options
marickvantuil Feb 18, 2024
0728bb5
Refactor
marickvantuil Feb 18, 2024
e6a795d
Refactor & cleanup
marickvantuil Feb 18, 2024
90359c0
Wip to remove Config class
marickvantuil Feb 18, 2024
abe5dbc
Simplify authorization with simple token instead of hacky signed audi…
marickvantuil Feb 18, 2024
19ac434
Refactor & cleanup
marickvantuil Feb 18, 2024
77c7545
wip readme
marickvantuil Feb 18, 2024
9ecafb6
wip
marickvantuil Feb 19, 2024
00b3343
Refactor some tests
marickvantuil Feb 20, 2024
dfb24c6
Get rid of LogFake and clean up test jobs
marickvantuil Feb 20, 2024
82dfb28
Cleanup
marickvantuil Feb 20, 2024
ef54f7e
wip
marickvantuil Feb 20, 2024
cfcf0eb
wip
marickvantuil Feb 21, 2024
ff0c7b1
Make task handler route customizable
marickvantuil Feb 21, 2024
c30151a
Make it possible to pass headers to task
marickvantuil Feb 21, 2024
0ba0b99
Fix "Metadata found in doc-comment"
marickvantuil Feb 21, 2024
8bb520a
[4.x-dev] Can use the task handler url from the job, if available. Ot…
aaajeetee Feb 27, 2024
b461e1e
[4.x-dev] A job can also have task headers, which are merged with any…
aaajeetee Feb 27, 2024
b5c9241
Add support for job timeouts
marickvantuil Mar 4, 2024
b377efa
Pint
marickvantuil Mar 4, 2024
841471c
Potentially improve task insert performance by using hashed random id…
marickvantuil Mar 6, 2024
7a0f7df
[4.x-dev] Can set the HandlerUrl using a Closure callback. If given, …
aaajeetee Mar 13, 2024
e642299
Merge branch '4.x-dev' into 4.x-dev
aaajeetee Mar 13, 2024
fa0d675
[4.x-dev] Import class
aaajeetee Mar 13, 2024
17a9d59
Improve task deduplication and name in Cloud Tasks
marickvantuil Mar 24, 2024
3ffdee8
Let Cloud Tasks handle task deletion
marickvantuil Mar 24, 2024
4cf5598
[4.x-dev] Added missing test attribute
aaajeetee Mar 27, 2024
829f720
[4.x-dev] Added "setTaskHeadersUsing()" method to allow a closure to …
aaajeetee Mar 27, 2024
4b224ae
Merge branch '4.x-dev' into 4.x-dev
aaajeetee Mar 27, 2024
52ae5ea
Merge pull request #139 from aaajeetee/4.x-dev
marickvantuil Mar 31, 2024
a6189b4
Refactor and cleanups
marickvantuil Mar 30, 2024
7ff3368
Update README.md
marickvantuil Mar 31, 2024
680cc37
Shorten environment names
marickvantuil Mar 31, 2024
adb22a3
Update UPGRADING.md
marickvantuil Mar 31, 2024
a84a2b5
Pint
marickvantuil Mar 31, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
224 changes: 105 additions & 119 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<p align="center">
<img src="/assets/logo.png" width="400">
<img src="/assets/logo.png" width="400">
</p>
<p align="center">
<img src="https://github.com/stackkit/laravel-google-cloud-tasks-queue/workflows/Run%20tests/badge.svg?branch=master" alt="Build Status">
Expand All @@ -12,161 +12,147 @@
This package allows Google Cloud Tasks to be used as the queue driver.

<p align="center">
<img src="/assets/cloud-tasks-home.png" width="100%">
<img src="/assets/cloud-tasks-home.png" width="100%">
</p>

<details>
<summary>
Requirements
</summary>
### Requirements

<br>
This package requires Laravel 10 or higher and supports MySQL 8 and PostgreSQL 15. Might support older database versions too, but package hasn't been tested for it.
This package requires Laravel 10 or 11.

Please check the [Laravel support policy](https://laravel.com/docs/master/releases#support-policy) table for supported Laravel and PHP versions.
</details>
<details>
<summary>Installation</summary>
<br>

Require the package using Composer

```console
composer require stackkit/laravel-google-cloud-tasks-queue
```

Add a new queue connection to `config/queue.php`

```php
'cloudtasks' => [
'driver' => 'cloudtasks',
'project' => env('STACKKIT_CLOUD_TASKS_PROJECT', ''),
'location' => env('STACKKIT_CLOUD_TASKS_LOCATION', ''),
'queue' => env('STACKKIT_CLOUD_TASKS_QUEUE', 'default'),

// Required when using AppEngine
'app_engine' => env('STACKKIT_APP_ENGINE_TASK', false),
'app_engine_service' => env('STACKKIT_APP_ENGINE_SERVICE', ''),

// Required when not using AppEngine
'handler' => env('STACKKIT_CLOUD_TASKS_HANDLER', ''),
'service_account_email' => env('STACKKIT_CLOUD_TASKS_SERVICE_EMAIL', ''),
'signed_audience' => env('STACKKIT_CLOUD_TASKS_SIGNED_AUDIENCE', true),

// Optional: The deadline in seconds for requests sent to the worker. If the worker
// does not respond by this deadline then the request is cancelled and the attempt
// is marked as a DEADLINE_EXCEEDED failure.
'dispatch_deadline' => null,
'backoff' => 0,
],
```

Update the `QUEUE_CONNECTION` environment variable

```dotenv
QUEUE_CONNECTION=cloudtasks
```
### Installation

Require the package using Composer

```console
composer require stackkit/laravel-google-cloud-tasks-queue
```

Publish the service provider:

```console
php artisan vendor:publish --provider=cloud-tasks
```

Add a new queue connection to `config/queue.php`

```php
'cloudtasks' => [
'driver' => 'cloudtasks',
'project' => env('CLOUD_TASKS_PROJECT', ''),
'location' => env('CLOUD_TASKS_LOCATION', ''),
'queue' => env('CLOUD_TASKS_QUEUE', 'default'),

// Required when using AppEngine
'app_engine' => env('APP_ENGINE_TASK', false),
'app_engine_service' => env('APP_ENGINE_SERVICE', ''),

// Required when not using AppEngine
'handler' => env('CLOUD_TASKS_HANDLER', ''),
'service_account_email' => env('CLOUD_TASKS_SERVICE_EMAIL', ''),

'backoff' => 0,
],
```

If you are using separate services for dispatching and handling tasks, and your application only dispatches jobs and should not be able to handle jobs, you may disable the task handler from `config/cloud-tasks.php`:

```php
'disable_task_handler' => env('CLOUD_TASKS_DISABLE_TASK_HANDLER', false),
```

Finally, change the `QUEUE_CONNECTION` to the newly defined connection.

```dotenv
QUEUE_CONNECTION=cloudtasks
```

Now that the package is installed, the final step is to set the correct environment variables.

Please check the table below on what the values mean and what their value should be.

| Environment variable | Description |Example
---------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|---
| `STACKKIT_CLOUD_TASKS_PROJECT` | The project your queue belongs to. |`my-project`
| `STACKKIT_CLOUD_TASKS_LOCATION` | The region where the project is hosted. |`europe-west6`
| `STACKKIT_CLOUD_TASKS_QUEUE` | The default queue a job will be added to. |`emails`
| **App Engine**
| `STACKKIT_APP_ENGINE_TASK` (optional) | Set to true to use App Engine task (else a Http task will be used). Defaults to false. |`true`
| `STACKKIT_APP_ENGINE_SERVICE` (optional) | The App Engine service to handle the task (only if using App Engine task). |`api`
| **Non- App Engine apps**
| `STACKKIT_CLOUD_TASKS_SERVICE_EMAIL` (optional) | The email address of the service account. Important, it should have the correct roles. See the section below which roles. |`[email protected]`
| `STACKKIT_CLOUD_TASKS_HANDLER` (optional) | The URL that Cloud Tasks will call to process a job. This should be the URL to your Laravel app. By default we will use the URL that dispatched the job. |`https://<your website>.com`
| `STACKKIT_CLOUD_TASKS_SIGNED_AUDIENCE` (optional) | True or false depending if you want extra security by signing the audience of your tasks. May misbehave in certain Cloud Run setups. Defaults to true. | `true`
| Environment variable | Description | Example
---------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------
| `CLOUD_TASKS_PROJECT` | The project your queue belongs to. | `my-project`
| `CLOUD_TASKS_LOCATION` | The region where the project is hosted. | `europe-west6`
| `CLOUD_TASKS_QUEUE` | The default queue a job will be added to. | `emails`
| **App Engine**
| `APP_ENGINE_TASK` (optional) | Set to true to use App Engine task (else a Http task will be used). Defaults to false. | `true`
| `APP_ENGINE_SERVICE` (optional) | The App Engine service to handle the task (only if using App Engine task). | `api`
| **Non- App Engine apps**
| `CLOUD_TASKS_SERVICE_EMAIL` (optional) | The email address of the service account. Important, it should have the correct roles. See the section below which roles. | `[email protected]`
| `CLOUD_TASKS_HANDLER` (optional) | The URL that Cloud Tasks will call to process a job. This should be the URL to your Laravel app. By default we will use the URL that dispatched the job. | `https://<your website>.com`

</details>
<details>
<summary>
How it works & Differences
</summary>
<br>
Using Cloud Tasks as a Laravel queue driver is fundamentally different than other Laravel queue drivers, like Redis.

Typically a Laravel queue has a worker that listens to incoming jobs using the `queue:work` / `queue:listen` command.
With Cloud Tasks, this is not the case. Instead, Cloud Tasks will schedule the job for you and make an HTTP request to your application with the job payload. There is no need to run a `queue:work/listen` command.
### How to

#### Good to know
#### Passing headers to a task

- The "Min backoff" and "Max backoff" options in Cloud Tasks are ignored. This is intentional: Laravel has its own backoff feature (which is more powerful than what Cloud Tasks offers) and therefore I have chosen that over the Cloud Tasks one.
- Similarly to the backoff feature, I have also chosen to let the package do job retries the 'Laravel way'. In Cloud Tasks, when a task throws an exception, Cloud Tasks will decide for itself when to retry the task (based on the backoff values). It will also manage its own state and knows how many times a task has been retried. This is different from Laravel. In typical Laravel queues, when a job throws an exception, the job is deleted and released back onto the queue. In order to support Laravel's backoff feature, this package must behave the same way about job retries.
You can pass headers to a task by using the `setTaskHeadersUsing` method on the `CloudTasksQueue` class.

</details>
<details>
<summary>Dashboard (beta)</summary>
<br>
The package comes with a beautiful dashboard that can be used to monitor all queued jobs.
```php
use Stackkit\LaravelGoogleCloudTasksQueue\CloudTasksQueue;

CloudTasksQueue::setTaskHeadersUsing(static fn() => [
'X-My-Header' => 'My-Value',
]);
```

<img src="/assets/dashboard.png" width="100%">
If necessary, the current payload being dispatched is also available:

---
```php
use Stackkit\LaravelGoogleCloudTasksQueue\CloudTasksQueue;

_Experimental_
CloudTasksQueue::setTaskHeadersUsing(static fn(array $payload) => [
'X-My-Header' => $payload['displayName'],
]);
```

The dashboard works by storing all outgoing tasks in a database table. When Cloud Tasks calls the application and this
package handles the task, we will automatically update the tasks' status, attempts
and possible errors.
#### Configure task handler url

There is probably a (small) performance penalty because each task dispatch and handling does extra database read and writes.
Also, the dashboard has not been tested with high throughput queues.
You can set the handler url for a task by using the `configureHandlerUrlUsing` method on the `CloudTasksQueue` class.

---
```php
use Stackkit\LaravelGoogleCloudTasksQueue\CloudTasksQueue;

CloudTasksQueue::configureHandlerUrlUsing(static fn() => 'https://example.com/my-url');
```

To make use of it, enable it through the `.env` file:
If necessary, the current job being dispatched is also available:

```dotenv
STACKKIT_CLOUD_TASKS_DASHBOARD_ENABLED=true
STACKKIT_CLOUD_TASKS_DASHBOARD_PASSWORD=MySecretLoginPasswordPleaseChangeThis
```
```php
use Stackkit\LaravelGoogleCloudTasksQueue\CloudTasksQueue;

Then publish its assets and migrations:
CloudTasksQueue::configureHandlerUrlUsing(static fn(MyJob $job) => 'https://example.com/my-url/' . $job->something());
```

```console
php artisan vendor:publish --tag=cloud-tasks
php artisan migrate
```
### How it works and differences

The dashboard is accessible at the URI: /cloud-tasks
Using Cloud Tasks as a Laravel queue driver is fundamentally different than other Laravel queue drivers, like Redis.

</details>
<details>
<summary>Authentication</summary>
<br>
Typically a Laravel queue has a worker that listens to incoming jobs using the `queue:work` / `queue:listen` command.
With Cloud Tasks, this is not the case. Instead, Cloud Tasks will schedule the job for you and make an HTTP request to
your application with the job payload. There is no need to run a `queue:work/listen` command.

#### Good to know

Cloud Tasks has it's own retry configuration options: maximum number of attempts, retry duration, min/max backoff and max doublings. All of these options are ignored by this package. Instead, you may configure max attempts, retry duration and backoff strategy right from Laravel.

### Authentication

Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable with a path to the credentials file.

More info: https://cloud.google.com/docs/authentication/production

If you're not using your master service account (which has all abilities), you must add the following roles to make it works:
If you're not using your master service account (which has all abilities), you must add the following roles to make it
works:

1. App Engine Viewer
2. Cloud Tasks Enqueuer
3. Cloud Tasks Viewer
4. Cloud Tasks Task Deleter
5. Service Account User
</details>
<details>
<summary>Security</summary>
<br>
The job handler requires each request to have an OpenID token. In the installation step we set the service account email, and with that service account, Cloud Tasks will generate an OpenID token and send it along with the job payload to the handler.

This package verifies that the token is digitally signed by Google. Only Google Tasks will be able to call your handler.
### Upgrading

More information about OpenID Connect:

https://developers.google.com/identity/protocols/oauth2/openid-connect
</details>
<details>
<summary>Upgrading</summary>
<br>
Read [UPGRADING.MD](UPGRADING.md) on how to update versions.
</details>
Read [UPGRADING.MD](UPGRADING.md) on how to update versions.
38 changes: 38 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,41 @@
# From 3.x to 4.x

## Renamed environment names (Impact: high)

The following environment variables have been shortened:
- `STACKKIT_CLOUD_TASKS_PROJECT` → `CLOUD_TASKS_PROJECT`
- `STACKKIT_CLOUD_TASKS_LOCATION` → `CLOUD_TASKS_LOCATION`
- `STACKKIT_CLOUD_TASKS_QUEUE` → `CLOUD_TASKS_QUEUE`
- `STACKKIT_CLOUD_TASKS_HANDLER` → `CLOUD_TASKS_HANDLER`
- `STACKKIT_CLOUD_TASKS_SERVICE_EMAIL` → `CLOUD_TASKS_SERVICE_EMAIL`

The following environment variables have been renamed to be more consistent:

- `STACKKIT_APP_ENGINE_TASK` → `CLOUD_TASKS_APP_ENGINE_TASK`
- `STACKKIT_APP_ENGINE_SERVICE` → `CLOUD_TASKS_APP_ENGINE_SERVICE`

The following environment variable has been removed:
- `STACKKIT_CLOUD_TASKS_SIGNED_AUDIENCE`

## Removed dashboard (Impact: high)

The dashboard has been removed to keep the package minimal. A separate composer package might be created with an updated version of the dashboard.

## New configuration file (Impact: medium)

The configuration file has been updated to reflect the removed dashboard and to add new configurable options.

Please publish the new configuration file:

```shell
php artisan vendor:publish --tag=cloud-tasks --force
```

## Dispatch deadline (Impact: medium)

The `dispatch_deadline` has been removed from the task configuration. You may now use Laravel's timeout configuration to control the maximum execution time of a task.


# From 2.x to 3.x

PHP 7.2 and 7.3, and Laravel 5.x are no longer supported.
Expand Down
Binary file removed assets/dashboard.png
Binary file not shown.
13 changes: 10 additions & 3 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,18 @@
}
],
"require": {
"php": "^8.1",
"ext-json": "*",
"phpseclib/phpseclib": "^3.0",
"google/auth": "^v1.29.1",
"google/cloud-tasks": "^1.10",
"thecodingmachine/safe": "^1.0|^2.0"
},
"require-dev": {
"orchestra/testbench": "^8.0",
"nunomaduro/larastan": "^1.0 || ^2.0",
"thecodingmachine/phpstan-safe-rule": "^1.2",
"laravel/legacy-factories": "^1.3"
"laravel/legacy-factories": "^1.3",
"laravel/pint": "^1.13",
"larastan/larastan": "^2.9"
},
"autoload": {
"psr-4": {
Expand All @@ -44,6 +45,12 @@
"l10": [
"composer require laravel/framework:10.* orchestra/testbench:8.* --no-interaction --no-update",
"composer update --prefer-stable --prefer-dist --no-interaction"
],
"pint": [
"pint"
],
"larastan": [
"@php -d memory_limit=-1 vendor/bin/phpstan"
]
}
}
9 changes: 5 additions & 4 deletions config/cloud-tasks.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@
declare(strict_types=1);

return [
'dashboard' => [
'enabled' => env('STACKKIT_CLOUD_TASKS_DASHBOARD_ENABLED', false),
'password' => env('STACKKIT_CLOUD_TASKS_DASHBOARD_PASSWORD', 'MyPassword1!')
],
// The URI of the endpoint that will handle the task
'uri' => env('CLOUD_TASKS_URI', 'handle-task'),

// If the application only dispatches jobs
'disable_task_handler' => env('CLOUD_TASKS_DISABLE_TASK_HANDLER', false),
];
1 change: 0 additions & 1 deletion dashboard/.env.production

This file was deleted.

4 changes: 0 additions & 4 deletions dashboard/.gitignore

This file was deleted.

6 changes: 0 additions & 6 deletions dashboard/.prettierignore

This file was deleted.

6 changes: 0 additions & 6 deletions dashboard/.prettierrc.js

This file was deleted.

4 changes: 0 additions & 4 deletions dashboard/.prettierrc.json

This file was deleted.

7 changes: 0 additions & 7 deletions dashboard/README.md

This file was deleted.

Loading