|
2 | 2 | // SPDX-License-Identifier: MIT
|
3 | 3 |
|
4 | 4 | // Package queue implements a specialized concurrent queue system for Gitea.
|
5 |
| - |
| 5 | +// |
6 | 6 | // Terminology:
|
7 |
| - |
| 7 | +// |
8 | 8 | // 1. Item:
|
9 |
| -// - An item can be a simple value, such as an integer, or a more complex structure that has multiple fields and |
10 |
| -// methods. Usually a item serves as a unit of work, sets of items will be sent to a handler to be processed. |
11 |
| -// - It's represented as a binary slice on the queue |
12 |
| -// - When an item serve as a unit of work generally it will be referred as a task |
| 9 | +// - An item can be a simple value, such as an integer, or a more complex structure that has multiple fields and |
| 10 | +// methods. Usually a item serves as a unit of work, sets of items will be sent to a handler to be processed. |
| 11 | +// - It's represented as a binary slice on the queue |
| 12 | +// - When an item serve as a unit of work generally it will be referred as a task |
13 | 13 | //
|
14 | 14 | // 2. Batch:
|
15 |
| -// - A collection of items that are grouped together for processing. Each worker receives a batch of items. |
| 15 | +// - A collection of items that are grouped together for processing. Each worker receives a batch of items. |
16 | 16 | //
|
17 | 17 | // 3. Worker:
|
18 |
| -// - Individual unit of execution designed to process items from the queue. It's a goroutine that calls the Handler. |
19 |
| -// - Workers will get new items through a channel (WorkerPoolQueue is responsible for the distribution). |
20 |
| -// - Workers operate in parallel. The default value of max workers is determined by the setting system. |
| 18 | +// - Individual unit of execution designed to process items from the queue. It's a goroutine that calls the Handler. |
| 19 | +// - Workers will get new items through a channel (WorkerPoolQueue is responsible for the distribution). |
| 20 | +// - Workers operate in parallel. The default value of max workers is determined by the setting system. |
21 | 21 | //
|
22 | 22 | // 4. Handler (represented by HandlerFuncT type):
|
23 |
| -// - It's the function responsible for processing items. Each active worker will call it. |
24 |
| -// - When processing batches, there might be instances where certain items remain unprocessed or "unhandled". |
25 |
| -// In such scenarios, the Handler ensures these unhandled items are returned to the base queue after a brief delay. |
26 |
| -// This mechanism is particularly beneficial in cases where the processing entity (like a document indexer) is |
27 |
| -// temporarily unavailable. It ensures that no item is skipped or lost due to transient failures in the processing |
28 |
| -// mechanism. |
| 23 | +// - It's the function responsible for processing items. Each active worker will call it. |
| 24 | +// - When processing batches, there might be instances where certain items remain unprocessed or "unhandled". |
| 25 | +// In such scenarios, the Handler ensures these unhandled items are returned to the base queue after a brief delay. |
| 26 | +// This mechanism is particularly beneficial in cases where the processing entity (like a document indexer) is |
| 27 | +// temporarily unavailable. It ensures that no item is skipped or lost due to transient failures in the processing |
| 28 | +// mechanism. |
29 | 29 | //
|
30 | 30 | // 5. Base queue:
|
31 |
| -// - Represents the underlying storage mechanism for the queue. There are several implementations: |
32 |
| -// - Channel: Uses Go's native channel constructs to manage the queue, suitable for in-memory queuing. |
33 |
| -// - LevelDB: Especially useful in persistent queues for single instances. |
34 |
| -// - Redis: Suitable for clusters, where we may have multiple nodes. |
35 |
| -// - Dummy: This is special, it's not a real queue, it's a immediate no-op queue, which is useful for tests. |
36 |
| -// - They all have the same abstraction, the same interface, and they are tested by the same testing code. |
| 31 | +// - Represents the underlying storage mechanism for the queue. There are several implementations: |
| 32 | +// - Channel: Uses Go's native channel constructs to manage the queue, suitable for in-memory queuing. |
| 33 | +// - LevelDB: Especially useful in persistent queues for single instances. |
| 34 | +// - Redis: Suitable for clusters, where we may have multiple nodes. |
| 35 | +// - Dummy: This is special, it's not a real queue, it's a immediate no-op queue, which is useful for tests. |
| 36 | +// - They all have the same abstraction, the same interface, and they are tested by the same testing code. |
37 | 37 | //
|
38 | 38 | // 6. WorkerPoolQueue:
|
39 | 39 | // - It's responsible to glue all together, using the "base queue" to provide "worker pool" functionality. It creates
|
40 |
| -// new workers if needed and can flush the queue, running all the items synchronously till it finishes. |
| 40 | +// new workers if needed and can flush the queue, running all the items synchronously till it finishes. |
41 | 41 | // - Its "Push" function doesn't block forever, it will return an error if the queue is full after the timeout.
|
42 | 42 | //
|
43 | 43 | // 7. Manager:
|
44 | 44 | // - The purpose of it is to serve as a centralized manager for multiple WorkerPoolQueue instances. Whenever we want
|
45 |
| -// to create a new queue, flush, or get a specific queue, we have to use it. |
| 45 | +// to create a new queue, flush, or get a specific queue, we have to use it. |
46 | 46 | //
|
47 | 47 | // A queue can be "simple" or "unique". A unique queue will try to avoid duplicate items.
|
48 | 48 | // Unique queue's "Has" function can be used to check whether an item is already in the queue,
|
|
53 | 53 | // If you want another kind of items to run, you would have to call the manager to create a new WorkerPoolQueue for you
|
54 | 54 | // with a different handler that works with this new type of item. As an example of this:
|
55 | 55 | //
|
56 |
| -// func Init() error { |
57 |
| -// itemQueue = queue.CreateSimpleQueue(graceful.GetManager().ShutdownContext(), "item", handler) |
58 |
| -// ... |
59 |
| -// } |
60 |
| -// func handler(items ...*admin_model.Item) []*admin_model.Item { ... } |
| 56 | +// func Init() error { |
| 57 | +// itemQueue = queue.CreateSimpleQueue(graceful.GetManager().ShutdownContext(), "item", handler) |
| 58 | +// ... |
| 59 | +// } |
| 60 | +// func handler(items ...*admin_model.Item) []*admin_model.Item { ... } |
61 | 61 | //
|
62 | 62 | // As you can see, the handler defined the admin_model.Item type for the queue
|
63 | 63 | package queue
|
|
0 commit comments