Skip to content

Commit 2544818

Browse files
sjbermansjberman
authored
Enhancement Proposal for UpstreamSettingsPolicy (#2489)
Problem: As a user, I want to be able to configure settings for connection from NGF to an upstream application. Solution: Submit the enhancement proposal for the UpstreamSettingsPolicy.
1 parent e3a53b3 commit 2544818

File tree

1 file changed

+232
-0
lines changed

1 file changed

+232
-0
lines changed

‎docs/proposals/upstream-settings.md

+232
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
# Enhancement Proposal-2467: Upstream Settings Policy
2+
3+
- Issue: https://github.com/nginxinc/nginx-gateway-fabric/issues/2467
4+
- Status: Implementable
5+
6+
## Summary
7+
8+
This Enhancement Proposal introduces the `UpstreamSettingsPolicy` API that allows Application developers to configure the behavior of the connection between NGINX and their upstream applications. This Policy will attach to a Service that is referenced in an HTTPRoute or GRPCRoute.
9+
10+
## Goals
11+
12+
- Define upstream settings.
13+
- Define an API for upstream settings.
14+
15+
## Non-Goals
16+
17+
- Provide implementation details for implementing the upstream settings policy.
18+
- Define an API for upstream settings for TLSRoute or other layer 4 routes.
19+
20+
## Introduction
21+
22+
### Upstream Settings
23+
24+
Upstream settings are NGINX directives that affect requests sent from NGINX Gateway Fabric to an upstream application.
25+
26+
To begin, the Upstream Settings Policy will include the following NGINX directives:
27+
28+
- [`zone`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone)
29+
- [`keepalive`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive)
30+
- [`keepalive_requests`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests)
31+
- [`keepalive_time`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_time)
32+
- [`keepalive_timeout`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout)
33+
34+
In the future, we can extend the Upstream Settings Policy to include more [upstream-related directives](nginx-extensions.md#upstream-settings).
35+
36+
## API, Customer Driven Interfaces, and User Experience
37+
38+
The `UpstreamSettingsPolicy` API is a CRD that is a part of the `gateway.nginx.org` Group. It adheres to the guidelines and requirements of a Direct Policy as outlined in the [Direct Policy Attachment GEP](https://gateway-api.sigs.k8s.io/geps/gep-2648/). It will target and be attached to a Service which is referenced in an HTTPRoute or GRPCRoute.
39+
40+
Below is the Golang API for the `UpstreamSettingsPolicy` API:
41+
42+
### Go
43+
44+
```go
45+
package v1alpha1
46+
47+
import (
48+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
49+
gatewayv1alpha2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
50+
)
51+
52+
type UpstreamSettingsPolicy struct {
53+
metav1.TypeMeta `json:",inline"`
54+
metav1.ObjectMeta `json:"metadata,omitempty"`
55+
56+
// Spec defines the desired state of the UpstreamSettingsPolicy.
57+
Spec UpstreamSettingsPolicySpec `json:"spec"`
58+
59+
// Status defines the state of the UpstreamSettingsPolicy.
60+
Status gatewayv1alpha2.PolicyStatus `json:"status,omitempty"`
61+
}
62+
63+
type UpstreamSettingsPolicySpec struct {
64+
// TargetRefs identifies API object(s) to apply the policy to.
65+
// Objects must be in the same namespace as the policy.
66+
// Support: Service
67+
TargetRefs []gatewayv1alpha2.LocalPolicyTargetReference `json:"targetRefs"`
68+
69+
// ZoneSize is the size of the shared memory zone used by the upstream. This memory zone is used to share
70+
// the upstream configuration between nginx worker processes. The more servers that an upstream has,
71+
// the larger memory zone is required.
72+
// Default: OSS: 512k, Plus: 1m.
73+
// Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone
74+
//
75+
// +optional
76+
ZoneSize *Size `json:"zoneSize,omitempty"`
77+
78+
// KeepAlive defines the keep-alive settings.
79+
//
80+
// +optional
81+
KeepAlive *UpstreamKeepAlive `json:"keepAlive,omitempty"`
82+
}
83+
84+
// UpstreamKeepAlive defines the keep-alive settings for upstreams.
85+
type UpstreamKeepAlive struct {
86+
// Connections sets the maximum number of idle keep-alive connections to upstream servers that are preserved
87+
// in the cache of each nginx worker process. When this number is exceeded, the least recently used
88+
// connections are closed.
89+
// Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
90+
//
91+
// +optional
92+
Connections *int32 `json"connections,omitempty"`
93+
94+
// Requests sets the maximum number of requests that can be served through one keep-alive connection.
95+
// After the maximum number of requests are made, the connection is closed.
96+
// Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests
97+
//
98+
// +optional
99+
Requests *int32 `json:"requests,omitempty"`
100+
101+
// Time defines the maximum time during which requests can be processed through one keep-alive connection.
102+
// After this time is reached, the connection is closed following the subsequent request processing.
103+
// Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_time
104+
//
105+
// +optional
106+
Time *Duration `json:"time,omitempty"`
107+
108+
// Timeout defines the keep-alive timeout for upstreams.
109+
// Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout
110+
//
111+
// +optional
112+
Timeout *Duration `json:"timeout,omitempty"`
113+
}
114+
115+
// Duration is a string value representing a duration in time.
116+
// The format is a subset of the syntax parsed by Golang time.ParseDuration.
117+
// Examples: 1h, 12m, 30s, 150ms.
118+
type Duration string
119+
120+
// Size is a string value representing a size. Size can be specified in bytes, kilobytes (suffix k),
121+
// or megabytes (suffix m).
122+
// Examples: 1024, 8k, 1m.
123+
type Size string
124+
```
125+
126+
### Global zone size setting
127+
128+
Since this Policy only applies to `http` upstreams, there's no way to set the zone size for `stream` upstreams. For now, we can introduce a global `zoneSize` variable in the `NginxProxy` resource that will set the zone size for all upstreams. Then this Policy would override that global setting on upstreams that it attaches to.
129+
130+
### Versioning and Installation
131+
132+
The version of the `UpstreamSettingsPolicy` API will be `v1alpha1`.
133+
134+
The `UpstreamSettingsPolicy` CRD will be installed by the Cluster Operator via Helm or with manifests. It will be required, and if the `UpstreamSettingsPolicy` CRD does not exist in the cluster, NGINX Gateway Fabric will log errors until it is installed.
135+
136+
### Status
137+
138+
#### CRD Label
139+
140+
According to the [Direct Policy Attachment GEP](https://gateway-api.sigs.k8s.io/geps/gep-2648/), the `UpstreamSettingsPolicy` CRD must have the `gateway.networking.k8s.io/policy: direct` label to specify that it is a direct policy.
141+
This label will help with discoverability and will be used by the planned Gateway API Policy [kubectl plugin](https://gateway-api.sigs.k8s.io/geps/gep-713/#kubectl-plugin-or-command-line-tool).
142+
143+
#### Conditions
144+
145+
#### Conditions/Policy Ancestor Status
146+
147+
According to the [Direct Policy Attachment GEP](https://gateway-api.sigs.k8s.io/geps/gep-2648/), the `UpstreamSettingsPolicy` CRD must include a `status` stanza with a slice of Conditions.
148+
149+
The `Accepted` Condition must be populated on the `UpstreamSettingsPolicy` CRD using the reasons defined in the [PolicyCondition API](https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1alpha2/policy_types.go). If these reasons are not sufficient, we can add implementation-specific reasons.
150+
151+
If the Policy is attached to a Service that is applied to a TLSRoute, it will not be Accepted.
152+
153+
The Condition stanza must be namespaced using the `controllerName` since more than one controller could reconcile the Policy.
154+
155+
In the [Direct Policy Attachment GEP](https://gateway-api.sigs.k8s.io/geps/gep-2648/), the `PolicyAncestorStatus` applies to Direct Policies.
156+
[`PolicyAncestorStatus`](https://github.com/kubernetes-sigs/gateway-api/blob/f1758d1bc233d78a3e1e6cfba34336526655d03d/apis/v1alpha2/policy_types.go#L156) contains a list of ancestor resources that are associated with the policy, and the status of the policy for each ancestor.
157+
This status provides a view of the resources the policy is affecting. It is beneficial for policies implemented by multiple controllers (e.g., BackendTLSPolicy) or that attach to resources with different capabilities.
158+
159+
#### Setting Status on Objects Affected by a Policy
160+
161+
The [Direct Policy Attachment GEP](https://gateway-api.sigs.k8s.io/geps/gep-2648/) mentions adding a Condition or label to all objects affected by a Policy.
162+
163+
This solution gives the object owners some knowledge that their object is affected by a policy but minimizes status updates by limiting them to when the affected object starts or stops being affected by a policy.
164+
165+
The first step is adding the `gateway.nginx.org/UpstreamSettingsPolicyAffected: true` label to the affected Service. We also must set this Condition on all Routes that reference a Service affected by an `UpstreamSettingsPolicy`.
166+
Below is an example of what this Condition may look like:
167+
168+
```yaml
169+
Conditions:
170+
Type: gateway.nginx.org/UpstreamSettingsPolicyAffected
171+
Message: Object affected by an UpstreamSettingsPolicy.
172+
Observed Generation: 1
173+
Reason: PolicyAffected
174+
Status: True
175+
```
176+
177+
Implementing this involves defining a new Condition type and reason:
178+
179+
```go
180+
package conditions
181+
182+
import (
183+
gatewayv1alpha2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
184+
)
185+
186+
const (
187+
UpstreamSettingsPolicyAffected gatewayv1alpha2.PolicyConditionType = "gateway.nginx.org/UpstreamSettingsPolicyAffected"
188+
PolicyAffectedReason gatewayv1alpha2.PolicyConditionReason = "PolicyAffected"
189+
)
190+
```
191+
192+
Some additional rules:
193+
194+
- This Condition and label should be added when the affected object starts being affected by an `UpstreamSettingsPolicy`.
195+
- When the last `UpstreamSettingsPolicy` affecting that object is removed, the Condition and label should be removed.
196+
- The Observed Generation is the generation of the affected object, not the generation of the `UpstreamSettingsPolicy`.
197+
198+
## Use Cases
199+
200+
- As an Application Developer, I want to be able to configure upstream settings for my application based on its behavior or requirements.
201+
- I may have a large number of Pods for my Service and therefore need a larger memory zone for it.
202+
- I may want to alter the keepalive settings for my upstream.
203+
204+
## Testing
205+
206+
- Unit tests
207+
- Functional tests that verify the attachment of the CRD to a Service, and that NGINX behaves properly based on the configuration.
208+
209+
## Security Considerations
210+
211+
Validating all fields in the `UpstreamSettingsPolicy` is critical to ensuring that the NGINX config generated by NGINX Gateway Fabric is correct and secure.
212+
213+
All fields in the `UpstreamSettingsPolicy` will be validated with Open API Schema. If the Open API Schema validation rules are not sufficient, we will use [CEL](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules).
214+
215+
RBAC via the Kubernetes API server will ensure that only authorized users can update the CRD.
216+
217+
## Future Work
218+
219+
- Add support for more [upstream-related directives](nginx-extensions.md#upstream-settings).
220+
- Add a `StreamUpstreamSettingsPolicy` that configures upstream settings for stream servers (TLSRoute, TCPRoute).
221+
222+
## Alternatives
223+
224+
- Single policy that contains `http` and `stream` fields for both types of upstreams. It was decided that this isn't as good of a user experience versus just having separate policies with a more targeted purpose.
225+
- Single policy with all top level fields, where some would only be applied depending on the type of upstream, and have conditions that say which fields were ignored. This was rejected for similar reasoning.
226+
227+
## References
228+
229+
- [NGINX Extensions Enhancement Proposal](nginx-extensions.md)
230+
- [Policy and Metaresources GEP](https://gateway-api.sigs.k8s.io/geps/gep-713/)
231+
- [Direct Policy Attachment GEP](https://gateway-api.sigs.k8s.io/geps/gep-2648/)
232+
- [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md)

0 commit comments

Comments
 (0)