Skip to content

Commit d3e47a9

Browse files
miledxzmiledxz
committed
Add reconfig tests setup and results (nginx#1116)
* Add reconfig tests setup and results * Review feedback * Add TOC, change bash -> console in commands
1 parent 667cd4b commit d3e47a9

10 files changed

+415
-0
lines changed

‎tests/reconfig/results/v1.0.0.md

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Reconfiguration testing Results
2+
3+
<!-- TOC -->
4+
- [Reconfiguration testing Results](#reconfiguration-testing-results)
5+
- [Test environment](#test-environment)
6+
- [Results Table](#results-table)
7+
- [NumResources -\> Total Resources](#numresources---total-resources)
8+
- [Observations](#observations)
9+
<!-- TOC -->
10+
11+
## Test environment
12+
13+
GKE cluster:
14+
15+
- Node count: 3
16+
- Instance Type: e2-medium
17+
- k8s version: 1.27.4-gke.900
18+
- Zone: europe-west2-b
19+
- Total vCPUs: 6
20+
- Total RAM: 12GB
21+
- Max pods per node: 110
22+
23+
NGF deployment:
24+
25+
- NGF version: edge - git commit 72b6c6ef8915c697626eeab88fdb6a3ce15b8da0
26+
- NGINX Version: 1.25.2
27+
28+
## Results Table
29+
30+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) |
31+
| ----------- | ------------ | -------------------- | ------------------------ | ------------- | -------------------------- |
32+
| 1 | 30 | 5 | 5 | 2 | 166 |
33+
| 1 | 150 | 7 | 7 | 2 | 353 |
34+
| 2 | 30 | 21 | <1 | 30 | 142 |
35+
| 2 | 150 | 123 | <1 | 46 | 190 |
36+
| 3 | 30 | <1 | <1 | 93 | 137 |
37+
| 3 | 150 | 1 | 1 | 453 | 127 |
38+
39+
## NumResources -> Total Resources
40+
| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Total Resources |
41+
| ------------ | -------- | ------- | --------------- | ---------- | ---------------- | -------------------- | ---------- | --------------- |
42+
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | <total> |
43+
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 244 |
44+
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 1204 |
45+
46+
## Observations
47+
48+
1. We are reloading after reconciling a ReferenceGrant even when there is no Gateway. This is because we treat every
49+
upsert/delete of a ReferenceGrant as a change. This means we will regenerate NGINX config every time a ReferenceGrant
50+
is created, updated (generation must change), or deleted, even if it does not apply to the accepted Gateway.
51+
52+
Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1124
53+
54+
2. We are reloading after reconciling a HTTPRoute even when there is no accepted Gateway and no config being generated.
55+
56+
Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1123
57+
58+
3. All reloads were in the <500ms bucket. A slight increase in the reload time based on number of configured resources
59+
resulting in NGINX configuration changes was observed.
60+
61+
4. No errors (NGF or NGINX) were observed in any test run.

‎tests/reconfig/scripts/cafe-routes.yaml

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
apiVersion: gateway.networking.k8s.io/v1beta1
2+
kind: HTTPRoute
3+
metadata:
4+
name: cafe-tls-redirect
5+
spec:
6+
parentRefs:
7+
- name: gateway.networking.k8s.io/v1beta1
8+
namespace: default
9+
sectionName: http
10+
hostnames:
11+
- "cafe.example.com"
12+
rules:
13+
- filters:
14+
- type: RequestRedirect
15+
requestRedirect:
16+
scheme: https
17+
port: 443
18+
---
19+
apiVersion: gateway.networking.k8s.io/v1beta1
20+
kind: HTTPRoute
21+
metadata:
22+
name: coffee
23+
spec:
24+
parentRefs:
25+
- name: gateway
26+
namespace: default
27+
sectionName: https
28+
hostnames:
29+
- "cafe.example.com"
30+
rules:
31+
- matches:
32+
- path:
33+
type: PathPrefix
34+
value: /coffee
35+
backendRefs:
36+
- name: coffee
37+
port: 80
38+
---
39+
apiVersion: gateway.networking.k8s.io/v1beta1
40+
kind: HTTPRoute
41+
metadata:
42+
name: tea
43+
spec:
44+
parentRefs:
45+
- name: gateway
46+
sectionName: https
47+
namespace: default
48+
hostnames:
49+
- "cafe.example.com"
50+
rules:
51+
- matches:
52+
- path:
53+
type: PathPrefix
54+
value: /tea
55+
backendRefs:
56+
- name: tea
57+
port: 80

‎tests/reconfig/scripts/cafe.yaml

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: coffee
5+
spec:
6+
replicas: 1
7+
selector:
8+
matchLabels:
9+
app: coffee
10+
template:
11+
metadata:
12+
labels:
13+
app: coffee
14+
spec:
15+
containers:
16+
- name: coffee
17+
image: nginxdemos/nginx-hello:plain-text
18+
ports:
19+
- containerPort: 8080
20+
---
21+
apiVersion: v1
22+
kind: Service
23+
metadata:
24+
name: coffee
25+
spec:
26+
ports:
27+
- port: 80
28+
targetPort: 8080
29+
protocol: TCP
30+
name: http
31+
selector:
32+
app: coffee
33+
---
34+
apiVersion: apps/v1
35+
kind: Deployment
36+
metadata:
37+
name: tea
38+
spec:
39+
replicas: 1
40+
selector:
41+
matchLabels:
42+
app: tea
43+
template:
44+
metadata:
45+
labels:
46+
app: tea
47+
spec:
48+
containers:
49+
- name: tea
50+
image: nginxdemos/nginx-hello:plain-text
51+
ports:
52+
- containerPort: 8080
53+
---
54+
apiVersion: v1
55+
kind: Service
56+
metadata:
57+
name: tea
58+
spec:
59+
ports:
60+
- port: 80
61+
targetPort: 8080
62+
protocol: TCP
63+
name: http
64+
selector:
65+
app: tea

‎tests/reconfig/scripts/certificate-ns-and-cafe-secret.yaml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: certificate
5+
---
6+
apiVersion: v1
7+
kind: Secret
8+
metadata:
9+
name: cafe-secret
10+
namespace: certificate
11+
type: kubernetes.io/tls
12+
data:
13+
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNzakNDQVpvQ0NRQzdCdVdXdWRtRkNEQU5CZ2txaGtpRzl3MEJBUXNGQURBYk1Sa3dGd1lEVlFRRERCQmoKWVdabExtVjRZVzF3YkdVdVkyOXRNQjRYRFRJeU1EY3hOREl4TlRJek9Wb1hEVEl6TURjeE5ESXhOVEl6T1ZvdwpHekVaTUJjR0ExVUVBd3dRWTJGbVpTNWxlR0Z0Y0d4bExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTHFZMnRHNFc5aStFYzJhdnV4Q2prb2tnUUx1ek10U1Rnc1RNaEhuK3ZRUmxIam8KVzFLRnMvQVdlS25UUStyTWVKVWNseis4M3QwRGtyRThwUisxR2NKSE50WlNMb0NEYUlRN0Nhck5nY1daS0o4Qgo1WDNnVS9YeVJHZjI2c1REd2xzU3NkSEQ1U2U3K2Vab3NPcTdHTVF3K25HR2NVZ0VtL1Q1UEMvY05PWE0zZWxGClRPL051MStoMzROVG9BbDNQdTF2QlpMcDNQVERtQ0thaEROV0NWbUJQUWpNNFI4VERsbFhhMHQ5Z1o1MTRSRzUKWHlZWTNtdzZpUzIrR1dYVXllMjFuWVV4UEhZbDV4RHY0c0FXaGRXbElweHlZQlNCRURjczN6QlI2bFF1OWkxZAp0R1k4dGJ3blVmcUVUR3NZdWxzc05qcU95V1VEcFdJelhibHhJZVVDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBcjkrZWJ0U1dzSnhLTGtLZlRkek1ISFhOd2Y5ZXFVbHNtTXZmMGdBdWVKTUpUR215dG1iWjlpbXQKL2RnWlpYVE9hTElHUG9oZ3BpS0l5eVVRZVdGQ2F0NHRxWkNPVWRhbUloOGk0Q1h6QVJYVHNvcUNOenNNLzZMRQphM25XbFZyS2lmZHYrWkxyRi8vblc0VVNvOEoxaCtQeDljY0tpRDZZU0RVUERDRGh1RUtFWXcvbHpoUDJVOXNmCnl6cEJKVGQ4enFyM3paTjNGWWlITmgzYlRhQS82di9jU2lyamNTK1EwQXg4RWpzQzYxRjRVMTc4QzdWNWRCKzQKcmtPTy9QNlA0UFlWNTRZZHMvRjE2WkZJTHFBNENCYnExRExuYWRxamxyN3NPbzl2ZzNnWFNMYXBVVkdtZ2todAp6VlZPWG1mU0Z4OS90MDBHUi95bUdPbERJbWlXMGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
14+
tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzZtTnJSdUZ2WXZoSE4KbXI3c1FvNUtKSUVDN3N6TFVrNExFeklSNS9yMEVaUjQ2RnRTaGJQd0ZuaXAwMFBxekhpVkhKYy92TjdkQTVLeApQS1VmdFJuQ1J6YldVaTZBZzJpRU93bXF6WUhGbVNpZkFlVjk0RlAxOGtSbjl1ckV3OEpiRXJIUncrVW51L25tCmFMRHF1eGpFTVBweGhuRklCSnYwK1R3djNEVGx6TjNwUlV6dnpidGZvZCtEVTZBSmR6N3Rid1dTNmR6MHc1Z2kKbW9RelZnbFpnVDBJek9FZkV3NVpWMnRMZllHZWRlRVJ1VjhtR041c09va3R2aGxsMU1udHRaMkZNVHgySmVjUQo3K0xBRm9YVnBTS2NjbUFVZ1JBM0xOOHdVZXBVTHZZdFhiUm1QTFc4SjFINmhFeHJHTHBiTERZNmpzbGxBNlZpCk0xMjVjU0hsQWdNQkFBRUNnZ0VBQnpaRE50bmVTdWxGdk9HZlFYaHRFWGFKdWZoSzJBenRVVVpEcUNlRUxvekQKWlV6dHdxbkNRNlJLczUyandWNTN4cU9kUU94bTNMbjNvSHdNa2NZcEliWW82MjJ2dUczYnkwaVEzaFlsVHVMVgpqQmZCcS9UUXFlL2NMdngvSkczQWhFNmJxdFRjZFlXeGFmTmY2eUtpR1dzZk11WVVXTWs4MGVJVUxuRmZaZ1pOCklYNTlSOHlqdE9CVm9Sa3hjYTVoMW1ZTDFsSlJNM3ZqVHNHTHFybmpOTjNBdWZ3ZGRpK1VDbGZVL2l0K1EvZkUKV216aFFoTlRpNVFkRWJLVStOTnYvNnYvb2JvandNb25HVVBCdEFTUE05cmxFemIralQ1WHdWQjgvLzRGY3VoSwoyVzNpcjhtNHVlQ1JHSVlrbGxlLzhuQmZ0eVhiVkNocVRyZFBlaGlPM1FLQmdRRGlrR3JTOTc3cjg3Y1JPOCtQClpoeXltNXo4NVIzTHVVbFNTazJiOTI1QlhvakpZL2RRZDVTdFVsSWE4OUZKZnNWc1JRcEhHaTFCYzBMaTY1YjIKazR0cE5xcVFoUmZ1UVh0UG9GYXRuQzlPRnJVTXJXbDVJN0ZFejZnNkNQMVBXMEg5d2hPemFKZUdpZVpNYjlYTQoybDdSSFZOcC9jTDlYbmhNMnN0Q1lua2Iwd0tCZ1FEUzF4K0crakEyUVNtRVFWNXA1RnRONGcyamsyZEFjMEhNClRIQ2tTazFDRjhkR0Z2UWtsWm5ZbUt0dXFYeXNtekJGcnZKdmt2eUhqbUNYYTducXlpajBEdDZtODViN3BGcVAKQWxtajdtbXI3Z1pUeG1ZMXBhRWFLMXY4SDNINGtRNVl3MWdrTWRybVJHcVAvaTBGaDVpaGtSZS9DOUtGTFVkSQpDcnJjTzhkUVp3S0JnSHA1MzRXVWNCMVZibzFlYStIMUxXWlFRUmxsTWlwRFM2TzBqeWZWSmtFb1BZSEJESnp2ClIrdzZLREJ4eFoyWmJsZ05LblV0YlhHSVFZd3lGelhNcFB5SGxNVHpiZkJhYmJLcDFyR2JVT2RCMXpXM09PRkgKcmppb21TUm1YNmxhaDk0SjRHU0lFZ0drNGw1SHhxZ3JGRDZ2UDd4NGRjUktJWFpLZ0w2dVJSSUpBb0dCQU1CVApaL2p5WStRNTBLdEtEZHUrYU9ORW4zaGxUN3hrNXRKN3NBek5rbWdGMU10RXlQUk9Xd1pQVGFJbWpRbk9qbHdpCldCZ2JGcXg0M2ZlQ1Z4ZXJ6V3ZEM0txaWJVbWpCTkNMTGtYeGh3ZEVteFQwVit2NzZGYzgwaTNNYVdSNnZZR08KditwVVovL0F6UXdJcWZ6dlVmV2ZxdStrMHlhVXhQOGNlcFBIRyt0bEFvR0FmQUtVVWhqeFU0Ym5vVzVwVUhKegpwWWZXZXZ5TW54NWZyT2VsSmRmNzlvNGMvMHhVSjh1eFBFWDFkRmNrZW96dHNpaVFTNkN6MENRY09XVWxtSkRwCnVrdERvVzM3VmNSQU1BVjY3NlgxQVZlM0UwNm5aL2g2Tkd4Z28rT042Q3pwL0lkMkJPUm9IMFAxa2RjY1NLT3kKMUtFZlNnb1B0c1N1eEpBZXdUZmxDMXc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K

‎tests/reconfig/scripts/create-resources-gw-last.sh

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/bin/bash
2+
3+
num_namespaces=$1
4+
5+
# Create namespaces
6+
for ((i=1; i<=$num_namespaces; i++)); do
7+
namespace_name="namespace$i"
8+
kubectl create namespace "$namespace_name"
9+
done
10+
11+
# Create single instance resources
12+
kubectl create -f certificate-ns-and-cafe-secret.yaml
13+
kubectl create -f reference-grant.yaml
14+
15+
# Create backend service and apps
16+
for ((i=1; i<=$num_namespaces; i++)); do
17+
namespace_name="namespace$i"
18+
sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe.yaml | kubectl apply -n "$namespace_name" -f -
19+
done
20+
21+
# Create routes
22+
for ((i=1; i<=$num_namespaces; i++)); do
23+
namespace_name="namespace$i"
24+
sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe-routes.yaml | kubectl apply -n "$namespace_name" -f -
25+
done
26+
27+
# Wait for apps to be ready
28+
sleep 60
29+
30+
# Create Gateway
31+
kubectl create -f gateway.yaml

‎tests/reconfig/scripts/create-resources-routes-last.sh

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/bin/bash
2+
3+
num_namespaces=$1
4+
5+
# Create namespaces
6+
for ((i=1; i<=$num_namespaces; i++)); do
7+
namespace_name="namespace$i"
8+
kubectl create namespace "$namespace_name"
9+
done
10+
11+
# Create backend service and apps
12+
for ((i=1; i<=$num_namespaces; i++)); do
13+
namespace_name="namespace$i"
14+
sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe.yaml | kubectl apply -n "$namespace_name" -f -
15+
done
16+
17+
# Wait for apps to be ready
18+
sleep 60
19+
20+
# Create single instance resources
21+
kubectl create -f certificate-ns-and-cafe-secret.yaml
22+
kubectl create -f reference-grant.yaml
23+
kubectl create -f gateway.yaml
24+
25+
# Create routes
26+
for ((i=1; i<=$num_namespaces; i++)); do
27+
namespace_name="namespace$i"
28+
sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe-routes.yaml | kubectl apply -n "$namespace_name" -f -
29+
done

‎tests/reconfig/scripts/delete-multiple.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/bash
2+
3+
num_namespaces=$1
4+
5+
# Delete namespaces
6+
for ((i=1; i<=$num_namespaces; i++)); do
7+
namespace_name="namespace$i"
8+
kubectl delete namespace "$namespace_name"
9+
done
10+
11+
# Delete single instance resources
12+
kubectl delete -f gateway.yaml
13+
kubectl delete -f reference-grant.yaml
14+
kubectl delete -f certificate-ns-and-cafe-secret.yaml

‎tests/reconfig/scripts/gateway.yaml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
apiVersion: gateway.networking.k8s.io/v1beta1
2+
kind: Gateway
3+
metadata:
4+
name: gateway
5+
spec:
6+
gatewayClassName: nginx
7+
listeners:
8+
- name: http
9+
port: 80
10+
protocol: HTTP
11+
allowedRoutes:
12+
namespaces:
13+
from: "All"
14+
- name: https
15+
port: 443
16+
protocol: HTTPS
17+
allowedRoutes:
18+
namespaces:
19+
from: "All"
20+
tls:
21+
mode: Terminate
22+
certificateRefs:
23+
- kind: Secret
24+
name: cafe-secret
25+
namespace: certificate

‎tests/reconfig/scripts/reference-grant.yaml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
apiVersion: gateway.networking.k8s.io/v1beta1
2+
kind: ReferenceGrant
3+
metadata:
4+
name: access-to-cafe-secret
5+
namespace: certificate
6+
spec:
7+
to:
8+
- group: ""
9+
kind: Secret
10+
name: cafe-secret # if you omit this name, then Gateways in default ns can access all Secrets in the certificate ns
11+
from:
12+
- group: gateway.networking.k8s.io
13+
kind: Gateway
14+
namespace: default

0 commit comments

Comments
 (0)