|
1 | 1 | # Reconfig tests |
2 | 2 |
|
| 3 | +## Goals |
| 4 | + |
| 5 | +- Measure how long it takes NGF to reconfigure NGINX when a number of Gateway API and referenced core Kubernetes |
| 6 | + resources are created at once. |
| 7 | +- Two runs of each test should be ran with differing numbers of resources. Each run will deploy: |
| 8 | + - a single Gateway, Secret, and ReferenceGrant resources |
| 9 | + - `x+1` number of namespaces |
| 10 | + - `2x` number of backend apps and services |
| 11 | + - `3x` number of HTTPRoutes. |
| 12 | +- Where x=30 OR x=150. |
| 13 | + |
| 14 | +## Test Environment |
| 15 | + |
| 16 | + The following cluster will be sufficient: |
| 17 | + |
| 18 | +- A Kubernetes cluster with 3 nodes on GKE |
| 19 | + - Node: e2-medium (2 vCPU, 4GB memory) |
| 20 | + |
3 | 21 | ## Setup |
4 | 22 |
|
5 | 23 | 1. Create cloud cluster |
|
16 | 34 | --create-namespace --wait -n nginx-gateway |
17 | 35 | ``` |
18 | 36 |
|
19 | | -4. Optional: Add pod scrape if running in GKE (see [GKE Pod scrape config](#gke-pod-scrape-config)). |
20 | | -5. Run tests: |
| 37 | +4. Run tests: |
21 | 38 | 1. There are 3 versions of the reconfiguration tests that need to be ran, with a low and high number of resources. |
22 | 39 | Therefore, a full test suite includes 6 test runs. |
23 | 40 | 2. There are scripts to generate the required resources and config changes. |
|
29 | 46 | - Note: Clean up after each test run for isolated results. There's a script provided for removing all the test |
30 | 47 | fixtures `scripts/delete-multiple.sh` which takes a number (needs to be the same number as what was used in the |
31 | 48 | create script.) |
32 | | -6. After each individual test run, grab logs of both NGF containers and grab metrics. |
| 49 | +5. After each individual test run, grab logs of both NGF containers and grab metrics. |
33 | 50 | Note: You can expose metrics by running the below snippet and then navigating to `127.0.0.1:9113/metrics`: |
34 | 51 |
|
35 | 52 | ```bash |
36 | 53 | GW_POD=$(k get pods -n nginx-gateway | sed -n '2s/^\([^[:space:]]*\).*$/\1/p') |
37 | 54 | kubectl port-forward $GW_POD -n nginx-gateway 9113:9113 & |
38 | 55 | ``` |
39 | 56 |
|
40 | | -7. Measure Time To Ready by as described in each test, get the reload count, and get the average NGINX reload duration. |
41 | | -8. For accuracy, repeat the test suite once or twice, take the averages, and look for any anomolies/ outliers. |
| 57 | +6. Measure Time To Ready as described in each test, get the reload count, and get the average NGINX reload duration. |
| 58 | + The average reload duration can be computed by taking the `nginx_gateway_fabric_nginx_reloads_milliseconds_sum` |
| 59 | + metric value and dividing it by the `nginx_gateway_fabric_nginx_reloads_milliseconds_count` metric value. |
| 60 | +7. For accuracy, repeat the test suite once or twice, take the averages, and look for any anomolies or outliers. |
42 | 61 |
|
43 | 62 | ## Tests |
44 | 63 |
|
|
48 | 67 | 1. Use either of the provided scripts with the required number of resources, |
49 | 68 | e.g. `cd scripts && bash create-resources-gw-last.sh 30`. The script will deploy backend apps and services, wait |
50 | 69 | 60 seconds for them to be ready, and deploy 1 Gateway, 1 RefGrant, 1 Secret, and HTTPRoutes. |
51 | | - 2. Deploy NFG |
52 | | - 3. Check logs for time takes from start-up -> config written and NGINX reloaded. Get reload count and average reload |
| 70 | + 2. Deploy NGF |
| 71 | + 3. Check logs for time it takes from start-up -> config written and NGINX reloaded. Get reload count and average reload |
53 | 72 | duration from metrics and logs. |
54 | 73 |
|
55 | 74 | ### Test 2: Start NGF, deploy Gateway, create many resources attached to GW |
56 | 75 |
|
57 | | -1. Deploy all Gateway resources, NFG running: |
58 | | - 1. Deploy NFG |
| 76 | +1. Deploy all Gateway resources, NGF running: |
| 77 | + 1. Deploy NGF |
59 | 78 | 2. Run the provided script with the required number of resources, |
60 | 79 | e.g. `cd scripts && bash create-resources-routes-last.sh 30`. The script will deploy backend apps and services, |
61 | 80 | wait 60 seconds for them to be ready, and deploy 1 Gateway, 1 Secret, 1 RefGrant, and HTTPRoutes at the same time. |
62 | | - 3. Check logs for time takes from NFG receiving first resource update -> final config written, and NGINX's final |
| 81 | + 3. Check logs for time it takes from NGF receiving first resource update -> final config written, and NGINX's final |
63 | 82 | reload. Check logs for average individual HTTPRoute TTR also. Get reload count and average reload duration from |
64 | 83 | metrics and logs. |
65 | 84 |
|
66 | 85 | ### Test 3: Start NGF, create many resources attached to a Gateway, deploy the Gateway |
67 | 86 |
|
68 | | -1. Deploy HTTPRoute resources, NFG running, Gateway last: |
69 | | - 1. Deploy NFG |
| 87 | +1. Deploy HTTPRoute resources, NGF running, Gateway last: |
| 88 | + 1. Deploy NGF |
70 | 89 | 2. Run the provided script with the required number of resources, |
71 | 90 | e.g. `cd scripts && bash create-resources-gw-last.sh 30`. |
72 | 91 | The script will deploy the namespaces, backend apps and services, 1 Secret, 1 ReferenceGrant, and the HTTPRoutes; |
73 | 92 | wait 60 seconds for the backend apps to be ready, and then deploy 1 Gateway for all HTTPRoutes. |
74 | | - 3. Check logs for time takes from NFG receiving gateway resource -> config written and NGINX reloaded. Get reload |
| 93 | + 3. Check logs for time it takes from NGF receiving gateway resource -> config written and NGINX reloaded. Get reload |
75 | 94 | count and average reload duration from metrics and logs. |
76 | | - |
77 | | -## GKE Pod scrape config |
78 | | - |
79 | | -To create a Pod scrape config, you can run the following: |
80 | | - |
81 | | - ```bash |
82 | | - cat <<EOF | kubectl apply -f - |
83 | | - apiVersion: monitoring.googleapis.com/v1 |
84 | | - kind: PodMonitoring |
85 | | - metadata: |
86 | | - name: prom-example |
87 | | - namespace: nginx-gateway |
88 | | - spec: |
89 | | - selector: |
90 | | - matchLabels: |
91 | | - app.kubernetes.io/name: nginx-gateway |
92 | | - endpoints: |
93 | | - - port: metrics |
94 | | - interval: 30s |
95 | | - EOF |
96 | | - ``` |
0 commit comments