Skip to content

Commit a2e9e7a

Browse files
chrisakkerchrisakker
committed
doc updates
1 parent 05567be commit a2e9e7a

6 files changed

+288
-46
lines changed

‎docs/Installation.md

+231
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
# Nginx Kubernetes Loadbalancer Solution
2+
3+
<br/>
4+
5+
## This is the Installation Guide for the Nginx Kubernetes Loadbalancer controller Solution. It contains the detailed instructions for implementing the different components for the Solution.
6+
7+
<br/>
8+
9+
## Pre-Requisites
10+
11+
- Working kubernetes cluster, with admin privleges
12+
- Running nginx-ingress controller, either OSS or Plus. This install guide follows the instructions for deploying an Nginx Ingress Controller here: https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
13+
- Optional: Demo application, this install guide uses the Nginx Cafe example, found here: https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/ingress-resources/complete-example
14+
- A bare metal Linux server or VM for the external LB Server, connected to a network external to the cluster.
15+
- NginxPlus software loaded on the Edge Server, this install guide follows the instructions for installing NginxPlus on Centos 7, located here: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/
16+
- The Nginx Kubernetes Loadbalancer (NKL) controller, new software for the Solution
17+
18+
<br/>
19+
20+
## Kubernetes Cluster
21+
22+
A standard K8s cluster is all that is required. There must be enough resources to run the Nginx Ingress Controller, and the Nginx Kubernetes Loadbalancer Controller. You must have administrative access to be able to create the namespace, services, and deployments for this Solution. This Solution was tested on Kubernetes version 1.23. Most recent versions => v1.21 should work just fine.
23+
24+
<br/>
25+
26+
## Nginx Ingress Controller
27+
28+
This is not part of the actual Solution, but it is the destination target for traffic (north-south) that is being sent to the cluster. The installation of the actual Ingress Controller is outside the scope of the installation guide, but we include the links to the docs for your reference. The NIC installation must follow the documents exactly as written, as this Solution refers to the `nginx-ingress` namespace and service objects. Only the very last step is changed.
29+
30+
>Important! The very last step in the NIC deployment with manifests, is to deploy the NodePort.yaml Service file. `This file must be changed! It is not the default nodeport file provided.` Use the `nodeport-nkl.yaml` manifest file that is provided here. The "ports name" in the Nodeport manifest `MUST` be in the correct format for this Solution to work correctly. The port name is the mapping from NodePort to the LB Server's upstream blocks. The port names are intentionally changed to avoid conflicts with other NodePort definitions.
31+
32+
Review the new NodePort Service defintion file:
33+
34+
```yaml
35+
# NKL Nodeport Service file
36+
# NodePort port name must be in the format of
37+
# nkl-<upstream-block-name>
38+
# Chris Akker, Jan 2023
39+
#
40+
apiVersion: v1
41+
kind: Service
42+
metadata:
43+
name: nginx-ingress
44+
namespace: nginx-ingress
45+
spec:
46+
type: NodePort
47+
ports:
48+
- port: 80
49+
targetPort: 80
50+
protocol: TCP
51+
name: nkl-nginx-lb-http # Must be changed
52+
- port: 443
53+
targetPort: 443
54+
protocol: TCP
55+
name: nkl-nginx-lb-https # Must be changed
56+
selector:
57+
app: nginx-ingress
58+
59+
```
60+
61+
62+
```bash
63+
64+
kubectl apply -f nodeport-nkl.yaml
65+
66+
```
67+
68+
<br/>
69+
70+
## Demo Application
71+
72+
This is not part of the actual Solution, but it is useful to have a well-known application running in the cluster, as a useful target for test commands. The example provided here is used by the Solution to demonstrate K8s application health check monitoring, to determine if the application is running in the cluster. If you choose a different Application to test with, the health checks provided will NOT work, and will need to be modified to work correctly.
73+
74+
- Do not use the `cafe-ingress.yaml` file. Rather, use the `cafe-virtualserver.yaml` file that is provided here. It uses the Nginx CRDs to define a VirtualServer, and the related Routes and Redirects needed. The redirects are required for the LB Server's health checks to work correctly!
75+
76+
```yaml
77+
#Example virtual server with routes for Cafe Demo
78+
#For NKL Solution, redirects required for LB Server health checks
79+
#Chris Akker, Jan 2023
80+
#
81+
apiVersion: k8s.nginx.org/v1
82+
kind: VirtualServer
83+
metadata:
84+
name: cafe-vs
85+
spec:
86+
host: cafe.example.com
87+
tls:
88+
secret: cafe-secret
89+
redirect:
90+
enable: true #Redirect from http > https
91+
code: 301
92+
upstreams:
93+
- name: tea
94+
service: tea-svc
95+
port: 80
96+
lb-method: round_robin
97+
slow-start: 20s
98+
healthCheck:
99+
enable: true
100+
path: /tea
101+
interval: 20s
102+
jitter: 3s
103+
fails: 5
104+
passes: 2
105+
connect-timeout: 30s
106+
read-timeout: 20s
107+
- name: coffee
108+
service: coffee-svc
109+
port: 80
110+
lb-method: round_robin
111+
healthCheck:
112+
enable: true
113+
path: /coffee
114+
interval: 10s
115+
jitter: 3s
116+
fails: 3
117+
passes: 2
118+
connect-timeout: 30s
119+
read-timeout: 20s
120+
routes:
121+
- path: /
122+
action:
123+
redirect:
124+
url: https://cafe.example.com/coffee
125+
code: 302 #Redirect from / > /coffee
126+
- path: /tea
127+
action:
128+
pass: tea
129+
- path: /coffee
130+
action:
131+
pass: coffee
132+
```
133+
134+
<br/>
135+
136+
## Linux VM or bare-metal LB Server
137+
138+
This is a standard Linux OS system, based on the Linux Distro and Technical Specs required for NginxPlus, which can be found here: https://docs.nginx.com/nginx/technical-specs/ This installation guide followed the "Installation of Nginx Plus on Centos/Redhat/Oracle" steps for installing Nginx Plus. Note: This solution will not work with Nginx OpenSource, as OpenSource does not have the API that is used in this Solution. Installation on unsupported Distros is not Supported.
139+
140+
<br/>
141+
142+
## NginxPlus LB Server
143+
144+
This is the configuration required for the LB Server, external to the cluster. It must be configured for the following.
145+
146+
- Move the Nginx default Welcome page from port 80 to port 8080. Port 80 will be used by the stream context, instead of the http context.
147+
- API write access enabled
148+
- Plus Dashboard enabled, used for testing and visualization of the solution working
149+
- Stream context enabled, for TCP loadbalancing
150+
- Stream TCP loadbalancing configured
151+
152+
After a new installation of Nginx Plus, make the following configuration changes:
153+
154+
- Change Nginx's http default server to port 8080. See the included `default.conf` file. After reloading nginx, the default `Welcome to Nginx` page will be located at http://localhost:8080.
155+
156+
- Use the dashboard.conf file provided. It will enable the /api endpoint, change the port to 9000, and provide access to the Plus dashboard. Place this file in the /etc/nginx/conf.d folder, and reload nginx. The Plus dashboard is now accessible at <server-ip>:9000/dashboard.html. It should look similar to this:
157+
158+
< ss here>
159+
160+
- Create a new folder for the stream config files. /etc/nginx/stream was used in this Solution.
161+
162+
```bash
163+
mkdir /etc/nginx/stream
164+
```
165+
166+
- Create 2 new `STATE` files for Nginx. These are used to backup the configuration, in case Nginx restarts/reloads.
167+
168+
Nginx State Files Required for Upstreams
169+
state file /var/lib/nginx/state/nginx-lb-http.state
170+
state file /var/lib/nginx/state/nginx-lb-https.state
171+
172+
```bash
173+
touch /var/lib/nginx/state/nginx-lb-http.state
174+
touch /var/lib/nginx/state/nginx-lp-https.state
175+
```
176+
177+
- Enable the `stream` context for Nginx, which provides TCP load balancing. See the included nginx.conf file. Notice that the stream context is no longer commented out, a new folder is included, and a new stream.log logfile is used to track requests/responses.
178+
179+
- Configure Nginx Stream for TCP loadbalancing for this Solution. Place this file in the /etc/nginx/stream folder.
180+
181+
```bash
182+
# NginxK8sLB Stream configuration, for L4 load balancing
183+
# Chris Akker, Jan 2023
184+
# TCP Proxy and load balancing block
185+
# Nginx Kubernetes Loadbalancer
186+
# State File for persistent reloads/restarts
187+
# Health Check Match example for cafe.example.com
188+
#
189+
#### nginxk8slb.conf
190+
191+
upstream nginx-lb-http {
192+
zone nginx-lb-http 256k;
193+
state /var/lib/nginx/state/nginx-lb-http.state;
194+
}
195+
196+
upstream nginx-lb-https {
197+
zone nginx-lb-https 256k;
198+
state /var/lib/nginx/state/nginx-lb-https.state;
199+
}
200+
201+
server {
202+
listen 80;
203+
status_zone nginx-lb-http;
204+
proxy_pass nginx-lb-http;
205+
health_check match=cafe;
206+
}
207+
208+
server {
209+
listen 443;
210+
status_zone nginx-lb-https;
211+
proxy_pass nginx-lb-https;
212+
health_check match=cafe;
213+
}
214+
215+
match cafe {
216+
send "GET cafe.example.com/ HTTP/1.0\r\n";
217+
expect ~ "30*";
218+
}
219+
220+
```
221+
222+
<br/>
223+
224+
## Nginx Kubernetes Loadbalancing Controller
225+
226+
This is the new Controller, which is configured to watch the k8s environment, and send API updates to the Nginx LB Server when there are changes. It only requires three things.
227+
- New kubernetes namespace
228+
- NKL Deployment, to start the Controller
229+
- NKL ConfigMap, to configure the Controller
230+
231+

‎docs/NginxKubernetesLoadbalancer.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
- This is will synchronize the K8s Service Endpoint list, with the Nginx LB server's Upstream block server list.
1616
- The primary use case is for tracking the NodePort IP:Port definitions for the Nginx Ingress Controller's `nginx-ingress Service`.
1717
- With the NginxPlus Server located external to the K8s cluster, this new controller LB function would provide an alternative TCP "Load Balancer Service" for On Premises k8s clusters, which do not have access to a Cloud providers "Service Type LoadBalancer".
18-
- Make the solution a native Kubernetes Component, configured and managed with standard K8s tools.
18+
- The solution works as a native Kubernetes Controller object, configured and managed with standard K8s tools.
1919

2020
<br/>
2121

‎docs/cafe-virtualserver.yaml

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
#Example virtual server with routes for Cafe Demo
2+
#For NKL Solution, redirects required for LB Server health checks
3+
#Chris Akker, Jan 2023
4+
#
5+
apiVersion: k8s.nginx.org/v1
6+
kind: VirtualServer
7+
metadata:
8+
name: cafe-vs
9+
spec:
10+
host: cafe.example.com
11+
tls:
12+
secret: cafe-secret
13+
redirect:
14+
enable: true #Redirect from http > https
15+
code: 301
16+
upstreams:
17+
- name: tea
18+
service: tea-svc
19+
port: 80
20+
lb-method: round_robin
21+
slow-start: 20s
22+
healthCheck:
23+
enable: true
24+
path: /tea
25+
interval: 20s
26+
jitter: 3s
27+
fails: 5
28+
passes: 2
29+
connect-timeout: 30s
30+
read-timeout: 20s
31+
- name: coffee
32+
service: coffee-svc
33+
port: 80
34+
lb-method: round_robin
35+
healthCheck:
36+
enable: true
37+
path: /coffee
38+
interval: 10s
39+
jitter: 3s
40+
fails: 3
41+
passes: 2
42+
connect-timeout: 30s
43+
read-timeout: 20s
44+
routes:
45+
- path: /
46+
action:
47+
redirect:
48+
url: https://cafe.example.com/coffee
49+
code: 302 #Redirect from / > /coffee
50+
- path: /tea
51+
action:
52+
pass: tea
53+
- path: /coffee
54+
action:
55+
pass: coffee

‎docs/nginxlb.conf

-44
This file was deleted.

‎docs/nodeport-nkl.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# NKL Nodeport Service file
2-
# NodePort name must be in the format of
2+
# NodePort port name must be in the format of
33
# nkl-<upstream-block-name>
44
# Chris Akker, Jan 2023
55
#

‎docs/~$Nginx-k8s-LB-Controller-v2.pptx

-165 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)