Add per-Gateway support to NginxProxy

[kate-osborn]

The nginx deployment should be configurable via the helm chart and NginxProxy CRD.

• The NginxProxy CRD needs to be enhanced to work at the Gateway level. The nginx Deployment/Service configuration can then live in the NginxProxy CRD and either be applied globally (GatewayClass) or per Gateway. Certain fields (like a Service's loadBalancerIP) would have to be applied per-Gateway, so a user needs to be aware of where to attach the NginxProxy resource for these types of cases.
• The helm chart should allow for setting the global NginxProxy configuration, which we'll create in the control plane namespace at installation and link to the GatewayClass (as we do today).
• For per-Gateway configuration, a user will need to manually create their NginxProxy resources in the proper namespaces and link to their Gateways.
• A user can update the NginxProxy at runtime to change the Deployment/Service config, and we'll attempt to patch the Deployment and/or Service. If it fails, logs, events, and status are written.
• If a user creates a Gateway resource that references an NginxProxy configuration that doesn't exist, then we should wait until that resource exists before deploying nginx. Status and logs should be written in this scenario.

A/C:

• Allow cluster operator to specify NginxProxy in the parameters ref field of the Gateway
• NginxProxy fields set on the Gateway override NginxProxy fields set on the GatewayClass

[kate-osborn]

NginxProxy is currently a cluster-scoped resource since it's referenced from a GatewayClass which is also cluster-scoped. To reference a NginxProxy on a Gateway, it must be namespaced.

Changing NginxProxy from a cluster-scope resource to a namespaced resource will be a breaking change.

Update: CRD scope is immutable, so this change will require a nuke and pave upgrade. This is acceptable since upgrading to 2.0 will also require nuke and pave.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.