Nginx ModSecurity WAF with OAuth2 Proxy does not work with http/2

[logicfox]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): waf, oauth2

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

NGINX Ingress controller version: 0.9.0

Kubernetes version (use kubectl version): 1.8.2

Environment:

• Cloud provider or hardware configuration: Azure AKS
• OS (e.g. from /etc/os-release): Ubuntu (on nodes, managed master)
• Kernel (e.g. uname -a):
• Install tools:
• Others: Nginx Ingress Controller 0.9.0, bit.ly oauth2_proxy, Google Chorme, Mozilla Firefox, curl

The AKS cluster is configured with nginx-ingress v0.9.0, installed from the master branch with the manifests. The cluster also runs github.com/kubernetes-incubator/external-dns to register sub-domains and github.com/jetstack/kube-lego to obtain Let's Encrypt certificates. We have set up external-auth based on the example provided at github.com/kubernetes/ingress-nginx/tree/master/docs/examples/external-auth to authenticate with Azure AD. To screen malicious requests, we enabled the ModSecurity WAF with the standard OWASP rules and paranoia level 1.

What happened: Requests made over HTTP 2.0 to applications that are secured by oauth2_proxy do not receive requests when the WAF is enabled. The request just hangs until the connection times out. However, everything works normally over HTTP 1.1. The behavior is completely normal if the WAF is disabled.

WAF	OAuth	HTTP	Does it work?
Enabled	Enabled	1.1	Yes
Disabled	Enabled	1.1	Yes
Enabled	Disabled	1.1	Yes
Disabled	Disabled	1.1	Yes
Enabled	Enabled	2	No
Disabled	Enabled	2	Yes
Enabled	Disabled	2	Yes
Disabled	Disabled	2	Yes

What you expected to happen: Application receives requests after being filtered by the WAF and authenticated by oauth2_proxy, irrespective of HTTP protocol version. All traffic is over SSL/TLS until termination at the ingress controller.

How to reproduce it (as minimally and precisely as possible): Set up a Kubernetes cluster with Nginx Ingress Controller, TLS termination and Oauth2_proxy as an external auth provider. Enable Modsecurity WAF per documentation and attempt accessing application over HTTP2. Notice that the request hangs and stdout logs of the Nginx and Oauth2_proxy pods will print only when the connection terminates.

Anything else we need to know: HTTP 1.1 works without issue. HTTP2 fails when modsecurity WAF and external authentication with oauth2_proxy is enabled.

[aledbf]

@logicfox can you please add --v=10 to the deployment and post the logs in a gist after a couple of requests?

[aledbf]

@logicfox also please test quay.io/aledbf/nginx-ingress-controller:0.311. It contains current master.

[logicfox]

@aledbf The issue is present in the quay.io/aledbf/nginx-ingress-controller:0.311 image as well. I had to terminate the request.

The log files were too big for Gist :) . So I pushed them to this Github repo: https://github.com/logicfox/nginx-ingress-logs

[aledbf]

@logicfox in the logs I see:

2018/01/19 18:20:57 [debug] 65#65: *1 SSL_read: 9
2018/01/19 18:20:57 [debug] 65#65: *1 SSL_read: -1
2018/01/19 18:20:57 [debug] 65#65: *1 SSL_get_error: 2

are you using a self generated SSL certificate for dvwa.demo.blargh-digital.net ?

[logicfox]

No, it's a let's encrypt certificate acquired with kube-lego

[logicfox]

@aledbf Let me clarify. Its only the POST requests that hang. Get requests work fine. The application in question is a test application called Damn Vulnerable Web Application that contains many vulnerabilities. In HTTP 1.1 mode, the oauth2_proxy authenticates the request and the user is presented with the application's login page. Logging into the application is a simple POST which fails over HTTP/2, but works with HTTP 1.1.

Curl with HTTP/2

 curl 'https://dvwa.demo.blargh-digital.net/login.php' -H 'Host: dvwa.demo.blargh-digital.net' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://dvwa.demo.blargh-digital.net/login.php' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Cookie: _oauth2_proxy=YWRpdHlhLnN1bmRhcmFtdXJ0aHlAbWFlcnNrLmNvbQ==|1516385436|T65hJkd1VXX9w3-IsYbEnaWTIww=; PHPSESSID=18dg3kv5eb8kpi6pd19cn6dal1; security=impossible' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' --data 'username=admin&password=password&Login=Login&user_token=dbc2c6230a4be9de291c7e60e159bd93' -v
*   Trying 40.68.210.189...
* TCP_NODELAY set
* Connected to dvwa.demo.blargh-digital.net (40.68.210.189) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=dvwa.demo.blargh-digital.net
*  start date: Jan 11 12:01:16 2018 GMT
*  expire date: Apr 11 12:01:16 2018 GMT
*  subjectAltName: host "dvwa.demo.blargh-digital.net" matched cert's "dvwa.demo.blargh-digital.net"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc86c808e00)
> POST /login.php HTTP/2
> Host: dvwa.demo.blargh-digital.net
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-US,en;q=0.5
> Referer: https://dvwa.demo.blargh-digital.net/login.php
> Content-Type: application/x-www-form-urlencoded
> Cookie: _oauth2_proxy=YWRpdHlhLnN1bmRhcmFtdXJ0aHlAbWFlcnNrLmNvbQ==|1516385436|T65hJkd1VXX9w3-IsYbEnaWTIww=; PHPSESSID=18dg3kv5eb8kpi6pd19cn6dal1; security=impossible
> DNT: 1
> Connection: keep-alive
> Upgrade-Insecure-Requests: 1
> Pragma: no-cache
> Cache-Control: no-cache
> Content-Length: 88
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* We are completely uploaded and fine

Curl with HTTP 1.1

curl 'https://dvwa.demo.blargh-digital.net/login.php' -H 'Host: dvwa.demo.blargh-digital.net' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://dvwa.demo.blargh-digital.net/login.php' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Cookie: _oauth2_proxy=YWRpdHlhLnN1bmRhcmFtdXJ0aHlAbWFlcnNrLmNvbQ==|1516385436|T65hJkd1VXX9w3-IsYbEnaWTIww=; PHPSESSID=18dg3kv5eb8kpi6pd19cn6dal1; security=impossible' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' --data 'username=admin&password=password&Login=Login&user_token=dbc2c6230a4be9de291c7e60e159bd93' -v --http1.1
*   Trying 40.68.210.189...
* TCP_NODELAY set
* Connected to dvwa.demo.blargh-digital.net (40.68.210.189) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=dvwa.demo.blargh-digital.net
*  start date: Jan 11 12:01:16 2018 GMT
*  expire date: Apr 11 12:01:16 2018 GMT
*  subjectAltName: host "dvwa.demo.blargh-digital.net" matched cert's "dvwa.demo.blargh-digital.net"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> POST /login.php HTTP/1.1
> Host: dvwa.demo.blargh-digital.net
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-US,en;q=0.5
> Referer: https://dvwa.demo.blargh-digital.net/login.php
> Content-Type: application/x-www-form-urlencoded
> Cookie: _oauth2_proxy=YWRpdHlhLnN1bmRhcmFtdXJ0aHlAbWFlcnNrLmNvbQ==|1516385436|T65hJkd1VXX9w3-IsYbEnaWTIww=; PHPSESSID=18dg3kv5eb8kpi6pd19cn6dal1; security=impossible
> DNT: 1
> Connection: keep-alive
> Upgrade-Insecure-Requests: 1
> Pragma: no-cache
> Cache-Control: no-cache
> Content-Length: 88
>
* upload completely sent off: 88 out of 88 bytes
< HTTP/1.1 302 Found
< Server: nginx/1.13.8
< Date: Fri, 19 Jan 2018 20:12:37 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Location: login.php
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
<
* Connection #0 to host dvwa.demo.blargh-digital.net left intact

[logicfox]

The behaviour is the same with version 0.10.0 released yesterday

[Stono]

I can confirm this same behaviour only when enable-owasp-modsecurity-crs is enabled. Going onto the pods and running top you'll see extremely high CPU usage and then in the logs I get:

127.0.0.1 - [127.0.0.1] - - [25/Jan/2018:21:43:34 +0000] "GET / HTTP/2.0" 200 30295

Have you tested differences with crs enabled/disabled too? I wasn't expecting it to hammer the CPU so much

For a single request:

[logicfox]

@Stono the behavior is the same either way - CRS enabled and disabled. The request just hangs until the connection is terminated by the client.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[Stono]

Hey!
I've just realised this seems to be exactly what I was observing in #2545

@logicfox did you get anywhere?