PATCH method dies with mod security with nginx configured as proxy/ingress

[PaulCharlton]

Version info:

version: libmodsecurity.so.3.0.3

name: nginx-ingress
repository: https://kubernetes-charts.storage.googleapis.com
version: 1.33.5
kubectl version
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"a17149e1a189050796ced469dbd78d380f2ed5ef", GitTreeState:"clean", BuildDate:"2020-04-16T23:15:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

===> cross-post from kubernetes/ingress-nginx#5723

Summary Observations:
PATCH method with JSON request body sent to ingress receives no response bytes on connection and connection is left open.

Connection is closed only when NGINX ingress is bounced due to server reload (on change of ConfigMap, pod deletion ... etc)

when Debug logging is enabled with
SecDebugLog /dev/stdout
SecDebugLogLevel 4

the debug log shows that phase 1 of the Modsecurity (a) recognized the "application/json", and that (b) phase 2 assigned the JSON attributes into ARGS for subsequent scanning.

** Changing from "DetectOnly" to rule enforcement does not change the behavior. Adding the CRS ruleset does not change the behavior.

** POST and PUT methods work without hanging the connection
** PATCH method works correctly when enable-modsecurity: "false"

====================================

[PaulCharlton]

open to any triage suggestions --- I realize there are several moving parts here between Nginx, modsecurity, and the Nginx-connector

[PaulCharlton]

Additional info:
The PATCH request body is a single JSON object, not terminated with a newline. The size in bytes of that JSON object matches exactly the Content-Length header. and Content-Type: application/json;charset=utf-8

Additional info:
adding EOL to the payload (and increasing the content length accordingly) is misbehaving in the same way

[martinhsv]

Hi @PaulCharlton ,

Probably the single most useful thing would be to increase your log levels. SecDebugLogLevel set to 9 will show a lot more detail about what is happening within ModSecurity. Similarly, if the error is occurring in nginx code, setting the error_log level to info, or even debug, might help.

[PaulCharlton]

@martinhsv thanks for the insights on how to proceed. Increasing the ModSecurity debug log level to 9 did not show anything useful -- all rules appear to be passing. I will continue some data gathering with the Nginx error_log level settings.

[PaulCharlton]

@martinhsv this may be the best clue thus far:
2020/06/20 15:11:03 [info] 34461#34461: *4116259 ModSecurity: Warning. Matched "Operator 'Rx' with parameter '^0?$' against variable 'REQUEST_HEADERS:content-length' (Value: '945' ) [file "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "161"] [id "920170"] [rev ""] [msg "GET or HEAD Request with Body Content."] [data "GET"] [severity "2"] [ver "OWASP_CRS/3.2.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"] [tag "CAPEC-272"] [hostname "10.240.0.26"] [uri "{redacted"] [unique_id "159266586361.523559"] [ref "o0,3v0,3v2318,3"], client: 10.240.0.35, server: _, request: "POST {redacted} HTTP/2.0", subrequest: "/_external-auth-L2NmLWVsbSgvfCQpKC4qKQ", host: "{redacted}"

For context:

1. I do not (yet) know the control flow through the various Nginx modules.
2. other than the level 4,9 modsecurity logging (initially passing) there is nothing in the Nginx info log, and the session is hanging ... UNTIL
3. UNTIL I break the session from the client side. When I break the session from the client side, it produces the log entry above, which appears to be a clue as to where NGINX was hanging in the session flow.
   3a) yes, we do have an external auth provider -- which works fine with POST/PATCH until modsecurity is injected into the control flow.

so I am still investigating the following: (a) why does the session hang, and where in the code is it hanging? (b) what happens if I disable external auth, (c) can I increase logging on the external auth to see how the session state differs if modsecurity is on/off (d) why does breaking the session from the client side produce any additional modsecurity logging AFTER session is closed?

[PaulCharlton]

continuing that investigation -- the entire control flow works when I disable our external auth by removing the auth related annotations from the k8s ingress resource.

combining that with the info from the log above -- Nginx appears to be sending the entire request body to the auth provider -- even on a POST/PATCH/PUT request ... and all our auth provider really cares about is a couple of the request headers.

[PaulCharlton]

@martinhsv so -- modsecurity flagging on a sub-request -- what is the control flow supposed to be? it definitely should not be hanging the session until client-side terminates.

[PaulCharlton]

more information:
disabling modsecurity at the "location" level does NOT change the behavior -- its presence, even when disabled at location, and even when all rules are disable, still produces the hang.

Looking at the issue I identified in the log above, I went looking for the configuration for the external auth, and contrary to the modsecurity log above, the external auth location does not pass the request body ...

                location = /_external-auth-L2NmLWFwcCgvfCQpKC4qKQ {                               
                        internal;                                                                 
                                                                                                  
                        # ngx_auth_request module overrides variables in the parent request,      
                        # therefore we have to explicitly set this variable again so that when the parent request
                        # resumes it has the correct value set for this variable so that Lua can pick backend correctly
                        set $proxy_upstream_name "cfk8s-cf-react-app-docker-80";                                       
                                                                                                                       
                        proxy_pass_request_body     off;                                                               
                        proxy_set_header            Content-Length "";                                                 
                        proxy_set_header            X-Forwarded-Proto "";                                              
                                                                                                                       
                        proxy_set_header            Host                    $host;                                     
                        proxy_set_header            X-Original-URL          $scheme://$http_host$request_uri;          
                        proxy_set_header            X-Original-Method       $request_method;                           
                        proxy_set_header            X-Sent-From             "nginx-ingress-controller";                
                        proxy_set_header            X-Real-IP               $remote_addr;                              
                                                                                                                       
                        proxy_set_header            X-Forwarded-For        $remote_addr;                               
                                                                                                                       
                        proxy_set_header            X-Auth-Request-Redirect $request_uri;                              
                                                                                                                       
                        proxy_buffering                         off;                                                   
                                                                                                                       
                        proxy_buffer_size                       8k;                                                    
                        proxy_buffers                           4 8k;                                                  
                        proxy_request_buffering                 on;                                          
                        proxy_http_version                      1.1;                                         
                                                                                                             
                        proxy_ssl_server_name       on;                                                      
                        proxy_pass_request_headers  on;                                                      
                                                                                                             
                        client_max_body_size        1m;                                                      
                                                                                                             
                        # Pass the extracted client certificate to the auth provider                   
                                                                                                       
                        set $target https://$host/oauth2/auth;                                        
                        proxy_pass $target;                                              
                } 

[PaulCharlton]

summary of what I have observed:

1. Nginx ingress with no modsecurity/external_auth. -> works
2. Nginx ingress with either modsecurity or external_auth. -> works
3. Nginx ingress with both modsecurity and external_auth. -> hangs session until broken by client
   3a) modsecurity logs "invalid request body" (Rule #920170) with external auth path upon session break by client

I am imagining that something in the control flow nginx->modsecurity->nginx->external_auth->nginx->proxied_server is corrupting session state at the boundary where modsecurity returns to nginx. [or doing a req_read, not realizing that req is already at EOF/(content-length) boundary]

[martinhsv]

Hi @PaulCharlton ,

Those are some good clues.

In some ways it sounds a bit like something akin to bitly/oauth2_proxy#442 . I.e. if no body is sent but a Content-Length header is present with a nonzero value, then a hang could be occurring due to a waiting-for-body state.

A few things you could try to further clarify what is happening:
A) Are you able to actually sniff what is being sent to the auth provider in that subrequest (auth endpoint logging, wireshark, etc. )? Can you confirm that it is actually receiving a Content-Length header with value 945? And a body is present also?
B) If A) is yes/yes, does the same thing happen when auth is on but ModSecurity is off entirely (i.e. a scenario that 'works' per number 2 in your previous posting). If this is yes also, then that suggests that at least part of the answer is not directly related to ModSecurity, but a problem that your settings within location = /_external-auth-L2NmLWFwcCgvfCQpKC4qKQ are not being respected.
C) With ModSecurity back on in general, can you modify the the behaviour within the auth location besides using 'modsecurity off'. E.g. can you use a modsecurity_rules directive to either i) use SecRuleEngine Off, or a SecRule or SecAction that performs a ctl:ruleRemoveById=920170. I.e. does that action appear in the debug log, and does the log entry for 920170 stop appearing?

Two additional possibilities to proceed:

• you mentioned that you are using ModSecurity v3.0.3. There is a newer version (v3.0.4) that has a fair number of fixes that you could try upgrading to. I haven't spotted any fixes that appear obviously applicable to this situation, but it could be worth a try.
• the ModSecurity-nginx recently has a newer version as (v1.0.1), so if you're still using v1.0.0, you could consider upgrading to see if that helps. Same caveat as in the previous bullet point

[PaulCharlton]

@martinhsv I will follow up on your ideas and let you know:
A) -- sniffing -- can do one better and inject a lightweight proxy server in between Nginx and the auth server and trace everything
B) the location policy for the auth server already has proxy_pass_request_body off; proxy_set_header Content-Length ""; so I would be surprised if there was any actual body or content-length sent out ... however, the truth may be stranger than fiction, so back to the sniffing.
C) re: versions, I am going to set up a standalone helm chart and k8s cluster to reproduce the issue with the minimal number of components and play with levers such as component version numbers. I will also do some visual code review/inspection before resorting to a debug build of Nginx / modsecurity-nginx / etc.