nginx + v3: high CPU usage on rule 200001

[theMiddleBlue]

I'm not sure if this issue was already reported. Anyway:

What reported here SpiderLabs/owasp-modsecurity-crs#1133 seems to be related to the rule 200001 and the requestBodyProcessor=JSON on each request with Content-Type: application/json.

With a long JSON string in the request body, the rule 200001 takes a very long time to process it, and the nginx worker CPU usage stuck at 100% until it finishes.

Logs and dumps
Using an 800 Kb JSON file as request body, and sent with content-type application/json the request took 83 seconds:

curl -s -H 'Content-Type: application/json' -d @test.json 'http://localhost' -w @curl-format.txt
host localhost upstream: 
    time_namelookup:  0.005550
       time_connect:  0.006125
    time_appconnect:  0.000000
   time_pretransfer:  0.011318
      time_redirect:  0.000000
 time_starttransfer:  0.014967
                    ----------
         time_total:  83.675212

Just replacing the content-type value with application/x-www-form-urlencoded, the request took just 2 seconds:

curl -s -H 'Content-Type: application/x-www-form-urlencoded' -d @test.json 'http://localhost' -w @curl-format.txt
host localhost upstream: 
    time_namelookup:  0.005729
       time_connect:  0.006164
    time_appconnect:  0.000000
   time_pretransfer:  0.006292
      time_redirect:  0.000000
 time_starttransfer:  0.012784
                    ----------
         time_total:  2.071719

To Reproduce

• download test.json
• create curl-format.txt as following

    time_namelookup:  %{time_namelookup}\n
       time_connect:  %{time_connect}\n
    time_appconnect:  %{time_appconnect}\n
   time_pretransfer:  %{time_pretransfer}\n
      time_redirect:  %{time_redirect}\n
 time_starttransfer:  %{time_starttransfer}\n
                    ----------\n
         time_total:  %{time_total}\n

• send request to nginx with content-type application/json and the test.json content as the request body:

$ curl -v -H 'Content-Type: application/json' -d @test.json 'http://localhost' -w @curl-format.txt

Server:

• ModSecurity - v3.0.3-45-g145f2f35 for Linux
• Modsecurity-nginx 1.0.0
• nginx 1.13.6

[victorhora]

Hey @theMiddleBlue,

We had a similar issue in the past that was brought on the Nginx-connector issue tracker: owasp-modsecurity/ModSecurity-nginx#89. This was found to be resolved a number of performance improvements added last year.

Can you re-run your tests without the CRS loaded? With CRS it was taking a significant amount to process, but with the basic configuration (which includes rule 200001 to activate JSON parsing), it was taking only 2.3 seconds here using that measurement model:

    time_namelookup:  0.000516
       time_connect:  0.002522
    time_appconnect:  0.000000
   time_pretransfer:  0.002735
      time_redirect:  0.000000
 time_starttransfer:  0.003224
                    ----------
         time_total:  2.324432

It's worth mentioning that this payload as is could trigger a 412 error due to the default value of SecRequestBodyNoFilesLimit.
But I still stumbled upon a weird behaviour (Empty reply from server!?) when running this against ModSec 2.9.x with the CRS. Without the CRS it runs fine.

Can you please let us know what you get if you run this against 2.9?

Maybe a good approach to identify any rule which might be using causing the slowdown is using the benchmark and SystemTap scripts and plot in a flame graph like this:

[jleproust]

I tested this with modsecurity 3.0.3, nginx 1.15.8, modsecurity-nginx 1.0.0 and crs 3.1.0.

Actually the problem is not the JSON parser itself. The real issue is that a relatively small JSON payload can generate thousands of arguments, which fill the ARGS and ARGS_NAMES collections, and then the OWASP CRS rules spend a lot of matching all of these values. This is why disabling the JSON processing makes modsecurity much faster. There is no single root cause, just the total complexity. (I can share the systemtap output to back this).

Unfortunately it makes the JSON processor completely unusable with the CRS. I see no other solution than completely disabling JSON processing, because my application relies on large arbitrary JSON requests, and the alternative is a huge DoS vulnerability in the http server.

[zimmerle]

Hi, @theMiddleBlue

Thank you for raise this up. Considering the comments from @victorhora and @jleproust I think it is safe to close that issue. If you think otherwise, please open it again within your concerns.

[jleproust]

Well, it's still an issue and the only usable fix for now is my PR #2060, right? Maybe we should keep it open as a reference until we merge it or find a better solution?

[goweiwen]

Thanks for the work on this issue @jleproust!

We had to stop using ModSecurity due to request timeouts caused by this. Is there an estimate when #2060 will be merged and released?

[jleproust]

@goweiwen You're welcome! I have debian 9 and ubuntu 16.04 packages if you want to test.