Problems with blocking if response is using compression

[azurit]

Describe the bug

When blocking a request, which uses output compression (for example Content-Encoding: gzip), in phase 4 (so response headers are already created by application), blocking is NOT done properly because Content-Encoding: gzip will already be send to client so it awaits compressed response - on the client side, browsers will show error similar to this (this one is from Firefox):
Content Encoding Error

To Reproduce

Problem can be easily reproduced using this PHP script and blocking rule:

<?php
@ob_start("ob_gzHandler");
header('X-test-blocking: block-me');
echo 'test';
?>

SecRule &RESPONSE_HEADERS:X-test-blocking "!@eq 0" \
    "id:99999999999,\
    phase:4,\
    deny"

Expected behavior

modsecurity should remove or properly set the Content-Encoding header if request is blocked.

[zimmerle]

Hi @azurit,

Indeed, once the response headers hit the client, there is no much we can do to change the parameters that were already sent. The best thing to do in that scenario is to block the request abruptly as opposed to respecting the disruptive action informed on the rule.

At a given phase N, there is no way to know that at phase N+1 a rule will match, leading to a block. As phase occurs independently and in sequence, their inst much we can do. The possibility to hold the response headers till response body is processed is often undesired. It leads to significantly increased latency and reduces the server capability in terms of requests per second.

Usually, the out coming from a body inspection that "matches" is not to allow the response to hit the client; in a sense, it is workable. What are you aiming to accomplish?

[azurit]

Hi @zimmerle,

are headers already sent in phase 4? I thought they are still only buffered somewhere.

I'm trying to block web shells which are identified by searching for patterns in resposne body (see coreruleset/coreruleset#1962). When implementing this feature i come accross few web shells which are doing response body compression on purpose to hide themself - and it's working :( i can't catch them and block. So i implemented a solution to prevent this bypass (see coreruleset/coreruleset#1968) which is working too except the ugly error message displayed by web browser (Content Encoding Error).

[zimmerle]

Is it running on a nginx or Apache? which version of ModSecurity are you targeting? and What kind of deployment do you have?

[azurit]

Apache, modsec 2.9.3 (but both mentioned PRs should work also in version 3). What do you mean by 'deployment'?

Anyway, if headers are already sent, we cannot change them. What about to check if client is expecting compression (Content-Encoding header was sent with gzip or similar values) and, if yes, compress our response?

[zimmerle]

The behavior of sending the response headers before inspecting the response body may vary from server to server. The deployment is how ModSecurity is working on your environment (e.g., as a proxy on an independent server, embedded in the app webserver).

The thing to consider is what type of disruptive action you want to take that will lead to the response body's delivery or response body modification. Those are the disruptive actions that we can have -

action: deny (together with status)

It needs to send the response headers with the correct status code (403?). Will generate the error on the web browser.

action: redirect

It needs to send 302 on the response headers. Will generate the error on the web browser.

action: allow

Not applicable in the context

action: drop

Drop and consequently will generate the error that you have mentioned on the web browser.

action: pass

Not applicable in the context

---

In which condition you foresee a body modification that demands re-compression of the response body content?

As for the suggestion for decompressing the body for inspection, it sounds promising to me. I am wondering if we could have a transformation t:unzip. What do you think?

[azurit]

It's embedded.

I'm using 'deny' action.

In which condition you foresee a body modification that demands re-compression of the response body content?

If headers were already sent to client (i.e. we are in phase 4 using embedded modsecurity) AND they included header Content-Encoding with value gzip, the client is expecting gzipped response. If we decide to deny this request and do NOT send original response body, we need to compress our new response body so client is not confused.

As for the suggestion for decompressing the body for inspection, it sounds promising to me. I am wondering if we could have a transformation t:unzip. What do you think?

This would be sooo cool! I'm really missing this feature. As you can see in PR coreruleset/coreruleset#1968, i implemented something like t:unzip using Lua script but a transformation function will probably be much faster.

[zimmerle]

If headers were already sent to client (i.e. we are in phase 4 using embedded modsecurity) AND they included header Content-Encoding with value gzip, the client is expecting gzipped response. If we decide to deny this request and do NOT send original response body, we need to compress our new response body so client is not confused.

Once the request was disruptive blocked by a deny action, there is no content to be delivered to the user -- a possible malicious player in that case.

This would be sooo cool! I'm really missing this feature. As you can see in PR coreruleset/coreruleset#1968, i implemented something like t:unzip using Lua script but a transformation function will probably be much faster.

Indeed it should be faster and more generic (e.g., available to other variables). One of the ideas is to support Lua is to encourage prototyping; it fits the implementation that you have made 👍

Looking at your code coreruleset/coreruleset@bfd4d49#diff-ec25e9162441fd2a294691ac0f3fff833c80e893f436feebd1ea86df2b19ac5d

require("m")
require("zlib")
function main()
local f = zlib.inflate()
local response_body_uncompressed = f(m.getvar("RESPONSE_BODY", "none"))
m.setvar("tx.response_body_uncompressed", response_body_uncompressed)
return nil
end

The c++ variant shouldn't be that different. The annoying part is to understand how to add the Zlib dependency on ModSecurity. Once that is done, the transformation should be easy to implement, something like the example below (not tested, just a draft) -

void GZipDecompress::execute(const Transaction *t, const ModSecString &in, ModSecString &out) noexcept {
    zlibcomplete::GZipDecompressor decompressor;
    std::string ret = decompressor.decompress(std::string(in.c_str(), in.size()));
    out.assign(ret.c_str(), ret.size());
}

Here is an example of the transformation t:phpArgsNames added by @marshal09 few weeks ago - bab9723 . In this example we don't have an external dependency but gives you an idea of what needs to be made.

Up for the challenge? :) 🚀 🚀

[azurit]

Once the request was disruptive blocked by a deny action, there is no content to be delivered to the user -- a possible malicious player in that case.

This isn't always true, HTTP specification is not disallowing content with status 403 (which is the modsecurity default/recommended status for deny action). Even more, default Apache installation is sending also content:

HTTP/1.1 403 Forbidden
Date: Thu, 07 Jan 2021 15:21:13 GMT
Server: Apache
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>

This is why browsers are showing Content Encoding Error error - original content was compressed and was replaced by uncompressed one.

[zimmerle]

I may be misreading the circumstances. From what you have described, the request was blocked after the headers being sent to the client. In this scenario, the browser receives a code 200 with headers signaling the response body will be compressed. But, no response body at all - leading to the browser error. That is the expected scenario.

In this last example, you are showing a request for which the error code was 403.

Can you share a tcdump containing the entire HTTP transaction that leads your browser to the error?

[azurit]

Just run PHP script with modsec rule in my original message above, problem is 100% reproducable.

EDIT: Tested with Apache 2.4.

[azurit]

Here is a response from server, which was modified (blocked) by modsecurity, catched by curl (as you can see, content is in clear text, not gzipped but contains header Content-Encoding: gzip [because original content, removed by modsec, was gzipped]):

HTTP/1.1 403 Forbidden
Date: Thu, 07 Jan 2021 17:00:59 GMT
Server: Apache
X-test-blocking: block-me
Content-Encoding: gzip
Vary: Accept-Encoding
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>

This one is also catched by curl but was NOT blocked by modsecurity, so original, gzipped, content was sent (as you can see, content is really binary so curl is complainig - just an example how gzipped content is shown by curl and a proof that content from request above was not gzipped):

HTTP/1.1 200 OK
Date: Thu, 07 Jan 2021 17:01:26 GMT
Server: Apache
Content-Encoding: gzip
Vary: Accept-Encoding
Content-Length: 24
Content-Type: text/html; charset=UTF-8

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

[azurit]

Am i beeing more clear?