segmentation fault with SecStreamInBodyInspection and chunked encoding

[marcstern]

On Redhat/CentOS/Rocky 8, httpd crashes with a segmentation fault when enabling SecStreamInBodyInspection and sending a chunked encoded request.
This happens with the platform httpd & mod_security2. Same with latest mod_security2 (v2/master 2.9.4).

Note that the crash doesn't happen in CentOS 7, nor in Fedora. I guess there's a memory problem that corrupts the memory, but not always in a critical location.

This can be reproduced with this simple request:
curl -v -H "Transfer-Encoding: chunked" -d hello http://x/a

[marcstern]

Minimal config to reproduce the problem:

LoadModule mpm_worker_module modules/mod_mpm_worker.so
LoadModule systemd_module    modules/mod_systemd.so
LoadModule unixd_module      modules/mod_unixd.so

LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so

LoadModule proxy_module      modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

LoadModule unique_id_module  modules/mod_unique_id.so
LoadModule security2_module  modules/mod_security2.so

User apache
Group apache
CoreDumpDirectory "/tmp"
PidFile /var/run/httpd/httpd.pid
DocumentRoot /var/www/htdocs
ServerName x.y.z
LogLevel warn

SecRuleEngine On
SecAuditLogType Serial
SecAuditLog /var/log/httpd/security.log
SecAuditLogParts ABIFHZ
SecTmpDir    "/tmp/modsecurity/TmpDir"
SecUploadDir "/tmp/modsecurity/UploadDir"
SecDataDir   "/tmp/modsecurity/DataDir"
SecHashKey rand KeyOnly
SecResponseBodyAccess off
SecRequestBodyAccess On
# This causes the segmentation fault
SecStreamInBodyInspection On

Listen 80
<VirtualHost "*:80">
  ServerName x
  <Location />
    ProxyPreserveHost Off
    ProxyPass "http://127.0.0.1:80/notexist/"
  </Location>
</VirtualHost>

[marcstern]

Example of stack trace:

Thread 3 "httpd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffda59e700 (LWP 3202)]
ap_proxy_http_prefetch (url=0x7fffcc325dc0 "/a", uri=0x7fffcc325c80,
req=0x7fffcc325ce0) at mod_proxy_http.c:640
640 origin->keepalive = AP_CONN_CLOSE;
(gdb) bt
#0 ap_proxy_http_prefetch (url=0x7fffcc325dc0 "/a", uri=0x7fffcc325c80,
req=0x7fffcc325ce0) at mod_proxy_http.c:640
#1 proxy_http_handler (r=0x7fffcc011840, worker=, conf=,
url=0x7fffd8492a6e "http://127.0.0.1/a", proxyname=,
proxyport=) at mod_proxy_http.c:2076
#2 0x00007fffeeb84094 in proxy_run_scheme_handler (r=r@entry=0x7fffcc011840, worker=0x5555f8139920,
conf=conf@entry=0x5555f8139778, url=0x7fffd8492a6e "http://127.0.0.1/a",
proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at mod_proxy.c:3102
#3 0x00007fffeeb84e49 in proxy_handler (r=0x7fffcc011840) at mod_proxy.c:1266
#4 0x000055555559cad8 in ap_run_handler (r=r@entry=0x7fffcc011840) at config.c:170
#5 0x000055555559d096 in ap_invoke_handler (r=r@entry=0x7fffcc011840) at config.c:444
#6 0x00005555555b4243 in ap_process_async_request (r=r@entry=0x7fffcc011840) at http_request.c:453
#7 0x00005555555b43b2 in ap_process_request (r=r@entry=0x7fffcc011840) at http_request.c:488
#8 0x00005555555b05d5 in ap_process_http_sync_connection (c=0x7fffd4003c68) at http_core.c:210
#9 ap_process_http_connection (c=0x7fffd4003c68) at http_core.c:251
#10 0x00005555555a6ab8 in ap_run_process_connection (c=c@entry=0x7fffd4003c68) at connection.c:42
#11 0x00005555555a7038 in ap_process_connection (c=c@entry=0x7fffd4003c68, csd=csd@entry=0x7fffd4003a50) at connection.c:219
#12 0x00007ffff392b41f in process_socket (bucket_alloc=0x7fffcc000b88, my_thread_num=0, my_child_num=0,
sock=0x7fffd4003a50, p=0x7fffd40039d8, thd=0x5555566286a0) at worker.c:479
#13 worker_thread (thd=0x5555566286a0, dummy=) at worker.c:808
#14 0x00007ffff6b0a14a in start_thread (arg=) at pthread_create.c:479
#15 0x00007ffff6635dc3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[martinhsv]

Hi @marcstern ,

I rather suspect that this has nothing to do with ModSecurity.

For one thing, there are no ModSecurity functions in the call stack. That alone is only an indicator rather than a guarantee of any sort, however ...

There have been other reports of a comparable failure using Redhat 8 unrelated to ModSecurity, see for examples these items from a thread ...
https://www.mail-archive.com/dev@httpd.apache.org/msg75786.html
https://www.mail-archive.com/dev@httpd.apache.org/msg75809.html
https://www.mail-archive.com/dev@httpd.apache.org/msg75788.html

What do you think?

[marcstern]

Hi @martinhsv ,

"SecStreamInBodyInspection on" generates the core dump, "SecStreamInBodyInspection off" works correctly.
This really looks like a problem linked to ModSecurity (maybe the way it register as a filter?).

[martinhsv]

Hi @marcstern ,

If the on/off setting toggles the outcome reliably, then that does suggest some interaction between ModSecurity and the non-ModSecurity call stack that you provided in your third comment. However, that doesn't necessarily mean that the actual code bug is in ModSecurity code -- it could equally be in non-ModSecurity apache code that is engaging in some memory misuse.

Nevertheless, I will re-open this at least temporarily.

However, given that:

• it's not certain that the bug is in ModSecurity
• it is only seen in some O/S environments
• it involves a setting that I doubt is widely used
• even if someone were inclined to use that setting, it can be turned off and worked-around

... I doubt I can justify spending a whole lot of time on this item (especially in the near term).

But perhaps some member of the community will feel inclined to dig into this further.

[marcstern]

Solved with httpd 2.4.48+