ModSecurity segfault

[adamjacobmuller]

Hello,

We are running ModSecurity@1edd3570e11e9bb2b6d86b249232b24917a4b0ac and ModSecurity-nginx@abbf2c47f6f3205484a1a9db618e067dce213b89 with nginx 1.13.1 and seeing the following segfault:

(gdb) bt
#0  0x00007fc6a040945b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007fc6a1c9934d in modsecurity::Rule::getFinalVars(modsecurity::Transaction*) () from /usr/lib/x86_64-linux-gnu/libmodsecurity.so.3
#2  0x00007fc6a1c9a4d2 in modsecurity::Rule::evaluate(modsecurity::Transaction*, std::shared_ptr<modsecurity::RuleMessage>) () from /usr/lib/x86_64-linux-gnu/libmodsecurity.so.3
#3  0x00007fc6a1c8d006 in modsecurity::Rules::evaluate(int, modsecurity::Transaction*) () from /usr/lib/x86_64-linux-gnu/libmodsecurity.so.3
#4  0x00007fc6a1c7dfb5 in modsecurity::Transaction::processLogging() () from /usr/lib/x86_64-linux-gnu/libmodsecurity.so.3
#5  0x0000000000523398 in ngx_http_modsecurity_log_handler (r=0x1bea9f310) at ../mod_security/src/ngx_http_modsecurity_log.c:72
#6  0x00000000004543f4 in ngx_http_log_request (r=r@entry=0x1ba5e3bd0) at src/http/ngx_http_request.c:3554
#7  0x0000000000455ae7 in ngx_http_free_request (r=0x1ba5e3bd0, rc=rc@entry=0) at src/http/ngx_http_request.c:3501
#8  0x000000000047fcba in ngx_http_v2_close_stream (stream=0x1ba5e4890, rc=rc@entry=0) at src/http/v2/ngx_http_v2.c:4057

This looks like an issue/bug in ModSecurity directly (and not the nginx connector) so filing the bug here, let me know if that's incorrect.

[victorhora]

Hi @adamjacobmuller,

Thanks for the bug report. Just compiling these versions with default modsecurity.conf and starting Nginx leads to this crash? Could you please described in more details the steps your performed that lead to the segfault?

Thanks.

[adamjacobmuller]

Hi @victorhora,

The segfault occurs during the request processing, in the logging phase. The request is still delivered to the client. At high request/segfault rates, this will, of course, cause lots of client requests to fail if they are served by a worker that is log/panicking for another request.

I'm attaching a copy of the mod_security ruleset in use, and I can provide an nginx.conf mockup that will demonstrate the issue as well if you need, but, basically we're proxying requests to a backend and the response bodies are coming from that backend. SSL and HTTP/2 are in-use but the issue is not isolated to them.
rules.zip

Thank you,
-Adam

[wergoth]

Hi @adamjacobmuller,
could you tell me about nginx configuration options you used before nginx build? Thanks.

[adamjacobmuller]

Hi @wergoth,

Do you mean nginx ./configure options or for ModSecurity itself, for the latter, its just basic ./configure && make && make install

For nginx things are a bit more complex, but, not too much:

configure arguments: --prefix=/usr --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --with-cc-opt=-I/usr/include --with-ld-opt=-L/usr/lib --with-ipv6 --with-pcre --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_limit_conn_module --without-http_memcached_module --with-http_geoip_module --with-http_stub_status_module --with-http_realip_module --with-http_ssl_module --with-http_auth_request_module --user=nobody --group=nobody --with-http_mp4_module --with-http_flv_module --add-module=../lua-nginx-module --add-module=../ngx_devel_kit --add-module=../brotli --with-threads --with-file-aio --with-http_v2_module --add-module=../mod_security

[adamjacobmuller]

@victorhora and @wergoth, if you're having issues reproducing the bug, I'm happy to give you access to a test system that I have setup to reproduce it.

The issue definitely happens much more with parallel requests. It's very difficult to reproduce with something like curl, but, trivial with a browser. Drop me an email so the credentials aren't in a public github issue, adam@belugacdn.com and I'll setup access for you.

[wergoth]

Hi @adamjacobmuller,
We reproduced this issue and got the next results. Possibly it would help:
##########################################################

REQUEST

##########################################################
curl 'http://10.8.4.213:8081/..%5c..%5c..%5c..%5c..%5c..%5c..%5c..%5c..%5c..%5c/windows/win.ini'

expected: 404 Not Found
got: nginx segfault

##########################################################

DEBUGGER OUTPUT

##########################################################

(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
igdb)
0x00007f2e60e67390 in __memcpy_ssse3 () from /lib64/libc.so.6

(gdb) where
#0 0x00007f2e60e67390 in __memcpy_ssse3 () from /lib64/libc.so.6
#1 0x00007f2e5fed48e0 in std::string::_Rep::_M_clone(std::allocator const&, unsigned long) () from /lib64/libstdc++.so.6
#2 0x00007f2e5fed4f8c in std::basic_string<char, std::char_traits, std::allocator >::basic_string(std::string const&) ()
from /lib64/libstdc++.so.6
#3 0x00007f2e622d7a76 in modsecurity::Rule::getFinalVars (this=0x31217a0, trans=0x39b5dd0) at rule.cc:572
#4 0x00007f2e622d8f02 in modsecurity::Rule::evaluate (this=0x31217a0, trans=0x39b5dd0,
ruleMessage=std::shared_ptr (count 2, weak 0) 0x3b57b20) at rule.cc:706
#5 0x00007f2e622d9705 in modsecurity::Rule::evaluate (this=0x3121650, trans=0x39b5dd0,
ruleMessage=std::shared_ptr (count 2, weak 0) 0x3b57b20) at rule.cc:758
#6 0x00007f2e622ccb40 in modsecurity::Rules::evaluate (this=0x16afed0, phase=3, transaction=0x39b5dd0) at rules.cc:212
#7 0x00007f2e622b5e9b in modsecurity::Transaction::processRequestBody (this=0x39b5dd0) at transaction.cc:799
#8 0x00007f2e622bc3a1 in modsecurity::msc_process_request_body (transaction=0x39b5dd0) at transaction.cc:1795
#9 0x00000000004aacfb in ngx_http_modsecurity_pre_access_handler (r=0x39ac500)
at /root/ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c:199
#10 0x000000000044e0ba in ngx_http_core_generic_phase (r=0x39ac500, ph=0x390e2f0) at src/http/ngx_http_core_module.c:882
#11 0x0000000000449eb3 in ngx_http_core_run_phases (r=r@entry=0x39ac500) at src/http/ngx_http_core_module.c:860
#12 0x0000000000449fc1 in ngx_http_handler (r=r@entry=0x39ac500) at src/http/ngx_http_core_module.c:843
#13 0x00000000004556d1 in ngx_http_process_request (r=r@entry=0x39ac500) at src/http/ngx_http_request.c:1940
#14 0x0000000000455aac in ngx_http_process_request_headers (rev=rev@entry=0x3911380) at src/http/ngx_http_request.c:1367
#15 0x0000000000455dc2 in ngx_http_process_request_line (rev=rev@entry=0x3911380) at src/http/ngx_http_request.c:1040
#16 0x0000000000456201 in ngx_http_wait_request_handler (rev=0x3911380) at src/http/ngx_http_request.c:506
#17 0x000000000043d430 in ngx_epoll_process_events (cycle=0x167c460, timer=, flags=)
at src/event/modules/ngx_epoll_module.c:902
#18 0x00000000004330ec in ngx_process_events_and_timers (cycle=cycle@entry=0x167c460) at src/event/ngx_event.c:242
#19 0x000000000043b40c in ngx_worker_process_cycle (cycle=0x167c460, data=) at src/os/unix/ngx_process_cycle.c:749
#20 0x0000000000439ac1 in ngx_spawn_process (cycle=cycle@entry=0x167c460, proc=0x43b379 <ngx_worker_process_cycle>, data=0x0,
name=0x4f161e "worker process", respawn=respawn@entry=0) at src/os/unix/ngx_process.c:198
#21 0x000000000043c10e in ngx_reap_children (cycle=0x167c460) at src/os/unix/ngx_process_cycle.c:621
#22 ngx_master_process_cycle (cycle=cycle@entry=0x167c460) at src/os/unix/ngx_process_cycle.c:174
#23 0x0000000000413c5e in main (argc=, argv=) at src/core/nginx.c:375

#3 0x00007f2e622d7a76 in modsecurity::Rule::getFinalVars (this=0x31217a0, trans=0x39b5dd0) at rule.cc:572
572 new std::string(*v->m_key),

(gdb) print *this
$1 = {m_accuracy = 0, m_actionsConf = std::vector of length 0, capacity 0, m_actionsRuntimePos = std::vector of length 0, capacity 0,
m_actionsRuntimePre = std::vector of length 0, capacity 0, m_chained = false, m_chainedRule = 0x0,
m_fileName = "/usr/local/cwaf/rules/rule-33062-Apps_WPPlugin.conf", m_lineNumber = 190, m_logData = "", m_marker = "",
m_maturity = 0, m_op = 0x3121740, m_phase = 3, m_rev = "", m_ruleId = 0, m_secMarker = false, m_variables = 0x3120f80, m_ver = "",
m_unconditional = false, m_referenceCount = 1}

(gdb) l
567 }
568 continue;
569 }
570
571 std::unique_ptrcollection::Variable var(new collection::Variable(
572 new std::string(*v->m_key),
573 new std::string(*v->m_value)));
574 var->m_dynamic_value = true;
575 var->m_dynamic_key = true;
576 for (auto &i : v->m_orign) {

(gdb) print *v
$2 = {m_key = 0x7ffce6153a70, m_value = 0x3c349d0, m_dynamic_value = false, m_dynamic_key = false, m_dynamic = true,
m_orign = empty std::list}

(gdb) printf "<%-64s>\n", (char*)(*v->m_key)
<6aaa80360ef1cc3640113a2aa4514aad::SESSION:wp_add >
##############################################################

NOTE: here *v->m_key seems not to be \0 terminated

##############################################################
(gdb) printf "<%-64s>\n", (char*)(*v->m_value)
<1' >

(gdb) up
#4 0x00007f2e622d8f02 in modsecurity::Rule::evaluate (this=0x31217a0, trans=0x39b5dd0,
ruleMessage=std::shared_ptr (count 2, weak 0) 0x3b57b20) at rule.cc:706
706 finalVars = getFinalVars(trans);

(gdb) print *this
$4 = {m_accuracy = 0, m_actionsConf = std::vector of length 0, capacity 0, m_actionsRuntimePos = std::vector of length 0, capacity 0,
m_actionsRuntimePre = std::vector of length 0, capacity 0, m_chained = false, m_chainedRule = 0x0,
m_fileName = "/usr/local/cwaf/rules/rule-33062-Apps_WPPlugin.conf", m_lineNumber = 190, m_logData = "", m_marker = "",
m_maturity = 0, m_op = 0x3121740, m_phase = 3, m_rev = "", m_ruleId = 0, m_secMarker = false, m_variables = 0x3120f80, m_ver = "",
m_unconditional = false, m_referenceCount = 1}
(gdb)

(gdb) print (Variable)this->m_variables
$8 = {_vptr.Variable = 0x3120ff0,
m_name = " \026\022\003\000\000\000\000 \000\000\000\000\000\000\000\061\000\000\000\000\000\000\000\060\377^b.\177\000\000\001\000\016a\001\000\000\000\330\017\022\003\000\000\000\000\370\343\021.\177\000\000\001\000\000\000H\017\267C1\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005", '\000' <repeats 15 times>, "nolog\000\000\000\004\000\000\005L\213t$1\000\000\000\000\000\000\000\260\355^b.\177\000\000\000\000\000\000\002\000\000\000\270\020\022\003\000\000\000\000\370\343\021.\177\000\000\001\000\000\000\000\000\003L1\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\004", '\000' <repeats 13 times>, "\aHpass\000\000\000\000"...,
m_collectionName = " \026\022\003\000\000\000\000 \000\000\000\000\000\000\000\061\000\000\000\000\000\000\000\060\377^b.\177\000\000\001\000\016a\001\000\000\000\330\017\022\003\000\000\000\000\370\343\021.\177\000\000\001\000\000\000H\017\267C1\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005", '\000' <repeats 15 times>, "nolog\000\000\000\004\000\000\005L\213t$1\000\000\000\000\000\000\000\260\355^b.\177\000\000\000\000\000\000\002\000\000\000\270\020\022\003\000\000\000\000\370\343\021.\177\000\000\001\000\000\000\000\000\003L1\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\004", '\000' <repeats 13 times>, "\aHpass\000\000\000\000"..., m_type = (modsecurity::Variables::Variable::MultipleMatches | unknown: 32), m_kind = modsecurity::Variables::Variable::DirectVariable,
m_isExclusion = 224, m_isCount = 8}

(gdb) print *this->m_op
$9 = {_vptr.Operator = 0x7f2e625f0fd0 <vtable for modsecurity::operators::Eq+16>, m_match_message = "", m_negation = false,
m_op = "Eq", m_param = "1"}

(gdb) print trans
$11 = {modsecurity::TransactionAnchoredVariables = {m_variableArgsNames = {m_transaction = 0x39b5dd0, m_var = 0x39c1850,
m_offset = 76, m_name = "ARGS_NAMES", m_value = "page action id"}, m_variableArgGetNames = {m_transaction = 0x39b5dd0,
m_var = 0x3964380, m_offset = 76, m_name = "ARGS_GET_NAMES", m_value = "page action id"}, m_variableArgPostNames = {
m_transaction = 0x39b5dd0, m_var = 0x3a18bc0, m_offset = 0, m_name = "ARGS_POST_NAMES", m_value = ""},
m_variableRequestHeadersNames = {m_transaction = 0x39b5dd0, m_var = 0x3a5f1e0, m_offset = 148, m_name = "REQUEST_HEADERS_NAMES",
m_value = "User-Agent Host Accept Cookie"}, m_variableResponseContentType = {m_transaction = 0x39b5dd0, m_var = 0x3ae37e0,
m_offset = 0, m_name = "RESPONSE_CONTENT_TYPE", m_value = ""}, m_variableResponseHeadersNames = {m_transaction = 0x39b5dd0,
m_var = 0x3ae3810, m_offset = 0, m_name = "RESPONSE_HEADERS_NAMES", m_value = ""}, m_variableARGScombinedSize = {
m_transaction = 0x39b5dd0, m_var = 0x3a6a4a0, m_offset = 79, m_name = "ARGS_COMBINED_SIZE", m_value = "45.000000"},
m_variableAuthType = {m_transaction = 0x39b5dd0, m_var = 0x3a25f80, m_offset = 0, m_name = "AUTH_TYPE", m_value = ""},
m_variableFilesCombinedSize = {m_transaction = 0x39b5dd0, m_var = 0x3af2150, m_offset = 0, m_name = "FILES_COMBINED_SIZE",
m_value = ""}, m_variableFullRequest = {m_transaction = 0x39b5dd0, m_var = 0x3aef060, m_offset = 656, m_name = "FULL_REQUEST",
m_value = "User-Agent: curl/7.29.0\nCookie: wordpress_6aaa80360ef1cc3640113a2aa4514aad=admin%7C1479613486%7Cl24GVlSh2KxHIDmtsbygMHW22CWFh7SUEuwQ0jKnXdH%7C85fafcb201f503724f94915ae9df1bd4762d307da6dbb9438fdfdcdf22"...}, m_variableFullRequestLength = {
m_transaction = 0x39b5dd0, m_var = 0x3af2180, m_offset = 656, m_name = "FULL_REQUEST_LENGTH", m_value = "568"},
m_variableInboundDataError = {m_transaction = 0x39b5dd0, m_var = 0x3a24ee0, m_offset = 0, m_name = "INBOUND_DATA_ERROR",
m_value = "0"}, m_variableMatchedVar = {m_transaction = 0x39b5dd0, m_var = 0x3976450, m_offset = 656, m_name = "MATCHED_VAR",
m_value = "/wp430/wp-admin/admin.php"}, m_variableMatchedVarName = {m_transaction = 0x39b5dd0, m_var = 0x3976480,
m_offset = 656, m_name = "MATCHED_VAR_NAME", m_value = "REQUEST_FILENAME"}, m_variableMultipartCrlfLFLines = {
m_transaction = 0x39b5dd0, m_var = 0x3a447c0, m_offset = 0, m_name = "MULTIPART_CRLF_LF_LINES", m_value = ""},
m_variableMultipartDataAfter = {m_transaction = 0x39b5dd0, m_var = 0x39c20d0, m_offset = 0, m_name = "MULTIPART_DATA_AFTER",
m_value = ""}, m_variableMultipartFileLimitExceeded = {m_transaction = 0x39b5dd0, m_var = 0x39c2140, m_offset = 0,
m_name = "MULTIPART_FILE_LIMIT_EXCEEDED", m_value = ""}, m_variableMultipartStrictError = {m_transaction = 0x39b5dd0,
m_var = 0x39c2170, m_offset = 0, m_name = "MULTIPART_STRICT_ERROR", m_value = ""}, m_variableMultipartHeaderFolding = {
m_transaction = 0x39b5dd0, m_var = 0x39a6c60, m_offset = 0, m_name = "MULTIPART_HEADER_FOLDING", m_value = ""},
m_variableMultipartInvalidQuoting = {m_transaction = 0x39b5dd0, m_var = 0x39549d0, m_offset = 0,
m_name = "MULTIPART_INVALID_QUOTING", m_value = ""}, m_variableMultipartInvalidHeaderFolding = {m_transaction = 0x39b5dd0,
m_var = 0x3bcd7d0, m_offset = 0, m_name = "MULTIPART_INVALID_HEADER_FOLDING", m_value = ""},
m_variableMultipartUnmatchedBoundary = {m_transaction = 0x39b5dd0, m_var = 0x3bcd840, m_offset = 0,
m_name = "MULTIPART_UNMATCHED_BOUNDARY", m_value = ""}, m_variableOutboundDataError = {m_transaction = 0x39b5dd0,
m_var = 0x3a0ad90, m_offset = 0, m_name = "OUTBOUND_DATA_ERROR", m_value = ""}, m_variablePathInfo = {m_transaction = 0x39b5dd0,
m_var = 0x3a0ae20, m_offset = 4, m_name = "PATH_INFO", m_value = "/wp430/wp-admin/admin.php"}, m_variableQueryString = {
m_transaction = 0x39b5dd0, m_var = 0x3c34af0, m_offset = 30, m_name = "QUERY_STRING",
m_value = "page=heat-trackr_abtest_admin.php&action=edit&id=1"}, m_variableRemoteAddr = {m_transaction = 0x39b5dd0,
m_var = 0x3c34b50, m_offset = 0, m_name = "REMOTE_ADDR", m_value = "10.100.72.104"}, m_variableRemoteHost = {
m_transaction = 0x39b5dd0, m_var = 0x3c34bb0, m_offset = 0, m_name = "REMOTE_HOST", m_value = "10.100.72.104"},
m_variableRemotePort = {m_transaction = 0x39b5dd0, m_var = 0x3a14050, m_offset = 0, m_name = "REMOTE_PORT", m_value = "62543"},
m_variableReqbodyError = {m_transaction = 0x39b5dd0, m_var = 0x3a140b0, m_offset = 656, m_name = "REQBODY_ERROR", m_value = "0"},
m_variableReqbodyErrorMsg = {m_transaction = 0x39b5dd0, m_var = 0x3a0adc0, m_offset = 0, m_name = "REQBODY_ERROR_MSG",
m_value = ""}, m_variableReqbodyProcessorError = {m_transaction = 0x39b5dd0, m_var = 0x3a26050, m_offset = 656,
m_name = "REQBODY_PROCESSOR_ERROR", m_value = "0"}, m_variableReqbodyProcessorErrorMsg = {m_transaction = 0x39b5dd0,
m_var = 0x3a260c0, m_offset = 0, m_name = "REQBODY_PROCESSOR_ERROR_MSG", m_value = ""}, m_variableReqbodyProcessor = {
m_transaction = 0x39b5dd0, m_var = 0x3a260f0, m_offset = 0, m_name = "REQBODY_PROCESSOR", m_value = ""},
m_variableRequestBasename = {m_transaction = 0x39b5dd0, m_var = 0x3947d60, m_offset = 20, m_name = "REQUEST_BASENAME",
m_value = "admin.php"}, m_variableRequestBody = {m_transaction = 0x39b5dd0, m_var = 0x3ae3870, m_offset = 0,
m_name = "REQUEST_BODY", m_value = ""}, m_variableRequestBodyLength = {m_transaction = 0x39b5dd0, m_var = 0x3947d90,
m_offset = 0, m_name = "REQUEST_BODY_LENGTH", m_value = ""}, m_variableRequestFilename = {m_transaction = 0x39b5dd0,
m_var = 0x3ae3920, m_offset = 4, m_name = "REQUEST_FILENAME", m_value = "/wp430/wp-admin/admin.php"}, m_variableRequestLine = {
m_transaction = 0x39b5dd0, m_var = 0x39c1990, m_offset = 0, m_name = "REQUEST_LINE",
m_value = "GET /wp430/wp-admin/admin.php?page=heat-trackr_abtest_admin.php&action=edit&id=1 HTTP/1.1"},
m_variableRequestMethod = {m_transaction = 0x39b5dd0, m_var = 0x39c19f0, m_offset = 0, m_name = "REQUEST_METHOD",
m_value = "GET"}, m_variableRequestProtocol = {m_transaction = 0x39b5dd0, m_var = 0x3ae3950, m_offset = 81,
m_name = "REQUEST_PROTOCOL", m_value = "HTTP/1.1"}, m_variableRequestURI = {m_transaction = 0x39b5dd0, m_var = 0x3af8360,
m_offset = 4, m_name = "REQUEST_URI", m_value = "/wp430/wp-admin/admin.php?page=heat-trackr_abtest_admin.php&action=edit&id=1"},
m_variableRequestURIRaw = {m_transaction = 0x39b5dd0, m_var = 0x3af83c0, m_offset = 4, m_name = "REQUEST_URI_RAW",
m_value = "/wp430/wp-admin/admin.php?page=heat-trackr_abtest_admin.php&action=edit&id=1"}, m_variableResource = {
m_transaction = 0x39b5dd0, m_var = 0x3af8420, m_offset = 0, m_name = "RESOURCE", m_value = ""}, m_variableResponseBody = {
m_transaction = 0x39b5dd0, m_var = 0x394da00, m_offset = 0, m_name = "RESPONSE_BODY", m_value = ""},
m_variableResponseContentLength = {m_transaction = 0x39b5dd0, m_var = 0x39c1a60, m_offset = 0, m_name = "RESPONSE_CONTENT_LENGTH",
m_value = ""}, m_variableResponseProtocol = {m_transaction = 0x39b5dd0, m_var = 0x394dab0, m_offset = 0,
m_name = "RESPONSE_PROTOCOL", m_value = ""}, m_variableResponseStatus = {m_transaction = 0x39b5dd0, m_var = 0x3aef3f0,
m_offset = 0, m_name = "RESPONSE_STATUS", m_value = ""}, m_variableServerAddr = {m_transaction = 0x39b5dd0, m_var = 0x3aef450,
m_offset = 0, m_name = "SERVER_ADDR", m_value = "10.100.72.104"}, m_variableServerName = {m_transaction = 0x39b5dd0,
m_var = 0x3aef4b0, m_offset = 120, m_name = "SERVER_NAME", m_value = "10.8.4.213"}, m_variableServerPort = {
m_transaction = 0x39b5dd0, m_var = 0x1684270, m_offset = 0, m_name = "SERVER_PORT", m_value = "8081"}, m_variableSessionID = {
m_transaction = 0x39b5dd0, m_var = 0x16842d0, m_offset = 656, m_name = "SESSIONID",
m_value = "6aaa80360ef1cc3640113a2aa4514aad"}, m_variableUniqueID = {m_transaction = 0x39b5dd0, m_var = 0x1684330, m_offset = 0,
m_name = "UNIQUE_ID", m_value = "15009866385.337469"}, m_variableUrlEncodedError = {m_transaction = 0x39b5dd0,
m_var = 0x394dae0, m_offset = 0, m_name = "URLENCODED_ERROR", m_value = "0"}, m_variableUserID = {m_transaction = 0x39b5dd0,
m_var = 0x399c360, m_offset = 0, m_name = "USERID", m_value = ""},
m_variableArgs = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 3 elements = {["id"] = 0x39b77c0,
["action"] = 0x39b75f0, ["page"] = 0x39a5390}, m_transaction = 0x39b5dd0, m_name = "ARGS"},
m_variableArgsGet = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 3 elements = {
["id"] = 0x3975e60, ["action"] = 0x39b7690, ["page"] = 0x39a5430}, m_transaction = 0x39b5dd0, m_name = "ARGS_GET"},
m_variableArgsPost = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "ARGS_POST"},
m_variableFilesSizes = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "FILES_SIZES"},
m_variableFilesNames = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "FILES_NAMES"},
m_variableFilesTmpContent = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "FILES_TMP_CONTENT"},
m_variableMultiPartFileName = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "MULTIPART_FILENAME"},
m_variableMultiPartName = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "MULTIPART_NAME"},
m_variableMatchedVarsNames = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 1 elements = {
["REQUEST_FILENAME"] = 0x3ad6dd0}, m_transaction = 0x39b5dd0, m_name = "MATCHED_VARS_NAMES"},
m_variableMatchedVars = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 1 elements = {
["REQUEST_FILENAME"] = 0x3ad6e30}, m_transaction = 0x39b5dd0, m_name = "MATCHED_VARS"},
m_variableFiles = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "FILES"},
m_variableRequestCookies = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 6 elements = {
["PHPSESSID"] = 0x39a58b0, ["wp-settings-time-1"] = 0x39bf140, ["wp-settings-1"] = 0x39bef20,
["wordpress_logged_in_6aaa80360ef1cc3640113a2aa4514aad"] = 0x3a312e0, ["wordpress_test_cookie"] = 0x3947260,
["wordpress_6aaa80360ef1cc3640113a2aa4514aad"] = 0x3a27350}, m_transaction = 0x39b5dd0, m_name = "REQUEST_COOKIES"},
m_variableRequestHeaders = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 4 elements = {
["Accept"] = 0x3976020, ["Host"] = 0x39a5490, ["Cookie"] = 0x3aef600, ["User-Agent"] = 0x3942770}, m_transaction = 0x39b5dd0,
m_name = "REQUEST_HEADERS"},
m_variableResponseHeaders = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "RESPONSE_HEADERS"},
m_variableGeo = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "GEO"},
m_variableRequestCookiesNames = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 6 elements = {
["PHPSESSID"] = 0x39a5800, ["wp-settings-time-1"] = 0x39bf000, ["wp-settings-1"] = 0x3a31480,
["wordpress_logged_in_6aaa80360ef1cc3640113a2aa4514aad"] = 0x3a31370, ["wordpress_test_cookie"] = 0x3a273b0,
["wordpress_6aaa80360ef1cc3640113a2aa4514aad"] = 0x3aef6a0}, m_transaction = 0x39b5dd0, m_name = "REQUEST_COOKIES_NAMES"},
m_variableRule = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 1868 elements = {
["logdata"] = 0x3b64940, ["logdata"] = 0x3b63160, ["logdata"] = 0x3c6dc20, ["logdata"] = 0x3bdfd80, ["logdata"] = 0x3b60e60,
["logdata"] = 0x3b60d80, ["logdata"] = 0x3b601d0, ["logdata"] = 0x3b5fa80, ["logdata"] = 0x3b5f1b0, ["logdata"] = 0x3b5ea30,
["logdata"] = 0x3bd8010, ["logdata"] = 0x3c37320, ["logdata"] = 0x3c369f0, ["logdata"] = 0x3c36230, ["logdata"] = 0x3c35c60,
["logdata"] = 0x3c352d0, ["logdata"] = 0x39bd660, ["logdata"] = 0x39bdc10, ["logdata"] = 0x39bceb0, ["logdata"] = 0x39bc700,
["logdata"] = 0x39bbe80, ["logdata"] = 0x39bb920, ["logdata"] = 0x3a3f3e0, ["logdata"] = 0x3a3ec90, ["logdata"] = 0x3a3e550,
["logdata"] = 0x3a3dcb0, ["logdata"] = 0x39a0160, ["logdata"] = 0x39a0860, ["logdata"] = 0x399f8c0, ["logdata"] = 0x399f020,
["logdata"] = 0x39a8cf0, ["logdata"] = 0x399ea30, ["logdata"] = 0x39a8450, ["logdata"] = 0x39a7c80, ["logdata"] = 0x39a74b0,
["logdata"] = 0x3af1e20, ["logdata"] = 0x3af20f0, ["logdata"] = 0x3af0e80, ["logdata"] = 0x3af05e0, ["logdata"] = 0x396f6b0,
["logdata"] = 0x3af0010, ["logdata"] = 0x396ee10, ["logdata"] = 0x396e570, ["logdata"] = 0x3aed4a0,
["logdata"] = 0x396dfc0...}, m_transaction = 0x39b5dd0, m_name = "RULE"},
m_variableFilesTmpNames = {<std::unordered_multimap<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Variable*, modsecurity::MyHash, modsecurity::MyEqual, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Variable*> > >> = std::unordered_multimap with 0 elements,
m_transaction = 0x39b5dd0, m_name = "FILES_TMPNAMES"}, m_variableOffset = 656}, m_creationTimeStamp = 45,
m_clientIpAddress = 0x39ad478 "10.100.72.104", m_httpVersion = 0x503072 "1.1", m_serverIpAddress = 0x7f2e62e5f818 "10.100.72.104",
m_uri = 0x1698310 "/wp430/wp-admin/admin.php?page=heat-trackr_abtest_admin.php&action=edit&id=1",
m_uri_no_query_string_decoded = "/wp430/wp-admin/admin.php", m_ARGScombinedSizeDouble = 45, m_clientPort = 62543,
m_highestSeverityAction = 2, m_httpCodeReturned = 200, m_serverPort = 8081, m_ms = 0x1689a10,
m_requestBodyType = modsecurity::Transaction::UnknownFormat, m_requestBodyProcessor = modsecurity::Transaction::UnknownFormat,
Python Exception <type 'exceptions.ValueError'> Cannot find type std::list<int, std::allocator >::_Node:
Python Exception <type 'exceptions.ValueError'> Cannot find type std::list<std::pair<std::basic_string<char, std::char_traits, std::allocator >, std::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator >, std::basic_string<char, std::char_traits, std::allocator > > > >::_Node:
m_rules = 0x16afed0, m_ruleRemoveById = empty std::list, m_ruleRemoveTargetByTag = empty std::list,
Python Exception <type 'exceptions.ValueError'> Cannot find type std::list<std::pair<int, std::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<int, std::basic_string<char, std::char_traits, std::allocator > > > >::_Node:
m_ruleRemoveTargetById = empty std::list, Python Exception <type 'exceptions.ValueError'> Cannot find type std::list<std::pair<int, std::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<int, std::basic_string<char, std::char_traits, std::allocator > > > >::_Node:
m_requestBodyAccess = 2, m_auditLogModifier = empty std::list,
m_rulesMessages = empty std::list, m_requestBody = , m_responseBody = ,
m_id = "15009866385.337469", m_marker = "", m_skip_next = 0, m_allowType = ,
m_uri_decoded = "/wp430/wp-admin/admin.php?page=heat-trackr_abtest_admin.php&action=edit&id=1",
m_actions = std::vector of length 0, capacity 0, m_it = {status = 200, pause = 0, url = 0x0, log = 0x0, disruptive = 0},
m_timeStamp = 1500986638,
m_collections = {<std::unordered_map<std::basic_string<char, std::char_traits, std::allocator >, modsecurity::collection::Collection*, std::hash<std::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, modsecurity::collection::Collection*> > >> = std::unordered_map with 1 elements = {["TX"] = 0x394c360},
m_transient = 0x394c290, m_global_collection_key = "global", m_ip_collection_key = "10.100.72.104_curl/7.29.0'",
m_session_collection_key = "6aaa80360ef1cc3640113a2aa4514aad", m_user_collection_key = "", m_resource_collection_key = "",
m_global_collection = 0x1689a60, m_ip_collection = 0x168dc20, m_session_collection = 0x168fcd0, m_user_collection = 0x1691d80,
m_resource_collection = 0x168bb70}, m_matched = empty std::list, m_xml = 0x3a130a0, m_json = 0x3bd1990, m_variableDuration = "",
m_variableEnvs = std::map with 0 elements, m_variableHighestSeverityAction = "", m_variableRemoteUser = "", m_variableTime = "",
m_variableTimeDay = "", m_variableTimeEpoch = "", m_variableTimeHour = "", m_variableTimeMin = "", m_variableTimeSec = "",
m_variableTimeWDay = "", m_variableTimeYear = "", m_logCbData = 0x3ad32b0}

(gdb) q

#############################################

PLATFORM AND CONFIGURATION

#############################################

ModSecurity - v3.0.0+e14dc60 for Linux

Mandatory dependencies

• libInjection ....v2.9.0-839-ge14dc60
• SecLang tests ....e14dc60

Optional dependencies

• GeoIP ....found v1.5.0
  -lGeoIP , -I/usr/include/
• LibCURL ....found v7.29.0
  -lcurl , -DWITH_CURL
• YAJL ....found v2.0.4
  -lyajl , -DWITH_YAJL
• LMDB ....not found
• LibXML2 ....found v2.9.1
  -lxml2 -lz -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2

Other Options

• Test Utilities ....enabled
• SecDebugLog ....enabled
• afl fuzzer ....disabled
• library examples ....enabled
• Building parser ....disabled

---

gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)

---

uname -a
Linux localhost.localdomain 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

---

nginx version: nginx/1.13.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --add-module=/root/ModSecurity-nginx --with-cc-opt=-I/usr/include --with-ld-opt=-L/usr/lib --with-ipv6 --with-pcre --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_limit_conn_module --without-http_memcached_module --with-http_geoip_module --with-http_stub_status_module --with-http_realip_module --with-http_ssl_module --with-http_auth_request_module --user=nobody --group=nobody --with-http_mp4_module --with-http_flv_module --add-module=../ngx_devel_kit --add-module=../ngx_brotli --with-threads --with-file-aio --with-http_v2_module --with-debug

[zimmerle]

Hi @adamjacobmuller,

I believe that the problem you are facing is a consequence of the rules being wrongly loaded. Not a run time issue. I am investigating.

[defanator]

Observing the very similar issue on Debian 8 "jessie" - this is the only OS where I was able to get 100% reproducible way to trigger segfault.

OS details:

root@vagrant:/# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 8.7 (jessie)
Release:	8.7
Codename:	jessie

root@vagrant:/# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10)

root@vagrant:/# dpkg-query -W libc6 
libc6:amd64	2.19-18+deb8u10

Minimal nginx configuration:

user  nginx;
worker_processes  1;
worker_rlimit_core 500000000;
working_directory /tmp;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;

load_module modules/ngx_http_modsecurity_module.so;

events {
    worker_connections  1024;
}

http {
    access_log  off;

    server {
        listen 80;

        modsecurity on;
        modsecurity_rules_file /etc/nginx/modsec/main.conf;

        location / {
            proxy_pass http://nginx.org;
        }
    }
}

/etc/nginx/modsec/main.conf:

include /etc/nginx/modsec/modsecurity.conf

# OWASP CRS v3.0.0 rules
include /etc/nginx/modsec/owasp-crs/crs-setup.conf
include /etc/nginx/modsec/owasp-crs/rules/*.conf

/etc/nginx/modsec/modsecurity.conf:

SecRuleEngine On
SecRequestBodyAccess On
SecRule REQUEST_HEADERS:Content-Type "(?:text|application)/xml" \
     "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type "application/json" \
     "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 131072
SecRequestBodyLimitAction Reject
SecRule REQBODY_ERROR "!@eq 0" \
"id:'200002', phase:2,t:none,log,deny,status:400,msg:'Failed to parse request body.',logdata:'%{reqbody_error_msg}',severity:2"
SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:400, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"
SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
"id:'200004',phase:2,t:none,log,deny,msg:'Multipart parser detected a possible unmatched boundary.'"
SecPcreMatchLimit 1000
SecPcreMatchLimitRecursion 1000
SecRule TX:/^MSC_/ "!@streq 0" \
        "id:'200005',phase:2,t:none,deny,msg:'ModSecurity internal error flagged: %{MATCHED_VAR_NAME}'"
SecResponseBodyAccess On
SecResponseBodyMimeType text/plain text/html text/xml
SecResponseBodyLimit 524288
SecResponseBodyLimitAction ProcessPartial
SecTmpDir /tmp/
SecDataDir /tmp/
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/modsec_audit.log
SecArgumentSeparator &
SecCookieFormat 0
SecUnicodeMapFile unicode.mapping 20127
SecStatusEngine On

In /etc/nginx/modsec/owasp-crs/ there's a list of OWASP CRS v3.0.0 rules without any modifications.

Request:

root@vagrant:/# curl -vi -H 'User-Agent: Mozilla/5.00 (Nikto/2.1.6)' http://localhost/modsec-full/
* Hostname was NOT found in DNS cache
*   Trying ::1...
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /modsec-full/ HTTP/1.1
> Host: localhost
> Accept: */*
> User-Agent: Mozilla/5.00 (Nikto/2.1.6)
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

Backtrace:

(gdb) bt full
#0  0x00007f07d482226b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#1  0x00007f07d57a2954 in modsecurity::Rule::getFinalVars (this=this@entry=0x7f07d982d1d0, trans=trans@entry=0x7f07d97d77a0) at rule.cc:622
        key = 0x7ffdde1e7e60
        var = std::unique_ptr<modsecurity::collection::Variable> containing 0x7ffdde1e8070
        v = <optimized out>
        __for_range = std::vector of length 1, capacity 15 = {0x7f07dab7fed0}
        variable = <optimized out>
        e = std::vector of length 1, capacity 15 = {0x7f07dab7fed0}
        ignoreVariable = false
        i = 0
        exclusions_update_by_tag_remove = empty std::list
        exclusions_update_by_id_remove = empty std::list
        variables = std::vector of length 1, capacity 1 = {0x7f07d982c9b0}
        exclusions = empty std::list
        finalVars = std::vector of length 0, capacity 0
#2  0x00007f07d57a44b4 in modsecurity::Rule::evaluate (this=this@entry=0x7f07d982d1d0, trans=trans@entry=0x7f07d97d77a0, 
    ruleMessage=std::shared_ptr (count 1, weak 0) 0x7f07dab7f7e0) at rule.cc:774
        variables = <optimized out>
        recursiveGlobalRet = <optimized out>
        containsDisruptive = false
        eparam = "\"1\""
        globalRet = false
        finalVars = std::vector of length 0, capacity 0
#3  0x00007f07d5794e07 in modsecurity::Rules::evaluate (this=0x7f07d97fe6a0, phase=phase@entry=3, transaction=transaction@entry=0x7f07d97d77a0) at rules.cc:219
        rule = 0x7f07d982d1d0
        i = 295
        rules = std::vector of length -17458961258336, capacity -17458961258336 = {0x7f07d980a950, 0x7f07d980ad40, 0x7f07d980b4e0, 0x7f07d980bb70, 0x7f07d987dbb0, 
          0x7f07d986d660, 0x7f07d986dc40, 0x7f07d986e2d0, 0x7f07d986e800, 0x7f07d986ef00, 0x7f07d986f720, 0x7f07d986fce0, 0x7f07d9870110, 0x7f07d987df50, 0x7f07d987e390, 
          0x7f07d987e890, 0x7f07d987ee20, 0x7f07d987f460, 0x7f07d987f960, 0x7f07d98800c0, 0x7f07d9887780, 0x7f07d9887df0, 0x7f07d9888630, 0x7f07d9888be0, 0x7f07d9889110, 
          0x7f07d98896f0, 0x7f07d9889d90, 0x7f07d988a280, 0x7f07d988a850, 0x7f07d988ac90, 0x7f07d98699e0, 0x7f07d9869f00, 0x7f07d986a390, 0x7f07d986b350, 0x7f07d986b7f0, 
          0x7f07d986c8d0, 0x7f07d988d370, 0x7f07d988d800, 0x7f07d988e350, 0x7f07d988f190, 0x7f07d9890040, 0x7f07d9890ef0, 0x7f07d9892430, 0x7f07d98927a0, 0x7f07d9893690, 
          0x7f07d9894950, 0x7f07d9895e20, 0x7f07d9898330, 0x7f07d9899cb0, 0x7f07d989a810, 0x7f07d989c260, 0x7f07d989c610, 0x7f07d989e010, 0x7f07d989e350, 0x7f07d989eb00, 
          0x7f07d98674c0, 0x7f07d9868290, 0x7f07d9860d80, 0x7f07d98621f0, 0x7f07d98627c0, 0x7f07d9863c80, 0x7f07d989f040, 0x7f07d98a0510, 0x7f07d98a1ad0, 0x7f07d98a3090, 
          0x7f07d98a4680, 0x7f07d98a49c0, 0x7f07d98a50f0, 0x7f07d98a5f90, 0x7f07d98a6a90, 0x7f07d98a75d0, 0x7f07d98a7a70, 0x7f07d985cf50, 0x7f07d985e400, 0x7f07d985ecf0, 
          0x7f07d985f810, 0x7f07d9860380, 0x7f07d98a7e70, 0x7f07d985a870, 0x7f07d98ab4e0, 0x7f07d98ad150, 0x7f07d98adca0, 0x7f07d98ae260, 0x7f07d98aeb70, 0x7f07d98549f0, 
          0x7f07d9856440, 0x7f07d991c4a0, 0x7f07d9926660, 0x7f07d9926fe0, 0x7f07d9938050, 0x7f07d994f4f0, 0x7f07d994fe70, 0x7f07d9966b70, 0x7f07d9966fe0, 0x7f07d9850b90, 
          0x7f07d98520b0, 0x7f07d9853310, 0x7f07d9967410, 0x7f07d9968530, 0x7f07d996e4f0, 0x7f07d996fab0, 0x7f07d99710e0, 0x7f07d9972980, 0x7f07d9973470, 0x7f07d99748c0, 
          0x7f07d9975ec0, 0x7f07d99777c0, 0x7f07d9978950, 0x7f07d9979b50, 0x7f07d997ac40, 0x7f07d997aec0, 0x7f07d997bee0, 0x7f07d997faf0, 0x7f07d9981720, 0x7f07d9982160, 
          0x7f07d9983d50, 0x7f07d9984800, 0x7f07d9985b70, 0x7f07d9986ed0, 0x7f07d9988260, 0x7f07d9989570, 0x7f07d998abf0, 0x7f07d998c430, 0x7f07d998e3a0, 0x7f07d998f590, 
          0x7f07d9990fc0, 0x7f07d9992090, 0x7f07d9993250, 0x7f07d99946e0, 0x7f07d999a3e0, 0x7f07d999b370, 0x7f07d999e000, 0x7f07d999f260, 0x7f07d999fa80, 0x7f07d99a0f10, 
          0x7f07d99a1780, 0x7f07d99a2780, 0x7f07d99a4210, 0x7f07d99a55d0, 0x7f07d99a6c60, 0x7f07d99a6f60, 0x7f07d984ca60, 0x7f07d984e220, 0x7f07d984f890, 0x7f07d99a8020, 
          0x7f07d99a96c0, 0x7f07d99aab00, 0x7f07d99abec0, 0x7f07d99ad4c0, 0x7f07d99adc60, 0x7f07d99af410, 0x7f07d99afbf0, 0x7f07d99b0870, 0x7f07d99b18e0, 0x7f07d99b2a60, 
          0x7f07d99b2ea0, 0x7f07d9848b90, 0x7f07d984a740, 0x7f07d984c180, 0x7f07d99b8b70, 0x7f07d9bec540, 0x7f07d9bece90, 0x7f07d9c00450, 0x7f07d9c00fe0, 0x7f07d9c01400, 
          0x7f07d9844b60, 0x7f07d9846150, 0x7f07d9847660, 0x7f07d9c01e50, 0x7f07d9c027f0, 0x7f07d9c03990, 0x7f07d9c04a60, 0x7f07d9c055f0, 0x7f07d9c05a10, 0x7f07d9840af0, 
          0x7f07d9c0a010, 0x7f07d9c11900, 0x7f07d9c1a3a0, 0x7f07d9c23160, 0x7f07d9c24dd0, 0x7f07d9c89fd0, 0x7f07d9c8ba80, 0x7f07d9c94210, 0x7f07d9c95bf0, 0x7f07d9ca3510, 
          0x7f07d9ca4c90, 0x7f07d9ca5580, 0x7f07d9ca60c0, 0x7f07d9ca6c50, 0x7f07d9ca71f0, 0x7f07d983ca40, 0x7f07d983f3c0, 0x7f07d9ca7d10, 0x7f07d9ca9090, 0x7f07d9d30b60, 
          0x7f07d9d3b620, 0x7f07d9d3ce60, 0x7f07d9d4ee30, 0x7f07d9d50620, 0x7f07d9d51e50...}
#4  0x00007f07d5781eca in modsecurity::Transaction::processRequestBody (this=<optimized out>) at transaction.cc:811
        a = std::unique_ptr<std::basic_string<char, std::char_traits<char>, std::allocator<char> >> containing 0x0
        fullRequest = "Host: localhost\nAccept: */*\nUser-Agent: Mozilla/5.00 (Nikto/2.1.6)\n\n\n"
        l = std::vector of length 3, capacity 4 = {0x7f07da910140, 0x7f07d9803760, 0x7f07d9803a20}
#5  0x00007f07d5aa0ba0 in ngx_http_modsecurity_pre_access_handler (r=0x7f07dab80040) at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c:199
        already_inspected = 0
        chain = 0x0
        ctx = 0x7f07d9791c58
        cf = 0x7f07dab80040
        old_pool = 0x0
#6  0x00007f07d7c470cf in ngx_http_core_generic_phase (r=0x7f07d9790f20, ph=0x7f07dab317f0) at src/http/ngx_http_core_module.c:873
        rc = <optimized out>
#7  0x00007f07d7c4292d in ngx_http_core_run_phases (r=r@entry=0x7f07d9790f20) at src/http/ngx_http_core_module.c:851
        rc = <optimized out>
        ph = 0x7f07dab31760
        cmcf = <optimized out>
#8  0x00007f07d7c42a22 in ngx_http_handler (r=r@entry=0x7f07d9790f20) at src/http/ngx_http_core_module.c:834
        cmcf = <optimized out>
#9  0x00007f07d7c4d60e in ngx_http_process_request (r=0x7f07d9790f20) at src/http/ngx_http_request.c:1948
        c = 0x7f07cfaf52e0
#10 0x00007f07d7c4deb0 in ngx_http_process_request_line (rev=0x7f07cf4f4130) at src/http/ngx_http_request.c:1048
        n = <optimized out>
        rc = <optimized out>
        rv = <optimized out>
        host = {len = 1, 
          data = 0x7f07d7c4e008 <ngx_http_wait_request_handler+264> "H\205\300H\211\003\017\204\020\377\377\377H\215\005E\373\377\377H\211\357H\211E\020H\203\304\030[]A\\A]\351/\373\377\377\017\037\200"}
        c = 0x7f07cfaf52e0
        r = 0x7f07d9790f20
#11 0x00007f07d7c36424 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:902
        events = 1
        revents = 1
        instance = <optimized out>
        i = 0
        level = <optimized out>
        err = <optimized out>
        rev = <optimized out>
        wev = <optimized out>
        queue = <optimized out>
        c = 0x7f07cfaf52e0
#12 0x00007f07d7c2d1ea in ngx_process_events_and_timers (cycle=cycle@entry=0x7f07d9778ec0) at src/event/ngx_event.c:242
        flags = <optimized out>
        timer = <optimized out>
        delta = 1503598642301
#13 0x00007f07d7c342b1 in ngx_worker_process_cycle (cycle=cycle@entry=0x7f07d9778ec0, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:749
        worker = 0
#14 0x00007f07d7c32be3 in ngx_spawn_process (cycle=0x7f07d9778ec0, proc=0x7f07d7c34230 <ngx_worker_process_cycle>, data=0x0, name=0x7f07d7ccaa97 "worker process", respawn=1)
    at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 1
#15 0x00007f07d7c355ea in ngx_reap_children (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:621
        i = 1
        n = 2
        ch = {command = 2, pid = 14165, slot = 1, fd = -1}
        ccf = 0x7f07dab80040
#16 ngx_master_process_cycle (cycle=0x7f07d9778ec0) at src/os/unix/ngx_process_cycle.c:174
        title = 0x7f07d7f2b990 <ngx_last_process> "\002"
        size = 140728330001264
        n = 140728330001264
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
#17 0x00007f07d7c0d30b in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:381
        b = <optimized out>
        log = 0x7f07d7f18300 <ngx_log>
        i = <optimized out>
        cycle = 0x7f07d9774eb0
        init_cycle = {conf_ctx = 0x0, pool = 0x7f07d9774970, log = 0x7f07d7f18300 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0, 
            handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0, free_connection_n = 0, 
          modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, reusable_connections_n = 0, listening = {elts = 0x0, 
            nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0, size = 0, 
            nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {key = 0, left = 0x0, right = 0x0, parent = 0x0, 
            color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {
            last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, 
          write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x7ffdde1eaf8b "ss"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, 
            data = 0x7ffdde1eaf8b "ss"}, prefix = {len = 11, data = 0x7f07d7cc6860 "/etc/nginx/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = <optimized out>
        ccf = 0x7f07d9776c78
(gdb) 

A Vagrant-based environment is available here for quick reproducing this segfault: https://github.com/defanator/modsecurity-performance

Steps to reproduce:

# git clone https://github.com/defanator/modsecurity-performance.git
# cd modsecurity-performance
# vagrant up debian
# vagrant ssh debian
vagrant@vagrant:~$ curl -vi -H 'User-Agent: Mozilla/5.00 (Nikto/2.1.6)' http://localhost/modsec-full/

Tested on the following Vagrant box:

debian/jessie64 (libvirt, 8.7.0)

[defanator]

Just finished testing on another box, debian/jessie64 (virtualbox, 8.9.0), the same results. OS details:

vagrant@vagrant:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 8.9 (jessie)
Release:	8.9
Codename:	jessie

vagrant@vagrant:~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10) 

vagrant@vagrant:~$ dpkg-query -W libc6
libc6:amd64	2.19-18+deb8u10

[zimmerle]

Should be ok at: cc1d220b408fe73a4e1950b71848772d505d6ce0

[zimmerle]

Fix confirmed. Thanks ;)