Apache with loaded modsecurity has crashed on my server with the signal 11, Segmentation fault. apr_global_mutex_lock is trying to lock the uncreated mutex

[rcbarnett-zz]

MODSEC-416: Hello.

I have an issue with modsecurity-apache_2.7.4.
Apache with loaded modsecurity has crashed on my server with the signal 11.

gdb says:
Program terminated with signal 11, Segmentation fault.
#0 0x00007fba3c835f3c in apr_global_mutex_lock (mutex=0x0) at locks/unix/global_mutex.c:97

97 locks/unix/global_mutex.c: No such file or directory.
in locks/unix/global_mutex.c
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 fontconfig-2.8.0-3.el6.x86_64 glibc-2.12-1.80.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 libgcrypt-1.4.5-9.el6_2.2.x86_64 libgpg-error-1.7-4.el6.x86_64 libidn-1.18-2.el6.x86_64 libpng-1.2.49-1.el6_2.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64
(gdb) bt
#0 0x00007fba3c835f3c in apr_global_mutex_lock (mutex=0x0) at locks/unix/global_mutex.c:97
#1 0x00007fba362a7f0e in sec_audit_logger (msr=0x7fba240041d0)

at src/modsecurity-apache_2.5.9/apache2/msc_logging.c:491

#2 0x00007fba3629bb89 in modsecurity_process_phase_logging (msr=0x7fba240041d0)

at src/modsecurity-apache_2.5.9/apache2/modsecurity.c:492

#3 0x00007fba3629bdaf in modsecurity_process_phase (msr=0x7fba240041d0, phase=5)

at src/modsecurity-apache_2.5.9/apache2/modsecurity.c:585

#4 0x00007fba3628c166 in hook_log_transaction (r=0x7fba24002970)

at src/modsecurity-apache_2.5.9/apache2/mod_security2.c:916

#5 0x000000000042ff36 in ap_run_log_transaction (r=0x7fba24002970) at protocol.c:1719
#6 0x0000000000450cb4 in ap_process_request (r=0x7fba24002970) at http_request.c:308
#7 0x000000000044d901 in ap_process_http_connection (c=0x7fba2c002d98) at http_core.c:190
#8 0x0000000000448d2f in ap_run_process_connection (c=0x7fba2c002d98) at connection.c:43
#9 0x0000000000449193 in ap_process_connection (c=0x7fba2c002d98, csd=0x7fba2c002b80)

at connection.c:190

#10 0x0000000000457ac2 in process_socket (p=0x7fba2c002b08, sock=0x7fba2c002b80, my_child_num=0,

my_thread_num=0, bucket_alloc=0x7fba240008e8) at worker.c:545

#11 0x00000000004583e5 in worker_thread (thd=0x1bed378, dummy=0x7fba2c0008c0) at worker.c:895
#12 0x00007fba3c847ae5 in dummy_worker (opaque=0x1bed378) at threadproc/unix/thread.c:142
#13 0x0000003c41607851 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003c412e767d in clone () from /lib64/libc.so.6

It looks like the sec_audit_logger function is trying to lock the uncreated mutex:
#0 0x00007fba3c835f3c in apr_global_mutex_lock (mutex=0x0) at locks/unix/global_mutex.c:97

So, I decided to check how that mutex is created, and I discovered that the semget system function fails with error 28 on my system for some reason:
static apr_status_t proc_mutex_sysv_create(apr_proc_mutex_t *new_mutex,
const char *fname)
{
union semun ick;
apr_status_t rv;

new_mutex->interproc = apr_palloc(new_mutex->pool, sizeof(*new_mutex->interproc));
new_mutex->interproc->filedes = semget(IPC_PRIVATE, 1, IPC_CREAT | 0600);

gdb ./bin/httpd
(gdb) set environment LD_LIBRARY_PATH=/import/home/ivan.voronin/tmp/linux/redhat6/022.01/apache2/ps/lib:/import/home/ivan.voronin/tmp/linux/redhat6/022.01/apache2/lib:/import/home/ivan.voronin/tmp/linux/redhat6/022.01/apache2/modules
(gdb) b modsecurity_init
Function "modsecurity_init" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (modsecurity_init) pending.
(gdb) run -X -f /import/home/ivan.voronin/tmp/linux/redhat6/022.01/apache2/conf/httpd.conf
Starting program: /import/home/ivan.voronin/tmp/linux/redhat6/022.01/apache2/bin/httpd -X -f /import/home/ivan.voronin/tmp/linux/redhat6/022.01/apache2/conf/httpd.conf
[Thread debugging using libthread_db enabled]
Detaching after fork from child process 23581.
Detaching after fork from child process 23582.
Detaching after fork from child process 23583.
Detaching after fork from child process 23584.
Detaching after fork from child process 23585.
Detaching after fork from child process 23586.
warning: Temporarily disabling breakpoints for unloaded shared library "/import/home/ivan.voronin/tmp/linux/redhat6/022.01/apache2/modules/mod_security.so"
Detaching after fork from child process 23596.
Detaching after fork from child process 23598.
Detaching after fork from child process 23599.
Detaching after fork from child process 23600.
Detaching after fork from child process 23601.
Detaching after fork from child process 23602.

Breakpoint 1, modsecurity_init (msce=0x862c18, mp=0x68b138) at src/modsecurity-apache_2.5.9/apache2/modsecurity.c:101
101 src/modsecurity-apache_2.5.9/apache2/modsecurity.c: No such file or directory.
in src/modsecurity-apache_2.5.9/apache2/modsecurity.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 libidn-1.18-2.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64
(gdb) p rc
$1 = 32767
(gdb) s
apr_global_mutex_create (mutex=0x862c20, fname=0x0, mech=APR_LOCK_DEFAULT, pool=0x68b138) at locks/unix/global_mutex.c:53
53 locks/unix/global_mutex.c: No such file or directory.
in locks/unix/global_mutex.c
(gdb) list
48 in locks/unix/global_mutex.c
(gdb) p *mutex
$3 = (apr_global_mutex_t *) 0x0
(gdb) n
54 in locks/unix/global_mutex.c
(gdb) p *m
$5 = {pool = 0x0, proc_mutex = 0x0, thread_mutex = 0x0}
(gdb) p pool
$6 = (apr_pool_t *) 0x68b138
(gdb) n
56 in locks/unix/global_mutex.c
(gdb) p pool
$7 = (apr_pool_t *) 0x68b138
(gdb) p *m
$8 = {pool = 0x68b138, proc_mutex = 0x0, thread_mutex = 0x0}
(gdb) p rv
$9 = 32767
(gdb) s
apr_proc_mutex_create (mutex=0xa726f0, fname=0x0, mech=APR_LOCK_DEFAULT, pool=0x68b138) at locks/unix/proc_mutex.c:887
887 locks/unix/proc_mutex.c: No such file or directory.
in locks/unix/proc_mutex.c
(gdb) list
882 in locks/unix/proc_mutex.c
(gdb) n
888 in locks/unix/proc_mutex.c
(gdb) p mech
$10 = APR_LOCK_DEFAULT
(gdb) p fname
$11 = 0x0
(gdb) list
883 in locks/unix/proc_mutex.c
(gdb) p *new_mutex
$13 = {pool = 0x0, meth = 0x0, inter_meth = 0x0, curr_locked = 0, fname = 0x0, interproc = 0x0, psem_interproc = 0x0,
pthread_interproc = 0x0}
(gdb) p pool
$14 = (apr_pool_t *) 0x68b138
(gdb) p new_mutex
$15 = (apr_proc_mutex_t *) 0xa72700
(gdb) n
890 in locks/unix/proc_mutex.c
(gdb) s
proc_mutex_create (new_mutex=0xa72700, mech=APR_LOCK_DEFAULT, fname=0x0) at locks/unix/proc_mutex.c:866
866 in locks/unix/proc_mutex.c
(gdb) p rv
$16 = 0
(gdb) p new_mutex
$17 = (apr_proc_mutex_t *) 0xa72700
(gdb) p *new_mutex
$18 = {pool = 0x68b138, meth = 0x0, inter_meth = 0x0, curr_locked = 0, fname = 0x0, interproc = 0x0, psem_interproc = 0x0,
pthread_interproc = 0x0}
(gdb) p mech
$19 = APR_LOCK_DEFAULT
(gdb) n
870 in locks/unix/proc_mutex.c
(gdb) n
872 in locks/unix/proc_mutex.c
(gdb) s
proc_mutex_sysv_create (new_mutex=0xa72700, fname=0x0) at locks/unix/proc_mutex.c:228
228 in locks/unix/proc_mutex.c
(gdb) n
229 in locks/unix/proc_mutex.c
(gdb) p errno
$20 = 2
(gdb) p new_mutex
$21 = (apr_proc_mutex_t *) 0xa72700
(gdb) p new_mutex->interproc
$22 = (apr_file_t *) 0xa72740
(gdb) p *new_mutex->interproc
$23 = {pool = 0x0, filedes = 0, fname = 0x0, flags = 0, eof_hit = 0, is_pipe = 0, timeout = 0, buffered = 0,
blocking = BLK_UNKNOWN, ungetchar = 0, buffer = 0x0, bufpos = 0, bufsize = 0, dataRead = 0, direction = 0, filePtr = 0,
thlock = 0x0}
(gdb) n
231 in locks/unix/proc_mutex.c
(gdb) p *new_mutex->interproc
$24 = {pool = 0x0, filedes = -1, fname = 0x0, flags = 0, eof_hit = 0, is_pipe = 0, timeout = 0, buffered = 0,
blocking = BLK_UNKNOWN, ungetchar = 0, buffer = 0x0, bufpos = 0, bufsize = 0, dataRead = 0, direction = 0, filePtr = 0,
thlock = 0x0}
(gdb) p errno
$25 = 28

apachectl -V says:
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/import/home/ivan.voronin/tmp/tmp/apache_project/distrib/apache2"
-D SUEXEC_BIN="/import/home/ivan.voronin/tmp/tmp/apache_project/distrib/apache2/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

When Apache is compiled with APR_USE_SYSVSEM_SERIALIZE and the apr_proc_mutex_create is launched with memch=APR_LOCK_DEFAULT,
the proc_mutex_sysv_create (and semget) function is called by the apr_proc_mutex_create function for creating the mutex.

It doesn't matter why semget fails on my system, but modsecurity_init is not logging it:

int modsecurity_init(msc_engine *msce, apr_pool_t *mp) {
apr_status_t rc;

/* Serial audit log mutext */
rc = apr_global_mutex_create(&msce->auditlog_lock, NULL, APR_LOCK_DEFAULT, mp);
if (rc != APR_SUCCESS) {
    //ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, "mod_security: Could not create modsec_auditlog_lock");
    //return HTTP_INTERNAL_SERVER_ERROR;
    return -1;
}

...

and geo_lock too:
rc = apr_global_mutex_create(&msce->geo_lock, NULL, APR_LOCK_DEFAULT, mp);
if (rc != APR_SUCCESS) {
return -1;
}

It isn't logged in the hook_post_config function, too:
static int hook_post_config(apr_pool_t *mp, apr_pool_t *mp_log, apr_pool_t *mp_temp, server_rec *s) {
void *init_flag = NULL;
int first_time = 0;

...

} else {
    modsecurity_init(modsecurity, mp);
}

And the result value of the modsecurity_init function is not checked.

Could you please check this and let me know why the logging is commented out in the modsecurity_init function and why the result of the modsecurity_init function is not checked?

Why the modsecurity module for apache has no directive for choosing the type of synchronization, like AcceptMutex or SSLMutex, or doesn't simply use the AcceptMutex directive?

It looks like a bug and I made the patch that fixes this issue. Let me know what you think about it.

Hope to hear from you soon,
Ivan Voronin

--- ./modsecurity-apache_2.7.4/apache2/modsecurity.c 2013-05-27 07:33:36.000000000 +0400
+++ ./modsecurity-apache_2.7.4_patched/apache2/modsecurity.c 2013-07-11 13:31:30.330985000 +0400
@@ -21,6 +21,8 @@
#include "msc_util.h"
#include "msc_xml.h"
#include "apr_version.h"
+#include "mpm_common.h"
+#include "util_mutex.h"

unsigned long int DSOLOCAL unicode_codepage = 0;

@@ -114,49 +116,59 @@

• after configuration processing is complete as Apache needs to know the

• username it is running as.
  */
  -int modsecurity_init(msc_engine *msce, apr_pool_t *mp) {

  • apr_status_t rc;
    +apr_status_t modsecurity_init(msc_engine *msce, apr_pool_t *mp, server_rec *s) {
  • apr_status_t rc=APR_SUCCESS;

  /* Serial audit log mutext */

  • rc = apr_global_mutex_create(&msce->auditlog_lock, NULL, APR_LOCK_DEFAULT, mp);
    +// *********************** BTS 0073283 ******************************
    +// rc = apr_global_mutex_create(&msce->auditlog_lock, NULL, APR_LOCK_DEFAULT, mp);
  • rc = ap_global_mutex_create(&msce->auditlog_lock, NULL, AP_ACCEPT_MUTEX_TYPE, NULL, s, mp, 0);
    if (rc != APR_SUCCESS) {

  •    ap_log_error(APLOG_MARK, APLOG_EMERG, rc, s,
    

  •                "mod_security: Could not create modsec_auditlog_lock");
    

    //ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, "mod_security: Could not create modsec_auditlog_lock");
    //return HTTP_INTERNAL_SERVER_ERROR;

  •    return -1;
    

  •    return rc;
    

    }

-#if !defined(MSC_TEST)
-#ifdef __SET_MUTEX_PERMS
-#if AP_SERVER_MAJORVERSION_NUMBER > 1 && AP_SERVER_MINORVERSION_NUMBER > 2

• rc = ap_unixd_set_global_mutex_perms(msce->auditlog_lock);
  -#else
• rc = unixd_set_global_mutex_perms(msce->auditlog_lock);
  -#endif
• if (rc != APR_SUCCESS) {
  +// *********************** BTS 0073283 ******************************
  +// #if !defined(MSC_TEST)
  +// #ifdef __SET_MUTEX_PERMS
  +// #if AP_SERVER_MAJORVERSION_NUMBER > 1 && AP_SERVER_MINORVERSION_NUMBER > 2
• // rc = ap_unixd_set_global_mutex_perms(msce->auditlog_lock);
  +// #else
• // rc = unixd_set_global_mutex_perms(msce->auditlog_lock);
  +// #endif
• // if (rc != APR_SUCCESS) {
  // ap_log_error(APLOG_MARK, APLOG_ERR, rc, s, "mod_security: Could not set permissions on modsec_auditlog_lock; check User and Group directives");
  // return HTTP_INTERNAL_SERVER_ERROR;

•    return -1;
  

• }
  -#endif /* SET_MUTEX_PERMS */
• rc = apr_global_mutex_create(&msce->geo_lock, NULL, APR_LOCK_DEFAULT, mp);

•    // return -1;
  

• // }
  +// #endif /* SET_MUTEX_PERMS _/

+// *_********************* BTS 0073283 ******************************
+// rc = apr_global_mutex_create(&msce->geo_lock, NULL, APR_LOCK_DEFAULT, mp);

• rc = ap_global_mutex_create(&msce->geo_lock, NULL, AP_ACCEPT_MUTEX_TYPE, NULL, s, mp, 0);
  if (rc != APR_SUCCESS) {

•    return -1;
  

• }

•    ap_log_error(APLOG_MARK, APLOG_EMERG, rc, s,
  

•                "mod_security: Could not create modsec_geo_lock");
  

•    return rc;
  

• }

+// *********************** BTS 0073283 ******************************
+// #ifdef __SET_MUTEX_PERMS
+// #if AP_SERVER_MAJORVERSION_NUMBER > 1 && AP_SERVER_MINORVERSION_NUMBER > 2

• // rc = ap_unixd_set_global_mutex_perms(msce->geo_lock);
  +// #else
• // rc = unixd_set_global_mutex_perms(msce->geo_lock);
  +// #endif
• // if (rc != APR_SUCCESS) {

•    // return -1;
  

• // }
  +// #endif /* SET_MUTEX_PERMS */
  +// #endif

-#ifdef __SET_MUTEX_PERMS
-#if AP_SERVER_MAJORVERSION_NUMBER > 1 && AP_SERVER_MINORVERSION_NUMBER > 2

• rc = ap_unixd_set_global_mutex_perms(msce->geo_lock);
  -#else
• rc = unixd_set_global_mutex_perms(msce->geo_lock);
  -#endif
• if (rc != APR_SUCCESS) {

•    return -1;
  

• }
  -#endif /* SET_MUTEX_PERMS */
  -#endif
• return 1;
• return rc;
  }

/**
--- ./modsecurity-apache_2.7.4/apache2/modsecurity.h 2013-05-27 07:33:36.000000000 +0400
+++ ./modsecurity-apache_2.7.4_patched/apache2/modsecurity.h 2013-07-11 13:28:29.859297000 +0400
@@ -653,7 +653,7 @@

msc_engine DSOLOCAL *modsecurity_create(apr_pool_t *mp, int processing_mode);

-int DSOLOCAL modsecurity_init(msc_engine *msce, apr_pool_t *mp);
+apr_status_t DSOLOCAL modsecurity_init(msc_engine *msce, apr_pool_t *mp, server_rec *s);

void DSOLOCAL modsecurity_child_init(msc_engine *msce);

--- ./modsecurity-apache_2.7.4/apache2/mod_security2.c 2013-05-27 07:33:36.000000000 +0400
+++ ./modsecurity-apache_2.7.4_patched/apache2/mod_security2.c 2013-07-11 13:33:05.845007000 +0400
@@ -622,6 +622,7 @@
static int hook_post_config(apr_pool_t *mp, apr_pool_t *mp_log, apr_pool_t *mp_temp, server_rec *s) {
void *init_flag = NULL;
int first_time = 0;

• apr_status_t rv=APR_SUCCESS;

/* ENH Figure out a way to validate config before we start

• - skipafter: need to make sure we found all of our targets
  @@ -634,7 +635,9 @@
  apr_pool_userdata_set((const void *)1, "modsecurity-init-flag",
  apr_pool_cleanup_null, s->process->pool);
  } else {

•    modsecurity_init(modsecurity, mp);
  

•    rv=modsecurity_init(modsecurity, mp, s);
  

•    if (rv != APR_SUCCESS)
  

•        return HTTP_INTERNAL_SERVER_ERROR;
  

  }

  /* Store the original server signature */

[rcbarnett-zz]

Original reporter: viv

[rcbarnett-zz]

viv: patch that fixes this issue

[rcbarnett-zz]

bpinto: Hello Ivan,

Thanks to report it. This could be a way to go. However if it is true, we will need some extra work to make it for standalone module.
Maybe we have an easy way to fix it. Did you try change the mech type for apr_global_mutex_create(.. , ..., mech ...); ?

http://apr.apache.org/docs/apr/1.4/group___a_p_r___global_mutex.html#ga7a5ade1ba8ea6c7a7a61e3e719683cb0

If it works, maybe we can create a directive like you said "AcceptMutex".

Breno

[rcbarnett-zz]

viv: I've made two patches.
The first one (patch_mod_security_2.7.4_crash_mutex.patch) is for modsecurity 2.7.4 + Apache 2.4.4
The other one (patch_mod_security_2.5.9_crash_mutex.patch) is for modsecurity 2.5.9 + Apache 2.2.23

They contain the following changes:
fix: checking result of the modsecurity_init function;
add: logging in the modsecurity_init function;
modify: using ap_global_mutex_create instead of apr_global_mutex_create for Apache 2.4;
modify: ap_accept_lock_mech as mech type in apr_global_mutex_create for Apache 2.2.

I’ve built Apache and the patched modsecurity in both cases (modsecurity 2.7.4 +patch_mod_security_2.7.4_crash_mutex.patch+ Apache 2.4.4 / modsecurity 2.5.9 +patch_mod_security_2.5.9_crash_mutex.patch+ Apache 2.2.23) and tested it.
It works well.
I modified httpd.conf as follows:
AcceptMutex pthread (for Apache 2.2)
Mutex pthread (for Apache 2.4)
and now modsecurity uses mech=APR_LOCK_PROC_PTHREAD :
Breakpoint 1, proc_mutex_create (new_mutex=0x68a698, mech=APR_LOCK_PROC_PTHREAD, fname=0x0) at locks/unix/proc_mutex.c:866
866 locks/unix/proc_mutex.c: No such file or directory.
in locks/unix/proc_mutex.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 libidn-1.18-2.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64

Ivan

[rcbarnett-zz]

bpinto: Hello Ivan,

I like you idea to fix it. I cannot release it for 2.7.5 because it is already during the release process and this fix will require some test not only for Apache 2.x but for Nginx and IIS too.

However this will be a priority fix for 2.7.6.

Thanks

Breno

[ngo]

I can confirm that this problem is still relevant in latest 2.8.0 modsecurity.

In my case the root cause of the faulty mutex initialization was also semget() which failed with errno=28 (No space left on the device).

The workaround for the semget fail is to list semaphorest with ipcs -s and clean unneeded ones with ipcrm -s $id.

But this still does not resolve the root cause - the lack of error checking and reporting.

[ngo]

In my opinion, the first problem to fix is not the "mutex type selection" part of the initial bug report, but the "lack of error checking" part. Fixing lack of error checking on its own should be a rather small patch and may also cover other failiures.

[ngo]

Also, the real root cause of the problem is the fact that the locks in question are never ever cleaned up!

While for some lock methods this is okay, because on process termination these get released anyway, for mutex_sysv_methods it is a disaster, because semaphores do not get released on process shutdown.

This means that on each invokation of modsecurity_init, to semaphores are created that will never be released. This, in turn, means that on systems where apr's default locking method is sysv, after, say, 30-40 restarts, modsecurity will constantly fail to properly initialize the mutex. Only manual cleanup or reboot will help that.

The resourse release should happen in modsecurity_shutdown (which is currently an empty func), but this will only help for apache. For iis and nginx (and other users of standalone API), the modsecurity_shutdown should be also exposed as a function in standalone/api.h (which is currently not the case).

[ngo]

After a little more debugging, I found that I'm a little bit wrong about not-released semaphores.

apr DOES automatically clean up the used mutexes when the corresponding pool is destroyed. In standalone modsecurity this happens in modsecTerminate - if you call that, the semaphores are released. So, my remark about semaphores that need to be manually cleaned up is invalid (though I would say that cleaning them up explicitly is still a decent idea), but all other remarks about error handling remain valid.

[shawnholt]

the mutex issue is very big - especially with mod_geoip (under CPANEL)
see: http://forums.cpanel.net/f185/modsecurity-rule-processing-failed-453321.html
this is the code:https://github.com/SpiderLabs/ModSecurity/blob/master/apache2/msc_geo.c
Perhaps this should call the NEW (http://webauth.stanford.edu/manual/developer/new_api_2_4.html) apache api for 2.4+ .

FYI - enabling geoip in server setup nearly crashed the server with the errors from the mutex fail!

related:#712

[kipras]

I'm experiencing the same issue, when running modsecurity 2.8.0 under nginx (more precisely this revision: https://github.com/SpiderLabs/ModSecurity/tree/33b8760e87b7441142a431175d5b459245551314 under nginx_refactoring branch).

The segfaults occurred randomly previously, under nginx 1.7.4.1, but now after upgrading nginx to 1.9.3.2 - they now happen for almost every request, or at least when there are at least two parallel requests it looks like.

However, changing APR_LOCK_DEFAULT to any value from https://apr.apache.org/docs/apr/2.0/group__apr__proc__mutex.html#ga75dd95a48a1e855a87b509b522746ed4 other than APR_LOCK_DEFAULT or APR_LOCK_SYSVSEM in the call to apr_global_mutex_create() in modsecurity_init() (see https://github.com/SpiderLabs/ModSecurity/blob/master/apache2/modsecurity.c#L136) fixes this issue.

The system is Ubuntu 14.04.1 LTS 64 bit

I'm guessing APR_LOCK_SYSVSEM no longer works correctly under nginx 1.9.3.2 for some reason and that it was also used as APR_LOCK_DEFAULT in this Ubuntu version.

I'm not sure which of APR_LOCK_FCNTL, APR_LOCK_FLOCK, APR_LOCK_PROC_PTHREAD to pick, but they all seem to work.

I tried upgrading modsecurity (using more recent code from nginx_refactoring branch) but that introduced other issues (requests don't respond until keepalive_timeout times out) so i reverted modsecurity back to the previously mentioned revision.

[victorhora]

The global mutex is optional since 112ba45 (included in 2.9.2 release)

Nginx support for ModSecurity 2.9.x branch is unsupported. This shouldn't be a concern with libModSecurity. Please upgrade.

Thanks!