php_opcache crash - memory access violation

[jjdunn]

Description

I don't know how to recreate the problem. it occurs on our production server, sporadically. 6x in past 4 days. never happened on old server (Windows 2012 R2) running same versions of PHP (8.0.14 x64 thread-safe) and apache httpd (2.4.51 x64), and the same opcache configuration. We are not using JIT.

Windows Event Log shows:

Faulting application name: php.exe, version: 8.0.14.0, time stamp: 0x61bb14bc
Faulting module name: php_opcache.dll, version: 8.0.14.0, time stamp: 0x61bb1862
Exception code: 0xc0000005
Fault offset: 0x0000000000002fe6
Faulting process id: 0x25d8
Faulting application start time: 0x01d8045afae2031a
Faulting application path: C:\Program Files\php\php.exe
Faulting module path: C:\Program Files\PHP\ext\php_opcache.dll
Report Id: d7c91f64-1d66-4a3c-a206-7f1efaf4d8cc
Faulting package full name:
Faulting package-relative application ID:

php.ini:

[PHP_OPCACHE]
zend_extension=php_opcache.dll
opcache.enable=1;
opcache.enable_cli=1;
opcache.memory_consumption=512;
opcache.interned_strings_buffer=32;
opcache.validate_timestamps=1;
opcache.save_comments=0;
opcache.revalidate_path=1;
opcache.enable_file_override=1;
opcache.max_file_size=0;
opcache.file_cache_only=0;
opcache.log_verbosity_level=2;
opcache.mmap_base = 0x20000000
opcache.jit_buffer_size=0;
opcache.jit=Off;

Running Windows DebugDiag tool, with PHP 8.0.14 symbols loaded, gives the report attached here:
opcache-crash-analysis.pdf

PHP Version

PHP 8.0.14

Operating System

Windows Server 2019; apache httpd 2.4.51 x64

[jjdunn]

additional information:

the crash has occurred 6x; each crash produces (2) .dmp files. Just ran DebugDiag on all 12 files.

Every one of the errors looks like this:

In php.exe_220104_074301.dmp the assembly instruction at php_opcache!accel_post_startup+1b4
in C:\Program Files\php\ext\php_opcache.dll from The PHP Group
has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000060 on thread 0

or like this:

In php.exe_220107_224305-1.dmp the assembly instruction at php_opcache!accel_new_interned_string+66
in C:\Program Files\php\ext\php_opcache.dll from The PHP Group
has caused an access violation exception (0xC0000005) when trying to read from memory location 0xa0000148 on thread 0

all of the errors are in accel_new_interned_string or accel_post_startup

in every case, the php process which crashed is a console application, NOT running in apache httpd.

[cmb69]

I guess the issues on Windows Server 2019 are due to more aggressive ASLR.

Are you running multiple PHP CLI process in parallel? Either toplevel process, or PHP processes spawned by a single toplevel PHP process. If so, would assigning a unique opcache.cache_id to these processes help?

Also consider to set opcache.log_verbosity_level=4 and check the opcache error log for any suspicious messages.

[jjdunn]

Thanks @cmb69 for the speedy reply. Yes we are running tons of CLI processes: ~2000/day on this server. However the schedule of these jobs is unchanged from the old server.

Keeping a separate opcache.cache_id would defeat some of the value of caching - since these processes use many of the same files - and would consume vastly more memory (there are currently 22 separate CLI processes), so I don't think that's a realistic solution.

I've changed verbosity_level=4 as suggested and added opcache.error_log configuration. Will post details here after the next crash...which I presume will occur.

[cmb69]

Yeah, makes probably sense to wait on hopefully helpful log messages.

However, sharing OPcache instances for "unrelated" (i.e. non forked) processes is a quirk on Windows, and can't generally work, since these processes (especially if dynamically loaded extensions are involved) can have different memory layout (ASLR), but OPcache partly relies on a fixed memory layout, and the only mitigation on Windows is checking that a single central function (zend_execute_ex) is at the same address when reattaching.

[jjdunn]

hmmm... I was not aware of the point you made in the previous comment. Is it possible for each CLI process to set opcache.cache_id dynamically, using something like ini_set("opcache.cache_id", <processID>); ?

And, if I did that, would the opcache configurations (especially opcache.memory_consumption=512;) apply to each of the CLI processes?? that's a lot of memory...unless I dial-down the memory_consumption for each of the CLI...