PHP8.1 Slow processing speed of string operations

[toshimitsu-sato]

Description

Does not occur in PHP8.0

The following code is slow when updated to PHP8.1
The functions used are below

• strlen
• mb_convert_encoding
• str_replace

The following code:

3500 data are set in $data
In PHP8.0 it takes 2 seconds to complete. It takes 3 minutes with PHP8.1

<?php
                foreach ($data as $record) {

                    $csvData = array();

                    foreach (array_values((array)$record) as $item) {

                        if (strlen($item) !== strlen(mb_convert_encoding(mb_convert_encoding($item, 'SJIS-win', 'UTF-8'), 'UTF-8', 'SJIS-win'))) {

                            $notExistsUnicodeValues[] = $item;

                        } else {

                              $csvData[] = str_replace(array("\r\n", "\r", "\n"), "\n", $item);

                        }

                    }
                }

PHP Version

PHP 8.1.0

Operating System

・Ubuntu 20.04
・CentOS Linux release 7.9.2009
・

[youkidearitai]

I tried benchmark below that using utf_ken_all.csv:

<?php

$fp = fopen("utf_ken_all.csv", "r");
$data = [];
while (($line = fgetcsv($fp) !== false)) {
    $data[] = mb_convert_encoding($line, "SJIS-win", "UTF-8");
}

$start = microtime(true);

foreach ($data as $record) {
    $csvData = array();
    foreach (array_values((array)$record) as $item) {
        if (strlen($item) !== strlen(mb_convert_encoding(mb_convert_encoding($item, 'SJIS-win', 'UTF-8'), 'UTF-8', 'SJIS-win'))) {
            $notExistsUnicodeValues[] = $item;
        } else {
            $csvData[] = str_replace(array("\r\n", "\r", "\n"), "\n", $item);
        }
    }
}

$end = microtime(true) - $start;

var_dump($end);

PHP 8.0.30

$ sapi/cli/php -v
PHP 8.0.30 (cli) (built: Sep 15 2023 17:34:19) ( NTS DEBUG )
Copyright (c) The PHP Group
Zend Engine v4.0.30, Copyright (c) Zend Technologies
$  sapi/cli/php benchmark.php
float(0.3169260025024414)

PHP 8.1.22

$ sapi/cli/php -v
PHP 8.1.22 (cli) (built: Oct  5 2023 13:39:10) (NTS DEBUG)
Copyright (c) The PHP Group
Zend Engine v4.1.22, Copyright (c) Zend Technologies
$ sapi/cli/php benchmark.php
float(0.3286919593811035)

Seems not slow so much.
Could you please more information?

[youkidearitai]

I'm sorry, my code is mistake.

<?php

$fp = fopen("utf_ken_all.csv", "r");
$data = [];
while (($line = fgetcsv($fp)) !== false) {
    $data[] = mb_convert_encoding($line, "SJIS-win", "UTF-8");
}

$start = microtime(true);

foreach ($data as $record) {
    $csvData = array();
    foreach (array_values((array)$record) as $item) {
        if (strlen($item) !== strlen(mb_convert_encoding(mb_convert_encoding($item, 'SJIS-win', 'UTF-8'), 'UTF-8', 'SJIS-win'))) {
            $notExistsUnicodeValues[] = $item;
        } else {
            $csvData[] = str_replace(array("\r\n", "\r", "\n"), "\n", $item);
        }
    }
}

$end = microtime(true) - $start;

var_dump($end);

PHP 8.0.30

$ sapi/cli/php -d'memory_limit=-1' benchmark.php
float(6.605648040771484)

PHP 8.1.22

I'm executing code, at least, very slow, not end.

[toshimitsu-sato]

@youkidearitai
thanks!

The update documentation for PHP8.1 doesn't specifically mention any changes, so I don't know why the processing speed has slowed down.

This is an important issue.
I have the same problem with PHP8.2

[youkidearitai]

@toshimitsu-sato Thanks for catch!

It seems mb_convert_encoding. As a test, without mb_convert_encoding, trying below code.

<?php

$fp = fopen("utf_ken_all.csv", "r");
$data = [];
while (($line = fgetcsv($fp)) !== false) {
    $data[] = mb_convert_encoding($line, "SJIS-win", "UTF-8");
}

$start = microtime(true);

foreach ($data as $record) {
    $csvData = array();
    foreach (array_values((array)$record) as $item) {
        if (strlen($item) !== strlen($item)) {
            $notExistsUnicodeValues[] = $item;
        } else {
            $csvData[] = str_replace(array("\r\n", "\r", "\n"), "\n", $item);
        }
    }
}

$end = microtime(true) - $start;

var_dump($end);

PHP 8.0.30

$ sapi/cli/php -d'memory_limit=-1' benchmark.php
float(0.6467161178588867)

PHP 8.1.22

$ sapi/cli/php -d memory_limit=-1 benchmark.php
float(0.6404819488525391)

Certainly, mb_convert_encoding seems slow.

@alexdowad What do you think?

[youkidearitai]

Hmm, mb_convert_encoding uses at once, almost same result.

<?php

$fp = fopen("utf_ken_all.csv", "r");
$data = [];
while (($line = fgetcsv($fp)) !== false) {
    $data[] = mb_convert_encoding($line, "SJIS-win", "UTF-8");
}

$start = microtime(true);

foreach ($data as $record) {
    $csvData = array();
    foreach (array_values((array)$record) as $item) {
        if (strlen($item) !== strlen(mb_convert_encoding($item, 'UTF-8', 'SJIS-win'))) {
            $notExistsUnicodeValues[] = $item;
        } else {
            $csvData[] = str_replace(array("\r\n", "\r", "\n"), "\n", $item);
        }
    }
}

$end = microtime(true) - $start;

var_dump($end);

PHP 8.0.30

$ sapi/cli/php -d'memory_limit=-1' benchmark.php
float(2.1218929290771484)

PHP 8.1.22

$ sapi/cli/php -d memory_limit=-1 benchmark.php
float(2.212764024734497)

However, 0.1 seconds slow.

[youkidearitai]

Just a moment, without strlen seems works fine.

<?php

$fp = fopen("utf_ken_all.csv", "r");
$data = [];
while (($line = fgetcsv($fp)) !== false) {
    $data[] = mb_convert_encoding($line, "SJIS-win", "UTF-8");
}

$start = microtime(true);

foreach ($data as $record) {
    $csvData = array();
    foreach (array_values((array)$record) as $item) {
        if ($item !== mb_convert_encoding(mb_convert_encoding($item, 'SJIS-win', 'UTF-8'), 'UTF-8', 'SJIS-win')) {
            $notExistsUnicodeValues[] = $item;
        } else {
            $csvData[] = str_replace(array("\r\n", "\r", "\n"), "\n", $item);
        }
    }
}

$end = microtime(true) - $start;

var_dump($end);

PHP 8.1.22 result

$ sapi/cli/php -d memory_limit=-1 benchmark.php
float(6.628969192504883)

Hmm... Is this strlen problem?

[toshimitsu-sato]

If you set $data to the value obtained from the database instead of the data read from the file and run the code below, mb_convert_encoding, str_len, and str_replace will all become slower.

It is still faster to loop using data read from a file.

foreach ($data as $record) {
    $csvData = array();
    foreach (array_values((array)$record) as $item) {
        if ($item !== mb_convert_encoding(mb_convert_encoding($item, 'SJIS-win', 'UTF-8'), 'UTF-8', 'SJIS-win')) {
            $notExistsUnicodeValues[] = $item;
        } else {
            $csvData[] = str_replace(array("\r\n", "\r", "\n"), "\n", $item);
        }
    }
}

PHP 8.1.22 result
・very late

[youkidearitai]

@toshimitsu-sato Excuse me, I tried PHP 8.2.13(dev version) and PHP 8.1.25, it seems works fine. Do you use linux distribution's package? and can you use newest version?

[toshimitsu-sato]

@youkidearitai

thanks!

I will try PHP 8.2.13(dev version) and PHP 8.1.25

I'm using Azure AppService, but I can't specify the php patch version.

[nielsdos]

I can't reproduce the problem with @youkidearitai 's script posted here #12684 (comment)
At least, I don't see a huge time gap between the execution in 8.0 and 8.1.
I have a feeling something else entirely is going on. Especially as OP says it only happens when the data comes from a database instead of a file.
@toshimitsu-sato What happens when you disable garbage collection?

[toshimitsu-sato]

@nielsdos
thank you for contacting！

What happens when you disable garbage collection?

I disabled it but the problem persists
php.ini settings

zend.enable_gc => Off => Off

However, there was a way to resolve this. mb_convert_encoding and str_replace also allow arrays.

Processing speed is improved by using arrays instead of strings

            $stream = fopen('php://output', 'w');

            header("Content-Type:text/csv;charset=UTF-8");
            header("Content-Disposition:attachment;filename=" . "text.csv");

            foreach ($data as $record) {

                $values = array_values((array)$record);

                $errorValues = array_diff($values, mb_convert_encoding(mb_convert_encoding($values, 'SJIS-win', 'UTF-8'), 'UTF-8', 'SJIS-win'));

                if (count($errorValues) > 0) {

                    $notExistsUnicodeValues[] = implode(",", $errorValues);
                } else {

                    fputcsv($stream, str_replace(array("\r\n", "\r", "\n"), " ", $values), ',', '"');
                }
            }

            fclose($stream);