The behavior of mb_strcut in mbstring has been changed in PHP8.1

[pakutoma]

Description

When using the mb_strcut function to cut out ISO-2022-JP encoded strings, depending on the number of bytes specified, the behavior is different in PHP 8.1 than before.

The following code:

<?php
$input = 'あaいb';
$bytes_length = 10;
$encoding = "ISO-2022-JP";
$converted_str = mb_convert_encoding($input, $encoding, mb_internal_encoding());
$cut_str = mb_strcut($converted_str, 0, $bytes_length, $encoding);
$reconverted_str = mb_convert_encoding($cut_str, mb_internal_encoding(), $encoding);
var_dump($reconverted_str);

Resulted in this output:

string(5) "あa?"

But I expected this output instead:

string(4) "あa"

This behavior has changed since PHP 8.1.
3v4l: https://3v4l.org/FaVWR

I believe the behavior up to PHP 8.0 is correct.
This is because PHP 8.1 and later output contains a "?", which is not supposed to be there.

PHP Version

PHP 8.1.10

Operating System

No response

[cmb69]

Although the behavior is not documented exactly, this looks like a bug. We likely want to truncate partial characters (code points) at the end. This apparently works nicely for UTF-8, see e.g. https://3v4l.org/oZgCU.

[youkidearitai]

Hi, I have a question. mb_strcut is unaffected of mb_substitute_character that manual written by below.

This setting affects mb_convert_encoding(), mb_convert_variables(), mb_output_handler(), and mb_send_mail().

However, there are other function affected by mb_substitute_character (e.g.: mb_convert_case https://3v4l.org/J7kVC )

For this issue, the 10th byte of 'あaいb' is 0x1b, meaning the escape sequence, then converted ?. (3v4l: https://3v4l.org/Intbd )
I'm curious why mb_substitute_character doesn't work.

If this other subject, then I open new feature request.

[NathanFreeman]

@youkidearitai Hi.
I think then converted ? is a valid character now so mb_substitute_character doesn't work.

[youkidearitai]

@NathanFreeman Thank you for reply.
My question was "Why mb_strcut is doesn't work of mb_substitute_character". But I understand that mb_substitute_character originally (since PHP 4.3.0 https://3v4l.org/s0gbn#v430) doesn't work.

Although the question remains why mb_convert_case is affected by mb_substitute_character which isn't in the manual(3v4l: https://3v4l.org/2lqgg#v430), this is another issue, thank you for your time.

[alexdowad]

@youkidearitai Can you read C code? I would be happy to point you to the lines in the source code which would answer your question. Or do you just want a high-level answer which does not require delving into the details of the implementation?

@pakutoma Thanks for reporting this issue. It is nice to see that @NathanFreeman is working on a fix. I will try to guide him so he can complete this successfully.

There are some very tricky issues involved with the implementation of mb_strcut, but I think we can work them out.

Just for information, the unexpected change in behavior which you have kindly reported was (indirectly) caused by 4299e2d.