mb_check_encoding is detecting SJIS as Windows-1252 in PHP 8.1

[rtio]

Description

Works as expected in 7.4

The following code:

<?php

$string = mb_convert_encoding('パーティーへ行かないか', 'SJIS', 'UTF-8');

var_dump(mb_check_encoding($string, 'SJIS')); // should be valid
var_dump(mb_check_encoding($string, 'Windows-1252')); // should be invalid

var_dump(mb_detect_encoding($string, 'SJIS', true)); // should be valid
var_dump(mb_detect_encoding($string, 'Windows-1252', true)); // should be invalid

var_dump(mb_convert_encoding($string, 'UTF-8', 'SJIS')); // should be correct
var_dump(mb_convert_encoding($string, 'UTF-8', 'Windows-1252')); // should be wrong

Resulted in this output:

bool(true)
bool(true)
string(4) "SJIS"
string(12) "Windows-1252"
string(33) "パーティーへ行かないか"
string(43) "ƒp�[ƒeƒB�[‚Ö�s‚©‚È‚¢‚©"

But I expected this output instead:

bool(true)
bool(false)
string(4) "SJIS"
bool(false)
string(33) "パーティーへ行かないか"
string(46) "ƒp�[ƒeƒB�[‚Ö�s‚©‚È‚¢‚©"

Online example here

PHP Version

PHP 8.1.27

Operating System

CentOS 7

[divinity76]

Dude is correct. Minimal reproducible example:

var_dump(mb_check_encoding("\x81", "Windows-1252"));

erroneously prints true, it should print false.
There are 5 unused bytes in Windows-1252, specifically \x81, \x8D, \x8F, \x90, and \x9D, and

$string = mb_convert_encoding('パーティーへ行かないか', 'SJIS', 'UTF-8');

returns

$string = "\x83\x70\x81\x5B\x83\x65\x83\x42\x81\x5B\x82\xD6\x8D\x73\x82\xA9\x82\xC8\x82\xA2\x82\xA9";

which contains both \x81 and \x8D , it's not valid Windows-1252 and mb_check_encoding is malfunctioning here.

[youkidearitai]

According to https://en.wikipedia.org/wiki/Windows-1252#Codepage_layout , it is wrote to below:

According to the information on Microsoft's and the Unicode Consortium's websites, positions 81, 8D, 8F, 90, and 9D are unused; however, the Windows API MultiByteToWideChar maps these to the corresponding C1 control codes. The "best fit" mapping documents this behavior, too.[18]

Therefore, \x81, \x8D, \x8F, \x90, and \x9D can see valid character.

And, please check to #10060 and #8744 (comment).

[rtio]

I agree with @youkidearitai probably that is not the reason. But definitely, something is off with this function. We have both Windows-1252 and SJIS as acceptable encodings and this behavior makes things difficult.

[divinity76]

Therefore, \x81, \x8D, \x8F, \x90, and \x9D can see valid character.

by that definition mb_check_encoding($str, 'Windows-1252') is a no-op, it's literally impossible to construct a windows-1252-incompatible byte sequence: windows-1252 is binary safe 🤔

[divinity76]

by that definition mb_check_encoding($str, 'Windows-1252') is a no-op, it's literally impossible to construct a windows-1252-incompatible byte sequence: windows-1252 is binary safe 🤔

that seems to be the definition from PHP8.1.15 (but not 8.1.14) and PHP8.2.1 (but not PHP8.2.0) and 8.3.x yeah: https://3v4l.org/rVDlt

... that means mb_check_encoding($str, 'Windows-1252'); is just an alias for true , the check is doing absolutely nothing...

[alexdowad]

@rtio, thanks very much for letting us know the challenge you are having with mbstring functions in PHP 8.1+.

@youkidearitai has already kindly explained that mbstring's handling of Windows-1252 now follows the behavior of the Win32 API. I think what we need to know next is: What is the actual problem which you are experiencing? The code sample which you shared well illustrates that there is a difference in behavior between PHP 7.4 and PHP 8.1, and we are aware of that. But it would be good to get the details of exactly how that difference is impacting your application.

[alexdowad]

... that means mb_check_encoding($str, 'Windows-1252'); is just an alias for true , the check is doing absolutely nothing...

That's true, not just for Windows-1252, but for some other single-byte text encodings as well. KOI-8R is an example, but there are others.

In 6fc8d01, @pakutoma did add some plumbing to mbstring so we can make mb_check_encoding stricter than mb_convert_encoding when this makes sense. Examples of where it does make sense include ISO-2022-JP and JIS encoded text.

We would need a very good argument to justify the change, though. "I don't think that mb_check_encoding should ever be a no-op" isn't a strong argument.

[rtio]

@alexdowad my example is quite simple, we have a function that takes an input and attempts to convert it to UTF-8, if not already UTF-8.

The code below is a simplification of that function, but it is enough to understand why this behavior is offending the code:

<?php

// this is ordered by likelihood/priority
$accepted_encodings = [
    'UTF-8',
    'Windows-1252',
    'SJIS',
    'ISO-8859-1',
];

$to_encoding = 'UTF-8';
$input = 'foo';

foreach ($accepted_encodings as $from_encoding) {
    if (mb_check_encoding($input, $from_encoding)) {
        $result = mb_convert_encoding($string, $to_encoding, $from_encoding);
        break;
    }
}

// validade the result and do something with it

[alexdowad]

@rtio OK, excellent.

You have put 'Windows-1252' ahead of 'SJIS' in $accepted_encodings. Is this because your application receives Windows-1252 input more frequently than SJIS?

Do your users provide Windows-1252 input so frequently that it is 'safe' to accept any string without 0x81, 0x8D, 0x8F, 0x90, and 0x9D as being a Windows-1252 string??

[rtio]

Is this because your application receives Windows-1252 input more frequently than SJIS?
Do your users provide Windows-1252 input so frequently that it is 'safe' to accept any string without 0x81, 0x8D, 0x8F, 0x90, and 0x9D as being a Windows-1252 string??

Yes, there is a lot of legacy stuff out there

Do your users provide Windows-1252 input so frequently that it is 'safe' to accept any string without 0x81, 0x8D, 0x8F, 0x90, and 0x9D as being a Windows-1252 string??

We have several ways of injecting content, like widgets, bots, and third-party clients. We do support those 4 input encodings on that order of precedence.

We could change that but even though I see the reason for this behavior it makes the logic behind mb_check_encoding quite hard to understand, honestly.

[divinity76]

The good news is that validating cp1252 in userland is easy,

function is_valid_windows1252(string $str): bool {
 return strlen($str) === strcspn($str, "\x81\x8D\x8F\x90\x9D");
}