mb_strtoupper converts non-alphabetic characters in Shift_JIS (CP932)

[masakielastic]

Description

mb_strtoupper converts non-alphabetic characters in Shift_JIS (CP932)

var_dump(
    "\x81\xE0" === mb_strtoupper("\x87\x90", 'cp932')
);

\x81\xE0 and \x87\x90 mean ≒ (U+2252: Approximately Equal to or the Image Of).

var_dump(
    "\u{2252}" === mb_convert_encoding("\x81\xE0", 'utf-8', 'cp932'),
    "\u{2252}" === mb_convert_encoding("\x87\x90", 'utf-8', 'cp932')
);

\x87\x90 is the one of NEC special characters and was registered in duplicate for historical reasons.

The unintended conversion is caused by a round-trip conversion between Unicode and Shift_JIS.

var_dump(
    "\x81\xE0" === roundtrip("\x87\x90", 'cp932'),
    "\x81\xE0" === roundtrip("\x81\xE0", 'cp932')
);

function roundtrip($char, $enc) {
    return mb_convert_encoding(mb_convert_encoding($char, 'utf-8', $enc), $enc, 'utf-8');
}

As far as I know, 398 characters in Shift_JIS are affected by round-trip conversion. The test code is here.

The same problem applies to mb_scrub, mb_strtolower, mb_convert_case.

var_dump(
    "\x81\xE0" === mb_scrub("\x87\x90", 'cp932'),
    "\x81\xE0" === mb_strtolower("\x87\x90", 'cp932'),
    "\x81\xE0" === mb_strtoupper("\x87\x90", 'cp932'),
    "\x81\xE0" === mb_convert_case("\x87\x90", MB_CASE_TITLE, 'cp932')
);

PHP Version

PHP 8.2.10

Operating System

Debian 11.7 (Google ChromeOS 117.0)

[zonuexe]

CP932 and these issues are also explained in the MySQL documentation.

• MySQL :: MySQL 8.1 Reference Manual :: 10.10.7.1 The cp932 Character Set

The MSDN goglobal page referenced from this page is a broken link, but can be found in the Web Archive.

• NEC special characters (lead byte 0x87):
  • https://web.archive.org/web/20130516080646/https://msdn.microsoft.com/en-us/goglobal/gg674964
• NEC selected—IBM extended characters (lead byte 0xED and 0xEE):
  • https://web.archive.org/web/20160329063327/https://msdn.microsoft.com/en-us/goglobal/gg671837
  • https://web.archive.org/web/20160329035459/https://msdn.microsoft.com/en-us/goglobal/gg671838
• IBM selected characters (lead byte 0xFA, 0xFB, 0xFC):
  • https://web.archive.org/web/20160329114926/https://msdn.microsoft.com/en-us/goglobal/gg671839
  • https://web.archive.org/web/20160329061434/https://msdn.microsoft.com/en-us/goglobal/gg671840
  • https://web.archive.org/web/20160329090150/https://msdn.microsoft.com/en-us/goglobal/gg671841

[zonuexe]

This code is consistent across all PHP versions, so it is not related to the recent mbstring overhaul.

<?php

var_dump(array(
    'mb_scrub' => function_exists('mb_scrub') ? "\x81\xE0" === mb_scrub("\x87\x90", 'cp932') : 'undefined',
    'mb_strtolower' => "\x81\xE0" === mb_strtolower("\x87\x90", 'cp932'),
    'mb_strtoupper' => "\x81\xE0" === mb_strtoupper("\x87\x90", 'cp932'),
    'mb_convert_case' => "\x81\xE0" === mb_convert_case("\x87\x90", MB_CASE_TITLE, 'cp932')
));

https://3v4l.org/O3jAS

[masakielastic]

mb_convert_kana and mb_encode_numericentity are also affected by this issue.

// ASCII range: 0x00..0x7F
$convmap = array(0x00, 0x7F, 0, 0xFF);

var_dump(array(
    'mb_convert_kana' => "\x81\xE0" === mb_convert_kana("\x87\x90", 'KV', 'cp932'),
    'mb_encode_numericentity' => "\x81\xE0" === mb_encode_numericentity("\x87\x90", $convmap, 'cp932')
));

https://3v4l.org/EB3tV

[masakielastic]

Sorry, I forget mb_decode_numericentity.

// ASCII range: 0x00..0x7F
$convmap = array(0x00, 0x7F, 0, 0xFF);

var_dump(array(
    'mb_decode_numericentity' => "\x81\xE0" === mb_decode_numericentity("\x87\x90", $convmap, 'cp932')
));

https://3v4l.org/nhosB

[iluuu1994]

/cc @alexdowad

[alexdowad]

@masakielastic, Thank you for sharing your findings here.

The behavior you have observed is inherent to mbstring's design. Internally, it converts all strings to Unicode codepoints before processing them. This means that inconsistencies like what you have observed will occur with all legacy encodings which are cannot be losslessly round-tripped to Unicode and back.

If you have a proposal on how to improve this behavior, please feel free to share it. Otherwise, this issue can be closed as a known limitation of mbstring.