Chinese numerals are not recognized by char::is_numeric

[wooster0]

I tried this code:

fn main() {
    assert!('一'.is_numeric());
}

I expected it to evaluate to true.

Instead, it evaluated to false.

I would expect at least 零/〇、一、二、三、四、五、六、七、八、九 (0-9) to be recognized. As for other numeral systems, like the Arabic numerals, after 9 the number wouldn't fit into a char anymore and thus can't be recognized, but with Chinese numerals, beyond 0-9 there's many other numbers represented with a single character too, like for example 10: 十, which could still be a char. I'm not sure whether this should be recognized, but perhaps it should.
There is also financial numbers and many others, see https://en.wikipedia.org/wiki/Chinese_numerals#Standard_numbers for a comprehensive list.

I've been told that, the numerals are covered in the UnicodeData.txt file mentioned in the docs of char::is_numeric, but they are listed in the Lo category which stands for Other Letter and so Rust doesn't consider them numeric, which doesn't make sense to me because clearly they are numerals and not letters. Rust should probably either recognize (some parts of) this category as numerals or the numerals should be added manually.

Adding support for this would in turn also mean support for numerals of other East Asian languages, like Japanese and Hokkien.

Meta

This happens on the stable 1.51.0 channel and all others.

[CryZe]

It seems like Unicode classifies it as a Lo / Letter Other with no numeric value. So I don't think this is a Rust specific problem and rather a problem with the Unicode standard (if even).

[ChrisDenton]

Unfortunately I think this would be a breaking change even if it is desirable. As you say, Unicode classifies them as "other letters, including syllables and ideographs" instead of one of the number categories (see Unicode categories). The Rust documentation says specifically that the code point must be in the Nd, Nl or No categories so any change to that would be breaking the current Rust API.

The only way I see around that is to do one of the following:

• Persuade Unicode to reclassify them for the next version of the Unicode specification.
• Create a new standard library function that isn't tied to the Unicode specification.
• Use a third pary crate to properly classify number regardless of the Unicode specification or Rust's std implementation. This could also handle number that take more than one code point.

[eggyal]

Whilst the General_Category of these characters is indeed Lo, their Numeric_Type property is Numeric. Perhaps that's what should be inspected instead, although as @ChrisDenton says this would be a breaking change to Rust.

[wooster0]

Whilst the General_Category of these characters is indeed Lo, their Numeric_Type property is Numeric. Perhaps that's what should be inspected instead

Then I suppose this is probably more of a Rust-side issue than Unicode's problem.

Persuade Unicode to reclassify them for the next version of the Unicode specification.

I contacted them about this.

Create a new standard library function that isn't tied to the Unicode specification.

What would this method be called? I believe it would cause more confusion than do good.

Yes, I suppose this will mean a breaking change, but is there not some kind of special exception for bugs? This is not a new feature but an important bug and correctness matters. I don't think we should leave this bug be just because it might break something. We should also consider how much of a breaking change this really is.

• I think what we can do maybe is to basically add a warning for some time to all is_numeric calls to tell the user about the incoming change and tell them that they should check if that doesn't break something for them?

But really, I don't know what this could actually break except for some very specific cases. I believe it would rather fix more things than break things.
Also, we have the next edition.

• Maybe this can make it into the next 2021 Edition? I think that makes this "breaking change" even more acceptable to be done. Maybe the warning for is_numeric calls can exist a few releases before the 2021 edition.
• Or perhaps when the user switches to the 2021 edition for the very first time (and that special behavior only exists in the first release of the new edition, or for a few more than that) there will be some kind of "special" dialog/message for is_numeric calls and the compiler tells the user to check those calls and make sure nothing breaks there?
  But perhaps that's a little too complicated and there's better ways, but I'm sure there's a very to get this change done.

[ChrisDenton]

What's been done before is for a new function to be created with a similar name and the old function be deprecated. Perhaps is_num, is_numeric_type, has_numeric_property or something else. Unless the libs team decides there's enough wiggle room to change the current function.

In either case the new functionality should, I think, be based on DerivedNumericType.txt which also tells how they should be derived from other Unicode data. DerivedNumericValues lists each value with the numeric value property.

[wooster0]

What's been done before is for a new function to be created with a similar name and the old function be deprecated. Perhaps is_num, is_numeric_type, has_numeric_property or something else.

If that's what the libs team decides for, how about is_numeral? In any case a new method name would mean less consistency with the other methods having numeric in their name such as is_alphanumeric and is_ascii_alphanumeric, so I hope it can be avoided and the existing method gets a change.

In either case the new functionality should, I think, be based on DerivedNumericType.txt which also tells how they should be derived from other Unicode data. DerivedNumericValues lists each value with the numeric value property.

That sounds good. And perhaps that doesn't just fix Chinese numerals not being detected but other specific cases too.

Let's wait for what the libs team thinks.

[workingjubilee]

In spite of being "breaking" changes, small fixes in the implementation of a type are permitted by RFCs 1105 and 1122, for the same reason that it would obviously behoove Rust to fix the compiler if rustc suddenly started solving 0 + 1 for -3. In saying such, I am not immediately weighing in on whether this is such an error in need of fixing, merely noting that in principle this could be deemed such a change.

[inquisitivecrystal]

@rustbot claim

It seems like everyone agrees that we either need a new API for this, or we need to change the existing one. It's for the libs team to decide which. In the meantime, there's some background work I can get started on.