support for unicode characters

[bobzhang]

let h = [':ship:', '世' , '界']

Assuming the source code is encoded in utf8, these characters should be parsed as Const_int (code_point).
I suggest we enforce that all source code must be encoded in utf8 which is similar to other languages like golang

[jfrolich]

How would you handle the difference in encoding in the file (utf-8) and make it compatible with the representation in the JS engine (utf-16)? The above example would be a single utf-8 character but in the JS runtime it would be represented as two characters as emojis don't fit the 16 bits.

[bobzhang]

there is a mis-conception. The source encoding is not relevant to string encoding, we choose utf8 just to follow the existing conventions, we do this which is exactly the same as ts/js: whose source are encoded in utf8 yet the string are utf16.
In rescript char will be int which is large enough for any code point.

[IwanKaramazow]

@jfrolich if you're interested you can read more about it here: https://262.ecma-international.org/5.1/#sec-6

If an actual source text is encoded in a form other than 16-bit code units it must be processed as if it was first converted to UTF-16.

The unicode topic is a bit confusing as there can be a difference between your javascript source file encoding and the encoding of strings at runtime. When you send JS files encoded as UTF-8 (which is for example the default when I open a JS file with VSCode) and send the appropriate headers, then the browser will decode the UTF-8.

@bobzhang will take care of this 👍

[jfrolich]

there is a mis-conception. The source encoding is not relevant to string encoding, we choose utf8 just to follow the existing conventions, we do this which is exactly the same as ts/js: whose source are encoded in utf8 yet the string are utf16.
In rescript char will be int which is large enough for any code point.

I didn't say anything that contradicts what you are saying here, and I agree what you are saying. The javascript engine will transparently encode the file and external inputs from utf-8 to utf-16 (internal representation).

The the thing to reconcile is character size (a rescript char as shown above is 4-byte but a js char is always two byte). This also means that the emoji as stated above will be represented as two characters in JavaScript. This will cause issues when calculating the length of the string, or iterating over characters of a string. So if you convert this list of chars back to a string, and count the number of characters it now has 6 characters (I think Chinese characters also fall outside of the 2 byte range, but not sure), iteration will also result in 6 characters to be iterated over.

Perhaps @bobzhang has a solution already for this, like a runtime conversion, but that was what I was alluding to.

[IwanKaramazow]

Would it be possible to give a concrete example of what you're trying to do? Forgive me, but I always get infinitely confused about what length and character means in the context of encoded strings.

Let's say you have a plain JS file in UTF8 with an emoji.

It is logical that x.length and x[i] will be wrong here. Wouldn't you write your custom function to get the length and characters in JS?

We could compile all this info into a forum post one day 🙌

[bobzhang]

@jfrolich I am not sure if I follow you, we are just doing things exactly the same as ts/js to avoid an abstraction layer for such primitive types. we are not saying we are doing unicode right, we are aiming to be the same as JS

[bloodyowl]

How would it work with emojis "composition", such as this one 👨‍👩‍👦? Would writing it as a char be a syntax error?

[bobzhang]

Writing as a char would be a syntax error, a char is a Unicode code point

[jfrolich]

Would it be possible to give a concrete example of what you're trying to do? Forgive me, but I always get infinitely confused about what length and character means in the context of encoded strings.

Let's say you have a plain JS file in UTF8 with an emoji.

It is logical that x.length and x[i] will be wrong here. Wouldn't you write your custom function to get the length and characters in JS?

We could compile all this info into a forum post one day 🙌

Yes I was talking about @hongbo example because he is using a char type. In a string this is absolutely correct behavior, x.length in JS also gives the utf16 length and not the real length in characters. That is just an inconvenience of the platform.

@jfrolich I am not sure if I follow you, we are just doing things exactly the same as ts/js to avoid an abstraction layer for such primitive types. we are not saying we are doing unicode right, we are aiming to be the same as JS

I agree that it is correct behavior with strings, but this becomes problematic if you have a char type as in the example. How would you handle chars here?

[jfrolich]

Writing as a char would be a syntax error, a char is a Unicode code point

Which unicode though? Utf-8 (outside world: files, network) or Utf-16 (internal representation).

[bobzhang]

Unicode is standardized which does not depend on the encoding method. For source code we will use utf8. This is all the same approach used by js/ts

[jfrolich]

Unicode is standardized which does not depend on the encoding method. For source code we will use utf8. This is all the same approach used by js/ts

So you'll have conversion functions between array of int32 codepoints to strings and the other way around? @bloodyowl's comments are interesting, because a single utf-8 character can actually be more than 4 byte (up to 17 byte) because it can be composed of multiple unicode scalars. But I guess that can be out of scope for the character type?

[jfrolich]

Unicode is standardized which does not depend on the encoding method.

Javascripts unicode codepoint units are 16-bit, so for an emoji you need two codepoint units while in utf-8 it fits in a single unit (or 4 bytes). (Relevant because if you start iterating over strings in JS you iterate over units instead of characters).

[bobzhang]

Let’s keep the discussion focused on the implementation details. We can discuss more about Unicode on the forum

[jfrolich]

Yes I agree. I am not keen in talking unicode in general at all. This is about how rescript is planning to reconcile the char type in the current proposal (four byte code points, because you used a four byte emoji as a char), and the javascript utf16 two byte codepoint units.