Unicode input on windows command line does not work

[retep998]

When using stdin with my rust program run from either the windows command line or git bash, any unicode characters are turned into question marks.
So "Алло" becomes [63, 63, 63, 63].
If I change the codepage of the windows command line to utf-8 via chcp 65001, then when I enter in any unicode characters stdin returns EndOfFile.

On the flip side, output using libgreen works fine (because libuv translates utf-8 to utf-16 and calls WriteConsoleW), while output using libnative produces garbage for unicode because it writes utf-8 as if it were ascii.

Using windows 8.1 x64, with 32-bit Rust nightly.

[retep998]

For anyone saying that windows command line does not support unicode, here is some quick code I wrote in C++ which correctly reads and writes unicode from and to the console:

wstring read() {
    auto in = GetStdHandle(STD_INPUT_HANDLE);
    array<wchar_t, 0x1000> buf;
    unsigned long read;
    ReadConsoleW(in, buf.data(), buf.size(), &read, nullptr);
    return{buf.begin(), buf.begin() + read};
}

void write(wstring p_str) {
    auto out = GetStdHandle(STD_OUTPUT_HANDLE);
    unsigned long wrote;
    WriteConsoleW(out, p_str.c_str(), p_str.size(), &wrote, nullptr);
}

int main() {
    write(read());
}

Here is a screenshot of that code working:

[retep998]

A patch for unicode input in libuv is currently going through the process, which will fix libgreen's unicode input.

joyent/libuv#1323

[retep998]

Currently working on writing up a patch for libnative unicode console support. Output works so far:

Still working on input.

[retep998]

Created a PR to address the issue in libnative: #15051

The libuv patch has been merged into libuv master, so that's waiting for libuv to release a new version and then rustuv to update to that new version

[retep998]

Since libnative has been fixed by #15051 and librustuv no longer exists, this issue is nearly done. Code still needs to be written to handle multi-codeunit codepoints broken across reads/writes.