stdio: Handle unicode boundaries better on Windows

[alexcrichton]

The stdio support on windows has two separate modes. One is when stdout is not a console and implies byte orientation, the other is when stdout is a console and implies [u16] orientation. In the stdout-is-a-console case we require that all input and output to be valid unicode so we can translate [u8] to [u16].

In this console case, however, our translation may not be quite right for a number of scenarios:

• When reading data, the console will give us a block of u16's which we need to translate to UTF-8. In theory the console can give us half of a surrogate pair (with the next half available on the next call to read), but we do not handle this case well currently. This should be fixed by simply adding a "buffer of size 1" to hold half of a surrogate pair if necessary.
• When reading data, we require the entire set of input to be valid UTF-16. We should instead attempt to read as much of the input as possible as valid UTF-16, only returning an error for the actual invalid elements. For example if we read 10 elements, 5 of which are valid UTF-16, the 6th is bad, and then the remaining are all valid UTF-16, we should probably return the first 5 on a call to read, then return an error, then return the remaining on the next call to read.
• When writing data, we require that the entire block of input is valid UTF-8. We should instead take a similar approach as above where we try to interpret as much of the input as possible as UTF-8, but we simply ignore the rest for that one call to read (returning an error if the first byte is invalid UTF-8).
• When writing data, we don't handle the case where a multibyte character straddles calls to write. Like the reading case, this could be alleviated with a 4-byte buffer for unwritten-but-valid-utf8 characters.
• When writing, we translate a block of valid UTF-8 to a block of UTF-16. Upon writing this UTF-16, however, not all characters may be written. There are two problems here:
  • We need to translate a length of UTF-16 characters to a number of UTF-8 bytes that were written.
  • If half of a surrogate pair is written, the other half should be "buffered" and the length of UTF-8 written should include the half-written character (as the extra data is buffered).

At this time I'm not sure what sort of guarantees Windows actually gives us on these APIs. For example does windows never deal with only half of a surrogate pair on a read/write? If Windows were to guarantee various properties like this (or that it always writes the entire buffer, for example), our lives would be a lot easier! For now, though, it is unclear what we can rely on, so this issue is written as if we rely on nothing.

[alexcrichton]

cc @retep998, @aturon

[alexcrichton]

The code for all this currently lives in src/libstd/sys/windows/stdio.rs, but it may move over time.

[retep998]

The Windows console just deals with an opaque sequence of WCHAR so it offers no guarantees regarding how it gives you surrogates.

[bluss]

Some notes on Stdin as it is today:

There seem to be some inefficiencies, the allocation of a 4 kB vector can possibly be turned into an array on the stack. A new String is then allocated again from that result.

The MemReader comment should go, it used to talk about an unwrap that's no longer there.

[alexcrichton]

As reported in #30399, an easy way to trigger this:

    let v = vec![255u8; 5000];
    let s = String::from_utf8_lossy(&v);
    // println!("Len {}", s.len());
    println!("{}", s);

[retep998]

I just ran into this issue while doing cargo list --tree in one of my projects presumably because it prints out some box drawing characters and one of them straddled a write boundary.

thread '<main>' panicked at 'failed printing to stdout: text was not valid unicode', ../src/libstd\io\stdio.rs:588

[luser]

This is a real bummer because you can't write non-UTF-8 data to stdout via the std::io APIs, which is not a limitation in C. The simplest example here would be a program that wants to write arbitrary binary data to stdout (expecting it to be piped somewhere), but I hit this today with my program that runs a command, captures its stdout, and then later writes that data to its own stdout. If the stdout from the command was not valid UTF-8 that fails. (The program in question was the Visual C++ installed with the French locale running in a Windows VM with the system locale set to US English.)

[luser]

I gather this code is written this way because we want to be able to write Rust strings (which can contain Unicode) to the console, but it's kind of limiting.

[alexcrichton]

@luser unfortunately (as I'm led to believe) that's actually a somewhat impossible operation on Windows. The captured output is all byte-oriented as that's how the stdout pipes are set up for a spawned program, but when printing that to a console (not a pipe) it's expected to be a u16-oriented stream, so a conversion needs to happen.

I'm not sure if there's a system API for writing bytes to a console, however, other than just the u16 units we're doing today.