Writing valid UTF-8 to the Windows terminal using stdout().write() can incorrectly return an error

[Count-Count]

I tried this code:

use std::io::Write;

fn standard_write_all<W: std::io::Write>(mut w: W, mut buf: &[u8]) -> Result<(), std::io::Error> {
    while !buf.is_empty() {
        match w.write(buf) {
            Ok(0) => {
                return Err(std::io::Error::new(
                    std::io::ErrorKind::WriteZero,
                    "failed to write whole buffer",
                ));
            }
            Ok(n) => buf = &buf[n..],
            Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {}
            Err(e) => return Err(e),
        }
    }
    Ok(())
}

fn main() {
    let poison = "Text here does not matter.\n".to_owned() + "x".repeat(1024 - 1).as_str() + "😀";

    // The special stdout().write_all() impl works.
    std::io::stdout().write_all(poison.as_bytes()).unwrap();
    println!();
    println!();

    // A standard write_all() implementation fails.
    //
    // 1. First write call
    // The bytes up to the newline are passed through to the console in one write.
    //
    // The next 1023 b'x' bytes and the first byte of "😀" are buffered leading to
    // an incomplete codepoint at the end of the buffer. The buffer is now poisoned
    // meaning that the next write() call will fail even if the missing UTF-8 bytes
    // are supplied.
    //
    // 2. Second write call
    // The buffer is flushed completely on the next call, the console writes 1023 b'x' bytes
    // and is then forced to write the remaining byte from the incomplete UTF-8 sequence.
    standard_write_all(std::io::stdout(), poison.as_bytes()).unwrap();
}

I expected to see this happen: Program does not panic.

Instead, this happened: Program panics with the error Windows stdio in console mode does not support writing non-UTF-8 byte sequences.

Meta

rustc --version --verbose:

rustc 1.50.0 (cb75ad5db 2021-02-10)
binary: rustc
commit-hash: cb75ad5db02783e8b0222fee363c5f63f7e2cf5b
commit-date: 2021-02-10
host: x86_64-pc-windows-msvc
release: 1.50.0

Backtrace

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidData, error: "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" }', src\main.rs:35:65
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\std\src\panicking.rs:493
   1: core::panicking::panic_fmt
             at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\core\src\panicking.rs:92
   2: core::option::expect_none_failed
             at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\core\src\option.rs:1268
   3: core::result::Result<tuple<>, std::io::error::Error>::unwrap<tuple<>,std::io::error::Error>
             at C:\Users\hans\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\result.rs:973
   4: termcolorbug::main
             at .\src\main.rs:35
   5: core::ops::function::FnOnce::call_once<fn(),tuple<>>
             at C:\Users\hans\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:227
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Update

To clarify: This problem occurs with all Write trait implementations that don't override the default Write::write_all() implementation, e.g. with this simple DelegatingWrite:

use std::io::Write;

struct DelegatingWrite<W> {
    delegate: W,
}

impl<W: Write> DelegatingWrite<W> {
    fn new(delegate: W) -> DelegatingWrite<W> {
        DelegatingWrite { delegate }
    }
}

impl<W: Write> Write for DelegatingWrite<W> {
    fn write(&mut self, b: &[u8]) -> std::io::Result<usize> {
        self.delegate.write(b)
    }

    fn flush(&mut self) -> std::io::Result<()> {
        self.delegate.flush()
    }
}

fn main() {
    let poison = "Text here does not matter.\n".to_owned() + "x".repeat(1024 - 1).as_str() + "😀";
    let mut w = DelegatingWrite::new(std::io::stdout());
    write!(w, "{}", poison).unwrap(); // <-- panics
}

[Count-Count]

I see two distinct possibilites to resolve this:

1. Treat incomplete write()s not ending on a codepoint boundary as a bug and fix that in the stdlib. Also clarify in the docs that indivual write() calls must be passed complete UTF-8 sequences if a Windows terminal is attached.

2. In the vain of "Be liberal in what you accept" handle incomplete UTF-8 passed in via write() calls appropriately, as long as the sequence is completed properly in the next write() call(s).

[ChrisDenton]

Hm, so this is ultimately a bug in the buffer std::io::stdout() uses? It attempts to write 1024 bytes but only 1023 bytes can written in the first OS call. It then has a leftover byte, which I guess it tries to write by itself thus causing this error?

[Count-Count]

@ChrisDenton The logic which errors if incomplete UTF-8 is passed in at index 0 is in the std lib at sys/windows/stdio.rs. That is where the error message comes from.

As far as I can see the stdio buffer implementation (in stable, it is in linewritershim.rs, apparently currently being revamped) can not know of the requirement that only complete UTF-8 may be written to the underlying Windows terminal so I think that it should be encapsulated in sys/windows/stdio.rs by just accepting and buffering incomplete UTF-8 when necessary.

[ChrisDenton]

I mean, a generic buffer can attempt to avoid small writes (say, less than 16 bytes) by copying those left over bytes to the front of the buffer and waiting for either another write to refill the buffer or a flush to empty it.

But I see what you mean. Introducing a small buffer to windows/stdio would fix the problem.