span information is mistaken in the presence of combining characters

[msullivan]

rustc reckons spans in terms of unicode code points, which causes problems when faced with combining characters. When the following code is compiled

fn main() {
    let s = ~"ZͨA͑ͦ͒͋ͤ͑̚L̄͑͋Ĝͨͥ̿͒̽̈́Oͥ͛ͭ!̏"; while true { break; }
    io::println(s);
    log(error, s);
}

rustc will produce the following screwed up diagnostic:

./nubs/utf8-combining.rs:2:45: 2:66 warning: denote infinite loops with loop { ... }
./nubs/utf8-combining.rs:2     let s = ~"ZͨA͑ͦ͒͋ͤ͑̚L̄͑͋Ĝͨͥ̿͒̽̈́Oͥ͛ͭ!̏"; while true { break; }
                                                                        ^~~~~~~~~~~~~~~~~~~~~

emacs tells me that the while loop starts in column 23. (Interestingly, emacs seems to split up the combining characters when it displays them, presumably for easier editing, but doesn't advance the column numbers.)

[graydon]

Yeah. Fixing this will have to wait on us binding to libICU in libstd, most likely. It's beyond the scope of the unicode-isms that libcore understands.

[sanxiyn]

Apart from combining characters, "wide" characters should be considered to occupy 2 columns. See http://www.unicode.org/reports/tr11/ for details.

[msullivan]

This seems to be fixed.

[felixc]

Apologies for reviving this, but I think there's either been a mistake, a regression, or I've completely misunderstood the bug.

When I compile

use std::io;

fn main() {
    let s = ~"ZͨA͑ͦ͒͋ͤ͑̚L̄͑͋Ĝͨͥ̿͒̽̈́Oͥ͛ͭ!̏"; while true { break; }
    io::println(s);
}

I get the a more or less equivalent diagnostic:

main.rs:4:46: 4:67 warning: denote infinite loops with loop { ... }, #[warn(while_true)] on by default
main.rs:4     let s = ~"ZͨA͑ͦ͒͋ͤ̚L̄͑͋Ĝͨͥ̿͒̽̈́Oͥ͛ͭ!̏"; while true { break; }
                                                       ^~~~~~~~~~~~~~~~~~~~~

Crucially, the character span is still off; though it's now 46-67. Unless I'm missing something, it seems like this should be reopened?

[huonw]

It's open as #8706.

(The issue is we compute spans with codepoints, not graphemes.)

[felixc]

Ah; thank you! This FIXME sent me to the wrong place, then: https://github.com/mozilla/rust/blob/master/src/libsyntax/diagnostic.rs#L316