support Unicode grapheme clusters

[tbu-]

The regex engine doesn't consider characters (graphemes) that consist of multiple code points correctly.

For example the letter 'ä' has two representations, that should both be matched by the regex ., howver only the latter is.

Bash                 | Rust       | Codepoints
echo $'\x61\xcc\x88' | "\u{e4}"   | U+00e4
echo $'\xc3\xa4'     | "a\u{308}" | U+0061 U+0308

[BurntSushi]

Yup. The regex crate assumes a single character is a Unicode scalar value.

Do you know of other regex libraries that handle graphemes correctly?

[kwantam]

@tbu- one way to handle this is to apply a normalization to strings before matching. For example, converting a string to NFC form (or was it NFKC? See UAX#15 and the unicode-normalizations crate) should convert equivalent representations like the one you've listed to a single, canonical form. Then, of course, your regex would have to match against the canonical version of the character.

[tbu-]

@kwantam I believe that this still does not work for the . pattern which should match a single grapheme (which might well be more than one codepoint, even after normalization).

[kwantam]

@tbu- UAX#18§2.2 discusses matching grapheme clusters. The rule here, RL2.2, does not say that . should match a grapheme cluster.

For example, an implementation could interpret \X as matching any
extended grapheme cluster, while interpreting "." as matching any
single code point. It could interpret \b{g} as a zero-width match
against any extended grapheme cluster boundary, and \B{g} as
the negation of that.

I agree that it would be nice to have \b{g} (and \b{w}, \b{l}, and \b{s}) defined. I haven't yet implemented UAX#14 or the UAX#29 sentence break boundary algorithms in the unicode-segmentation crate, but it's on the roadmap. At that point, it would be pretty straightforward to implement \X as .+?\b{g}, as discussed in UAX#18§2.2.

As regards other languages, the perlunicode man page has a nice summary of perl's support for UAX#18. One assumes if perl's implementation is missing it, most everyone else's is, too, and indeed they do not support RL2.2 at least as of perl5.20.1.

[tbu-]

Oh, I incorrectly assumed it would be described in "UAX#18§2.2.1 Grapheme Cluster Mode".

[kwantam]

Nothing incorrect about that; it would be nice to have that mode available, too! :)

(And, I think, once RL2.2 is implemented, it would be pretty easy to add a switch for grapheme cluster mode.)

[BurntSushi]

I changed the title of this issue because this crate doesn't support grapheme clusters, so what you're seeing is intended behavior.

I actually don't know much about graphemes and how they're encoded, but if they can be made to fit into the Input abstraction, then it should be relatively easy to actually experiment with this. (The representation of InputAt would have to change.)

[BurntSushi]

While I'm not against potentially adding this, I do think it would be a mighty big undertaking and it's not quite clear it could be feasibly done in the DFA. If someone wanted to move forward on working on this, then we can re-open this issue. Before writing any code, I would like to see a plan for how it would be added to the DFA though and at least some analysis on the memory required.

[Trolldemorted]

Any chance for grapheme cluster support in the near future? One of our projects requires that we match them, and I really would like to stick with rust and not switch to python, which unfortunately can do it:

import regex

haystack = "🏳️‍🌈F́́̂꣩⃧꣩ᷭ⳱꣪L⃒̲̀̀̀ᷛᷩ⃰᷃À̀̀̀꣢᪱̅̾ᷥG︪̯̰̀̀̀̀ͭ̇"
needle = "🏳️‍🌈\\X\\X\\X\\X"

foo = regex.search(needle, haystack)
print(foo)

prints

PS C:\Users\Benni\repositories\graphemeclusters> python .\sample.py
<regex.Match object; span=(0, 44), match='🏳️\u200d🌈F́́̂꣩⃧꣩ᷭ⳱꣪L⃒̲̀̀̀ᷛᷩ⃰᷃À̀̀̀꣢᪱̅̾ᷥG︪̯̰̀̀̀̀ͭ̇'>

[BurntSushi]

No, I don't see it happening. You can write the regex yourself if you want, which is here: https://github.com/BurntSushi/bstr/blob/master/scripts/regex/grapheme.sh

The regex is quite gnarly though, but that's exactly what this crate would do if it supported \X. My guess is that it will be pretty slow, but you can try it.

If you describe the high level problem you're trying to solve, then there are almost certainly other approaches you could take here. It would be pretty surprising to me to switch your entire programming language based on this one thing.

You could also use the pcre2 crate, which I believe supports the \X syntax explicitly.