[libc++] `<regex>`: Character class `[\W\D]` fails to match alphabetic characters

[muellerj2]

The (ECMAScript) regular expression [\W\D] describes a character class that matches the union of (a) all non-alphanumeric characters and (b) all non-digits. So effectively, the character class should be equivalent to [\D] and thus match all non-digits. However, libc++'s regex implementation only matches non-alphanumeric characters.

Test case:

#include <iostream>
#include <regex>

using namespace std;

int main()
{
    regex re(R"([\W\D])");
    cout << "matches alphabetic: " << regex_match("a", re) << '\n'
         << "matches digit: " << regex_match("0", re) << '\n' 
         << "matches non-alphanumeric: " << regex_match(".", re);
    
    return 0;
}

https://godbolt.org/z/YdvY4Pb6a

This prints:

matches alphabetic: 0
matches digit: 0
matches non-alphanumeric: 1

But it should print (as MSVC STL and libstdc++ do here):

matches alphabetic: 1
matches digit: 0
matches non-alphanumeric: 1

The problem lies here:

llvm-project/libcxx/include/regex

Lines 2139 to 2141 in 215c0d2

	_LIBCPP_HIDE_FROM_ABI void __add_neg_class(typename regex_traits<_CharT>::char_class_type __mask) {
	__neg_mask_ |= __mask;
	}

The negated character classes are bitwise or'ed, but De Morgan's law says that (not w) or (not d) = not (w and d), so the bit masks should really be bitwise and'ed.

But bitwise and'ing is problematic as well, because the standard only provides a guarantee that bitwise or'ing works, but doesn't state that bitwise and'ing corresponds to the intersection of the character classes (see [re.grammar/9]). Maybe and'ing will still work for libc++'s std::regex_traits<char> and std::regex_traits<wchar_t> traits classes (although I haven't checked that), but it might not do the right thing for some user-provided traits classes.

[frederick-vs-ja]

But bitwise and'ing is problematic as well, because the standard only provides a guarantee that bitwise or'ing works, but doesn't state that bitwise and'ing corresponds to the intersection of the character classes (see [re.grammar/9]). Maybe and'ing will still work for libc++'s std::regex_traits<char> and std::regex_traits<wchar_t> traits classes (although I haven't checked that), but it might not do the right thing for some user-provided traits classes.

I think bitwise and'ing is fine as __add_neg_class only plays with constants from ctype_base.

I agree on that __add_neg_class isn't doing right thing for user-provided regex trait classes because a user-provided traits possibly uses different constants. However, a large surgery seems necessary for this. It's unclear to me what a canonical customized trait class which is sufficiently different from std::regex_traits would be like.

[muellerj2]

I think bitwise and'ing is fine as __add_neg_class only plays with constants from ctype_base.

And'ing is also not guaranteed to work for ctype_base. For example, if you use MSVC's STL (which inherits the bitmasks from the underlying Win32 API), then and'ing xdigit and digit yields 0. (I haven't checked if there are similar issues in libc++'s ctype_base bitmasks.)

It's unclear to me what a canonical customized trait class which is sufficiently different from std::regex_traits would be like.

I think the most likely case is that a custom traits class reassigns some escapes to some other character classes for convenience. For example, \S could be remapped to "not a hexadecimal digit" by a custom traits class, while \D stays the same. Then [\S\D] should be "not a digit", but and'ing the masks will match "not nothing", so "everything", on Windows at least.