Problems in token pasting

[kyuupichan]

The token pasting code gets several tricky cases wrong. Consider the following for c++23:

#define concat1 \ ## Udeadbeef
concat1
#define concat2 R ## "averylongdelimiter()averylongdelimiter"
concat2
#define concat3 __VA_ARGS_ ## _
concat3
#define concat4 L'b' ## ud_suffix
concat4
#define concat5 R ## "aわ()aわ"
concat5

concat1 is invalid because the UCN is not a valid codepoint. Clang accepts it.
concat2 should diagnose the long raw literal delimiter. Clang does not.
concat3 should complain about an invalid use of __VA_ARGS__. Clang does not.
concat4 is a wide character literal with a user-defined suffix, clang rejects it saying a space is required. I believe it is valid.
concat5 contains an invalid character in the raw string delimiter, clang accepts it.

I used clang -E --std=c++23

$ clang --version
Apple clang version 16.0.0 (clang-1600.0.26.6)
Target: arm64-apple-darwin24.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[EugeneZelenko]

Could you please try LLVM's (not Apple) 20 main branch? https://godbolt.org should be helpful.

[kyuupichan]

Tested - it makes no difference

[frederick-vs-ja]

I think Clang correctly rejects some case despite the error messages not being good.

#define WRONG_ID \ ## Udeadbeef
auto WRONG_ID = 0;

#define RIGHT_ID \ ## U00008753
auto RIGHT_ID = 0;

Godbolt link

#define WRONG R ## "averylongdelimiter()averylongdelimiter"
const char arr2[]{WRONG};

#define WRONG2 R ## "aわ()aわ"
const char arr5[]{WRONG2};

#define RIGHT R ## "delimiter()delimiter"
const char arr3[]{RIGHT};

Godbolt link

---

And this is rejected with suitable message. Godbolt link.

#define concat3 __VA_ARGS_ ## _
concat3

---

An adjusted example of concat4 is correctly accepted. Godbolt link.

consteval auto operator""_ud_suffix(wchar_t c) noexcept { return c; }

#define concat4 L'b' ## _ud_suffix
constexpr auto c = concat4;

[kyuupichan]

So these are only caught in the front-end and not the preprocessor? It seems like a bit of a layering violation, which is perhaps why the diagnostics are poor. I think the 4 malformed ones should all be caught with -E.

concat4 is rejected by -E, so it's impossible to preprocess and then compile. I cannot understand how the compiler can accept something the preprocessor rejects? It seems very weird.

https://godbolt.org/z/rTheKoaPY

[shafik]

CC @cor3ntin @tahonermann