Better coalesce adjacent scalars #574
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
milseman
reviewed
Jul 14, 2022
hamishknight
force-pushed
the
better-together
branch
from
73ebb61 to
a32f08c
Compare
|
@rctcwyvrn mind taking a look at 203868f? |
hamishknight
changed the title
hamishknight
force-pushed
the
better-together
branch
2 times, most recently
from
ef29854 to
3595070
Compare
hamishknight
force-pushed
the
better-together
branch
from
3595070 to
3995e96
Compare
Previously we would emit a series of scalars written in the DSL as a series of individual characters in grapheme semantic mode. Change the behavior such that we coalesce any adjacent scalars and characters, including those in regex literals and nested concatenations. We then perform grapheme breaking over the result, and can emit character matches for scalars that coalesced into a grapheme. This transform subsumes a similar transform we performed for regex literals when converting them to a DSLTree. This has the nice side effect of allowing us to better preserve scalar syntax in the DSL transform. rdar://96942688
Previously we would only match entire characters. Update to use the generic Character consumer logic that can handle scalar semantic mode. rdar://97209131
In grapheme semantic mode, coalesce adjacent character and scalar members of a custom character class, over which we can perform grapheme breaking. This involves potentially re-writing ranges such that they contain a complete grapheme of adjacent scalars.
Make sure we throw the right error for ranges that are invalid in grapheme mode, but are valid in scalar mode.
I also noticed that `lexQuantifier` could silently eat trivia if it failed to lex a quantification, so also fix that.
hamishknight
force-pushed
the
better-together
branch
from
3995e96 to
dc4171f
Compare
milseman
approved these changes
Jul 26, 2022
|
@swift-ci please test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously we would only coalesce adjacent scalars in regex literals outside of custom character classes. Change the behavior such that we start coalescing scalars inside custom character classes in grapheme mode, e.g
[e\u{301}]matchese\u{301}, and start coalescing adjacent scalars in DSL.Previously for the DSL we would emit a series of scalars as a series of individual characters in grapheme semantic mode. Change the behavior such that we coalesce any adjacent scalars and characters, including those in regex literals and nested concatenations. We then perform grapheme breaking over the result, and can emit character matches for scalars that coalesced into a grapheme.
This transform subsumes a similar transform we performed for regex literals when converting them to a DSLTree. This has the nice side effect of allowing us to better preserve scalar syntax in the DSL transform.
Resolves #572 (rdar://96942688)
Resolves #586 (rdar://97209131)
Resolves #573