Remove "commit N does X" and "currently wrong / pre-fix" narrative from
test.cpp and locations.ql; where appropriate, replace with concise
process-free descriptions of the code (e.g. "Raw string literal;
content offset 3"). Line count and column positions of every regex
literal are preserved (empty lines take the place of removed comment
lines), so no .expected file changes. Remaining "Note:" comments in the
.qll files describe code semantics (excluded forms / handling), not the
build process, and are kept.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
cpp/ql/test/library-tests/regex/regexp.rb was the commit-1 verbatim
Ruby reference and has been fully replaced by test.cpp. Nothing in the
C++ regex tests references it. Deletion has no effect on .expected.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
Add u8"..." and u8R"(...)" cases to test_locations in test.cpp, guarded
by #if __cplusplus >= 201703L. Add an options file forcing
-std=c++17 so the u8 char8_t-era rows are actually extracted.
Plain "..." rows are unchanged. New u8/u8R rows in locations.expected
show correct columns computed by regexpContentOffset: offset 3 for
`u8"` and offset 5 for `u8R"(`. No change to regexpContentOffset was
needed.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
New cpp/ql/test/library-tests/regex/Consistency.ql selects, for each
corpus RegExp, characters that failedToParse. The empty
Consistency.expected asserts that no corpus regex has any unparsed
character, mirroring PR #22200's Consistency test as the primary
regression guard for the parser.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
POSIX bracket expressions ([:name:], [.x.], [=x=]) are only valid nested
inside another character class in std::regex. The three predicates now
additionally require that at position `start` we are inside an outer
character class, approximated by requiring more non-escaped `[` than
non-escaped `]` before `start`. A well-formed POSIX bracket contributes
one `[` and one `]` at/after `start`, so the check is unaffected by
earlier POSIX brackets.
Also stop emitting individual charSetTokens for characters inside a
POSIX bracket (`inAnyPosixBracket` guard), which removes the spurious
`[RegExpCharacterRange] :]-z` on [[:alpha:]-z].
Post-fix, unnested `[:digit:]`, `[:alpha:]`, and `a[🅱️]c` are correctly
parsed as ordinary character classes / literal sequences rather than
`RegExpNamedCharacterProperty`; and the [[:alpha:]-z] range no longer
crosses the POSIX bracket boundary.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
Add corpus cases: bare [:alpha:], mid-pattern a[🅱️]c, POSIX class as
range endpoint [[:alpha:]-z], malformed [[:alpha], leading literal ]
combined with a POSIX class []a[:alpha:]], three POSIX classes in one
class, additional names [[:xdigit:]] / [[:blank:]] / [[:cntrl:]] /
[[:graph:]], and the integration case combining all of the above.
The regenerated parse.expected reveals genuine mis-parses that this
commit deliberately records (fixes in Commit 6):
- unnested [:alpha:] and [🅱️] are treated as RegExpNamedCharacterProperty
even though POSIX brackets are only valid inside a character class,
- [[:alpha:]-z] emits a spurious [RegExpCharacterRange] ":]-z" instead
of a POSIX class + literal '-' + 'z'.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
In default ECMAScript std::regex (no multiline flag; this parser cannot
detect construction-site flags), ^ and $ anchor to string start/end, not
line boundaries. Update RegExpCaret, RegExpDollar, and RegExpAnchor doc
comments to describe string-start/string-end semantics and note that
multiline is not modelled, mirroring PR #22200. Tokenization unchanged.
.expected files unchanged.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
\U is a Ruby/PCRE-ism, not ECMAScript. Restrict
RegExpEscape.isUnicode()/getUnicode() to lowercase \uHHHH only. No \U
path exists in ParseRegExp.qll's escapedCharacter, and no corpus case
references \U. .expected files unchanged.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
In ECMAScript std::regex, \0 matches NUL. The previous escapedCharacter
arms all rejected it: the final arm's `not exists(getChar(start+1).toInt())`
guard fails for "0", and the numbered-backref arm excludes 0. Add an
explicit \0 case (end=start+2) guarded so the following character is not
a digit, mirroring EcmaRegExp.escapedCharacter in PR #22200. Add corpus
case "a\\0b" to test.cpp; \0 now parses as RegExpEscape.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
std::regex ECMAScript mode does not support \p{Name}; \p tokenizes as an
identity escape. Remove pStyleNamedCharacterProperty and its call sites;
namedCharacterProperty now covers only POSIX brackets ([:name:], [.x.],
[=x=]) and namedCharacterPropertyIsInverted keeps only the [[:^name:]]
case (fixing the offset from start+3 to start+2 for POSIX). Update
RegExpNamedCharacterProperty getName/isInverted docs to describe POSIX
brackets. Remove the four \p{...}/\P{...} corpus lines from test.cpp;
keep a plain [a-f\d]+ case for the class-with-escape shape.
Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).
Add posixCollatingSymbol([.x.]) and posixEquivalenceClass([=x=]) predicates
to ParseRegExp.qll, enabling correct tokenization of all three POSIX bracket
atom types inside character classes:
- [:name:] — already handled, unchanged
- [.x.] — new: posixCollatingSymbol predicate
- [=x=] — new: posixEquivalenceClass predicate
Refactored:
- charSetDelimiter: use inAnyPosixBracket helper (covers all three types)
- charSet closing: use inAnyPosixBracket helper
- inPosixBracket: updated to cover all three types
- simpleCharacter: updated to exclude all three types
- namedCharacterProperty: includes posixCollatingSymbol and posixEquivalenceClass
- inAnyPosixBracket: new private helper; uses pos in [x..y-1] for QL binding
Before this commit (commit 6):
[[.a.]] → RegExpCharacterClass([[.a.]) + stray ] (WRONG)
[[=a=]] → RegExpCharacterClass([[=a=]) + stray ] (WRONG)
After this commit:
[[.a.]] → RegExpCharacterClass containing RegExpNamedCharacterProperty [.a.]
[[=a=]] → RegExpCharacterClass containing RegExpNamedCharacterProperty [=a=]
Regenerated parse.expected and regexp.expected via codeql test run --learn
(CodeQL 2.26.1); all 2 tests pass.
Add new test cases to test.cpp exercising POSIX bracket expressions inside
character classes as supported by std::regex ECMAScript mode:
- Single POSIX classes: [[:alpha:]], [[:digit:]], [[:space:]], [[:upper:]],
[[:lower:]], [[:alnum:]], [[:print:]], [[:punct:]]
- Negated outer bracket: [^[:space:]]
- Mixed: regular char + POSIX class [a[:space:]]
- POSIX collating symbol [[.a.]]
- POSIX equivalence class [[=a=]]
- Mixed POSIX + range [[:alpha:]0-9]
The existing r_posix1-r_posix4 cases remain untouched. New cases use distinct
names (r_posix_alpha, r_posix_digit, etc.) to avoid redeclaration.
Generated .expected via codeql test run --learn (CodeQL 2.26.1); all 2 tests
pass. The expected output captures the CURRENT (pre-fix) tokenization — commit 7
will change these rows when correct POSIX nesting is implemented.
Content-only edits to test.cpp to remove Ruby-only syntax that has no
C++ ECMAScript equivalent:
1. Removed: "a{,8}" — Ruby-only "{,n}" no-lower-bound quantifier
(ECMAScript requires an explicit lower bound in {n,m})
2. Removed: second ".*" for /.*/m — mode flags in C++ are construction-site
arguments (e.g. std::regex::multiline), not part of the pattern string.
This mode variant was the exact same pattern string as r_meta1 so it
added no coverage.
3. Removed: "(?'foo'fo+)" — Ruby single-quote named-group form.
ECMAScript only supports the angle-bracket form (?<name>...).
Left in place for now (removed with the parser in commit 5):
\\A, \\z, \\G, \\h, \\H — Ruby-only anchor/escape classes.
The POSIX-bracket cases are kept through commit 7.
.expected regenerated by:
codeql test run --learn --search-path=. cpp/ql/test/library-tests/regex/
(CodeQL CLI 2.26.1)
All 2 tests passed.
Introduces cpp/ql/test/library-tests/regex/test.cpp with the Ruby regex
corpus mechanically translated to C++ string literals and wrapped in
std::regex construction calls using fully self-contained in-file stubs
(no #include of any external/standard header).
Stub surface:
std::basic_regex<CharT> constructor(const char*), constructor(const char*, int),
assign(const char*)
std::regex typedef for basic_regex<char>
std::regex_match / regex_search / regex_replace free functions
1:1 corpus mapping from regexp.rb:
- All Ruby /regex/ literals translated to C++ "string" literals
(each backslash doubled: /\d/ -> "\\d")
- Dropped: /#{A}bc/ (Ruby string interpolation, no string-literal form)
- Kept: a{,8}, .*m, \A, \z, \G, \h\H, (?'foo'...) — these are removed
in commits 4 and 5 (keeping them here preserves 1:1 mapping)
Also removes `abstract` from RegExp class in ParseRegExp.qll so that all
StringLiterals are regex candidates (trivial syntactic gate; no dataflow).
.expected generated by:
codeql test run --learn --search-path=. cpp/ql/test/library-tests/regex/
(CodeQL CLI 2.26.1, bundled C/C++ extractor)
All 2 tests passed.
This is a verbatim copy of the Ruby ParseRegExp.qll and RegExpTreeView.qll,
with only the file-header comments updated to reflect the new path.
The test files (parse.ql, regexp.ql, regexp.rb, *.expected) are also copied
verbatim from the Ruby test suite as a baseline reference.
This commit does NOT compile against C++: it still imports codeql.ruby.AST
and references Ruby AST types (RegExpLiteral, StringlikeLiteral, etc.).
The adaptation to compile against C++ is in the next commit.
Also adds the codeql/regex dependency to cpp/ql/lib/qlpack.yml and the
unreleased change-note at cpp/ql/lib/change-notes/2026-07-21-cpp-regex-tree-view.md.
