Three cumulative fixes for Harbour's preprocessor stringify forms
surfaced by harbour-core/tests/pp.prg:
1. Token alignment — tokenizePattern and tokenizeLine now both
split on parens and brackets, so `DUMB(a)` (no space) tokenises
as `DUMB`, `(`, `a`, `)` on both sides. Previously the line
tokenizer kept `DUMB(a)` as one token while the pattern split
it three ways, and the match never engaged. Fixes `_DUMB_(a)`-
style calls in pp.prg line 57+.
2. Substitution order — applyResult was replacing the bare `<z>`
marker first, eating the inner `<z>` of `#<z>`, `<"z">`, `<(z)>`
and `<.z.>` and leaving stray `#` / `<` / `.` characters that
the lexer reported as ILLEGAL tokens. Run all compound forms
first, bare `<z>` last.
3. Quote delimiter picker — ppQuote wraps a captured value in a
legal PRG string literal by trying `"..."` first, then `'...'`,
then `[...]`. Harbour's #<z> dumb-stringify needs this because
the capture may already contain `"`, and Five was producing
malformed `""world""` literals.
Bonus: smart-stringify `<(z)>` now recognises input that's already
a string literal (`"x"` / `'x'` / `[x]`) and keeps it verbatim
instead of double-quoting.
pp.prg 26 parse errors → 2 (remaining: `USE &b ALIAS &a.1` macro-
inside-command at line 21 and one related line, unrelated to this
fix). FiveSql2 43/43, Harbour compat 56/56, Go test ALL PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CharlesKWON
e9522772a7