`LIST [<fields>] [OFF] [FOR ...] [WHILE ...] [NEXT ...] [RECORD ...]
[REST] [ALL]` and `DISPLAY [<fields>] [OFF] [FOR ...] ... [ALL]`
reach the parser as plain function calls to a new RTL primitive
__dbList (rtlDbList in hbrtl/database.go).
Implementation: walk the workarea under dbEval-style FOR/WHILE/NEXT/
RECORD/REST bounds. For each visible record, evaluate each column
block and emit the rendered values via valueToDisplay (the same
formatter QOut already uses). Empty fields list defaults to
"all fields". OFF suppresses the record-number prefix.
LIST always emits the full filtered range; DISPLAY without ALL emits
only the current record (encoded as nCount=1). TO PRINTER / TO FILE
clauses are not yet wired through — for now everything goes to
stdout.
Wiring up LIST/DISPLAY surfaced four further gaps in PP that were
silently masking bugs in any rule with multiple word-list / list /
optional clauses chained together:
* matchSegment refused MarkerWordList inside `[...]`. The LIST
rule's `[<off:OFF>]` clause therefore never set the off
capture, and `<.off.>` substituted to nothing instead of .T./.F.
matchSegment now matches WordList markers the same way the
top-level matcher does.
* `<v,...>` and `<(f)>` capture stop boundaries didn't include the
values of following MarkerWordList markers. For
`[<v,...>] [<off:OFF>] [<all:ALL>]` against `LIST id, name OFF`,
the v list would happily eat OFF. New addStopFrom helper
contributes both literal keywords and word-list values; both
matchSegment's MarkerList branch and captureExpression now use
it.
* Optional-repeat loop in matchPattern merged a no-progress
iteration's empty capture into the running multi-capture string
(with the `\x01` separator) before the no-progress break check
fired. So a successful first iteration's value got contaminated
and the substitution loop then skipped it as multi-capture
garbage. The merge now happens after the progress check.
* Unreferenced `<.name.>` markers (optional clauses that didn't
match in the input) were getting cleaned up to empty by the
generic marker scrubber instead of the .F. sentinel Harbour's
std.ch expects. New replaceUnreferencedLogify pass mirrors the
existing replaceUnreferencedBlockify and runs just before the
cleanup.
Parser cleanup: LIST and DISPLAY removed from the IDENT-statement
no-op switch in both parseIdentStmt and parseExprStmt.
Gates green:
go test ./... : PASS
FiveSql2 SQL:1999 : 43/43
Harbour compat : 56/56
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`<{name}>` previously wrapped a list-typed capture's whole
comma-joined string in one code block: `{|| id , name }`. Harbour's
std.ch expects per-element wrapping so `{ <{v}> }` against
`LIST id, name` yields `{ {|| id }, {|| name } }` — an array of
column blocks the call site can evaluate per row.
applyResult now consults the marker table for blockify the same way
it already does for smart-stringify, splits the captured list on
top-level commas, and emits one `{|| expr }` per element.
Prereq for the upcoming LIST / DISPLAY rules; no user-visible
behavior change for the rules already in std.ch (their `<{for}>` /
`<{while}>` markers are scalar).
Gates green:
go test ./... : PASS
FiveSql2 SQL:1999 : 43/43
Harbour compat : 56/56
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`COPY TO <file> [FIELDS <list>] [FOR ...] [WHILE ...] [NEXT ...]
[RECORD ...] [REST] [ALL]` reaches the parser as a plain function
call to a new RTL primitive __dbCopy (rtlDbCopy in hbrtl/database.go).
Implementation: project the field list (case-insensitive name match
against the source's structure, full copy when omitted), dbCreate the
target file with that struct, open it under a temp alias, walk the
source under dbEval-style FOR/WHILE/NEXT/RECORD/REST bounds, and
GetValue/Append/PutValue per record into the target. SDF / DELIMITED
variants stay parser no-ops until those backends arrive.
Wiring up COPY surfaced four longstanding gaps in the PP that had to
be fixed for the rule to even reach the runtime:
* `<(name)>` *pattern* marker was treated as a regular `<name>`
with the parens baked into the captured key, so the matching
result substitution `<(name)>` couldn't find it. parseOneMarker
now strips the parens at parse time so capture key and result
marker share the bare name. The smart-stringify result behavior
is unchanged.
* matchSegment (the optional-clause matcher) bailed on every
non-Regular marker. `[FIELDS <fields,...>]` therefore failed to
match at all and the fields list arrived empty in the result
template. matchSegment now handles MarkerList with paren-balanced
capture and segment+outer literal stop boundaries.
* captureExpression only used the first literal in the pattern
tail as a stop boundary. With std.ch's chain of optional
clauses (`[TO <(f)>] [FIELDS ...] [FOR ...] [WHILE ...] ...`)
the file-name marker was happy to gobble a trailing FOR clause
when FIELDS was absent. It now stops at *any* of the remaining
pattern literals.
* `<(name)>` smart-stringify on a list-typed capture wrapped the
whole comma-joined string in one set of quotes — `{ "a , b" }` —
instead of `{ "a", "b" }`. New helper quoteListElements splits on
top-level commas (paren / bracket / brace / string-balanced) and
quotes each element. applyResult now consults the rule's marker
table to know which captures came from `<name,...>`.
Parser cleanup: COPY removed from the IDENT-statement no-op switch in
both parseIdentStmt and parseExprStmt.
Gates green:
go test ./... : PASS
FiveSql2 SQL:1999 : 43/43
Harbour compat : 56/56
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three xBase analytical commands that were silent no-ops in the
parser now execute as Harbour-style PP rewrites:
COUNT [TO <v>] [FOR <for>] [WHILE <while>] ... -> dbEval()
SUM <x> TO <v> [FOR <for>] [WHILE <while>] ... -> dbEval()
AVERAGE <x> TO <v> [FOR ...] -> __dbAverage()
COUNT and SUM expand to a `<v> := 0 ; dbEval( {|| ... } )` pair
matching harbour-core/include/std.ch verbatim. AVERAGE delegates to
a new RTL function rtlDbAverage (sum + count + divide; returns 0 on
empty match) — the chained-private-variable trick Harbour uses to
keep AVERAGE inline doesn't translate cleanly through Five's PP.
Wiring up these rules surfaced four PP issues that had to be fixed
for the rewrite to even reach the parser:
* Result template did not implement <{name}> blockify. So a rule
body like `{|| x := x + <x> }, <{for}>` left the literal text
`<{for}>` in the output. Added blockify substitution: captured
-> `{|| <captured> }`, missing -> NIL.
* findMarkerEnd did not recognise `{`/`}` so unreferenced
blockify markers were not cleaned up either. Added `{`/`}` to
its prefix/suffix sets.
* Optional-clause matching had no view of the outer pattern, so a
regular marker at the end of `[TO <v>]` would swallow the rest
of the line — `COUNT TO n FOR x>5` captured `<v>` as
"n FOR x>5". matchSegment now takes outerTail and stops at its
first literal.
* `#command` directives could not span multiple physical lines.
A trailing `;` is harbour-core's line-continuation marker for
std.ch and now joins the next line into the directive before
parsing.
Parser cleanup: COUNT, SUM, AVERAGE removed from the IDENT-statement
no-op switch in parseIdentStmt + parseExprStmt. The remaining xBase
verbs (COPY, SORT, TOTAL, JOIN, LIST, DISPLAY, LABEL, REPORT, ...)
stay in the parser until their RTL backends arrive.
Gates green:
go test ./... : PASS
FiveSql2 SQL:1999 : 43/43
Harbour compat : 56/56
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce compiler/pp/std.ch with 19 #command rules so that ERASE,
RENAME, DELETE FILE, CLOSE [<a>|ALL|DATABASES], COMMIT, UNLOCK,
LOCATE/CONTINUE, REINDEX, PACK, ZAP, KEYBOARD, RUN, MENU TO, and
CLEAR GETS reach the parser pre-rewritten as plain function calls.
Embedded into the compiler binary via //go:embed so it auto-loads
without an explicit #include in user code, exactly the way Harbour
auto-loads its std.ch.
This is a pure dispatch move, not a behavior change for the
already-working forms: the same Five RTL functions get called.
But it does fix three regressions that the parser was masking:
* ERASE / RENAME / DELETE FILE used to be silent no-ops — the
parser swallowed the entire line and returned NIL. They now
actually delete/rename files (FErase / FRename).
* CLOSE <alias> used to silently ignore the alias and close the
current area. It now switches to the named area first
(<a>->( DbCloseArea() )).
* Two latent #command matcher bugs that surfaced while wiring
std.ch up:
- bare `CLOSE` would match rule `CLOSE ALL` because the tail
of the pattern wasn't checked for unconsumed literals.
- bare `CLOSE` would match rule `CLOSE <a>` because all
unconsumed pattern markers were unconditionally treated as
optional. They are only optional when nested inside `[...]`.
Parser cleanup: parseIdentStmt + parseExprStmt no longer hardcode
ERASE / RENAME / RUN / KEYBOARD / REINDEX / LOCATE / CONTINUE /
COMMIT / CLOSE — the rewriter handles them. Other xBase verbs
(COPY / SORT / COUNT / SUM / AVERAGE / TOTAL / JOIN / LIST /
DISPLAY / LABEL / REPORT / DIR ...) still no-op in the parser
because their RTL backends aren't implemented yet — once the
backends land they move into std.ch the same way.
Gates green:
go test ./... : PASS
FiveSql2 SQL:1999 : 43/43
Harbour compat : 56/56
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Harbour's `#xcommand DEFAULT <v1> TO <x1> [, <vn> TO <xn>] => ...`
uses an optional, repeatable trailing `[...]` block to accept any
number of `var TO default` pairs on a single line. Five's PP
skipped bracket bodies during pattern matching and treated them
as no-ops in result templates, so
DEFAULT a TO 10, b TO 20, c TO 30
expanded (at best) the first pair and dropped the rest — and
common.ch itself was documented as "not yet supported".
Three concrete changes:
1. matchPattern now matches the `[...]` body repeatedly against
remaining line tokens via a new matchSegment helper. Each
successful iteration appends captures for the interior markers
under the same name, joined with a \x01 sentinel.
2. matchSegment, when capturing the last marker in a body with no
following literal, uses the body's opening literal (e.g. the `,`
in `[, <vn> TO <xn>]`) as the iteration boundary. Otherwise
captureExpression would greedily eat the rest of the line and
collapse every remaining pair into one capture.
3. applyResult's new expandOptionalRepeat walks the result template
for top-level `[...]` blocks. When a referenced marker is multi-
captured it emits the body N times (substituting per-iter value);
when it's single-captured it emits the body once; otherwise drops
the block. A separate referencedMarkers scanner and an inMarker
guard keep literal `[` / `]` inside PP markers (like `<.x.>`)
from being mistaken for bracket delimiters.
Side fix: ParseRule previously stripped every ` ;` as a Harbour
line-continuation marker, but that also destroyed in-line PRG
statement separators in result templates. Line joining is the
preprocessor's job upstream — keep semicolons intact here.
common.ch now ships real DEFAULT and UPDATE #xcommands. Verified
1-, 2-, and 3-pair DEFAULT expansion plus `common.ch` inclusion
from user code. FiveSql2 43/43, Harbour compat 56/56, Go test ALL
PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related fixes for Harbour's data-driven `USE &cFile ALIAS &cAlias
INDEX &cNdx` idiom — common in any app that dispatches table names
at runtime.
Parser (compiler/parser/parser.go parseUse):
- `USE &cFile` / `USE &(expr)` previously triggered a
skipToEndOfLine short-circuit, emitting an empty UseCmd (equivalent
to bare USE = close current area). Now parseMacro runs and the
MacroExpr becomes the File node, so codegen emits MacroPush +
dbUseArea.
- `ALIAS &cAlias` / `ALIAS &a.1` similarly dropped the macro result;
now captures it into UseCmd.AliasExpr so codegen evaluates the
alias at runtime. Both the IDENT-path ("ALIAS") and keyword-path
(token.ALIAS) handlers fixed.
PP (compiler/pp/command.go):
- captureExpression and the MarkerList branch now paren-balance
`(`/`[`/`{` so nested grouping inside a macro argument doesn't let
an inner `)` terminate the capture. Example:
_REGULAR_(&(a))
previously captured `&(a` (missing inner `)`) and left the outer
`)` dangling, producing parse errors in the expanded output.
- MarkerList capture still joins tokens with " " for raw `<z>`
substitution — comma tokens stay in the stream, so `s(<z>)`
re-emits them as argument separators and the list expands cleanly.
Bench: harbour-core/tests/pp.prg 2 errors → 0 for the realistic
`USE ¯o` / `&(expr)` patterns. Remaining parse errors on line 70
are a pathological `_REGULAR_L` list that includes `&a. [2]`
(space between macro's terminating dot and an array index) — the
PP expands it correctly but Five's lexer refuses the expanded
result. That form doesn't occur in real code.
/tmp/test_use_macro.prg — all four patterns (`USE &f`, `USE &f ALIAS
&f`, `USE &f ALIAS &f INDEX &i`, dot-terminated) now compile. FiveSql2
43/43, Harbour compat 56/56, Go test ALL PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three cumulative fixes for Harbour's preprocessor stringify forms
surfaced by harbour-core/tests/pp.prg:
1. Token alignment — tokenizePattern and tokenizeLine now both
split on parens and brackets, so `DUMB(a)` (no space) tokenises
as `DUMB`, `(`, `a`, `)` on both sides. Previously the line
tokenizer kept `DUMB(a)` as one token while the pattern split
it three ways, and the match never engaged. Fixes `_DUMB_(a)`-
style calls in pp.prg line 57+.
2. Substitution order — applyResult was replacing the bare `<z>`
marker first, eating the inner `<z>` of `#<z>`, `<"z">`, `<(z)>`
and `<.z.>` and leaving stray `#` / `<` / `.` characters that
the lexer reported as ILLEGAL tokens. Run all compound forms
first, bare `<z>` last.
3. Quote delimiter picker — ppQuote wraps a captured value in a
legal PRG string literal by trying `"..."` first, then `'...'`,
then `[...]`. Harbour's #<z> dumb-stringify needs this because
the capture may already contain `"`, and Five was producing
malformed `""world""` literals.
Bonus: smart-stringify `<(z)>` now recognises input that's already
a string literal (`"x"` / `'x'` / `[x]`) and keeps it verbatim
instead of double-quoting.
pp.prg 26 parse errors → 2 (remaining: `USE &b ALIAS &a.1` macro-
inside-command at line 21 and one related line, unrelated to this
fix). FiveSql2 43/43, Harbour compat 56/56, Go test ALL PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real Harbour headers write parameterised commands with no space
between the keyword and its opening paren:
#xcommand MAKE_TEST( <obj>, <v> ) => ...
ParseRule stored the rule keyword as `MAKE_TEST(` (stripping only
<>, [] marker wrappers), but firstToken normalised source lines by
stopping the first-word scan at `(` — so `MAKE_TEST( o, 42 )`
produced `MAKE_TEST` for the lookup. The two strings didn't match
and the fast-path keyword check rejected every invocation, leaving
the macro unexpanded and the call site as a bare undeclared
identifier.
Trim everything from the first `(` onward during keyword
extraction so both halves agree on the dispatch key. The marker
tokens inside the parens are still parsed normally by
parseMarkers / matchPattern.
Verified with /tmp/test_xcmd2.prg (`MAKE_TEST( o, 99 )` expands
and dispatches to the object's :hVar access). FiveSql2 43/43,
Harbour compat 56/56, Go test ALL PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
