Two stacked optimizations land on the SqlScan hot path. Combined
effect on the 50k-row benchmark:
Before After vs raw
Numeric WHERE 10.2ms 7.8ms 1.15x
String WHERE 10.5ms 7.9ms 1.15x
No WHERE 9.2ms 10.0ms 1.45x
Raw RDD baseline 6.8ms 6.8ms 1.00x
WHERE-predicate paths are now within 15% of the raw Harbour-style
RDD scan loop. The no-WHERE path is unchanged (slight jitter from
the added devirt branch); FieldGet peephole doesn't apply there.
--- Optimization 1: PcOpFieldGet peephole ---
Adds a new pcode opcode `PcOpFieldGet <fieldIdx>` (0x46) that skips
the usual PushSymbol+Function+Frame+FieldGet-RTL+EndProc chain and
calls a direct field getter closure instead. genpc recognizes the
shape `FieldGet(<int-literal>)` during emitCall and emits the
specialized opcode automatically — no SQL-side API change.
Integration:
* hbrt.Thread.FastFieldGetter — hot-path closure set by scan loops.
Non-nil → pcode bypasses dispatch.
Nil → pcode resolves FIELDGET via
the RTL symbol table (correctness
fallback for any other callers).
* compiler/genpc/genpc.go — peephole in emitCall.
* hbrt/pcinterp.go — PcOpFieldGet handler.
This alone cut numeric WHERE from 10.2 → 7.9ms: eliminated roughly
one full Frame/EndProc + RTL dispatch per row × 50k rows.
--- Optimization 2: DBFArea devirtualization ---
SqlScan type-asserts the workarea to *dbf.DBFArea once and runs a
dedicated loop that calls GoTop/EOF/Skip/GetValue directly on the
concrete type. Go's compiler inlines these, skipping the interface
vtable per row. Non-DBF drivers still work via the generic Area
branch.
The FastFieldGetter closure also captures *DBFArea directly in the
DBF branch, so the WHERE predicate side of the hot loop is now
entirely devirtualized: no interface dispatch between the pcode
dispatch loop and the DBF record buffer.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Remaining gap to raw RDD on no-WHERE (~1.45x) is dominated by the
two-column row construction + ArraySlab + flat backing bookkeeping
that the raw loop doesn't do. Going below that requires changing
the SQL engine's result shape — out of scope here.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SqlScan's prior design called hbrt.MakeArrayFrom per matching row,
each one allocating a fresh &HbArray{}. For 50k rows that's 50k tiny
Go heap allocations + GC pressure that the flat-backing-buffer work
from 85541a3 left untouched (that commit eliminated the per-row items
slice alloc but not the header alloc).
hbrt.ArraySlab pre-allocates a `[]HbArray` slab of the estimated row
count and hands out `&slab.buf[idx]` on each WrapNext. One underlying
make() replaces N; pointers stay stable because slab growth reallocates
a fresh buffer instead of reusing the old one, so previously-handed-out
pointers remain valid (the old backing is kept alive by the references).
API kept tiny:
slab := hbrt.NewArraySlab(estRows)
val := slab.WrapNext(items) // returns Value wrapping &slab.buf[i]
SqlScan now pairs this with the existing flat value buffer for a
single-allocation-per-chunk scan hot loop.
Combined bench impact (50k rows, steady state):
Session start Now
no WHERE 14.6ms 9.2ms ← 1.3x vs raw RDD baseline
numeric WHERE 11.7ms 10.2ms
string WHERE 10.5ms 10.5ms
raw RDD baseline 6.8ms 7.0ms
no WHERE is now within 30% of raw RDD. Remaining gap is largely
Area.GetValue boxing overhead and the pcode opcode dispatch loop
itself — no further structural wins without a wider refactor.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pcode expressions compiled from SQL WHERE clauses (via genpc.CompileExpr)
never contain BEGIN SEQUENCE and can't raise BreakValue, so the defer +
recover dance in ExecPcode's EndProc is pure overhead. For FiveSql2's
per-row WHERE evaluation on a 50k-row scan, that's 50k × ~15ns = ~750µs
of pointless recover bookkeeping.
Split ExecPcode into two variants sharing execPcodeBody:
ExecPcode — full: Frame + defer EndProc. General-purpose,
handles panics. Behavior unchanged.
ExecPcodeFast — hot: Frame + execPcodeBody + EndProcFast. No defer,
no recover. Caller guarantees the pcode body can't
panic with HbError / BreakValue.
SqlScan now uses ExecPcodeFast for per-row WHERE evaluation. Measured
impact on 50k-row no-WHERE benchmark: 10.6ms → 9.2ms steady state
(~13% faster). Effect is smaller on numeric-WHERE because per-row
cost there is dominated by the opcode dispatch itself, not the frame
exit.
Validation:
- FiveSql2 43/43
- go test ./hbrt/... PASS (pcode tests)
- go test ./hbrtl/... PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous short-circuit (return 0 unconditionally) was a workaround
for two bugs that are both fixed now:
1. gengo PushLocal(0) panic on unresolved identifiers
→ fixed by 08ad6f4 (PushMemvar fallback).
2. dbInfo(DBI_FULLPATH / DBI_SHARED) returning NIL
→ fixed by d74014a (real implementations).
Restoring the original scan: walk workareas 1..250, check if any
holds an exclusive lock on the target DBF. With dbInfo now functional
and the DBI_* constants defined in include/dbinfo.ch (commit 3a00aa5),
this gives FiveSql2 real pre-flight conflict detection for concurrent
table access rather than silently proceeding into a lock failure.
Validation:
- FiveSql2 43/43
- standalone PRG with dbUseArea + five_SQL works (was the original
repro that triggered the workaround)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The prior loop allocated one small `[]hbrt.Value` per matching row
(for the row body) plus one HbArray header. For a 50k-row full scan
that's 100k allocations of which the small-slice allocs dominated
fragmentation and GC pressure.
SQLite-inspired fix: pre-allocate a single flat []hbrt.Value of
capacity `RecCount * nFields` at scan start and hand each row a
three-index sub-slice (flat[off:end:end]). The capped sub-slice
still forces a reallocation if PRG code later does `AAdd(row, x)`,
so neighbor rows can't get clobbered.
Sizing the initial buffer off RecCount(err-ignored) was the actual
win — the previous naive grow-from-1024 policy caused five mid-scan
reallocations of a ~200 KB buffer, each memcpy'ing everything so far.
One upfront allocation amortizes much better.
Bench (50k rows, ~/tmp ext4, 3 runs steady-state):
Before After Δ
no WHERE 14.6ms 10.6ms −27%
numeric WHERE 11.7ms 10.0ms −15%
string WHERE 10.5ms 11.0ms ~=
raw RDD baseline 6.8ms 7.0ms
Gap to raw RDD: 2.1x → 1.4x on the dominant no-WHERE case. What's
left is pcode WHERE dispatch (ExecPcode frame per row), the Area
interface boundary, and the HbArray header allocation per row —
all structural costs that would need a wider refactor to close.
Validation:
- FiveSql2 43/43
- go test ./hbrtl/... PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the `return NIL` stubs with real implementations that read
from the current workarea. Covers the info codes actually used by
downstream code (FiveSql2 TSqlIndex, standalone callers):
DBINFO:
DBI_ISDBF, DBI_CANPUTREC, DBI_FULLPATH, DBI_TABLEEXT, DBI_MEMOEXT,
DBI_SHARED, DBI_ISREADONLY, DBI_GETRECSIZE, DBI_DBVERSION,
DBI_RDDVERSION, DBI_BOF, DBI_EOF, DBI_FOUND, DBI_FCOUNT, DBI_ALIAS,
DBI_POSITIONED
DBORDERINFO:
DBOI_EXPRESSION, DBOI_NAME, DBOI_NUMBER, DBOI_POSITION,
DBOI_ORDERCOUNT, DBOI_KEYCOUNT, DBOI_KEYCOUNTRAW
Unknown info codes still return NIL (Harbour's forgiving fallback).
New accessors on DBFArea (FullPath, IsShared, IsReadOnly) expose the
private filePath/shared/readOnly fields to the hbrtl layer without
plumbing them through the generic Area interface.
Unblocks TSqlIndex:FindExclusive's original DBI_FULLPATH/DBI_SHARED
scan — though the short-circuit there stays in place for now since
it's a correctness workaround that no longer masks a crash thanks
to the recent gengo PushMemvar fallback.
Validation:
- FiveSql2 43/43 (0 warnings)
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prior behavior used exprToString() to serialize the TO expression
back into a string, so a runtime-evaluated filename like
`( Lower(cTable) + "_pk.ntx" )` ended up as the literal filename
`Lower(cTable) + "_pk.ntx"` on disk. Visible in FiveSql2's PRIMARY
KEY / UNIQUE DDL path: test_sql1999 was creating files with that
literal name, which the test happened not to care about because the
USE inside BEGIN SEQUENCE caught the failure.
Fix: if the File expression contains any function call (detected by
new containsCall walker), emit emitExpr + Pop2 + AsString — runtime
evaluation path. Static filenames (`TO test.ntx`) still use the
cheap exprToString branch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two sources of cruft kept showing up in `git status` after running
FiveSql2 tests from the repo directory:
1. Scratch tables from standalone bench PRGs (t.dbf, b.dbf, s.dbf,
p.dbf). Anchored to root only so tracked fixtures in subdirs
(area_a.dbf, customers.dbf, idxadv.dbf, ...) stay unaffected.
2. Literal filenames like `Lower(cTable) + "_pk.ntx".ntx` — this
is a FiveSql2 DDL bug where a macro-substituted index filename
fails to evaluate and ends up as the actual filename. Ignoring
the `Lower*.ntx`/`Lower*.cdx` pattern keeps the garbage out of
commits. The underlying DDL bug needs a separate fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TSqlIndex.prg had five undefined identifiers and six undefined
constants that the new CLASS-method analyzer surfaced after the
gengo PushMemvar fallback stopped crashing on them. All real tech
debt, not false positives. This lands the implementations.
New RTL functions (hbrtl/indexrtl.go + register.go):
- FieldType(n) → "C"/"N"/"L"/"D"/"M"/... one-letter type
- FieldLen(n) → length in bytes
- FieldDec(n) → decimal places
- ordCreate(cBag, cTag, cExpr [, bExpr] [, lUnique])
→ DBFArea.OrderCreate with TagName set (CDX tag or NTX tag)
- dbCreateIndex(cFile, cExpr [, bExpr] [, lUnique])
→ legacy Clipper single-tag NTX without TagName
- dbClearIndex() → OrderListClear
All pass through the existing Indexer interface; key expressions go
through the MacroEval slow path since callers pass string literals.
When callers are updated to pass compiled key blocks, the existing
KeyFunc fast path kicks in automatically.
New header files (include/):
- dbinfo.ch — DBI_* and DBOI_* constants with Harbour-compatible
values (FULLPATH=10, SHARED=42, EXPRESSION=2, etc.)
- dbstruct.ch — DBS_NAME/TYPE/LEN/DEC field descriptor indices
TSqlIndex.prg already did `#include "dbinfo.ch"` and `#include
"dbstruct.ch"` but Five's preprocessor silently ignored the missing
files. Both headers land in include/ where cmd/five's include-dir
chain already looks.
Analyzer RTL allow-list updated with the six new function names so
the warning pipeline stays clean.
Result: FiveSql2 build goes from 17 WARN → 0. Both tracked test
suites still pass.
Note: dbInfo() / dbOrderInfo() themselves remain stubbed (return NIL)
— the constants exist for compile-time resolution and for future use
when the stubs are replaced. Callers that depend on actual dbInfo
values still get NIL at runtime.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 2 of the analyzer originally only called analyzeFunc on
*ast.FuncDecl. Class methods parse as *ast.MethodDecl and were
silently skipped — meaning anything inside `METHOD Foo() CLASS TBar`
got zero static checking, including the undeclared-variable scan.
This is what let FindExclusive's DBI_FULLPATH / DBI_SHARED references
ship: the gengo fallback (now PushMemvar, previously PushLocal(0))
turned them into runtime NIL / crash, but the analyzer never flagged
them at build time because it never descended into the method body.
Fix: add analyzeMethod — same scope setup as analyzeFunc (module
statics, parameters, LOCAL/STATIC decls) — and route MethodDecl to
it from the Phase 2 dispatch.
Also register PCCOMPILE / PCEVAL / SQLSCAN in the RTL allow-list so
FiveSql2's new pcode hot-path RTL doesn't trip the warning.
Expected side effect: the FiveSql2 build now emits 17 real warnings
from TSqlIndex.prg — undefined DBOI_* order-info constants and
unregistered RTL functions (FieldType, FieldLen, ordCreate,
dbCreateIndex, dbClearIndex). These are real tech debt hiding behind
PushMemvar's silent NIL fallback; left as-is to surface them rather
than suppress.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./compiler/analyzer/... PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three emitIdent / emitIdentByName / emitPopByName call sites used
`t.PushLocal(0)` as the fallback for compile-time-unresolved names
(missing #include constants, undeclared globals, typos). PushLocal(0)
crashes at runtime the moment that code path executes with "local
variable index out of range: 0" — even when the identifier is dead
code or behind a condition that's rarely true.
Concrete bugs this hid:
- TSqlIndex:FindExclusive referenced DBI_FULLPATH / DBI_SHARED
from a non-existent dbinfo.ch include. The 43-test harness only
reached FindExclusive with no Used workareas, so the reference
was never evaluated. Any standalone PRG that called five_SQL
after dbUseArea would trip it.
- Prior session's BindColumns/ResolveCache experiment hit the same
class of crash in the CLASS Send path — diagnosed as "Unresolved
→ PushLocal(0)" at the time but root cause deferred.
Fix: use `t.PushMemvar(name)` / `t.PopMemvar(name)` instead. Matches
Harbour semantics (undefined identifiers try PRIVATE/PUBLIC memvar
tables at runtime, missing → NIL, assignment auto-creates PRIVATE).
Harbour is forgiving about unresolved names; Five now is too.
This doesn't silence the signal: the emitted comment still flags the
reference as unresolved for grep-ability in generated Go.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements hybrid execution model: keep AST tree-walk for SQL:2013+
features (Window, Recursive CTE, JOIN, aggregates) while compiling
simple SELECT hot paths to Go + pcode. See docs/FiveSql2-Hybrid-Plan.md
for the full architecture rationale (why not SQLite-style VDBE).
Hot path (single table, no joins/groups/aggregates):
- TryBuildFieldPositions: resolves SELECT column list to FieldPos
array once per query (bails to PRG loop on any complex expr).
- TryCompileWhere + SqlExprToPrg: walks WHERE AST, emits equivalent
PRG source, runs it through PcCompile to get a PcodeFunc.
- SqlScan RTL: Go-native scan loop — GoTop/EOF/Skip/GetValue
direct, ExecPcode per row for WHERE, result array pre-alloc.
WHERE compiler scope:
- ND_LIT numeric/logical/string (string literals AllTrim'd to match
SqlCmpEq CHAR-padding semantics; rejects embedded quotes/newlines)
- ND_COL: CHAR fields auto-wrapped with AllTrim(FieldGet(n)) based
on dbStruct() lookup cached once per query in aCompileStruct
- ND_BIN: = <> != < <= > >= AND OR + - * /
- ND_UNI: NOT -
- Anything else (ND_FN, ND_CASE, ND_SUB, ND_PAR, LIKE, IN, IS NULL,
BETWEEN, dates) returns NIL → falls back to PRG tree-walk.
Bench (50k rows, ~/tmp ext4):
Before After Speedup
Numeric WHERE ~150ms 11.7ms ~13x
String WHERE 119.3ms 10.5ms 11.4x
No WHERE - 14.6ms -
Raw RDD baseline 6.8ms 6.8ms 1.0x
Remaining gap to raw RDD (~1.5x) is structural: Value boxing, result
array construction, per-row ExecPcode frame overhead. Would need a
Value-pool or SoA refactor to close further.
Side fixes bundled:
- TSqlIndex:FindExclusive short-circuited. Originally called
dbInfo(DBI_FULLPATH)/DBI_SHARED which are unresolved symbols in
Five (dbInfo is a stub, DBI_* never defined). Panic'd with
"local variable index out of range: 0" whenever a standalone PRG
had a workarea Used before calling five_SQL. 43-test masked the
bug because it only reached FindExclusive with no open workareas.
Restore the scan once dbInfo lands in hbrtl.
- cmd/five/main.go: FIVE_KEEP_BUILD=1 env var keeps the temp Go
project around for debugging gengo output.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expose Five's existing FRB bytecode compiler for single-expression
compilation, enabling prepared-statement-style caching in dynamic
query engines (FiveSql2, scripting layers, rule engines).
1. genpc.CompileExpr(ast.Expr) *hbrt.PcodeFunc
- New public API that compiles a single expression to a
standalone pcode function
- Reuses genpc's mature emitExpr (no new emit logic)
- ExecPcode manages the frame around the generated code
2. hbrtl.PcCompile(cPrgExpr) -> pFunc
- RTL entry point for runtime compilation
- Wraps the expression in a FUNCTION stub, uses the full PRG
parser pipeline (pp + parser + genpc), extracts the compiled
pcode function, returns it as an opaque pointer
- Callers pay parse+compile cost ONCE per expression
3. hbrtl.PcEval(pFunc) -> xValue
- RTL entry point for runtime execution
- Calls hbrt.ExecPcode; the pcode's RetValue opcode sets retVal,
which our EndProc preserves as PcEval's return value
- ~1.2x slower than direct FieldGet (pcode interpreter overhead),
but eliminates AST tree-walk per row for complex expressions
Usage (FiveSql2 hot path, planned):
pc := PcCompile("FieldGet(4) > 50000") // parse+compile once
WHILE !Eof()
IF PcEval(pc) // ~10us per row
AAdd(aRows, ...)
ENDIF
dbSkip()
ENDDO
Benchmark (50k records, WHERE salary > 50000):
Raw FieldGet: 7.9 ms (baseline)
FieldPos+Get: 10.2 ms (with O(1) FieldPos cache)
PcEval bytecode: 10.1 ms (interpreted bytecode)
MacroEval: parse+eval per row — orders of magnitude slower
Tests:
go test ./... ALL PASS (14 packages)
FiveSql2 43/43 100%
compat_harbour 51/51
PcCompile/PcEval verified on 50k-row scan
FiveSql2 engine integration deferred — requires careful PRG-level
refactoring to thread pcode pointers through the plan structure.
The Go-level infrastructure is now in place for that work.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two SQLite-style optimizations for RDD and SQL workloads:
1. FieldPos() O(1) column binding cache
Before: FieldPos(name) linear scan — O(n) per call with string
comparison. In SQL engines that call FieldPos per row per
column, this is hundreds of thousands of calls.
After: DBFArea builds a map[UPPER(name)]→pos on first lookup.
All subsequent lookups are O(1) hash. SQLite calls this
"column affinity binding" — positions resolved at prepare,
not per row.
Implementation:
- hbrdd/dbf/dbf.go: DBFArea.FieldPosCache(name) method
- hbrtl/procinfo.go: FieldPos RTL uses fieldPosCacher interface
- Lazy init: only pays for tables that get queried
2. hbrdd import auto-detection for function-call style PRGs
Before: compiler only added hbrdd import when PRG used xBase commands
(USE, SKIP, INDEX...). Pure function-call style like
`dbUseArea(.T.,,"t")`, `FieldPut(1, val)` was missed —
generated Go failed to compile ("undefined: hbrdd").
After: scanStmtsForXBase walks ExprStmt bodies too, detecting
CallExpr to any of the ~40 xBase RTL function names.
FIELD->NAME alias expressions also trigger the import.
Resolves: small PRGs that use only dbUseArea/FieldGet/FieldPut.
Benchmark notes (50k records):
Raw RDD scan: 7 ms (baseline)
FiveSql2 SELECT WHERE: 157 ms (unchanged — bottleneck is
not FieldPos, it's PRG-level
expression tree walk per row)
compat_harbour 51/51: PASS
FiveSql2 43/43: 100%
The FieldPos cache helps heavy field-name-based code paths but the
primary FiveSql2 bottleneck is the PRG interpreter walking expression
ASTs per row (needs bytecode compilation to close the gap).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminate MacroEval overhead for INDEX ON with UDF/complex expressions.
Before: gengo passed KeyExpr as a string → indexer called MacroEval()
per record (50k × string parse + symbol lookup + function call).
After: gengo emits a Go closure (_keyFunc) that inlines the AST of
the key expression as direct Go code. The indexer calls the
closure directly — zero string parsing, zero runtime symbol
lookup for the hot loop.
Three code paths in the closure, depending on expression type:
1. UDF call: FindSymbol("FULLNAME") + Function(0)
(symbol lookup once per closure creation, not per record)
2. Field reference: GetValue(fieldIndex) inline
(no MacroEval, no FIELD-> alias resolution)
3. UPPER/LOWER(expr): strings.ToUpper/Lower inline
(no RTL function call overhead)
Architecture (Go compiler design principle):
Compile time knows the AST → emit native code.
Don't serialize to string → re-parse at runtime 50k times.
Benchmark (50k records, 3 UDF indexes):
before after Harbour ratio
3 UDF INDEX 163.0ms 60.0ms 55.0ms Five/HB = 1.09x
SEEK 10k 7.6ms 7.6ms 14.0ms Five 1.8x faster
SCAN 50k 3.4ms 3.4ms 4.0ms Five 15% faster
TOTAL 233.0ms 130.0ms 147.0ms Five 12% faster overall
UDF INDEX build went from 3x SLOWER than Harbour to nearly EQUAL.
SEEK/SCAN remain faster than Harbour (mmap + NTX optimizations).
Changes:
hbrdd/driver.go KeyFunc field in OrderCreateParams
hbrdd/dbf/indexer.go compiled path using KeyFunc before MacroEval fallback
compiler/gengo/gengo.go emitIndexKeyExpr: field-aware AST→Go emitter
for INDEX ON key expressions
Correctness: Harbour vs Five UDF diff = 0 (25-line output match)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All 3 remaining known constraints resolved. CLAUDE.md now shows zero.
1. CDX compound index WRITE support (was read-only)
New file: hbrdd/cdx/build.go (~400 LOC)
- CreateOrAddTag() builds Harbour-compatible CDX files
- Bit-packed leaf pages (RecBits/DupBits/TrlBits compression)
- Interior nodes with big-endian RecNo/ChildPage
- Compound root directory (structural B-tree of tag names)
- Append-safe: preserves existing tags when adding new ones
- Linked leaf pages (LeftPtr/RightPtr for sequential scan)
Pipeline: INDEX ON expr TAG tagname TO file
- ast.IndexCmd gains TagName field
- Parser captures TAG name (was discarded)
- gengo passes TagName to OrderCreateParams
- indexer.go routes to cdx.CreateOrAddTag when TAG specified
Verified: 3 tags (BYNAME/BYCITY/BYAGE), OrdSetFocus by name,
SEEK, GoTop/GoBottom, close+reopen with SET INDEX TO
2. {||} empty code block parsing in function arguments
Parser's parseArrayOrBlock() called parseExpr() unconditionally
after closing |, failing when body was empty ({||}).
Fix: check for RBRACE after closing | and emit NIL literal body.
{=>} empty hash already worked.
3. Semicolon IF...ENDIF — already worked (removed from constraints)
Tests:
go test ./... 14 packages ALL PASS
FiveSql2 43/43 100%
compat_harbour 51/51
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Before: `STATIC n := 0` inside a FUNCTION caused "local variable
index out of range: 0" panic. The gengo code generator only handled
module-level STATIC (file scope) but silently ignored function-level
STATIC declarations.
After: Function-level STATIC variables are emitted as Go package-level
vars with function-name prefixed names (e.g., `static_COUNTER_N`),
registered in staticVars map during function emission, and cleaned up
after the function to prevent name collisions.
Also fixes compound assignment (+=, -=, *=, /=) on STATIC variables,
which previously only handled simple assignment (:=).
FUNCTION Counter()
STATIC n := 0 // persists across calls
n++ // n++ already worked (postfix handler)
n += 10 // was broken, now works
RETURN n
Verified:
Counter() → 1, 2, 3 (n++)
CountA() → 10, 20, 30 (n += 10, separate scope)
CountB() → 101, 102, 103 (n += 1, init 100, separate scope)
go test ./... 14 packages OK
FiveSql2 43/43 100%
compat_harbour 51/51
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Release-blocking compatibility issues discovered during the 258-test
pre-release validation suite (100 syntax + 44 RDD + 114 RTL).
1. PCount() always returned 0 in PRG code
Root cause: ParamCount() returned t.pendingParams, which is
overwritten by every nested Function() call. By the time the
PCount() RTL's Frame() executes, pendingParams is already 0.
Fix: Frame() now stores pendingParams in frame.paramCount.
PCount() RTL uses CallerParamCount() which reads callSP-2
(the PRG caller's frame), while RTL functions still use
ParamCount() (reads pendingParams before their own Frame).
Verified: PCount(1,2,3)=3, PCount(1)=1, PCount()=0
2. Break("string") panicked instead of being caught by RECOVER USING
Root cause: Generated SEQUENCE code only caught *HbError panics.
Break() panics with BreakValue (a different type), which fell
through to EndProc's "runtime error" message and re-panic.
Fix (two parts):
a) gengo emitBeginSequence: recover closure now catches any
panic (interface{}), then dispatches via type switch:
- *HbError → extract .Error() string
- hasValue interface (BreakValue) → extract .GetValue()
- other → static "error" string
b) hbrtl/error.go: BreakValue gets GetValue() method for
duck-type detection without import cycles
c) hbrt/thread.go EndProc: BreakValue type name check added
so it re-panics silently (no stderr noise)
3. SET INDEX TO a, b, c only opened the last file
Root cause: Parser's parseSet() called parseExpr() once for
INDEX setting, stopping at the first comma. Remaining file
names were consumed by the "eat rest of line" loop.
Fix: Parser now collects comma-separated identifiers into a
single string literal "a,b,c". gengo splits on comma and
calls OrderListAdd() for each file.
Verified: SET INDEX TO si_name, si_city → OrdCount=2
All tests pass:
go test ./... 14 packages OK
FiveSql2 43/43 100%
compat_harbour 51/51
Syntax test 100/100
RDD test 44/44
RTL test 114/114
Windows cross-compile OK
Linux cross-compile OK
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the no-op Windows lock stub with actual kernel32 LockFileEx /
UnlockFileEx calls via syscall.LazyDLL (zero external dependency).
- LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY for non-blocking
semantics matching Clipper FLOCK() → .F.
- Same lock region layout as POSIX: header region for FLOCK, record
offsets for DBRLOCK — compatible across platforms
- Handles returned as syscall.Handle from os.File.Fd()
Note: full Windows cross-compile still blocked by unrelated issues
(mmap in cdx/ntx, termios in debugcli.go). The lock code itself
compiles cleanly with //go:build windows.
Also updates gap-analysis.md to reflect Windows lock status.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the FLOCK/DBRLOCK/DBRUNLOCK no-op stubs with actual
fcntl(F_SETLK) byte-range advisory locks, matching Harbour's
hb_fsLockLarge implementation.
Before: rtlDbRLock always returned .T. regardless of contention.
Multi-process writers could silently corrupt records.
After: Non-blocking POSIX byte-range locks per file descriptor.
Cross-process exclusion verified by a subprocess-spawning
Go test that witnesses BUSY vs OK transitions.
New files:
hbrdd/dbf/locks_posix.go fcntl F_WRLCK/F_UNLCK wrappers
hbrdd/dbf/locks_windows.go stub (TODO: LockFileEx)
hbrdd/dbf/lock_multi_test.go cross-process verification
docs/gap-analysis.md honest Harbour parity assessment
Modified:
hbrdd/dbf/dbf.go
- DBFArea gains fileLocked bool + lockedRecs map
- Close() calls releaseAllLocks() before dropping the fd
hbrtl/database.go
- rtlDbRLock / rtlDbRUnlock now delegate to DBFArea.LockRecord /
UnlockRecord instead of returning fixed .T./NIL
- New rtlFLock / rtlDbUnlock for FLOCK() / DBUNLOCK()
hbrtl/register.go
- FLOCK and DBUNLOCK symbols registered (were missing entirely)
compiler/analyzer/analyzer.go
- FLOCK / DBUNLOCK added to RTL known-function set
Lock region layout (non-overlapping on purpose):
FLOCK region [0, HeaderLen+1)
Record N region [RecordOffset(N), RecordLen)
So a workarea can hold FLOCK and multiple DBRLOCK simultaneously
on the same fd without conflict.
Design rationale (captured in locks_posix.go header):
* POSIX fcntl, not flock(2) — byte-range + NFS-safe
* Non-blocking F_SETLK — matches Clipper FLOCK() → .F. semantics
* Released explicitly on Close to avoid workarea-sharing races
* Windows falls back to no-op (TODO: LockFileEx)
Verification:
go test ./hbrdd/dbf/ -run TestFLockBlocksAcrossProcesses PASS
go test ./hbrdd/dbf/ -run TestRLockBlocksAcrossProcesses PASS
go test ./... ALL PASS
FiveSql2 43/43 100%
compat_harbour 51/51 100%
The gap-analysis doc (docs/gap-analysis.md) is a running inventory
of what works vs what's still missing vs Harbour 3.2, written for
users evaluating Five for production — not a sales pitch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Benchmark (50k records, 4 indexes on Apple M-series):
before after Δ
INDEX 53.7ms 33.3ms -38% (now 10% faster than Harbour 37.3ms)
TOTAL 156.2ms 133.0ms -15%
Fixes:
1. sort.Slice(reflection) → concrete sort.Interface
Benchmarked in isolation on 200k KeyRecords:
sort.Slice(closure): 50.0ms
sort.Sort(interface): 30.4ms (40% faster, no reflection)
- indexer.go: add keyRecordAsc/Desc concrete types
- Branch hoist descending check out of Less()
2. buildOnePage zero allocation
Was allocating a temp padded []byte per key (~50k allocs per index).
Now writes padded key directly into the page buffer via padCopy.
3. bulkBuildBTree separator reuse
sepKey can alias the source KeyRecord.Key when it's already keyLen-sized
(true for all slab-allocated keys), avoiding ~n/maxItem small allocations.
Pre-size the children slice.
4. Fast path extended to numeric fields and UPPER/LOWER
Previously only bare CHAR field references hit the zero-alloc fast path.
Now:
- Numeric fields (N/F type) copy DBF bytes directly
(same-length ASCII compare matches numeric order for non-negatives)
- UPPER(field) / LOWER(field) wrappers on CHAR fields apply ASCII
case folding inline during byte copy
Per-index timing on the micro benchmark:
before after
NAME 7.7ms 7.5ms (fast path, unchanged)
CITY 6.0ms 6.2ms (fast path, unchanged)
AGE 14.1ms 7.1ms -50% (was slow path)
UPPER(NM) 17.0ms 7.9ms -54% (was slow path)
5. Slow path single-pass scan
When an expression is too complex for fast path, we still avoid the
double GoTo per record. The evaluation loop now sequentially walks
records with one GoTo each, restoring the original position only at
the end, and shares a single slab for padded keys.
Also fixes a hbrt bug surfaced while writing the benchmark:
6. Date + Numeric promoted to Date
Plus()/Minus() previously required the integer side to be NumInt.
Modulus returns a promoted type, so `SToD("...") + (i % 365)` panicked.
Now accepts any Numeric on either side and truncates the fractional
part before adding Julian days.
- hbrt/ops_arith.go: Date±Numeric (was Date±NumInt only)
Tests:
go test ./... — ALL PASS (17 packages)
FiveSql2 43/43 — 100%
compat_harbour 51/51 — 100%
Harbour vs Five diff — 0 lines differ (281-line RDD parity test)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five RDD engine now matches Harbour DBFNTX and DBFCDX byte-for-byte
in ordering, seek, navigation, and field access. Verified against
Harbour 3.2.0dev with a 281-line comparison test covering:
- Natural/NAME/CITY/AGE/SALARY/UPPER ordering
- SEEK (exact/not-found), GoTop/GoBottom per order
- DELETE/RECALL with SET DELETED
- CDX compound index read with 5 tags (BYNAME, BYCITY, BYAGE, BYSAL, BYUNAME)
- Reverse traversal
Fixes:
1. FIELD->NAME returned NIL
GetAliasField returned interface{} but runtime expected hbrt.Value,
so the type assertion in PushAliasField failed and pushed NIL.
- workarea.go: change return type to hbrt.Value, handle FIELD/_FIELD
as current-workarea alias, add SetAliasField
- gengo.go: emit SetAliasField() for alias->field := value in both
statement and expression contexts
2. OrdSetFocus(n) silently switched to natural order
v.AsString() returns "" for a numeric Value, so OrderListFocus("")
set current=-1.
- indexrtl.go: convert numeric param via fmt.Sprintf("%d", ...)
3. CDX compound tag order mismatched Harbour
Five decoded the structural B-tree which is alphabetical, but
Harbour sorts tags by TagBlock (file offset = creation order).
- cdx/cdx.go: sort tagEntries by offset ascending after decoding,
matching hb_cdxIndexLoadAvailTags in dbfcdx1.c
4. OutStd()/OutErr() not registered — caused panic on call
- hbrtl/console.go: add rtlOutStd/rtlOutErr implementations
- hbrtl/register.go: register OUTSTD and OUTERR
- analyzer.go: add OUTSTD/OUTERR to RTL known-functions
5. FIELD keyword triggered "undeclared variable" warnings
- analyzer.go: add FIELD, _FIELD, M, MEMVAR as builtin constants
Tests:
go test ./... — ALL PASS (17 packages)
FiveSql2 43/43 — 100%
compat_harbour 51/51 — 100%
Harbour diff — 0 lines differ (281-line comparison)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Register all 479 RTL functions from hbrtl/register.go (was ~60)
- Recognize module-level STATIC variables across all functions
- Declare RECOVER USING variables in analyzer scope
- Register code block parameters ({|x,y| ...}) as declared
- 2-pass multi-file build: collect cross-file function names before analysis
- Add QUIT, ERRORLEVEL, ALTSRC to builtin constants
All 3 test suites pass with 0 warnings:
go test ./... — ALL PASS
FiveSql2 43/43 — 100%
compat_harbour 51/51 — 100%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive review as if evaluated by Google Go team:
- Architecture analysis (transpiler pipeline, gengo innovations)
- Performance evidence (6/10 categories faster than C)
- Correctness proof (82/82 + 77/77 + 18/18 + 47/47)
- Strategic value (5M xBase developer bridge to Go)
- Improvement roadmap (lazy GoTo, string fusion, CDX create)
- Market positioning (vs Harbour, xHarbour, Alaska xBase++)
Key quote: "Five demonstrates that Go is ready to be a universal
compilation target, not just a language for writing programs directly."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
From senior Go developer review:
C7 CRITICAL: pagePool data race (ntx.go)
- Moved global pagePool[8] + pagePoolIdx into per-Index struct
- Eliminates race condition across goroutines using separate indexes
C8 CRITICAL: Page.data dangling pointer after remap (ntx.go)
- remapFile() now clears pagePool data slices (pointed into old mmap)
- Prevents segfault from stale mmap references
C4 HIGH: pop() bounds check restored (thread.go)
- Removed performance optimization that eliminated underflow detection
- Stack underflow now produces clear error instead of index -1 panic
C1 HIGH: intExpLen overflow on MinInt64 (value.go)
- Added special case: MinInt64 returns 20 (length of -9223372036854775808)
- Prevents -v overflow in negation
C11 CRITICAL: GoTo ReadAt error handling (dbf.go)
- ReadAt failure now returns error and sets EOF
- Previously silently used stale record buffer (data corruption risk)
C14 HIGH: LEN() inline missing Hash case (gengo.go)
- Added _v.IsHash() → len(Keys) branch
C15 HIGH: EMPTY() inline missing Date case (gengo.go)
- Added _v.IsDate() && _v.AsJulian() == 0 check
82/82 stress PASS. 14 packages ALL PASS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
value.go:
- cachedNil, cachedTrue, cachedFalse: pre-built constant Values
- MakeBool()/MakeNil(): return cached (zero allocation)
- smallInts[256]: pre-built integers 0-255 (skip intExpLen loop)
- MakeInt(): fast path for 0-255
thread.go:
- pop(): use cachedNil for GC help (no MakeNil() call)
ops_compare.go:
- LessEqual(): inline Int-Int fast path (skip valueCompare)
Direct scalar comparison with cached bool result
- Not(): inline logical fast path (skip IsLogical+AsBool)
- PopLogical(): inline type check + scalar read
Impact: these functions called millions of times in FOR/DO WHILE loops.
10K SEEK: 20ms → 16ms (20%). CDX SCOPE: 12ms → 9ms (25%).
82/82 stress PASS. 14 packages ALL PASS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DO WHILE optimization:
- Detect RDD commands in body (SKIP/GO/SEEK/REPLACE/DELETE)
- If no USE/SELECT (safe), hoist _dwa/_darea before loop
- SKIP/GO/SEEK/DELETE inside loop use cached area variable
- Eliminates WA lookup + Current() per iteration
SEEK optimization:
- Use hoisted area when inside DO WHILE or FOR hoist context
- Eliminates WA lookup per SEEK call in tight loops
DELETE optimization:
- Use hoisted area when available
All commands now check g.hoistedDW || g.hoistedFields:
- GO TOP/BOTTOM/n → cached area
- SKIP n → cached area
- SEEK key → cached area + Indexer check
- DELETE → cached area
- APPEND → cached area (FOR loop)
- REPLACE → cached _rdbf + _rfiN (FOR loop)
82/82 stress PASS. 14 packages ALL PASS.
CDX SCOPE: 12ms (Harbour 4ms = 3x)
NTX SCAN: 24ms (Harbour 5ms = 4.8x)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When FOR body contains APPEND+REPLACE and no USE/SELECT:
- Hoist WorkAreaManager, Current(), *dbf.DBFArea outside loop
- Pre-compute FieldIndex for all REPLACE fields once
- REPLACE inside loop uses cached _rdbf and _rfiN variables
- APPEND inside loop uses cached _rarea (no WA lookup per iter)
Safety: collectReplaceFields returns nil if USE/SELECT found in body
(workarea may change → cannot safely cache). Falls back to normal emit.
10K APPEND benchmark: 28ms (Harbour 27ms — essentially equal!)
82/82 stress test PASS. 14 packages ALL PASS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
