Commit Graph

24 Commits

Author SHA1 Message Date
f9ffd4050e perf(FiveSql2): FieldGet peephole + DBFArea devirt — WHERE at ~1.15x raw RDD
Two stacked optimizations land on the SqlScan hot path. Combined
effect on the 50k-row benchmark:

                       Before    After   vs raw
  Numeric WHERE        10.2ms    7.8ms   1.15x
  String WHERE         10.5ms    7.9ms   1.15x
  No WHERE              9.2ms   10.0ms   1.45x
  Raw RDD baseline      6.8ms    6.8ms   1.00x

WHERE-predicate paths are now within 15% of the raw Harbour-style
RDD scan loop. The no-WHERE path is unchanged (slight jitter from
the added devirt branch); FieldGet peephole doesn't apply there.

--- Optimization 1: PcOpFieldGet peephole ---

Adds a new pcode opcode `PcOpFieldGet <fieldIdx>` (0x46) that skips
the usual PushSymbol+Function+Frame+FieldGet-RTL+EndProc chain and
calls a direct field getter closure instead. genpc recognizes the
shape `FieldGet(<int-literal>)` during emitCall and emits the
specialized opcode automatically — no SQL-side API change.

Integration:
  * hbrt.Thread.FastFieldGetter  — hot-path closure set by scan loops.
                                   Non-nil → pcode bypasses dispatch.
                                   Nil → pcode resolves FIELDGET via
                                   the RTL symbol table (correctness
                                   fallback for any other callers).
  * compiler/genpc/genpc.go      — peephole in emitCall.
  * hbrt/pcinterp.go             — PcOpFieldGet handler.

This alone cut numeric WHERE from 10.2 → 7.9ms: eliminated roughly
one full Frame/EndProc + RTL dispatch per row × 50k rows.

--- Optimization 2: DBFArea devirtualization ---

SqlScan type-asserts the workarea to *dbf.DBFArea once and runs a
dedicated loop that calls GoTop/EOF/Skip/GetValue directly on the
concrete type. Go's compiler inlines these, skipping the interface
vtable per row. Non-DBF drivers still work via the generic Area
branch.

The FastFieldGetter closure also captures *DBFArea directly in the
DBF branch, so the WHERE predicate side of the hot loop is now
entirely devirtualized: no interface dispatch between the pcode
dispatch loop and the DBF record buffer.

Validation:
  - FiveSql2 43/43
  - Harbour compat 51/51
  - go test ./... ALL PASS

Remaining gap to raw RDD on no-WHERE (~1.45x) is dominated by the
two-column row construction + ArraySlab + flat backing bookkeeping
that the raw loop doesn't do. Going below that requires changing
the SQL engine's result shape — out of scope here.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 12:23:31 +09:00
5c067f35a4 perf(hbrt): ExecPcodeFast — pcode variant without defer/recover
Pcode expressions compiled from SQL WHERE clauses (via genpc.CompileExpr)
never contain BEGIN SEQUENCE and can't raise BreakValue, so the defer +
recover dance in ExecPcode's EndProc is pure overhead. For FiveSql2's
per-row WHERE evaluation on a 50k-row scan, that's 50k × ~15ns = ~750µs
of pointless recover bookkeeping.

Split ExecPcode into two variants sharing execPcodeBody:

  ExecPcode     — full: Frame + defer EndProc. General-purpose,
                  handles panics. Behavior unchanged.

  ExecPcodeFast — hot: Frame + execPcodeBody + EndProcFast. No defer,
                  no recover. Caller guarantees the pcode body can't
                  panic with HbError / BreakValue.

SqlScan now uses ExecPcodeFast for per-row WHERE evaluation. Measured
impact on 50k-row no-WHERE benchmark: 10.6ms → 9.2ms steady state
(~13% faster). Effect is smaller on numeric-WHERE because per-row
cost there is dominated by the opcode dispatch itself, not the frame
exit.

Validation:
  - FiveSql2 43/43
  - go test ./hbrt/... PASS (pcode tests)
  - go test ./hbrtl/... PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 12:07:54 +09:00
85541a3035 perf(sqlscan): flat backing buffer — 30% faster no-WHERE scan
The prior loop allocated one small `[]hbrt.Value` per matching row
(for the row body) plus one HbArray header. For a 50k-row full scan
that's 100k allocations of which the small-slice allocs dominated
fragmentation and GC pressure.

SQLite-inspired fix: pre-allocate a single flat []hbrt.Value of
capacity `RecCount * nFields` at scan start and hand each row a
three-index sub-slice (flat[off:end:end]). The capped sub-slice
still forces a reallocation if PRG code later does `AAdd(row, x)`,
so neighbor rows can't get clobbered.

Sizing the initial buffer off RecCount(err-ignored) was the actual
win — the previous naive grow-from-1024 policy caused five mid-scan
reallocations of a ~200 KB buffer, each memcpy'ing everything so far.
One upfront allocation amortizes much better.

Bench (50k rows, ~/tmp ext4, 3 runs steady-state):

                          Before        After       Δ
  no WHERE                14.6ms       10.6ms     −27%
  numeric WHERE           11.7ms       10.0ms     −15%
  string WHERE            10.5ms       11.0ms     ~=
  raw RDD baseline         6.8ms        7.0ms

Gap to raw RDD: 2.1x → 1.4x on the dominant no-WHERE case. What's
left is pcode WHERE dispatch (ExecPcode frame per row), the Area
interface boundary, and the HbArray header allocation per row —
all structural costs that would need a wider refactor to close.

Validation:
  - FiveSql2 43/43
  - go test ./hbrtl/... PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:57:05 +09:00
d74014a235 feat(rdd): dbInfo / dbOrderInfo — implement the stubs
Replaces the `return NIL` stubs with real implementations that read
from the current workarea. Covers the info codes actually used by
downstream code (FiveSql2 TSqlIndex, standalone callers):

DBINFO:
  DBI_ISDBF, DBI_CANPUTREC, DBI_FULLPATH, DBI_TABLEEXT, DBI_MEMOEXT,
  DBI_SHARED, DBI_ISREADONLY, DBI_GETRECSIZE, DBI_DBVERSION,
  DBI_RDDVERSION, DBI_BOF, DBI_EOF, DBI_FOUND, DBI_FCOUNT, DBI_ALIAS,
  DBI_POSITIONED

DBORDERINFO:
  DBOI_EXPRESSION, DBOI_NAME, DBOI_NUMBER, DBOI_POSITION,
  DBOI_ORDERCOUNT, DBOI_KEYCOUNT, DBOI_KEYCOUNTRAW

Unknown info codes still return NIL (Harbour's forgiving fallback).

New accessors on DBFArea (FullPath, IsShared, IsReadOnly) expose the
private filePath/shared/readOnly fields to the hbrtl layer without
plumbing them through the generic Area interface.

Unblocks TSqlIndex:FindExclusive's original DBI_FULLPATH/DBI_SHARED
scan — though the short-circuit there stays in place for now since
it's a correctness workaround that no longer masks a crash thanks
to the recent gengo PushMemvar fallback.

Validation:
  - FiveSql2 43/43 (0 warnings)
  - Harbour compat 51/51
  - go test ./... ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:42:18 +09:00
3a00aa5435 feat(hbrtl): field metadata + index creation RTL — TSqlIndex warnings to zero
TSqlIndex.prg had five undefined identifiers and six undefined
constants that the new CLASS-method analyzer surfaced after the
gengo PushMemvar fallback stopped crashing on them. All real tech
debt, not false positives. This lands the implementations.

New RTL functions (hbrtl/indexrtl.go + register.go):
  - FieldType(n) → "C"/"N"/"L"/"D"/"M"/... one-letter type
  - FieldLen(n)  → length in bytes
  - FieldDec(n)  → decimal places
  - ordCreate(cBag, cTag, cExpr [, bExpr] [, lUnique])
      → DBFArea.OrderCreate with TagName set (CDX tag or NTX tag)
  - dbCreateIndex(cFile, cExpr [, bExpr] [, lUnique])
      → legacy Clipper single-tag NTX without TagName
  - dbClearIndex() → OrderListClear

All pass through the existing Indexer interface; key expressions go
through the MacroEval slow path since callers pass string literals.
When callers are updated to pass compiled key blocks, the existing
KeyFunc fast path kicks in automatically.

New header files (include/):
  - dbinfo.ch  — DBI_* and DBOI_* constants with Harbour-compatible
                 values (FULLPATH=10, SHARED=42, EXPRESSION=2, etc.)
  - dbstruct.ch — DBS_NAME/TYPE/LEN/DEC field descriptor indices

TSqlIndex.prg already did `#include "dbinfo.ch"` and `#include
"dbstruct.ch"` but Five's preprocessor silently ignored the missing
files. Both headers land in include/ where cmd/five's include-dir
chain already looks.

Analyzer RTL allow-list updated with the six new function names so
the warning pipeline stays clean.

Result: FiveSql2 build goes from 17 WARN → 0. Both tracked test
suites still pass.

Note: dbInfo() / dbOrderInfo() themselves remain stubbed (return NIL)
— the constants exist for compile-time resolution and for future use
when the stubs are replaced. Callers that depend on actual dbInfo
values still get NIL at runtime.

Validation:
  - FiveSql2 43/43
  - Harbour compat 51/51
  - go test ./... ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:11:57 +09:00
8aaed994f4 perf(FiveSql2): hybrid fast path — 11x speedup on string WHERE scans
Implements hybrid execution model: keep AST tree-walk for SQL:2013+
features (Window, Recursive CTE, JOIN, aggregates) while compiling
simple SELECT hot paths to Go + pcode. See docs/FiveSql2-Hybrid-Plan.md
for the full architecture rationale (why not SQLite-style VDBE).

Hot path (single table, no joins/groups/aggregates):
  - TryBuildFieldPositions: resolves SELECT column list to FieldPos
    array once per query (bails to PRG loop on any complex expr).
  - TryCompileWhere + SqlExprToPrg: walks WHERE AST, emits equivalent
    PRG source, runs it through PcCompile to get a PcodeFunc.
  - SqlScan RTL: Go-native scan loop — GoTop/EOF/Skip/GetValue
    direct, ExecPcode per row for WHERE, result array pre-alloc.

WHERE compiler scope:
  - ND_LIT numeric/logical/string (string literals AllTrim'd to match
    SqlCmpEq CHAR-padding semantics; rejects embedded quotes/newlines)
  - ND_COL: CHAR fields auto-wrapped with AllTrim(FieldGet(n)) based
    on dbStruct() lookup cached once per query in aCompileStruct
  - ND_BIN: = <> != < <= > >= AND OR + - * /
  - ND_UNI: NOT -
  - Anything else (ND_FN, ND_CASE, ND_SUB, ND_PAR, LIKE, IN, IS NULL,
    BETWEEN, dates) returns NIL → falls back to PRG tree-walk.

Bench (50k rows, ~/tmp ext4):
                        Before      After     Speedup
  Numeric WHERE         ~150ms     11.7ms     ~13x
  String WHERE          119.3ms    10.5ms     11.4x
  No WHERE               -         14.6ms      -
  Raw RDD baseline        6.8ms     6.8ms      1.0x

Remaining gap to raw RDD (~1.5x) is structural: Value boxing, result
array construction, per-row ExecPcode frame overhead. Would need a
Value-pool or SoA refactor to close further.

Side fixes bundled:
  - TSqlIndex:FindExclusive short-circuited. Originally called
    dbInfo(DBI_FULLPATH)/DBI_SHARED which are unresolved symbols in
    Five (dbInfo is a stub, DBI_* never defined). Panic'd with
    "local variable index out of range: 0" whenever a standalone PRG
    had a workarea Used before calling five_SQL. 43-test masked the
    bug because it only reached FindExclusive with no open workareas.
    Restore the scan once dbInfo lands in hbrtl.
  - cmd/five/main.go: FIVE_KEEP_BUILD=1 env var keeps the temp Go
    project around for debugging gengo output.

Validation:
  - FiveSql2 43/43
  - Harbour compat 51/51
  - go test ./... ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 09:15:08 +09:00
6b26f1b642 feat: genpc.CompileExpr + PcCompile/PcEval runtime bytecode API
Expose Five's existing FRB bytecode compiler for single-expression
compilation, enabling prepared-statement-style caching in dynamic
query engines (FiveSql2, scripting layers, rule engines).

1. genpc.CompileExpr(ast.Expr) *hbrt.PcodeFunc
   - New public API that compiles a single expression to a
     standalone pcode function
   - Reuses genpc's mature emitExpr (no new emit logic)
   - ExecPcode manages the frame around the generated code

2. hbrtl.PcCompile(cPrgExpr) -> pFunc
   - RTL entry point for runtime compilation
   - Wraps the expression in a FUNCTION stub, uses the full PRG
     parser pipeline (pp + parser + genpc), extracts the compiled
     pcode function, returns it as an opaque pointer
   - Callers pay parse+compile cost ONCE per expression

3. hbrtl.PcEval(pFunc) -> xValue
   - RTL entry point for runtime execution
   - Calls hbrt.ExecPcode; the pcode's RetValue opcode sets retVal,
     which our EndProc preserves as PcEval's return value
   - ~1.2x slower than direct FieldGet (pcode interpreter overhead),
     but eliminates AST tree-walk per row for complex expressions

Usage (FiveSql2 hot path, planned):
   pc := PcCompile("FieldGet(4) > 50000")  // parse+compile once
   WHILE !Eof()
      IF PcEval(pc)                         // ~10us per row
         AAdd(aRows, ...)
      ENDIF
      dbSkip()
   ENDDO

Benchmark (50k records, WHERE salary > 50000):
   Raw FieldGet:      7.9 ms  (baseline)
   FieldPos+Get:     10.2 ms  (with O(1) FieldPos cache)
   PcEval bytecode:  10.1 ms  (interpreted bytecode)
   MacroEval:        parse+eval per row — orders of magnitude slower

Tests:
   go test ./...        ALL PASS (14 packages)
   FiveSql2 43/43       100%
   compat_harbour       51/51
   PcCompile/PcEval     verified on 50k-row scan

FiveSql2 engine integration deferred — requires careful PRG-level
refactoring to thread pcode pointers through the plan structure.
The Go-level infrastructure is now in place for that work.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 07:57:52 +09:00
ed33af41c5 perf: FieldPos O(1) cache + xbase import detection for function-call PRGs
Two SQLite-style optimizations for RDD and SQL workloads:

1. FieldPos() O(1) column binding cache

   Before: FieldPos(name) linear scan — O(n) per call with string
           comparison. In SQL engines that call FieldPos per row per
           column, this is hundreds of thousands of calls.

   After:  DBFArea builds a map[UPPER(name)]→pos on first lookup.
           All subsequent lookups are O(1) hash. SQLite calls this
           "column affinity binding" — positions resolved at prepare,
           not per row.

   Implementation:
     - hbrdd/dbf/dbf.go: DBFArea.FieldPosCache(name) method
     - hbrtl/procinfo.go: FieldPos RTL uses fieldPosCacher interface
     - Lazy init: only pays for tables that get queried

2. hbrdd import auto-detection for function-call style PRGs

   Before: compiler only added hbrdd import when PRG used xBase commands
           (USE, SKIP, INDEX...). Pure function-call style like
           `dbUseArea(.T.,,"t")`, `FieldPut(1, val)` was missed —
           generated Go failed to compile ("undefined: hbrdd").

   After:  scanStmtsForXBase walks ExprStmt bodies too, detecting
           CallExpr to any of the ~40 xBase RTL function names.
           FIELD->NAME alias expressions also trigger the import.

   Resolves: small PRGs that use only dbUseArea/FieldGet/FieldPut.

Benchmark notes (50k records):
  Raw RDD scan:              7 ms    (baseline)
  FiveSql2 SELECT WHERE:   157 ms    (unchanged — bottleneck is
                                      not FieldPos, it's PRG-level
                                      expression tree walk per row)
  compat_harbour 51/51:    PASS
  FiveSql2 43/43:          100%

The FieldPos cache helps heavy field-name-based code paths but the
primary FiveSql2 bottleneck is the PRG interpreter walking expression
ASTs per row (needs bytecode compilation to close the gap).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 07:42:00 +09:00
3adc9d7d59 fix: PCount, Break/RECOVER, SET INDEX TO — 3 Harbour compat fixes
Release-blocking compatibility issues discovered during the 258-test
pre-release validation suite (100 syntax + 44 RDD + 114 RTL).

1. PCount() always returned 0 in PRG code

   Root cause: ParamCount() returned t.pendingParams, which is
   overwritten by every nested Function() call. By the time the
   PCount() RTL's Frame() executes, pendingParams is already 0.

   Fix: Frame() now stores pendingParams in frame.paramCount.
   PCount() RTL uses CallerParamCount() which reads callSP-2
   (the PRG caller's frame), while RTL functions still use
   ParamCount() (reads pendingParams before their own Frame).

   Verified: PCount(1,2,3)=3, PCount(1)=1, PCount()=0

2. Break("string") panicked instead of being caught by RECOVER USING

   Root cause: Generated SEQUENCE code only caught *HbError panics.
   Break() panics with BreakValue (a different type), which fell
   through to EndProc's "runtime error" message and re-panic.

   Fix (two parts):
   a) gengo emitBeginSequence: recover closure now catches any
      panic (interface{}), then dispatches via type switch:
      - *HbError → extract .Error() string
      - hasValue interface (BreakValue) → extract .GetValue()
      - other → static "error" string
   b) hbrtl/error.go: BreakValue gets GetValue() method for
      duck-type detection without import cycles
   c) hbrt/thread.go EndProc: BreakValue type name check added
      so it re-panics silently (no stderr noise)

3. SET INDEX TO a, b, c only opened the last file

   Root cause: Parser's parseSet() called parseExpr() once for
   INDEX setting, stopping at the first comma. Remaining file
   names were consumed by the "eat rest of line" loop.

   Fix: Parser now collects comma-separated identifiers into a
   single string literal "a,b,c". gengo splits on comma and
   calls OrderListAdd() for each file.

   Verified: SET INDEX TO si_name, si_city → OrdCount=2

All tests pass:
  go test ./...          14 packages OK
  FiveSql2               43/43  100%
  compat_harbour         51/51
  Syntax test           100/100
  RDD test               44/44
  RTL test              114/114
  Windows cross-compile  OK
  Linux cross-compile    OK

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 18:06:28 +09:00
fc1dca9551 feat(rdd): real POSIX file/record locking + gap analysis doc
Replaces the FLOCK/DBRLOCK/DBRUNLOCK no-op stubs with actual
fcntl(F_SETLK) byte-range advisory locks, matching Harbour's
hb_fsLockLarge implementation.

Before: rtlDbRLock always returned .T. regardless of contention.
        Multi-process writers could silently corrupt records.

After:  Non-blocking POSIX byte-range locks per file descriptor.
        Cross-process exclusion verified by a subprocess-spawning
        Go test that witnesses BUSY vs OK transitions.

New files:
  hbrdd/dbf/locks_posix.go    fcntl F_WRLCK/F_UNLCK wrappers
  hbrdd/dbf/locks_windows.go  stub (TODO: LockFileEx)
  hbrdd/dbf/lock_multi_test.go   cross-process verification
  docs/gap-analysis.md        honest Harbour parity assessment

Modified:
  hbrdd/dbf/dbf.go
    - DBFArea gains fileLocked bool + lockedRecs map
    - Close() calls releaseAllLocks() before dropping the fd
  hbrtl/database.go
    - rtlDbRLock / rtlDbRUnlock now delegate to DBFArea.LockRecord /
      UnlockRecord instead of returning fixed .T./NIL
    - New rtlFLock / rtlDbUnlock for FLOCK() / DBUNLOCK()
  hbrtl/register.go
    - FLOCK and DBUNLOCK symbols registered (were missing entirely)
  compiler/analyzer/analyzer.go
    - FLOCK / DBUNLOCK added to RTL known-function set

Lock region layout (non-overlapping on purpose):
  FLOCK region       [0, HeaderLen+1)
  Record N region    [RecordOffset(N), RecordLen)

So a workarea can hold FLOCK and multiple DBRLOCK simultaneously
on the same fd without conflict.

Design rationale (captured in locks_posix.go header):
  * POSIX fcntl, not flock(2) — byte-range + NFS-safe
  * Non-blocking F_SETLK — matches Clipper FLOCK() → .F. semantics
  * Released explicitly on Close to avoid workarea-sharing races
  * Windows falls back to no-op (TODO: LockFileEx)

Verification:
  go test ./hbrdd/dbf/ -run TestFLockBlocksAcrossProcesses  PASS
  go test ./hbrdd/dbf/ -run TestRLockBlocksAcrossProcesses  PASS
  go test ./...                                             ALL PASS
  FiveSql2 43/43                                            100%
  compat_harbour 51/51                                      100%

The gap-analysis doc (docs/gap-analysis.md) is a running inventory
of what works vs what's still missing vs Harbour 3.2, written for
users evaluating Five for production — not a sales pitch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 17:58:03 +09:00
e95afad4ee feat: Harbour RDD parity — NTX/CDX 100% compatible, FIELD-> works
Five RDD engine now matches Harbour DBFNTX and DBFCDX byte-for-byte
in ordering, seek, navigation, and field access. Verified against
Harbour 3.2.0dev with a 281-line comparison test covering:
  - Natural/NAME/CITY/AGE/SALARY/UPPER ordering
  - SEEK (exact/not-found), GoTop/GoBottom per order
  - DELETE/RECALL with SET DELETED
  - CDX compound index read with 5 tags (BYNAME, BYCITY, BYAGE, BYSAL, BYUNAME)
  - Reverse traversal

Fixes:

1. FIELD->NAME returned NIL
   GetAliasField returned interface{} but runtime expected hbrt.Value,
   so the type assertion in PushAliasField failed and pushed NIL.
   - workarea.go: change return type to hbrt.Value, handle FIELD/_FIELD
     as current-workarea alias, add SetAliasField
   - gengo.go: emit SetAliasField() for alias->field := value in both
     statement and expression contexts

2. OrdSetFocus(n) silently switched to natural order
   v.AsString() returns "" for a numeric Value, so OrderListFocus("")
   set current=-1.
   - indexrtl.go: convert numeric param via fmt.Sprintf("%d", ...)

3. CDX compound tag order mismatched Harbour
   Five decoded the structural B-tree which is alphabetical, but
   Harbour sorts tags by TagBlock (file offset = creation order).
   - cdx/cdx.go: sort tagEntries by offset ascending after decoding,
     matching hb_cdxIndexLoadAvailTags in dbfcdx1.c

4. OutStd()/OutErr() not registered — caused panic on call
   - hbrtl/console.go: add rtlOutStd/rtlOutErr implementations
   - hbrtl/register.go: register OUTSTD and OUTERR
   - analyzer.go: add OUTSTD/OUTERR to RTL known-functions

5. FIELD keyword triggered "undeclared variable" warnings
   - analyzer.go: add FIELD, _FIELD, M, MEMVAR as builtin constants

Tests:
  go test ./...        — ALL PASS (17 packages)
  FiveSql2 43/43       — 100%
  compat_harbour 51/51 — 100%
  Harbour diff         — 0 lines differ (281-line comparison)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:37:47 +09:00
486e466592 feat: FiveSql2 43/43, @byref, mutable closure, RTL 479, DateTime fix
Major changes since last commit:
- FiveSql2 SQL:1999 engine (10,458 LOC) — 43/43 ALL PASS
- 21 compiler/runtime bugs fixed (short-circuit AND/OR, FOR LOOP, etc.)
- @byref pass-by-reference via RefCell pattern
- Mutable closure capture (EnsureLocalRef + RefCell sharing)
- RTL: 400 → 479 functions (+79: file, string, datetime, hash, UTF-8)
- DateTime/Timestamp fully working (hb_DateTime, hb_Hour/Min/Sec, display)
- Reserved word guard (39 keywords blocked from function calls)
- AEval arg order fix (element before index)
- Closure capture redecl fix (unique _cap_ names per block)
- Hash/string indexing in ArrayPush/ArrayPop
- Harbour compat test suite: 51/51
- 4 docs: Porting Report, Implementation Plan, Optimization Plan, Commercialization

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 11:35:37 +09:00
d451b836a6 perf: inline Str/PadR/PadL/SubStr/Left/Right/At/IIF in gengo
13 more RTL functions inlined — no Frame/EndProc, no VM dispatch:
- Str(n,w,d) → fmt.Sprintf("%*.*f", w, d, n)
- PadR(s,n) → s + hbrtl.Spaces(n-len(s))
- PadL(s,n[,fill]) → Spaces(pad) + s or Repeat(fill, pad) + s
- SubStr(s,p,l) → s[p:p+l] with bounds check
- Left(s,n) → s[:n], Right(s,n) → s[len-n:]
- At(search,target) → strings.Index + 1
- IIF(cond,a,b) → if/else without function call

Also: Spaces() exported for generated code access.

50K SEEK random: 62ms (Harbour 67ms — Five FASTER!)
82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 23:16:38 +09:00
05ccef05e2 perf: EndProcFast — eliminate defer recover() from RTL hot paths
Problem: every RTL function calls defer t.EndProc() which does recover().
50K SEEK loop = 250K recover() calls = ~12ms wasted.

Solution: EndProcFast() skips recover (only needs endFrame restore).
Applied to ALL RTL functions in strings.go, rdd.go, missing.go, database.go.
EndProc() with recover kept for generated PRG code (needs BEGIN SEQUENCE).

Analysis (50K sequential SEEK breakdown):
  Go NTX Seek direct: 7ms (faster than Harbour 27ms!)
  PRG VM overhead:    38ms (Frame + RTL calls + key generation)
  Key generation:     25ms (Str+LTrim+PadL+PadR = 5 RTL Frame/EndProc per iter)

With EndProcFast: RTL overhead reduced ~30%.

CDX SCOPE: 2ms (Harbour 4ms — 2x FASTER!)
82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 21:43:39 +09:00
5b378318a0 perf: RTL optimization — cached WA, spaces pool, stack-alloc fmt_int64
rdd.go:
- getWA() cached type assertion (avoid repeated interface check)
- waCache stores last WA pointer → O(1) for repeated calls

strings.go:
- spacesCache[257]: pre-built space strings for pad sizes 0-256
- spaces(n) returns cached string (no Repeat allocation)
- PadR/PadL use spaces() for fill=" " (most common case)
- Str() uses spaces() for right-padding

missing.go:
- fmt_int64: stack-allocated [20]byte array (was heap make([]byte))
- Reverse iteration (no prepend overhead)
- PadC uses spaces() for left/right padding

Benchmark (ext4, home dir):
  10K APPEND: 28ms → 26ms (Harbour 27ms!)
  50K APPEND: 130ms → 113ms (13% improvement)
  50K SCAN: 24ms → 23ms
  50K DUPKEY: 42ms → 35ms (17% improvement)
  CDX SCOPE: 12ms → 10ms (17% improvement)

82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 18:16:22 +09:00
b7028791d6 fix: 5 seek/dbf bugs — 77/77 thorough Harbour compatibility
1. SOFTSEEK: use idx.CurRecNo() for positioning (was checking recNo > 0)
   - SEEK with SET SOFTSEEK ON now positions at next higher key
   - SEEK command reads SET SOFTSEEK at runtime (was compile-time only)
   - rtlDbSeek defaults to GetSetSoftSeek() when no explicit param

2. SET DELETED ON + INDEX: SkipIndexed skips deleted records
   - GoTopIndexed: skip deleted record at top position
   - SkipIndexed: inner loop continues past deleted records

3. Compound key (CITY+NAME): field name TrimSpace before lookup
   - evalKeyExprInner: TrimSpace on fieldName after FIELD-> strip
   - Fixed "CITY " != "CITY" mismatch from + operator splitting

4. SET INDEX TO filename: treated as string, not variable
   - gengo uses exprToString for SET INDEX TO (was emitExpr)
   - Prevents identifier being resolved as local variable

5. hasXBaseCommands: recursive scan into nested blocks
   - BEGIN SEQUENCE, IF, FOR, DO WHILE, SWITCH bodies now scanned
   - Fixes missing hbrdd import for DB commands inside blocks

Thorough test: 77 items (14 sections) covering exact/partial/soft seek,
SET DELETED, duplicate keys, numeric keys, compound keys, empty/single
table, state consistency, order switching, full traversal — all identical.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 14:08:51 +09:00
7e2a159b88 feat: CDX support + ORDSCOPE + cross-read Harbour compatibility
CDX Integration:
- IndexEngine interface: common for NTX Index and CDX Tag
- OrderListAdd: auto-detects .cdx/.ntx extension, opens CDX tags
- decodeCompoundLeaf: proper bit-packed tag directory decoding
  (was stub falling through to scanCompoundLeaves with wrong names)
- CDX Tag: added KeyLen(), KeyExpr(), ForExpr(), IsDescending(), Close()
- CDX compound recNo = direct byte offset (not page number)

ORDSCOPE:
- SetScope/ClearScope/SetScopeTop/SetScopeBottom on DBFArea
- GoTopIndexed: seeks to scopeTop, validates within scopeBottom
- GoBottomIndexed: seeks to scopeBottom boundary
- SkipIndexed: stops at scope boundaries (top and bottom)
- OrdScope RTL function registered (nScope: 0=TOP, 1=BOTTOM)
- scopeKeyFromValue: converts Value to padded key bytes

Index Order Management:
- OrderListFocus: handles numeric order ("2" → order 2)
- SET ORDER TO n: gengo emits hbrt.NtoS for int-to-string conversion
- IndexOrd/OrdCount/OrdName/OrdKey: real implementations (were stubs)
- OrderCount/CurrentOrder/OrderName/OrderKeyExpr accessors on DBFArea
- ClearScope on order switch (prevents stale scope)

Cross-read test: Harbour-created CDX → Five reads, 20/20 items match:
  NAME/CITY/ID seek, ORDSCOPE count, GoTop/GoBottom all identical

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 12:21:26 +09:00
6e78d12cc2 fix: 3 RDD compat bugs — FIELD->, AsNumInt Double, PACK/ZAP with index
Bug 1: FIELD->NAME in INDEX ON expression
- evalKeyExprInner: strip FIELD->/alias-> prefix before field lookup
- exprToString: handle AliasExpr (FIELD->NAME → "FIELD->NAME")

Bug 2: AsNumInt() on Double returned IEEE 754 raw bits
- Value.AsNumInt(): check tDouble and convert via Float64frombits
- Fixed array index crash when index is result of % modulo

Bug 3: PACK/ZAP crash with open indexes
- OrderListRebuild: fully implemented (was TODO stub)
  Saves index info, closes all, sets idxState=nil, recreates
- OrderCreate: set current=-1 during key evaluation (natural GoTo)
- PACK/ZAP: save/restore idxState, rebuild after operation
- Register __DBPACK, __DBZAP, DBRECALL symbol aliases

Harbour vs Five: 45/47 match (96%), 2 diffs are duplicate-key sort order

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 04:41:19 +09:00
21fd9dc65c feat: SET DELETED filtering, SEEK/LOCATE/CONTINUE, SET command codegen
- skipFilter: skip deleted records in GoTop/GoBottom/Skip when SET DELETED ON
- hbrdd.IsSetDeleted callback: avoids circular import hbrdd→hbrtl
- Parser: capture ON/OFF for boolean SET commands (DELETED, EXACT, SOFTSEEK, etc.)
- Parser: capture TO expr for SET DATE/DECIMALS/EPOCH
- Gengo: emit proper t.Do() calls for 11 SET toggles + 3 value SETs
- stmtSet: was stub (skipToEOL), now calls parseSet()
- RTL: register 11 SET toggle functions (SETDELETED, SETEXACT, etc.)
- RTL: DBLOCATE/DBCONTINUE for sequential search
- RTL: DBSETFILTER/DBCLEARFILTER/DBFILTER
- PadL/PadR: support 3rd param fill character
- Area interface: added SetFound, SetLocate, LocateBlock, filter methods
- MemRDD: implements new Area interface methods
- Comprehensive PRG test: test_search.prg (7 test suites all pass)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 22:33:59 +09:00
827adeeb99 feat: SET commands + ErrorBlock/Break error handling
SET commands (setcmd.go):
  - SetDateFunc: __SetDateFormat([cNew]) → cOld
  - SetDecimalsFunc: SET DECIMALS TO n
  - SetEpochFunc: SET EPOCH TO n
  - 11 toggle functions: SetExact, SetDeleted, SetSoftSeek, SetExclusive,
    SetFixed, SetCancel, SetBell, SetConfirm, SetInsert, SetEscape, SetWrap
  - SET constants: _SET_EXACT, _SET_DELETED, etc. for PRG code
  - GetSetDateFormat(), GetSetDecimals(), GetSetEpoch() helpers
  - Default: DATE="mm/dd/yy", EPOCH=1900, DECIMALS=2

Error handling (error.go):
  - Break(xValue): panics with BreakValue, caught by BEGIN SEQUENCE
  - BreakBlock(): returns {|e| Break(e)} code block
  - LaunchError(): dispatches error through ErrorBlock handler
  - RuntimeError(): creates + launches standard runtime error
  - IsBreak(): checks if recovered panic is a BreakValue
  - createErrorHash(): builds Harbour-compatible error hash

Registration: ErrorBlock, ErrorNew, DosError, FError, Break + all SET functions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:33:07 +09:00
f950cb0784 fix: Phase 2 — HIGH #6,9,10,11,12,19,23,32,46,47
Files modified (5):
  compiler/gengo/gengo.go — #6,#32: Deduplicate 3 Generate functions into 1
    doGenerate(file, debug, library) replaces 170 lines of copy-paste
    Dead GenerateDebug method removed
  cmd/five/main.go — #9,10,11,12: Fix 5 silently ignored errors
    filepath.Abs, tidy.Run, tidyCmd.CombinedOutput now checked
  hbrtl/strings.go — #19: Str() now reads caller's nWidth/nDec params
    Was ignoring explicit Str(123, 10, 2) arguments
  compiler/pp/pp.go — #46: Fix stale "NOT implemented" comment
    #47: Extract maxIncludeDepth constant
  compiler/lexer/lexer.go — #23: Remove unused LookupKeyword result

Issues resolved: 10 (HIGH: 7, MEDIUM: 3)
Total fixed: 26/53

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 11:47:26 +09:00
7c61db70c3 fix: Critical code review fixes — race conditions, panic recovery, LTRIM/RTRIM
CRITICAL fixes:
- fileio.go: Add sync.Mutex to file handle table (race condition #2)
  allocHandle/getHandle/removeHandle thread-safe helpers
- goroutine.go: Add defer/recover to GoLaunch/GoLaunchBlock
  Goroutine panic no longer crashes entire process (#5)

HIGH fixes:
- strings.go: Implement proper LTrim (TrimLeft) and RTrim (TrimRight)
  Previously both aliased to AllTrim — silent semantic bug (#18)
- register.go: TRIM = RTrim (Harbour compatible)

From 53-issue senior code review.
Remaining: 47 issues (HIGH: 10, MEDIUM: 18, LOW: 16, CRITICAL: 3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 10:17:30 +09:00
272576f6ce feat: Harbour-compatible VM shutdown sequence
Implements full cleanup on program exit (normal, Ctrl+C, crash):
- EXIT PROCEDURE auto-execution (reverse module order)
- AtExit callback registry (LIFO order)
- All WorkAreas auto-close (child before parent)
- Terminal restore (raw → normal) on signal/exit
- Static variables clear
- Signal handlers (SIGINT, SIGTERM) for clean shutdown
- shutdown.go: Harbour hb_vmQuit() 25-step sequence adapted for Five
- vm.go: Run() now calls Shutdown() via defer
- rawtty.go: terminal restore registered with shutdown system

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 10:07:42 +09:00
59568f3301 Five v0.9 — Harbour + Go fusion language
- Compiler: PP → Lexer → Parser → Analyzer → Gengo pipeline
- Parser: 232/236 (98%) Harbour compatibility, registry-based dispatch
- RTL: 351 Harbour-compatible functions
- RDD: DBF/NTX/CDX engines with Rushmore bitmap optimization
- Go Interop: IMPORT + pkg.Func() + obj:Method() with FastPath (15M calls/sec)
- HB_FUNC API: Full Harbour C API compatible Go bridge
- Concurrency: SPAWN/LAUNCH/GOROUTINE, <-, WATCH, PARALLEL FOR, ASYNC/AWAIT
- Extensions: Multi-return, DEFER, Slice, f-string, Nil-safe ?:, CONST
- Macro Compiler: Runtime AST parsing and evaluation
- Debugger: TUI debugger with source display, breakpoints, stepping
- FRB: Native + Pcode dual mode runtime binary
- Tests: 13 packages ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 09:41:50 +09:00