Commit Graph

73 Commits

Author SHA1 Message Date
3ed246c47e feat(rdd): Windows LockFileEx implementation — real byte-range locks
Replace the no-op Windows lock stub with actual kernel32 LockFileEx /
UnlockFileEx calls via syscall.LazyDLL (zero external dependency).

- LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY for non-blocking
  semantics matching Clipper FLOCK() → .F.
- Same lock region layout as POSIX: header region for FLOCK, record
  offsets for DBRLOCK — compatible across platforms
- Handles returned as syscall.Handle from os.File.Fd()

Note: full Windows cross-compile still blocked by unrelated issues
(mmap in cdx/ntx, termios in debugcli.go). The lock code itself
compiles cleanly with //go:build windows.

Also updates gap-analysis.md to reflect Windows lock status.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 11:57:33 +09:00
fc1dca9551 feat(rdd): real POSIX file/record locking + gap analysis doc
Replaces the FLOCK/DBRLOCK/DBRUNLOCK no-op stubs with actual
fcntl(F_SETLK) byte-range advisory locks, matching Harbour's
hb_fsLockLarge implementation.

Before: rtlDbRLock always returned .T. regardless of contention.
        Multi-process writers could silently corrupt records.

After:  Non-blocking POSIX byte-range locks per file descriptor.
        Cross-process exclusion verified by a subprocess-spawning
        Go test that witnesses BUSY vs OK transitions.

New files:
  hbrdd/dbf/locks_posix.go    fcntl F_WRLCK/F_UNLCK wrappers
  hbrdd/dbf/locks_windows.go  stub (TODO: LockFileEx)
  hbrdd/dbf/lock_multi_test.go   cross-process verification
  docs/gap-analysis.md        honest Harbour parity assessment

Modified:
  hbrdd/dbf/dbf.go
    - DBFArea gains fileLocked bool + lockedRecs map
    - Close() calls releaseAllLocks() before dropping the fd
  hbrtl/database.go
    - rtlDbRLock / rtlDbRUnlock now delegate to DBFArea.LockRecord /
      UnlockRecord instead of returning fixed .T./NIL
    - New rtlFLock / rtlDbUnlock for FLOCK() / DBUNLOCK()
  hbrtl/register.go
    - FLOCK and DBUNLOCK symbols registered (were missing entirely)
  compiler/analyzer/analyzer.go
    - FLOCK / DBUNLOCK added to RTL known-function set

Lock region layout (non-overlapping on purpose):
  FLOCK region       [0, HeaderLen+1)
  Record N region    [RecordOffset(N), RecordLen)

So a workarea can hold FLOCK and multiple DBRLOCK simultaneously
on the same fd without conflict.

Design rationale (captured in locks_posix.go header):
  * POSIX fcntl, not flock(2) — byte-range + NFS-safe
  * Non-blocking F_SETLK — matches Clipper FLOCK() → .F. semantics
  * Released explicitly on Close to avoid workarea-sharing races
  * Windows falls back to no-op (TODO: LockFileEx)

Verification:
  go test ./hbrdd/dbf/ -run TestFLockBlocksAcrossProcesses  PASS
  go test ./hbrdd/dbf/ -run TestRLockBlocksAcrossProcesses  PASS
  go test ./...                                             ALL PASS
  FiveSql2 43/43                                            100%
  compat_harbour 51/51                                      100%

The gap-analysis doc (docs/gap-analysis.md) is a running inventory
of what works vs what's still missing vs Harbour 3.2, written for
users evaluating Five for production — not a sales pitch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 17:58:03 +09:00
6c5374778a perf(rdd): index build 38% faster — sort.Interface + fast path for numeric/UPPER
Benchmark (50k records, 4 indexes on Apple M-series):
             before   after   Δ
  INDEX     53.7ms  33.3ms  -38%  (now 10% faster than Harbour 37.3ms)
  TOTAL    156.2ms 133.0ms  -15%

Fixes:

1. sort.Slice(reflection) → concrete sort.Interface
   Benchmarked in isolation on 200k KeyRecords:
   sort.Slice(closure):  50.0ms
   sort.Sort(interface): 30.4ms  (40% faster, no reflection)

   - indexer.go: add keyRecordAsc/Desc concrete types
   - Branch hoist descending check out of Less()

2. buildOnePage zero allocation
   Was allocating a temp padded []byte per key (~50k allocs per index).
   Now writes padded key directly into the page buffer via padCopy.

3. bulkBuildBTree separator reuse
   sepKey can alias the source KeyRecord.Key when it's already keyLen-sized
   (true for all slab-allocated keys), avoiding ~n/maxItem small allocations.
   Pre-size the children slice.

4. Fast path extended to numeric fields and UPPER/LOWER
   Previously only bare CHAR field references hit the zero-alloc fast path.
   Now:
     - Numeric fields (N/F type) copy DBF bytes directly
       (same-length ASCII compare matches numeric order for non-negatives)
     - UPPER(field) / LOWER(field) wrappers on CHAR fields apply ASCII
       case folding inline during byte copy

   Per-index timing on the micro benchmark:
               before   after
     NAME       7.7ms   7.5ms  (fast path, unchanged)
     CITY       6.0ms   6.2ms  (fast path, unchanged)
     AGE       14.1ms   7.1ms  -50%  (was slow path)
     UPPER(NM) 17.0ms   7.9ms  -54%  (was slow path)

5. Slow path single-pass scan
   When an expression is too complex for fast path, we still avoid the
   double GoTo per record. The evaluation loop now sequentially walks
   records with one GoTo each, restoring the original position only at
   the end, and shares a single slab for padded keys.

Also fixes a hbrt bug surfaced while writing the benchmark:

6. Date + Numeric promoted to Date
   Plus()/Minus() previously required the integer side to be NumInt.
   Modulus returns a promoted type, so `SToD("...") + (i % 365)` panicked.
   Now accepts any Numeric on either side and truncates the fractional
   part before adding Julian days.

   - hbrt/ops_arith.go: Date±Numeric (was Date±NumInt only)

Tests:
  go test ./...        — ALL PASS (17 packages)
  FiveSql2 43/43       — 100%
  compat_harbour 51/51 — 100%
  Harbour vs Five diff — 0 lines differ (281-line RDD parity test)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 17:24:49 +09:00
e95afad4ee feat: Harbour RDD parity — NTX/CDX 100% compatible, FIELD-> works
Five RDD engine now matches Harbour DBFNTX and DBFCDX byte-for-byte
in ordering, seek, navigation, and field access. Verified against
Harbour 3.2.0dev with a 281-line comparison test covering:
  - Natural/NAME/CITY/AGE/SALARY/UPPER ordering
  - SEEK (exact/not-found), GoTop/GoBottom per order
  - DELETE/RECALL with SET DELETED
  - CDX compound index read with 5 tags (BYNAME, BYCITY, BYAGE, BYSAL, BYUNAME)
  - Reverse traversal

Fixes:

1. FIELD->NAME returned NIL
   GetAliasField returned interface{} but runtime expected hbrt.Value,
   so the type assertion in PushAliasField failed and pushed NIL.
   - workarea.go: change return type to hbrt.Value, handle FIELD/_FIELD
     as current-workarea alias, add SetAliasField
   - gengo.go: emit SetAliasField() for alias->field := value in both
     statement and expression contexts

2. OrdSetFocus(n) silently switched to natural order
   v.AsString() returns "" for a numeric Value, so OrderListFocus("")
   set current=-1.
   - indexrtl.go: convert numeric param via fmt.Sprintf("%d", ...)

3. CDX compound tag order mismatched Harbour
   Five decoded the structural B-tree which is alphabetical, but
   Harbour sorts tags by TagBlock (file offset = creation order).
   - cdx/cdx.go: sort tagEntries by offset ascending after decoding,
     matching hb_cdxIndexLoadAvailTags in dbfcdx1.c

4. OutStd()/OutErr() not registered — caused panic on call
   - hbrtl/console.go: add rtlOutStd/rtlOutErr implementations
   - hbrtl/register.go: register OUTSTD and OUTERR
   - analyzer.go: add OUTSTD/OUTERR to RTL known-functions

5. FIELD keyword triggered "undeclared variable" warnings
   - analyzer.go: add FIELD, _FIELD, M, MEMVAR as builtin constants

Tests:
  go test ./...        — ALL PASS (17 packages)
  FiveSql2 43/43       — 100%
  compat_harbour 51/51 — 100%
  Harbour diff         — 0 lines differ (281-line comparison)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:37:47 +09:00
02026a1966 fix: analyzer zero warnings — complete RTL coverage, cross-file awareness
- Register all 479 RTL functions from hbrtl/register.go (was ~60)
- Recognize module-level STATIC variables across all functions
- Declare RECOVER USING variables in analyzer scope
- Register code block parameters ({|x,y| ...}) as declared
- 2-pass multi-file build: collect cross-file function names before analysis
- Add QUIT, ERRORLEVEL, ALTSRC to builtin constants

All 3 test suites pass with 0 warnings:
  go test ./...        — ALL PASS
  FiveSql2 43/43       — 100%
  compat_harbour 51/51 — 100%

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 12:11:08 +09:00
468aa1efbd fix: add cmd/five/main.go to repo (was excluded by .gitignore)
- Changed .gitignore: "five" → "/five" to only ignore root binary
- cmd/five/main.go (702 LOC): Five CLI entry point (run, build, gen, debug, frb)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 11:57:56 +09:00
c35f456785 docs: add README with build instructions
- Go installation guide (Linux/WSL, macOS Intel, macOS Apple Silicon, Windows)
- Five compiler build steps
- PRG compilation examples (single file, multi-file)
- Test execution commands
- SQL demo example
- Project structure overview

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 11:45:35 +09:00
486e466592 feat: FiveSql2 43/43, @byref, mutable closure, RTL 479, DateTime fix
Major changes since last commit:
- FiveSql2 SQL:1999 engine (10,458 LOC) — 43/43 ALL PASS
- 21 compiler/runtime bugs fixed (short-circuit AND/OR, FOR LOOP, etc.)
- @byref pass-by-reference via RefCell pattern
- Mutable closure capture (EnsureLocalRef + RefCell sharing)
- RTL: 400 → 479 functions (+79: file, string, datetime, hash, UTF-8)
- DateTime/Timestamp fully working (hb_DateTime, hb_Hour/Min/Sec, display)
- Reserved word guard (39 keywords blocked from function calls)
- AEval arg order fix (element before index)
- Closure capture redecl fix (unique _cap_ names per block)
- Hash/string indexing in ArrayPush/ArrayPop
- Harbour compat test suite: 51/51
- 4 docs: Porting Report, Implementation Plan, Optimization Plan, Commercialization

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 11:35:37 +09:00
d451b836a6 perf: inline Str/PadR/PadL/SubStr/Left/Right/At/IIF in gengo
13 more RTL functions inlined — no Frame/EndProc, no VM dispatch:
- Str(n,w,d) → fmt.Sprintf("%*.*f", w, d, n)
- PadR(s,n) → s + hbrtl.Spaces(n-len(s))
- PadL(s,n[,fill]) → Spaces(pad) + s or Repeat(fill, pad) + s
- SubStr(s,p,l) → s[p:p+l] with bounds check
- Left(s,n) → s[:n], Right(s,n) → s[len-n:]
- At(search,target) → strings.Index + 1
- IIF(cond,a,b) → if/else without function call

Also: Spaces() exported for generated code access.

50K SEEK random: 62ms (Harbour 67ms — Five FASTER!)
82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 23:16:38 +09:00
7d44488d39 docs: Five technical evaluation — Google/Go team perspective
Comprehensive review as if evaluated by Google Go team:
- Architecture analysis (transpiler pipeline, gengo innovations)
- Performance evidence (6/10 categories faster than C)
- Correctness proof (82/82 + 77/77 + 18/18 + 47/47)
- Strategic value (5M xBase developer bridge to Go)
- Improvement roadmap (lazy GoTo, string fusion, CDX create)
- Market positioning (vs Harbour, xHarbour, Alaska xBase++)

Key quote: "Five demonstrates that Go is ready to be a universal
compilation target, not just a language for writing programs directly."

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 23:01:04 +09:00
279a16a88c refactor: pure Go — recursion→iteration, COW records, zero alloc
CDX Seek iterative (cdx.go):
- Converted recursive seekPage → iterative loop
- Single buf reused across all B-tree levels (was: make per level)
- Internal node: binary search (was: linear O(n))
- Eliminates 3 heap allocations per CDX SEEK

DBF Copy-on-Write records (dbf.go):
- GoTo: recBuf = mmap slice reference (zero-copy read)
- PutValue/Delete/Recall: promote to ownBuf before write
- Eliminates memcpy per GoTo for read-only SCAN operations
- recOwned flag tracks COW state

NTX build.go:
- setKeyEntry: write directly to page (no temp make([]byte))
- padCopy: copy+fill (no pre-fill entire buffer)

CDX DecodeLeafKeys slab (cdx.go):
- Single slab allocation for all keys per page

82/82 stress PASS. All unit tests PASS.

50K SEEK random: 63ms (Harbour 67ms — FASTER!)
50K DELSCAN: 2ms (Harbour 12ms — 6x FASTER!)
CDX SCOPE: 2ms (Harbour 4ms — 2x FASTER!)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:56:20 +09:00
0102c3c94e perf: CGo review — slab alloc, compareKeys simplify, zero-alloc padCopy
From CGo expert review (verdict: stay pure Go, CGo would be slower):

CDX DecodeLeafKeys slab allocation (cdx.go):
- Single make() for all keys + prevKey (was 30+ allocs per page)
- Keys are slices into pre-allocated slab (zero copy)

NTX compareKeys simplified (ntx.go):
- bytes.Compare already returns normalized -1/0/+1
- Removed redundant normalization branches

NTX build.go zero-alloc:
- padCopy: copy+fill instead of make+fill+copy
- setKeyEntry: write directly to page data (no temp buffer)

82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:44:56 +09:00
197720f869 fix: Go code review — 7 critical issues resolved
From senior Go developer review:

C7 CRITICAL: pagePool data race (ntx.go)
- Moved global pagePool[8] + pagePoolIdx into per-Index struct
- Eliminates race condition across goroutines using separate indexes

C8 CRITICAL: Page.data dangling pointer after remap (ntx.go)
- remapFile() now clears pagePool data slices (pointed into old mmap)
- Prevents segfault from stale mmap references

C4 HIGH: pop() bounds check restored (thread.go)
- Removed performance optimization that eliminated underflow detection
- Stack underflow now produces clear error instead of index -1 panic

C1 HIGH: intExpLen overflow on MinInt64 (value.go)
- Added special case: MinInt64 returns 20 (length of -9223372036854775808)
- Prevents -v overflow in negation

C11 CRITICAL: GoTo ReadAt error handling (dbf.go)
- ReadAt failure now returns error and sets EOF
- Previously silently used stale record buffer (data corruption risk)

C14 HIGH: LEN() inline missing Hash case (gengo.go)
- Added _v.IsHash() → len(Keys) branch

C15 HIGH: EMPTY() inline missing Date case (gengo.go)
- Added _v.IsDate() && _v.AsJulian() == 0 check

82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:26:34 +09:00
44d3c7385c perf: fused opcodes + inline EOF/BOF/Found/RecNo/Deleted
Fused opcodes (ops_compare.go):
- LocalLessEqualInt: FOR i<=N without Push+LessEqual+PopLogical
- LocalGreaterEqualInt: FOR STEP -1
- Direct local access + int comparison (no stack, no Value boxing)

gengo FOR loop:
- Detects literal TO value → emits LocalLessEqualInt (3 calls → 1)
- Falls back to stack-based for variable limits

Inline RDD functions (gengo tryEmitInlineRTL):
- EOF/BOF/Found/Deleted/RecNo/RecCount: direct area method call
- No FindSymbol + PushNil + Do(0) + Frame/EndProc overhead
- Uses hoisted _darea when inside DO WHILE context

Results (50K, ext4):
  SEEK random: 63ms (Harbour 67ms — FASTER!)
  SEEK seq: 44ms (Harbour 27ms — 1.6x)
  CDX SEEK NAME: 47ms (Harbour 27ms — 1.7x)
  CDX SEEK ID: 24ms (Harbour 17ms — 1.4x)

All counts correct. 82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:07:34 +09:00
ad1bc23e36 perf: inline RTL + symbol cache infrastructure + EndProcFast
gengo inline RTL (tryEmitInlineRTL):
- LTrim/RTrim/AllTrim/Upper/Lower/Len/Empty/Chr/Asc
- Skip Frame/EndProc/VM dispatch entirely
- Emit direct Go code (strings.TrimLeft, etc.)

Symbol cache infrastructure (collectSymbols):
- AST walker collects all referenced symbol names
- symCache field ready for future per-function hoisting
- Currently disabled (function-level hoisting caused side effects)

NTX TestGetMmap helper for profiling.

82/82 stress PASS. 14 packages ALL PASS.
50K SEEK random: 64-66ms (Harbour 67ms — equal or faster)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:01:24 +09:00
77562d4645 perf: inline RTL functions in gengo — skip Frame/EndProc entirely
tryEmitInlineRTL: recognized RTL functions emit direct Go code
instead of PushSymbol → PushNil → Push args → Function(n) dispatch.

Inlined functions (most common in SEEK key generation):
- LTrim → strings.TrimLeft(s, " ")
- RTrim/Trim → strings.TrimRight(s, " ")
- AllTrim → strings.TrimSpace(s)
- Upper → strings.ToUpper(s)
- Lower → strings.ToLower(s)
- Len → len(s) / len(arr.Items)
- Empty → nil/zero/empty check
- Chr → string(byte(n))
- Asc → int(s[0])

Each inlined call saves: FindSymbol + PushNil + Frame + locals copy
+ function body + EndProcFast + return handling = ~0.14ms per call.

In 50K SEEK loop with 5 string functions: saves ~35ms.

50K SEEK random: 64ms (Harbour 67ms — Five FASTER!)
82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 21:50:25 +09:00
05ccef05e2 perf: EndProcFast — eliminate defer recover() from RTL hot paths
Problem: every RTL function calls defer t.EndProc() which does recover().
50K SEEK loop = 250K recover() calls = ~12ms wasted.

Solution: EndProcFast() skips recover (only needs endFrame restore).
Applied to ALL RTL functions in strings.go, rdd.go, missing.go, database.go.
EndProc() with recover kept for generated PRG code (needs BEGIN SEQUENCE).

Analysis (50K sequential SEEK breakdown):
  Go NTX Seek direct: 7ms (faster than Harbour 27ms!)
  PRG VM overhead:    38ms (Frame + RTL calls + key generation)
  Key generation:     25ms (Str+LTrim+PadL+PadR = 5 RTL Frame/EndProc per iter)

With EndProcFast: RTL overhead reduced ~30%.

CDX SCOPE: 2ms (Harbour 4ms — 2x FASTER!)
82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 21:43:39 +09:00
9644b5469a perf: BoltDB BCE pattern — inline page access, eliminate bounds checks
NTX Page accessors (ntx.go):
- keyOffset/KeyChild/KeyRecNo: removed redundant bounds checks
- Use open-ended slice (data[off:]) for BCE — compiler proves safety
- pageKeyFind: inline offset table + key access in hot loop
  (was: compareKeys → KeyValue → keyOffset → LittleEndian)
  (now: compareKeys(data[off:off+kl]) — single slice expression)

CDX Seek (cdx.go):
- Binary search with leftmost match (correctly finds first duplicate)
- Cache hit path: skip DecodeLeafKeys entirely

50K NTX SEEK random: 67ms = Harbour 67ms (EQUAL!)
82/82 stress PASS. CDX 18/18. All unit tests PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 20:27:37 +09:00
b72623f79c perf: CDX binary search + leaf cache hit + DBF/NTX zero-copy
CDX Seek (cdx.go — ported from rddfive/cdx_engine.c):
- Linear search → binary search on decoded leaf keys (O(N) → O(log N))
- Leftmost match: continues searching left after match (duplicate key correctness)
- Leaf cache hit: skip decode if same page (SEEK loop optimization)

NTX zero-copy Page (ntx.go — BoltDB pattern):
- Page.data: []byte slice into mmap (was [1024]byte copy)
- cachedLoadPage: p.data = mmap[offset:offset+1024] (no memcpy!)
- pagePool: 8-slot ring for Page struct reuse

DBF mmap (dbf.go):
- GoTo: copy from mmap instead of file.ReadAt syscall
- Unmap before Append/Close/Flush (file growth), re-mmap after

Results (50K, ext4, Harbour comparison):
┌──────────────┬──────────┬──────────┬──────────────┐
│              │ Harbour  │ Five     │              │
├──────────────┼──────────┼──────────┼──────────────┤
│ CDX SEEK     │ 27ms     │ 49ms     │ 1.8x (was 6.5x!)│
│ CDX SEEK ID  │ 17ms     │ 24ms     │ 1.4x (was 8.4x!)│
│ CDX SCAN     │ 5ms      │ 4ms      │  FASTER    │
│ CDX SCOPE    │ 4ms      │ 3ms      │  FASTER    │
│ NTX SCAN     │ 4ms      │ 3ms      │  FASTER    │
│ NTX DELSCAN  │ 12ms     │ 3ms      │  4x FASTER │
│ NTX SEEK rnd │ 67ms     │ 69ms     │ ≈ equal      │
└──────────────┴──────────┴──────────┴──────────────┘

82/82 stress PASS. CDX 18/18 cross-read PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 20:18:54 +09:00
1d9b364df8 perf: BoltDB-style zero-copy Page — NTX SEEK 2x, SCAN 5x faster
Page.data changed from [1024]byte (copied) to []byte (mmap slice reference).
Inspired by BoltDB's zero-copy page access pattern.

cachedLoadPage: returns slice into mmap memory (no 1024-byte copy!)
- Before: copy(p.data[:], mmap[offset:offset+1024]) — memcpy per page
- After:  p.data = mmap[offset:offset+1024] — pointer assignment only

pagePool: reuses Page structs (8-slot ring) to reduce GC pressure.

Benchmark (ext4, home dir) — Harbour comparison:
┌──────────────────┬──────────┬──────────┬──────────┐
│ 50K              │ Harbour  │ Five     │          │
├──────────────────┼──────────┼──────────┼──────────┤
│ SEEK seq         │ 23ms     │ 43ms     │ 1.9x     │
│ SEEK random      │ 63ms     │ 65ms     │ ≈ equal! │
│ SCAN             │ 5ms      │ 3ms      │ FASTER!  │
│ DUPKEY scan      │ 23ms     │ 12ms     │ FASTER!  │
│ DELSCAN          │ 17ms     │ 2ms      │ 8.5x!    │
│ PACK             │ 16ms     │ 21ms     │ 1.3x     │
└──────────────────┴──────────┴──────────┴──────────┘

82/82 stress PASS. All unit tests PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 20:07:06 +09:00
3fa553d3ed perf: DBF mmap — zero-copy record reads, SCAN 2-3x faster
syscall.Mmap on DBF open for zero-copy GoTo:
- GoTo: copy from mmap slice instead of file.ReadAt syscall
- Eliminates kernel context switch per record read
- Unmap before Append (file grows), Close, Flush
- Re-mmap after Flush

Benchmark (ext4, home dir):
  10K SCAN: 4ms → 2ms (Harbour 1ms = 2x)
  10K DUPKEY: 6ms → 4ms (Harbour 4ms = 1x!)
  10K DELSCAN: 6ms → 2ms (Harbour 2ms = 1x!)
  50K SCAN: 26ms → 15ms (42% faster)
  50K DELSCAN: 29ms → 17ms (Harbour 17ms = 1x!)
  50K DUPKEY: 41ms → 29ms (Harbour 23ms = 1.3x)
  CDX SCAN: 13ms → 4ms (Harbour 6ms — FASTER!)
  CDX SCOPE: 9ms → 3ms (Harbour 4ms — FASTER!)

82/82 stress PASS. All unit tests PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 19:06:21 +09:00
3974333372 perf: VM hot-path optimization — cached values + inline stack ops
value.go:
- cachedNil, cachedTrue, cachedFalse: pre-built constant Values
- MakeBool()/MakeNil(): return cached (zero allocation)
- smallInts[256]: pre-built integers 0-255 (skip intExpLen loop)
- MakeInt(): fast path for 0-255

thread.go:
- pop(): use cachedNil for GC help (no MakeNil() call)

ops_compare.go:
- LessEqual(): inline Int-Int fast path (skip valueCompare)
  Direct scalar comparison with cached bool result
- Not(): inline logical fast path (skip IsLogical+AsBool)
- PopLogical(): inline type check + scalar read

Impact: these functions called millions of times in FOR/DO WHILE loops.
10K SEEK: 20ms → 16ms (20%). CDX SCOPE: 12ms → 9ms (25%).
82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 18:57:20 +09:00
5b378318a0 perf: RTL optimization — cached WA, spaces pool, stack-alloc fmt_int64
rdd.go:
- getWA() cached type assertion (avoid repeated interface check)
- waCache stores last WA pointer → O(1) for repeated calls

strings.go:
- spacesCache[257]: pre-built space strings for pad sizes 0-256
- spaces(n) returns cached string (no Repeat allocation)
- PadR/PadL use spaces() for fill=" " (most common case)
- Str() uses spaces() for right-padding

missing.go:
- fmt_int64: stack-allocated [20]byte array (was heap make([]byte))
- Reverse iteration (no prepend overhead)
- PadC uses spaces() for left/right padding

Benchmark (ext4, home dir):
  10K APPEND: 28ms → 26ms (Harbour 27ms!)
  50K APPEND: 130ms → 113ms (13% improvement)
  50K SCAN: 24ms → 23ms
  50K DUPKEY: 42ms → 35ms (17% improvement)
  CDX SCOPE: 12ms → 10ms (17% improvement)

82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 18:16:22 +09:00
48cd4f9e5c perf: DO WHILE/SEEK/DELETE WA hoisting — reduce per-iteration lookups
DO WHILE optimization:
- Detect RDD commands in body (SKIP/GO/SEEK/REPLACE/DELETE)
- If no USE/SELECT (safe), hoist _dwa/_darea before loop
- SKIP/GO/SEEK/DELETE inside loop use cached area variable
- Eliminates WA lookup + Current() per iteration

SEEK optimization:
- Use hoisted area when inside DO WHILE or FOR hoist context
- Eliminates WA lookup per SEEK call in tight loops

DELETE optimization:
- Use hoisted area when available

All commands now check g.hoistedDW || g.hoistedFields:
- GO TOP/BOTTOM/n → cached area
- SKIP n → cached area
- SEEK key → cached area + Indexer check
- DELETE → cached area
- APPEND → cached area (FOR loop)
- REPLACE → cached _rdbf + _rfiN (FOR loop)

82/82 stress PASS. 14 packages ALL PASS.
CDX SCOPE: 12ms (Harbour 4ms = 3x)
NTX SCAN: 24ms (Harbour 5ms = 4.8x)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 17:25:38 +09:00
bb6cf7c612 perf: FOR loop RDD hoisting — WA/FieldIndex cached outside loop
When FOR body contains APPEND+REPLACE and no USE/SELECT:
- Hoist WorkAreaManager, Current(), *dbf.DBFArea outside loop
- Pre-compute FieldIndex for all REPLACE fields once
- REPLACE inside loop uses cached _rdbf and _rfiN variables
- APPEND inside loop uses cached _rarea (no WA lookup per iter)

Safety: collectReplaceFields returns nil if USE/SELECT found in body
(workarea may change → cannot safely cache). Falls back to normal emit.

10K APPEND benchmark: 28ms (Harbour 27ms — essentially equal!)
82/82 stress test PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 17:07:33 +09:00
8f354ae24d perf: gengo RDD optimization — reduce VM overhead per operation
REPLACE (gen_cmd.go):
- Cache area.(*dbf.DBFArea) once per command (was N times for N fields)
- Remove _fi >= 0 check (FieldIndex returns -1, PutValue handles it)
- Reduces type assertions from N+1 to 2

GO/SKIP (gen_cmd.go):
- Literal integer parameters emit directly (no Push/Pop stack ops)
- GO 5 → _area.GoTo(uint32(5)) instead of Push(5) → Pop → AsNumInt
- SKIP -1 → _area.Skip(-1) instead of Push(1) → Negate → Pop

Benchmark impact (50K, ext4):
- SEEK random: 138ms → 121ms (12% improvement)
- DUPKEY scan: 41ms → 37ms (10% improvement)
- DELSCAN: 32ms → 28ms (13% improvement)

82/82 stress test PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 16:39:11 +09:00
30dfc0728d perf: CDX SCAN 4ms, SCOPE 3ms — faster than Harbour
CDX optimizations (from rddfive/cdx_engine.c):
- Byte-level leaf decode (vs bit-by-bit extractBits)
- Leaf page decode cache in Tag struct
- Zero-alloc internal node traversal (direct mmap slice read)

NTX/CDX + DBF:
- loadRecord() helper for future lazy-load optimization
- recLoaded flag in DBFArea (currently always true for safety)

Benchmark (50K, ext4):
  CDX SCAN:  276ms → 4ms  (Harbour 6ms — Five is FASTER!)
  CDX SCOPE: 238ms → 3ms  (Harbour 4ms — Five is FASTER!)
  CDX SEEK:  362ms → 185ms (49% improvement)
  NTX SCAN:  24ms → 14ms  (42% improvement)

82/82 stress test PASS. CDX 18/18 cross-read PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 15:44:10 +09:00
96d72a456c perf: CDX zero-alloc internal node seek — SEEK 45% faster
Internal node traversal: read directly from mmap/buf slice
- No DecodeIntKeys allocation (was nKeys+1 IntKeyEntry structs)
- No key byte slice copy (compare directly against buf)
- Big-endian child/recNo read inline

CDX 50K benchmark:
  SEEK NAME: 362ms → 199ms (45% faster)
  SEEK ID:   320ms → 184ms (42% faster)
  SCAN:      14ms (unchanged — leaf cache handles this)
  SCOPE:     20ms → 14ms

Harbour comparison:
  SEEK: 27ms (Harbour) vs 199ms (Five) = 7.4x
  SCAN: 6ms (Harbour) vs 14ms (Five) = 2.3x

CDX cross-read: 18/18 PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 14:05:20 +09:00
40935b6103 perf: CDX byte-level decode + leaf cache — SCAN 20x faster
Ported from rddfive/cdx_engine.c cdx_leaf_decode_all():
- Replaced bit-by-bit extractBits loop with byte-level shift/mask
- Read reqByte as little-endian integer, extract recNo/dup/trl via masks
- 10x+ faster than per-bit extraction

Leaf page decode cache:
- Tag.cachedLeafOff/cachedLeafKeys: avoid re-decoding same leaf page
- SkipNext/SkipPrev use getLeafKeys() with cache
- GoTop/Seek populate cache on first decode

CDX 50K benchmark (ext4):
┌──────────────┬──────────┬──────────┬──────────┐
│ CDX 50K      │ Harbour  │ Before   │ After    │
├──────────────┼──────────┼──────────┼──────────┤
│ SCAN 50K     │ 6ms      │ 276ms    │ 14ms     │ ← 20x faster
│ SCOPE 35K    │ 4ms      │ 238ms    │ 20ms     │ ← 12x faster
│ SEEK NAME    │ 27ms     │ 362ms    │ 239ms    │
│ SEEK ID      │ 18ms     │ 320ms    │ 195ms    │
└──────────────┴──────────┴──────────┴──────────┘

CDX cross-read: 18/18 PASS. All unit tests PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 13:37:57 +09:00
1b41384675 fix: CDX mmap + internal node format (BE key-first) — 50K works
CDX internal node format fix:
- Was: [child LE][recNo LE][key] (NTX-style)
- Now: [key][recNo BE][child BE] (correct CDX format)
- Fixes GoTop/Seek/Scan for large CDX files (50K+ records)

CDX mmap:
- syscall.Mmap on OpenIndex for zero-copy reads
- idx.readAt() helper: mmap slice or file fallback
- All ReadAt calls in Tag navigation replaced
- Close: munmap

CDX 50K benchmark (all counts correct):
  SEEK NAME 50K: 362ms (f=50000)
  SCAN 50K: 276ms (c=50000)
  SCOPE 35K: 238ms (c=35000)
  SEEK ID 50K: 320ms (f=50000)

CDX is slower than NTX due to bit-packed leaf decompression per page.
Cross-read test: 18/18 still PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 13:32:06 +09:00
a9600ad45c perf: proper 3-level bulk build — INDEX 50K: 180ms → 28ms (6.4x)
bulkBuildBTree: distributes sorted keys as [M leaf] [sep] [M leaf] [sep] ...
- Separator exists ONLY in parent, never in leaf (proper B-tree)
- Works for any depth (tested 10 to 50000 keys, all correct)
- Edge case: absorb trailing 1-key into previous leaf

Eliminated per-key insertion fallback (rebuildWithInsert).
All sizes now use O(N) bulk build instead of O(N log N) insertion.

Benchmark on ext4 (home dir):
┌──────────────┬──────────┬──────────┬───────┐
│ 50K Items    │ Harbour  │ Five     │ Ratio │
├──────────────┼──────────┼──────────┼───────┤
│ APPEND 50K   │ 61ms     │ 124ms    │ 2x    │
│ INDEX NAME   │ 6ms      │ 28ms     │ 4.7x  │
│ INDEX CITY   │ 5ms      │ 36ms     │ 7.2x  │
│ SEEK 50K seq │ 23ms     │ 97ms     │ 4.2x  │
│ SEEK 50K rnd │ 63ms     │ 122ms    │ 1.9x  │
│ SCAN 50K     │ 5ms      │ 24ms     │ 4.8x  │
│ DUPKEY 50K   │ 23ms     │ 38ms     │ 1.7x  │
│ PACK 50K     │ 16ms     │ 20ms     │ 1.25x │
└──────────────┴──────────┴──────────┴───────┘

All counts correct: 50000/50000/40000

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 12:08:27 +09:00
c5ed5612fb perf: mmap zero-copy page access — Go-native optimization
Replaced LRU page cache with syscall.Mmap:
- OpenIndex: mmap entire file read-only (MAP_SHARED)
- cachedLoadPage: copy from mmap slice (no syscall per page)
- Close: munmap + file close
- insertKeyBTree: munmap before modify, mmapFile after complete
- remapFile: re-mmap after file size changes

Results on ext4 (50K records):
- SEEK random: 188ms → 138ms (26% improvement)
- SCAN: 35ms → 23ms (34% improvement)
- DUPKEY: 53ms → 41ms (23% improvement)
- INDEX: 180ms (unchanged — per-key insertion, no mmap during build)

Go-native approach:
- syscall.Mmap instead of C-style LRU cache
- OS page cache handles eviction automatically
- Simpler code (60 lines removed, 30 added)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 11:42:44 +09:00
103f0d8b64 perf: NTX LRU page cache (256 slots) — reduces syscalls
LRU page cache ported from rddfive/ntx_engine.c:
- 256-slot cache with MRU fast-path (O(1) for repeated access)
- LRU eviction when all slots full
- cachedLoadPage replaces LoadPage for all navigation
- invalidateCache called before insertKeyBTree (pages modified)

10K benchmark improvement (ext4 home dir):
- SCAN FWD: 6ms → 5ms
- SEEK NUM: 18ms → 14ms (22% improvement)
- DUPKEY SCAN: 9ms → 8ms
- All counts correct: 10000/10000/8000

50K benchmark:
- SCAN: 35ms → 31ms
- DUPKEY: 50ms → 40ms (20% improvement)
- DELSCAN: 41ms → 33ms (20% improvement)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 11:35:26 +09:00
dadb97ee88 fix: 3-level NTX correctness + CDX SET INDEX TO string quoting
NTX 3-level tree (build.go):
- Hybrid approach: bulk build for ≤2 levels, insertKeyBTree for 3+
- rebuildWithInsert: creates proper B-tree via per-key insertion
- 5000-key test: Count=5000 Found=5000 (was 5004/4868)

CDX SET INDEX TO (gengo.go):
- Strip surrounding quotes from string literal in OrderListAdd
- Was: idx.OrderListAdd("\"path\"") → file not found
- Now: idx.OrderListAdd("path") → correct

All tests:
- 14 packages ALL PASS
- 82/82 NTX stress test
- 18/18 CDX cross-read
- 50K benchmark: all counts correct

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 11:04:07 +09:00
7fec4ce150 perf: 50K benchmark — Harbour vs Five on ext4
50K records benchmark on native ext4 (home directory):
- APPEND 50K: Five 140ms / Harbour 61ms (2.3x)
- INDEX 50K:  Five 31ms / Harbour 6ms (5.2x)
- SEEK 50K:   Five 142ms / Harbour 23ms (6.2x)
- SCAN 50K:   Five 35ms / Harbour 5ms (7x)
- PACK 50K:   Five 19ms / Harbour 16ms (1.2x)

All within acceptable Go vs C overhead (2-7x).
PACK nearly identical. APPEND close (2.3x).

Known issue: 3-level NTX bulk build has separator duplication
at interior→root level (count=50083 vs 50000).
Does not affect correctness for <= 2-level trees (100 records OK).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 09:36:12 +09:00
adede5cd69 perf: REPLACE remove Flush + bulk build + deferred write = 1600x faster
Critical fix: REPLACE was calling area.Flush() after every field write!
- gengo gen_cmd.go: removed Flush() from emitReplaceCmd
- Harbour defers write until DBCOMMIT/CLOSE/GoTo, not per-REPLACE

Combined with bulk build + deferred APPEND:
- B1 APPEND 10K:  72,228ms → 30ms  (2,400x improvement!)
- B2 INDEX NAME:  34ms → 5ms       (6.8x improvement)
- Harbour comparison: Five 30ms vs Harbour 27ms (1.1x)

Also: OrderCreate flushes dirty record + EOF + header before index build

Benchmark on ext4 (home dir):
┌─────────────┬──────────┬────────┬───────┐
│ Benchmark   │ Harbour  │ Five   │ Ratio │
├─────────────┼──────────┼────────┼───────┤
│ APPEND 10K  │ 27ms     │ 30ms   │ 1.1x  │
│ INDEX NAME  │ 2ms      │ 5ms    │ 2.5x  │
│ INDEX CITY  │ 0ms      │ 7ms    │ -     │
│ SEEK 10K    │ 6ms      │ 25ms   │ 4.2x  │
│ SCAN FWD    │ 1ms      │ 6ms    │ 6x    │
│ SCAN BWD    │ 0ms      │ 6ms    │ -     │
│ PACK        │ 4ms      │ 3ms    │ 0.75x │
└─────────────┴──────────┴────────┴───────┘

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 09:22:05 +09:00
b1e868f01e perf: NTX bulk build + APPEND deferred write (from rddfive C port)
NTX Bulk Build (build.go — ported from rddfive/ntx_engine.c):
- pageBuffer: dynamic memory buffer for all pages
- Phase 1: Build leaf pages in sequential memory (zero disk I/O)
- Phase 2: Build interior levels from cached leaf data (zero I/O)
- Separator promotion: remove last key from leaf only (not interior)
- Single bulk WriteAt for all pages at end
- INDEX ON 10K: 34ms → 5-8ms (4-6x improvement)

NTX Seek (ntx.go):
- Always descend to leaf on match (find first occurrence)
- fStop flag tracks path match, verified at leaf

APPEND Buffering (dbf.go):
- Append marks dirty without immediate disk write
- flushRecord writes record data only (no header/EOF per record)
- Close/Flush writes EOF marker + header once

Results: 14 packages ALL PASS, 82/82 stress test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 09:10:18 +09:00
1d3f897daf bench: RDD performance benchmark — Harbour vs Five comparison
10,000 records, 3 indexes, 12 benchmarks:
- APPEND: Five 72s vs Harbour 16ms (flush-per-record — optimization needed)
- INDEX: Five 30-36ms vs Harbour 1-2ms (per-key insert vs bulk)
- SEEK: Five 35ms vs Harbour 5ms (7x — acceptable)
- SCAN: Five 8-11ms vs Harbour 1-4ms (3-9x — acceptable)
- PACK: Five 4ms = Harbour 4ms (identical!)

B6 correctness: Five found=10000 (all), Harbour found=1 (hash collision)
All counts match: 10000 records, 8000 after SET DELETED, 8000 after PACK

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 08:38:19 +09:00
3fe8021e9e fix: NTX Seek descent + SET DELETED seek + BOF — 82/82 stress test PASS
NTX Seek (ntx.go):
- Always descend to leaf even on internal match (Harbour behavior)
  Prevents SEEK returning internal separator instead of first leaf entry
  Fixes duplicate key SEEK (NYC=9→10, Paris=8→10)
- fStop flag tracks path match, verified at leaf with key comparison
- Handle fStop at page end: ascend via nextKey to find actual match

SET DELETED + SEEK (indexer.go):
- When SEEK finds a deleted record with SET DELETED ON:
  Skip forward through matching deleted records
  If all matching records deleted → return not found (EOF)
  Fixes H04: deleted record now correctly returns .F.

BOF (indexer.go + dbf.go):
- Set a.FBof AFTER a.GoTo returns (GoTo resets FBof=false at line 393)
- Fixes infinite loop in DO WHILE !BOF() ... SKIP -1

Results:
- Unit tests: 14 packages ALL PASS
- 77-item thorough test: 77/77 (100%)
- 82-item stress test: 82/82 (100%) — Harbour identical

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 08:33:37 +09:00
9b9f87fd88 fix: NTX B-tree — proper pageSplit from Harbour + BOF detection
NTX Build (build.go):
- pageSplit: exact port of Harbour hb_ntxPageSplit
  - NewPage = LEFT half (lower keys), OldPage = RIGHT half (offset-swapped)
  - Proper offset table initialization for all pages
  - setKeyEntry/copyKeyEntry helpers for clean data writing
- insertKeyBTree: new root creation matches Harbour exactly
  - child[0] = newPage (left), child[1] = old root (right)

NTX Traversal (ntx.go):
- prevKey: guard iKey < keyCount before checking KeyChild
  (prevents infinite loop at rightmost child position)

BOF Detection (indexer.go):
- Set a.FBof AFTER GoTo returns (GoTo line 393 resets FBof=false)
- Previously: set FBof before GoTo → immediately cleared

Results: Unit tests ALL PASS, Stress test 82 items 79/82 match (96%)
Remaining 3 diffs: duplicate key count edge case + SET DELETED seek

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 08:13:42 +09:00
d2c17c7898 refactor: NTX B-tree rewrite — proper insertion with page splitting
Major rewrite based on Harbour dbfntx1.c analysis:

NTX B-tree traversal (ntx.go):
- nextKey: rewritten to match hb_ntxTagNextKey exactly
  - Advance iKey, check right child, descend via goLeftmost
  - Walk up stack on page exhaustion, truncate stackLevel
- prevKey: rewritten to match hb_ntxTagPrevKey
  - Check left child (only if iKey < keyCount), descend via goRightmost
  - Walk up stack for BOF detection
- goRightmost: internal nodes get iKey=keyCount (rightmost child),
  leaf nodes get iKey=keyCount-1 (last key) — matches Harbour

NTX B-tree build (build.go):
- CreateIndex: proper B-tree insertion (insert keys one by one)
- insertKeyBTree: search → insert at leaf → propagate splits up
- pageInsertKey: Harbour-style offset swapping (not data moving)
- pageSplit: collect all entries, split at midpoint, promote separator
- Proper offset table initialization for all pages

Unit tests: all 5 RDD packages PASS
Stress test: partial progress (Seek issues with split pages)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 07:49:31 +09:00
cc46ad2832 fix: SOFTSEEK, SET DELETED+INDEX, compound key, SET INDEX TO + stress test
Fixes from 77/77 thorough test:
- SOFTSEEK uses CurRecNo() (was requiring recNo>0)
- SEEK reads SET SOFTSEEK at runtime (was compile-time only)
- SkipIndexed skips deleted records when SET DELETED ON
- GoTopIndexed skips deleted at top position
- evalKeyExprInner TrimSpace on fieldName (compound key fix)
- SET INDEX TO uses exprToString (was emitExpr treating as variable)
- hasXBaseCommands scans nested blocks (BEGIN SEQUENCE, IF, FOR, etc.)

77/77 thorough test PASS. Stress test (82 items) in progress.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 15:11:17 +09:00
b7028791d6 fix: 5 seek/dbf bugs — 77/77 thorough Harbour compatibility
1. SOFTSEEK: use idx.CurRecNo() for positioning (was checking recNo > 0)
   - SEEK with SET SOFTSEEK ON now positions at next higher key
   - SEEK command reads SET SOFTSEEK at runtime (was compile-time only)
   - rtlDbSeek defaults to GetSetSoftSeek() when no explicit param

2. SET DELETED ON + INDEX: SkipIndexed skips deleted records
   - GoTopIndexed: skip deleted record at top position
   - SkipIndexed: inner loop continues past deleted records

3. Compound key (CITY+NAME): field name TrimSpace before lookup
   - evalKeyExprInner: TrimSpace on fieldName after FIELD-> strip
   - Fixed "CITY " != "CITY" mismatch from + operator splitting

4. SET INDEX TO filename: treated as string, not variable
   - gengo uses exprToString for SET INDEX TO (was emitExpr)
   - Prevents identifier being resolved as local variable

5. hasXBaseCommands: recursive scan into nested blocks
   - BEGIN SEQUENCE, IF, FOR, DO WHILE, SWITCH bodies now scanned
   - Fixes missing hbrdd import for DB commands inside blocks

Thorough test: 77 items (14 sections) covering exact/partial/soft seek,
SET DELETED, duplicate keys, numeric keys, compound keys, empty/single
table, state consistency, order switching, full traversal — all identical.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 14:08:51 +09:00
c04c9aeaa8 feat: INDEX ON with UDF support — user functions in key expressions
Core change:
- dbf.KeyEvalFunc: global callback set by gengo before OrderCreate
- evalKeyExprInner default case: calls KeyEvalFunc for unknown functions
- Final fallback: any unresolvable expression → KeyEvalFunc → MacroEval
- valueToKeyBytes: converts MacroEval result to index key bytes
- gengo: sets dbf.KeyEvalFunc = t.MacroEval before OrderCreate, clears after

Examples that now work:
  INDEX ON MyFunc(FIELD->NAME) TO idx    // UDF in key expression
  INDEX ON CityKey(FIELD->CITY, NAME) TO idx  // multi-param UDF
  INDEX ON Left(MyFunc(NAME), 15) TO idx // nested built-in + UDF

Also fixed:
- SET ORDER TO n: int→string via hbrt.NtoS (was empty string)
- CDX compound leaf decoder: proper bit-packed tag name extraction
- CDX compound recNo = direct byte offset (not page number)

All existing tests pass, NTX 47/47 + CDX 20/20 Harbour compat maintained.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 13:36:21 +09:00
7e2a159b88 feat: CDX support + ORDSCOPE + cross-read Harbour compatibility
CDX Integration:
- IndexEngine interface: common for NTX Index and CDX Tag
- OrderListAdd: auto-detects .cdx/.ntx extension, opens CDX tags
- decodeCompoundLeaf: proper bit-packed tag directory decoding
  (was stub falling through to scanCompoundLeaves with wrong names)
- CDX Tag: added KeyLen(), KeyExpr(), ForExpr(), IsDescending(), Close()
- CDX compound recNo = direct byte offset (not page number)

ORDSCOPE:
- SetScope/ClearScope/SetScopeTop/SetScopeBottom on DBFArea
- GoTopIndexed: seeks to scopeTop, validates within scopeBottom
- GoBottomIndexed: seeks to scopeBottom boundary
- SkipIndexed: stops at scope boundaries (top and bottom)
- OrdScope RTL function registered (nScope: 0=TOP, 1=BOTTOM)
- scopeKeyFromValue: converts Value to padded key bytes

Index Order Management:
- OrderListFocus: handles numeric order ("2" → order 2)
- SET ORDER TO n: gengo emits hbrt.NtoS for int-to-string conversion
- IndexOrd/OrdCount/OrdName/OrdKey: real implementations (were stubs)
- OrderCount/CurrentOrder/OrderName/OrderKeyExpr accessors on DBFArea
- ClearScope on order switch (prevents stale scope)

Cross-read test: Harbour-created CDX → Five reads, 20/20 items match:
  NAME/CITY/ID seek, ORDSCOPE count, GoTop/GoBottom all identical

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 12:21:26 +09:00
441d6c184f test: cross-read Harbour NTX from Five — 100% binary compatible
Five reads DBF + NTX files created by Harbour:
- NAME index: exact/partial seek, GoTop/Bottom, Skip, SoftSeek
- CITY index: duplicate key seek with correct RecNo order
- ID index: numeric key (Str(ID,6)) seek

17/17 items match Harbour output exactly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:55:55 +09:00
af146f03f7 fix: NTX duplicate key sort — RecNo tiebreak for Harbour compatibility
sort.Slice is unstable: equal keys had random record order.
Harbour NTX B-tree orders equal keys by ascending RecNo.
Added RecNo tiebreak to sort comparator.

Result: 47/47 (100%) Harbour compatibility on rdd_compat test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:38:45 +09:00
6e78d12cc2 fix: 3 RDD compat bugs — FIELD->, AsNumInt Double, PACK/ZAP with index
Bug 1: FIELD->NAME in INDEX ON expression
- evalKeyExprInner: strip FIELD->/alias-> prefix before field lookup
- exprToString: handle AliasExpr (FIELD->NAME → "FIELD->NAME")

Bug 2: AsNumInt() on Double returned IEEE 754 raw bits
- Value.AsNumInt(): check tDouble and convert via Float64frombits
- Fixed array index crash when index is result of % modulo

Bug 3: PACK/ZAP crash with open indexes
- OrderListRebuild: fully implemented (was TODO stub)
  Saves index info, closes all, sets idxState=nil, recreates
- OrderCreate: set current=-1 during key evaluation (natural GoTo)
- PACK/ZAP: save/restore idxState, rebuild after operation
- Register __DBPACK, __DBZAP, DBRECALL symbol aliases

Harbour vs Five: 45/47 match (96%), 2 diffs are duplicate-key sort order

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 04:41:19 +09:00
53370e7cbc test: Harbour vs Five RDD compatibility test (91% match)
47 test items comparing Harbour and Five output:
- T01-T28: 100% match (CRUD, navigation, SET DELETED)
- T29-T39: 100% match (SEEK exact/partial/softseek)
- T40-T41: Found matches, RecNo differs (duplicate key sort stability)
- T42-T43: 100% match
- T44-T47: Five crashes (PACK with open index)

Known issues found:
- FIELD->NAME syntax not supported in INDEX ON expression
- Modulo % returns Double causing array index hang (Int() workaround)
- PACK crashes when NTX index is open

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 19:42:50 +09:00
21fd9dc65c feat: SET DELETED filtering, SEEK/LOCATE/CONTINUE, SET command codegen
- skipFilter: skip deleted records in GoTop/GoBottom/Skip when SET DELETED ON
- hbrdd.IsSetDeleted callback: avoids circular import hbrdd→hbrtl
- Parser: capture ON/OFF for boolean SET commands (DELETED, EXACT, SOFTSEEK, etc.)
- Parser: capture TO expr for SET DATE/DECIMALS/EPOCH
- Gengo: emit proper t.Do() calls for 11 SET toggles + 3 value SETs
- stmtSet: was stub (skipToEOL), now calls parseSet()
- RTL: register 11 SET toggle functions (SETDELETED, SETEXACT, etc.)
- RTL: DBLOCATE/DBCONTINUE for sequential search
- RTL: DBSETFILTER/DBCLEARFILTER/DBFILTER
- PadL/PadR: support 3rd param fill character
- Area interface: added SetFound, SetLocate, LocateBlock, filter methods
- MemRDD: implements new Area interface methods
- Comprehensive PRG test: test_search.prg (7 test suites all pass)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 22:33:59 +09:00