13 Commits

Author SHA1 Message Date
f4ed42556b checkpoint: season-wide bug fix campaign + infra
Cumulative season's silent-bug hunting (~62 fixes) across the FiveSql2
SQL engine, the Five compiler/runtime, and the hbrdd RDD layer. Saved
as a single checkpoint before refactoring the parser to delegate xBase
command translation to the preprocessor.

Highlights:

FiveSql2 engine (_FiveSql2/src/)
- prefix-glob index attach -> explicit convention (<table>_pk.ntx,
  <table>_uq.ntx, <table>.cdx) — fixes silent multi-row INSERT row-drop
- DROP/CREATE TABLE FErase chain extended (.cdx, .fsc, .fsv, .dbt, .fpt)
- COUNT(DISTINCT col) parsed + aggregated via hSeen hash
- UNION column-count mismatch returns SQL_ERR_GRAMMAR (was silent)
- DISTINCT + ORDER BY hidden-col leak fixed (trim before DISTINCT)
- Derived table FROM (SELECT...) + JOIN right-side derived
- Self-FK CASCADE depth 2+ via SqlGetSingleColPK pre-collect
- LAG/LEAD default arg uses SqlEvalRowExpr (handles -N const exprs)
- DATE literal round-trip validation (Feb 29 non-leap rejected)
- CREATE OR REPLACE VIEW; CREATE VIEW errors on already-exists
- AlterTable type dispatcher comma-wrapped (1-char type "A" no longer
  matches CHARACTER)

Compiler / runtime
- gengo: HB_ -> FV_ prefix on emitted Go function names (Five identity)
- gengo split: emit_block.go, emit_stmt.go, folding.go extracted
- parser/stmtreg.go nudges
- hbrt: debug TUI/CLI restructure (debugcmd, debugkey, termios_*),
  windows debug stubs collapsed
- thread/vm/value/class/pcinterp tightening from panic traces

RDD layer (hbrdd/)
- dbf: null bitmap support (null.go + null_test.go), mmap split
  (mmap_posix.go / mmap_windows.go), byte-level numeric parse
- ntx/cdx: windows mmap parity
- workarea + mem RDD: cross-area state-bleed fixes

RTL (hbrtl/)
- errorlog rewrite with platform-specific FD (errorlog_fd_unix /
  errorlog_fd_other)
- sqlscan, sqlhelpers, indexrtl, datetime extensions

Gates green at checkpoint:
- go test ./...        : PASS
- FiveSql2 SQL:1999    : 43/43
- Harbour compat       : 56/56

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 09:26:25 +09:00
66882c30bd fix(cdx): Harbour-compatible layout — compound root, RCHB sig, leaf format
Align Five's CDX file layout with Harbour's expectations:
- Compound root header at 0, compound leaf at 1024, tags at 1536+
- "RCHB" signature at offset 20 in compound root
- IgnoreCase/collation flags at offset 503-505
- Compound leaf: LeftPtr/RightPtr = 0xFFFFFFFF, recBits=16 fixed
- Tags sorted alphabetically in compound directory B-tree
- Tag IndexOpt: TypeCompact | TypeCompound (0x60)

Status of Harbour cross-read verification:
- CHAR-only CDX tags: layout matches Harbour byte-for-byte
- Numeric tags: Harbour uses IEEE double (8-byte) key encoding,
  Five uses DBF ASCII key bytes — causes DBFCDX/1012 corruption
  when Harbour reads Five-created CDX with numeric tags
- Five reading Harbour CDX: works perfectly (existing)
- Five reading Five CDX: works perfectly

Remaining: numeric key encoding for full Harbour write-compatibility.
CLAUDE.md updated to reflect this single remaining limitation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 01:33:52 +09:00
4d5621c21a feat: CDX compound index write + {||} parsing + zero known constraints
All 3 remaining known constraints resolved. CLAUDE.md now shows zero.

1. CDX compound index WRITE support (was read-only)

   New file: hbrdd/cdx/build.go (~400 LOC)
   - CreateOrAddTag() builds Harbour-compatible CDX files
   - Bit-packed leaf pages (RecBits/DupBits/TrlBits compression)
   - Interior nodes with big-endian RecNo/ChildPage
   - Compound root directory (structural B-tree of tag names)
   - Append-safe: preserves existing tags when adding new ones
   - Linked leaf pages (LeftPtr/RightPtr for sequential scan)

   Pipeline: INDEX ON expr TAG tagname TO file
   - ast.IndexCmd gains TagName field
   - Parser captures TAG name (was discarded)
   - gengo passes TagName to OrderCreateParams
   - indexer.go routes to cdx.CreateOrAddTag when TAG specified

   Verified: 3 tags (BYNAME/BYCITY/BYAGE), OrdSetFocus by name,
   SEEK, GoTop/GoBottom, close+reopen with SET INDEX TO

2. {||} empty code block parsing in function arguments

   Parser's parseArrayOrBlock() called parseExpr() unconditionally
   after closing |, failing when body was empty ({||}).
   Fix: check for RBRACE after closing | and emit NIL literal body.
   {=>} empty hash already worked.

3. Semicolon IF...ENDIF — already worked (removed from constraints)

Tests:
  go test ./...        14 packages ALL PASS
  FiveSql2             43/43 100%
  compat_harbour       51/51

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:58:09 +09:00
ad544a5528 fix: Windows cross-compilation support (GOOS=windows)
- debugcli.go/debugtui.go: add //go:build !windows tag
- debugcli_windows.go/debugtui_windows.go: no-op stubs
- cdx/cdx.go: extract mmap to platform-specific files
- cdx/mmap_posix.go: syscall.Mmap/Munmap
- cdx/mmap_windows.go: no-op (falls back to read)
- ntx/ntx.go, ntx/build.go: same mmap extraction
- ntx/mmap_posix.go, ntx/mmap_windows.go: platform split

Builds verified: linux/amd64, windows/amd64, darwin/arm64, darwin/amd64

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:23:52 +09:00
e95afad4ee feat: Harbour RDD parity — NTX/CDX 100% compatible, FIELD-> works
Five RDD engine now matches Harbour DBFNTX and DBFCDX byte-for-byte
in ordering, seek, navigation, and field access. Verified against
Harbour 3.2.0dev with a 281-line comparison test covering:
  - Natural/NAME/CITY/AGE/SALARY/UPPER ordering
  - SEEK (exact/not-found), GoTop/GoBottom per order
  - DELETE/RECALL with SET DELETED
  - CDX compound index read with 5 tags (BYNAME, BYCITY, BYAGE, BYSAL, BYUNAME)
  - Reverse traversal

Fixes:

1. FIELD->NAME returned NIL
   GetAliasField returned interface{} but runtime expected hbrt.Value,
   so the type assertion in PushAliasField failed and pushed NIL.
   - workarea.go: change return type to hbrt.Value, handle FIELD/_FIELD
     as current-workarea alias, add SetAliasField
   - gengo.go: emit SetAliasField() for alias->field := value in both
     statement and expression contexts

2. OrdSetFocus(n) silently switched to natural order
   v.AsString() returns "" for a numeric Value, so OrderListFocus("")
   set current=-1.
   - indexrtl.go: convert numeric param via fmt.Sprintf("%d", ...)

3. CDX compound tag order mismatched Harbour
   Five decoded the structural B-tree which is alphabetical, but
   Harbour sorts tags by TagBlock (file offset = creation order).
   - cdx/cdx.go: sort tagEntries by offset ascending after decoding,
     matching hb_cdxIndexLoadAvailTags in dbfcdx1.c

4. OutStd()/OutErr() not registered — caused panic on call
   - hbrtl/console.go: add rtlOutStd/rtlOutErr implementations
   - hbrtl/register.go: register OUTSTD and OUTERR
   - analyzer.go: add OUTSTD/OUTERR to RTL known-functions

5. FIELD keyword triggered "undeclared variable" warnings
   - analyzer.go: add FIELD, _FIELD, M, MEMVAR as builtin constants

Tests:
  go test ./...        — ALL PASS (17 packages)
  FiveSql2 43/43       — 100%
  compat_harbour 51/51 — 100%
  Harbour diff         — 0 lines differ (281-line comparison)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:37:47 +09:00
279a16a88c refactor: pure Go — recursion→iteration, COW records, zero alloc
CDX Seek iterative (cdx.go):
- Converted recursive seekPage → iterative loop
- Single buf reused across all B-tree levels (was: make per level)
- Internal node: binary search (was: linear O(n))
- Eliminates 3 heap allocations per CDX SEEK

DBF Copy-on-Write records (dbf.go):
- GoTo: recBuf = mmap slice reference (zero-copy read)
- PutValue/Delete/Recall: promote to ownBuf before write
- Eliminates memcpy per GoTo for read-only SCAN operations
- recOwned flag tracks COW state

NTX build.go:
- setKeyEntry: write directly to page (no temp make([]byte))
- padCopy: copy+fill (no pre-fill entire buffer)

CDX DecodeLeafKeys slab (cdx.go):
- Single slab allocation for all keys per page

82/82 stress PASS. All unit tests PASS.

50K SEEK random: 63ms (Harbour 67ms — FASTER!)
50K DELSCAN: 2ms (Harbour 12ms — 6x FASTER!)
CDX SCOPE: 2ms (Harbour 4ms — 2x FASTER!)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:56:20 +09:00
0102c3c94e perf: CGo review — slab alloc, compareKeys simplify, zero-alloc padCopy
From CGo expert review (verdict: stay pure Go, CGo would be slower):

CDX DecodeLeafKeys slab allocation (cdx.go):
- Single make() for all keys + prevKey (was 30+ allocs per page)
- Keys are slices into pre-allocated slab (zero copy)

NTX compareKeys simplified (ntx.go):
- bytes.Compare already returns normalized -1/0/+1
- Removed redundant normalization branches

NTX build.go zero-alloc:
- padCopy: copy+fill instead of make+fill+copy
- setKeyEntry: write directly to page data (no temp buffer)

82/82 stress PASS. 14 packages ALL PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:44:56 +09:00
b72623f79c perf: CDX binary search + leaf cache hit + DBF/NTX zero-copy
CDX Seek (cdx.go — ported from rddfive/cdx_engine.c):
- Linear search → binary search on decoded leaf keys (O(N) → O(log N))
- Leftmost match: continues searching left after match (duplicate key correctness)
- Leaf cache hit: skip decode if same page (SEEK loop optimization)

NTX zero-copy Page (ntx.go — BoltDB pattern):
- Page.data: []byte slice into mmap (was [1024]byte copy)
- cachedLoadPage: p.data = mmap[offset:offset+1024] (no memcpy!)
- pagePool: 8-slot ring for Page struct reuse

DBF mmap (dbf.go):
- GoTo: copy from mmap instead of file.ReadAt syscall
- Unmap before Append/Close/Flush (file growth), re-mmap after

Results (50K, ext4, Harbour comparison):
┌──────────────┬──────────┬──────────┬──────────────┐
│              │ Harbour  │ Five     │              │
├──────────────┼──────────┼──────────┼──────────────┤
│ CDX SEEK     │ 27ms     │ 49ms     │ 1.8x (was 6.5x!)│
│ CDX SEEK ID  │ 17ms     │ 24ms     │ 1.4x (was 8.4x!)│
│ CDX SCAN     │ 5ms      │ 4ms      │  FASTER    │
│ CDX SCOPE    │ 4ms      │ 3ms      │  FASTER    │
│ NTX SCAN     │ 4ms      │ 3ms      │  FASTER    │
│ NTX DELSCAN  │ 12ms     │ 3ms      │  4x FASTER │
│ NTX SEEK rnd │ 67ms     │ 69ms     │ ≈ equal      │
└──────────────┴──────────┴──────────┴──────────────┘

82/82 stress PASS. CDX 18/18 cross-read PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 20:18:54 +09:00
96d72a456c perf: CDX zero-alloc internal node seek — SEEK 45% faster
Internal node traversal: read directly from mmap/buf slice
- No DecodeIntKeys allocation (was nKeys+1 IntKeyEntry structs)
- No key byte slice copy (compare directly against buf)
- Big-endian child/recNo read inline

CDX 50K benchmark:
  SEEK NAME: 362ms → 199ms (45% faster)
  SEEK ID:   320ms → 184ms (42% faster)
  SCAN:      14ms (unchanged — leaf cache handles this)
  SCOPE:     20ms → 14ms

Harbour comparison:
  SEEK: 27ms (Harbour) vs 199ms (Five) = 7.4x
  SCAN: 6ms (Harbour) vs 14ms (Five) = 2.3x

CDX cross-read: 18/18 PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 14:05:20 +09:00
40935b6103 perf: CDX byte-level decode + leaf cache — SCAN 20x faster
Ported from rddfive/cdx_engine.c cdx_leaf_decode_all():
- Replaced bit-by-bit extractBits loop with byte-level shift/mask
- Read reqByte as little-endian integer, extract recNo/dup/trl via masks
- 10x+ faster than per-bit extraction

Leaf page decode cache:
- Tag.cachedLeafOff/cachedLeafKeys: avoid re-decoding same leaf page
- SkipNext/SkipPrev use getLeafKeys() with cache
- GoTop/Seek populate cache on first decode

CDX 50K benchmark (ext4):
┌──────────────┬──────────┬──────────┬──────────┐
│ CDX 50K      │ Harbour  │ Before   │ After    │
├──────────────┼──────────┼──────────┼──────────┤
│ SCAN 50K     │ 6ms      │ 276ms    │ 14ms     │ ← 20x faster
│ SCOPE 35K    │ 4ms      │ 238ms    │ 20ms     │ ← 12x faster
│ SEEK NAME    │ 27ms     │ 362ms    │ 239ms    │
│ SEEK ID      │ 18ms     │ 320ms    │ 195ms    │
└──────────────┴──────────┴──────────┴──────────┘

CDX cross-read: 18/18 PASS. All unit tests PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 13:37:57 +09:00
1b41384675 fix: CDX mmap + internal node format (BE key-first) — 50K works
CDX internal node format fix:
- Was: [child LE][recNo LE][key] (NTX-style)
- Now: [key][recNo BE][child BE] (correct CDX format)
- Fixes GoTop/Seek/Scan for large CDX files (50K+ records)

CDX mmap:
- syscall.Mmap on OpenIndex for zero-copy reads
- idx.readAt() helper: mmap slice or file fallback
- All ReadAt calls in Tag navigation replaced
- Close: munmap

CDX 50K benchmark (all counts correct):
  SEEK NAME 50K: 362ms (f=50000)
  SCAN 50K: 276ms (c=50000)
  SCOPE 35K: 238ms (c=35000)
  SEEK ID 50K: 320ms (f=50000)

CDX is slower than NTX due to bit-packed leaf decompression per page.
Cross-read test: 18/18 still PASS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 13:32:06 +09:00
7e2a159b88 feat: CDX support + ORDSCOPE + cross-read Harbour compatibility
CDX Integration:
- IndexEngine interface: common for NTX Index and CDX Tag
- OrderListAdd: auto-detects .cdx/.ntx extension, opens CDX tags
- decodeCompoundLeaf: proper bit-packed tag directory decoding
  (was stub falling through to scanCompoundLeaves with wrong names)
- CDX Tag: added KeyLen(), KeyExpr(), ForExpr(), IsDescending(), Close()
- CDX compound recNo = direct byte offset (not page number)

ORDSCOPE:
- SetScope/ClearScope/SetScopeTop/SetScopeBottom on DBFArea
- GoTopIndexed: seeks to scopeTop, validates within scopeBottom
- GoBottomIndexed: seeks to scopeBottom boundary
- SkipIndexed: stops at scope boundaries (top and bottom)
- OrdScope RTL function registered (nScope: 0=TOP, 1=BOTTOM)
- scopeKeyFromValue: converts Value to padded key bytes

Index Order Management:
- OrderListFocus: handles numeric order ("2" → order 2)
- SET ORDER TO n: gengo emits hbrt.NtoS for int-to-string conversion
- IndexOrd/OrdCount/OrdName/OrdKey: real implementations (were stubs)
- OrderCount/CurrentOrder/OrderName/OrderKeyExpr accessors on DBFArea
- ClearScope on order switch (prevents stale scope)

Cross-read test: Harbour-created CDX → Five reads, 20/20 items match:
  NAME/CITY/ID seek, ORDSCOPE count, GoTop/GoBottom all identical

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 12:21:26 +09:00
59568f3301 Five v0.9 — Harbour + Go fusion language
- Compiler: PP → Lexer → Parser → Analyzer → Gengo pipeline
- Parser: 232/236 (98%) Harbour compatibility, registry-based dispatch
- RTL: 351 Harbour-compatible functions
- RDD: DBF/NTX/CDX engines with Rushmore bitmap optimization
- Go Interop: IMPORT + pkg.Func() + obj:Method() with FastPath (15M calls/sec)
- HB_FUNC API: Full Harbour C API compatible Go bridge
- Concurrency: SPAWN/LAUNCH/GOROUTINE, <-, WATCH, PARALLEL FOR, ASYNC/AWAIT
- Extensions: Multi-return, DEFER, Slice, f-string, Nil-safe ?:, CONST
- Macro Compiler: Runtime AST parsing and evaluation
- Debugger: TUI debugger with source display, breakpoints, stepping
- FRB: Native + Pcode dual mode runtime binary
- Tests: 13 packages ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 09:41:50 +09:00