Commit Graph

10 Commits

Author SHA1 Message Date
a9600ad45c perf: proper 3-level bulk build — INDEX 50K: 180ms → 28ms (6.4x)
bulkBuildBTree: distributes sorted keys as [M leaf] [sep] [M leaf] [sep] ...
- Separator exists ONLY in parent, never in leaf (proper B-tree)
- Works for any depth (tested 10 to 50000 keys, all correct)
- Edge case: absorb trailing 1-key into previous leaf

Eliminated per-key insertion fallback (rebuildWithInsert).
All sizes now use O(N) bulk build instead of O(N log N) insertion.

Benchmark on ext4 (home dir):
┌──────────────┬──────────┬──────────┬───────┐
│ 50K Items    │ Harbour  │ Five     │ Ratio │
├──────────────┼──────────┼──────────┼───────┤
│ APPEND 50K   │ 61ms     │ 124ms    │ 2x    │
│ INDEX NAME   │ 6ms      │ 28ms     │ 4.7x  │
│ INDEX CITY   │ 5ms      │ 36ms     │ 7.2x  │
│ SEEK 50K seq │ 23ms     │ 97ms     │ 4.2x  │
│ SEEK 50K rnd │ 63ms     │ 122ms    │ 1.9x  │
│ SCAN 50K     │ 5ms      │ 24ms     │ 4.8x  │
│ DUPKEY 50K   │ 23ms     │ 38ms     │ 1.7x  │
│ PACK 50K     │ 16ms     │ 20ms     │ 1.25x │
└──────────────┴──────────┴──────────┴───────┘

All counts correct: 50000/50000/40000

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 12:08:27 +09:00
c5ed5612fb perf: mmap zero-copy page access — Go-native optimization
Replaced LRU page cache with syscall.Mmap:
- OpenIndex: mmap entire file read-only (MAP_SHARED)
- cachedLoadPage: copy from mmap slice (no syscall per page)
- Close: munmap + file close
- insertKeyBTree: munmap before modify, mmapFile after complete
- remapFile: re-mmap after file size changes

Results on ext4 (50K records):
- SEEK random: 188ms → 138ms (26% improvement)
- SCAN: 35ms → 23ms (34% improvement)
- DUPKEY: 53ms → 41ms (23% improvement)
- INDEX: 180ms (unchanged — per-key insertion, no mmap during build)

Go-native approach:
- syscall.Mmap instead of C-style LRU cache
- OS page cache handles eviction automatically
- Simpler code (60 lines removed, 30 added)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 11:42:44 +09:00
103f0d8b64 perf: NTX LRU page cache (256 slots) — reduces syscalls
LRU page cache ported from rddfive/ntx_engine.c:
- 256-slot cache with MRU fast-path (O(1) for repeated access)
- LRU eviction when all slots full
- cachedLoadPage replaces LoadPage for all navigation
- invalidateCache called before insertKeyBTree (pages modified)

10K benchmark improvement (ext4 home dir):
- SCAN FWD: 6ms → 5ms
- SEEK NUM: 18ms → 14ms (22% improvement)
- DUPKEY SCAN: 9ms → 8ms
- All counts correct: 10000/10000/8000

50K benchmark:
- SCAN: 35ms → 31ms
- DUPKEY: 50ms → 40ms (20% improvement)
- DELSCAN: 41ms → 33ms (20% improvement)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 11:35:26 +09:00
dadb97ee88 fix: 3-level NTX correctness + CDX SET INDEX TO string quoting
NTX 3-level tree (build.go):
- Hybrid approach: bulk build for ≤2 levels, insertKeyBTree for 3+
- rebuildWithInsert: creates proper B-tree via per-key insertion
- 5000-key test: Count=5000 Found=5000 (was 5004/4868)

CDX SET INDEX TO (gengo.go):
- Strip surrounding quotes from string literal in OrderListAdd
- Was: idx.OrderListAdd("\"path\"") → file not found
- Now: idx.OrderListAdd("path") → correct

All tests:
- 14 packages ALL PASS
- 82/82 NTX stress test
- 18/18 CDX cross-read
- 50K benchmark: all counts correct

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 11:04:07 +09:00
adede5cd69 perf: REPLACE remove Flush + bulk build + deferred write = 1600x faster
Critical fix: REPLACE was calling area.Flush() after every field write!
- gengo gen_cmd.go: removed Flush() from emitReplaceCmd
- Harbour defers write until DBCOMMIT/CLOSE/GoTo, not per-REPLACE

Combined with bulk build + deferred APPEND:
- B1 APPEND 10K:  72,228ms → 30ms  (2,400x improvement!)
- B2 INDEX NAME:  34ms → 5ms       (6.8x improvement)
- Harbour comparison: Five 30ms vs Harbour 27ms (1.1x)

Also: OrderCreate flushes dirty record + EOF + header before index build

Benchmark on ext4 (home dir):
┌─────────────┬──────────┬────────┬───────┐
│ Benchmark   │ Harbour  │ Five   │ Ratio │
├─────────────┼──────────┼────────┼───────┤
│ APPEND 10K  │ 27ms     │ 30ms   │ 1.1x  │
│ INDEX NAME  │ 2ms      │ 5ms    │ 2.5x  │
│ INDEX CITY  │ 0ms      │ 7ms    │ -     │
│ SEEK 10K    │ 6ms      │ 25ms   │ 4.2x  │
│ SCAN FWD    │ 1ms      │ 6ms    │ 6x    │
│ SCAN BWD    │ 0ms      │ 6ms    │ -     │
│ PACK        │ 4ms      │ 3ms    │ 0.75x │
└─────────────┴──────────┴────────┴───────┘

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 09:22:05 +09:00
b1e868f01e perf: NTX bulk build + APPEND deferred write (from rddfive C port)
NTX Bulk Build (build.go — ported from rddfive/ntx_engine.c):
- pageBuffer: dynamic memory buffer for all pages
- Phase 1: Build leaf pages in sequential memory (zero disk I/O)
- Phase 2: Build interior levels from cached leaf data (zero I/O)
- Separator promotion: remove last key from leaf only (not interior)
- Single bulk WriteAt for all pages at end
- INDEX ON 10K: 34ms → 5-8ms (4-6x improvement)

NTX Seek (ntx.go):
- Always descend to leaf on match (find first occurrence)
- fStop flag tracks path match, verified at leaf

APPEND Buffering (dbf.go):
- Append marks dirty without immediate disk write
- flushRecord writes record data only (no header/EOF per record)
- Close/Flush writes EOF marker + header once

Results: 14 packages ALL PASS, 82/82 stress test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 09:10:18 +09:00
3fe8021e9e fix: NTX Seek descent + SET DELETED seek + BOF — 82/82 stress test PASS
NTX Seek (ntx.go):
- Always descend to leaf even on internal match (Harbour behavior)
  Prevents SEEK returning internal separator instead of first leaf entry
  Fixes duplicate key SEEK (NYC=9→10, Paris=8→10)
- fStop flag tracks path match, verified at leaf with key comparison
- Handle fStop at page end: ascend via nextKey to find actual match

SET DELETED + SEEK (indexer.go):
- When SEEK finds a deleted record with SET DELETED ON:
  Skip forward through matching deleted records
  If all matching records deleted → return not found (EOF)
  Fixes H04: deleted record now correctly returns .F.

BOF (indexer.go + dbf.go):
- Set a.FBof AFTER a.GoTo returns (GoTo resets FBof=false at line 393)
- Fixes infinite loop in DO WHILE !BOF() ... SKIP -1

Results:
- Unit tests: 14 packages ALL PASS
- 77-item thorough test: 77/77 (100%)
- 82-item stress test: 82/82 (100%) — Harbour identical

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 08:33:37 +09:00
9b9f87fd88 fix: NTX B-tree — proper pageSplit from Harbour + BOF detection
NTX Build (build.go):
- pageSplit: exact port of Harbour hb_ntxPageSplit
  - NewPage = LEFT half (lower keys), OldPage = RIGHT half (offset-swapped)
  - Proper offset table initialization for all pages
  - setKeyEntry/copyKeyEntry helpers for clean data writing
- insertKeyBTree: new root creation matches Harbour exactly
  - child[0] = newPage (left), child[1] = old root (right)

NTX Traversal (ntx.go):
- prevKey: guard iKey < keyCount before checking KeyChild
  (prevents infinite loop at rightmost child position)

BOF Detection (indexer.go):
- Set a.FBof AFTER GoTo returns (GoTo line 393 resets FBof=false)
- Previously: set FBof before GoTo → immediately cleared

Results: Unit tests ALL PASS, Stress test 82 items 79/82 match (96%)
Remaining 3 diffs: duplicate key count edge case + SET DELETED seek

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 08:13:42 +09:00
d2c17c7898 refactor: NTX B-tree rewrite — proper insertion with page splitting
Major rewrite based on Harbour dbfntx1.c analysis:

NTX B-tree traversal (ntx.go):
- nextKey: rewritten to match hb_ntxTagNextKey exactly
  - Advance iKey, check right child, descend via goLeftmost
  - Walk up stack on page exhaustion, truncate stackLevel
- prevKey: rewritten to match hb_ntxTagPrevKey
  - Check left child (only if iKey < keyCount), descend via goRightmost
  - Walk up stack for BOF detection
- goRightmost: internal nodes get iKey=keyCount (rightmost child),
  leaf nodes get iKey=keyCount-1 (last key) — matches Harbour

NTX B-tree build (build.go):
- CreateIndex: proper B-tree insertion (insert keys one by one)
- insertKeyBTree: search → insert at leaf → propagate splits up
- pageInsertKey: Harbour-style offset swapping (not data moving)
- pageSplit: collect all entries, split at midpoint, promote separator
- Proper offset table initialization for all pages

Unit tests: all 5 RDD packages PASS
Stress test: partial progress (Seek issues with split pages)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 07:49:31 +09:00
59568f3301 Five v0.9 — Harbour + Go fusion language
- Compiler: PP → Lexer → Parser → Analyzer → Gengo pipeline
- Parser: 232/236 (98%) Harbour compatibility, registry-based dispatch
- RTL: 351 Harbour-compatible functions
- RDD: DBF/NTX/CDX engines with Rushmore bitmap optimization
- Go Interop: IMPORT + pkg.Func() + obj:Method() with FastPath (15M calls/sec)
- HB_FUNC API: Full Harbour C API compatible Go bridge
- Concurrency: SPAWN/LAUNCH/GOROUTINE, <-, WATCH, PARALLEL FOR, ASYNC/AWAIT
- Extensions: Multi-return, DEFER, Slice, f-string, Nil-safe ?:, CONST
- Macro Compiler: Runtime AST parsing and evaluation
- Debugger: TUI debugger with source display, breakpoints, stepping
- FRB: Native + Pcode dual mode runtime binary
- Tests: 13 packages ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 09:41:50 +09:00