Benchmark (50k records, 4 indexes on Apple M-series):
before after Δ
INDEX 53.7ms 33.3ms -38% (now 10% faster than Harbour 37.3ms)
TOTAL 156.2ms 133.0ms -15%
Fixes:
1. sort.Slice(reflection) → concrete sort.Interface
Benchmarked in isolation on 200k KeyRecords:
sort.Slice(closure): 50.0ms
sort.Sort(interface): 30.4ms (40% faster, no reflection)
- indexer.go: add keyRecordAsc/Desc concrete types
- Branch hoist descending check out of Less()
2. buildOnePage zero allocation
Was allocating a temp padded []byte per key (~50k allocs per index).
Now writes padded key directly into the page buffer via padCopy.
3. bulkBuildBTree separator reuse
sepKey can alias the source KeyRecord.Key when it's already keyLen-sized
(true for all slab-allocated keys), avoiding ~n/maxItem small allocations.
Pre-size the children slice.
4. Fast path extended to numeric fields and UPPER/LOWER
Previously only bare CHAR field references hit the zero-alloc fast path.
Now:
- Numeric fields (N/F type) copy DBF bytes directly
(same-length ASCII compare matches numeric order for non-negatives)
- UPPER(field) / LOWER(field) wrappers on CHAR fields apply ASCII
case folding inline during byte copy
Per-index timing on the micro benchmark:
before after
NAME 7.7ms 7.5ms (fast path, unchanged)
CITY 6.0ms 6.2ms (fast path, unchanged)
AGE 14.1ms 7.1ms -50% (was slow path)
UPPER(NM) 17.0ms 7.9ms -54% (was slow path)
5. Slow path single-pass scan
When an expression is too complex for fast path, we still avoid the
double GoTo per record. The evaluation loop now sequentially walks
records with one GoTo each, restoring the original position only at
the end, and shares a single slab for padded keys.
Also fixes a hbrt bug surfaced while writing the benchmark:
6. Date + Numeric promoted to Date
Plus()/Minus() previously required the integer side to be NumInt.
Modulus returns a promoted type, so `SToD("...") + (i % 365)` panicked.
Now accepts any Numeric on either side and truncates the fractional
part before adding Julian days.
- hbrt/ops_arith.go: Date±Numeric (was Date±NumInt only)
Tests:
go test ./... — ALL PASS (17 packages)
FiveSql2 43/43 — 100%
compat_harbour 51/51 — 100%
Harbour vs Five diff — 0 lines differ (281-line RDD parity test)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
From senior Go developer review:
C7 CRITICAL: pagePool data race (ntx.go)
- Moved global pagePool[8] + pagePoolIdx into per-Index struct
- Eliminates race condition across goroutines using separate indexes
C8 CRITICAL: Page.data dangling pointer after remap (ntx.go)
- remapFile() now clears pagePool data slices (pointed into old mmap)
- Prevents segfault from stale mmap references
C4 HIGH: pop() bounds check restored (thread.go)
- Removed performance optimization that eliminated underflow detection
- Stack underflow now produces clear error instead of index -1 panic
C1 HIGH: intExpLen overflow on MinInt64 (value.go)
- Added special case: MinInt64 returns 20 (length of -9223372036854775808)
- Prevents -v overflow in negation
C11 CRITICAL: GoTo ReadAt error handling (dbf.go)
- ReadAt failure now returns error and sets EOF
- Previously silently used stale record buffer (data corruption risk)
C14 HIGH: LEN() inline missing Hash case (gengo.go)
- Added _v.IsHash() → len(Keys) branch
C15 HIGH: EMPTY() inline missing Date case (gengo.go)
- Added _v.IsDate() && _v.AsJulian() == 0 check
82/82 stress PASS. 14 packages ALL PASS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NTX 3-level tree (build.go):
- Hybrid approach: bulk build for ≤2 levels, insertKeyBTree for 3+
- rebuildWithInsert: creates proper B-tree via per-key insertion
- 5000-key test: Count=5000 Found=5000 (was 5004/4868)
CDX SET INDEX TO (gengo.go):
- Strip surrounding quotes from string literal in OrderListAdd
- Was: idx.OrderListAdd("\"path\"") → file not found
- Now: idx.OrderListAdd("path") → correct
All tests:
- 14 packages ALL PASS
- 82/82 NTX stress test
- 18/18 CDX cross-read
- 50K benchmark: all counts correct
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NTX Bulk Build (build.go — ported from rddfive/ntx_engine.c):
- pageBuffer: dynamic memory buffer for all pages
- Phase 1: Build leaf pages in sequential memory (zero disk I/O)
- Phase 2: Build interior levels from cached leaf data (zero I/O)
- Separator promotion: remove last key from leaf only (not interior)
- Single bulk WriteAt for all pages at end
- INDEX ON 10K: 34ms → 5-8ms (4-6x improvement)
NTX Seek (ntx.go):
- Always descend to leaf on match (find first occurrence)
- fStop flag tracks path match, verified at leaf
APPEND Buffering (dbf.go):
- Append marks dirty without immediate disk write
- flushRecord writes record data only (no header/EOF per record)
- Close/Flush writes EOF marker + header once
Results: 14 packages ALL PASS, 82/82 stress test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NTX Seek (ntx.go):
- Always descend to leaf even on internal match (Harbour behavior)
Prevents SEEK returning internal separator instead of first leaf entry
Fixes duplicate key SEEK (NYC=9→10, Paris=8→10)
- fStop flag tracks path match, verified at leaf with key comparison
- Handle fStop at page end: ascend via nextKey to find actual match
SET DELETED + SEEK (indexer.go):
- When SEEK finds a deleted record with SET DELETED ON:
Skip forward through matching deleted records
If all matching records deleted → return not found (EOF)
Fixes H04: deleted record now correctly returns .F.
BOF (indexer.go + dbf.go):
- Set a.FBof AFTER a.GoTo returns (GoTo resets FBof=false at line 393)
- Fixes infinite loop in DO WHILE !BOF() ... SKIP -1
Results:
- Unit tests: 14 packages ALL PASS
- 77-item thorough test: 77/77 (100%)
- 82-item stress test: 82/82 (100%) — Harbour identical
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major rewrite based on Harbour dbfntx1.c analysis:
NTX B-tree traversal (ntx.go):
- nextKey: rewritten to match hb_ntxTagNextKey exactly
- Advance iKey, check right child, descend via goLeftmost
- Walk up stack on page exhaustion, truncate stackLevel
- prevKey: rewritten to match hb_ntxTagPrevKey
- Check left child (only if iKey < keyCount), descend via goRightmost
- Walk up stack for BOF detection
- goRightmost: internal nodes get iKey=keyCount (rightmost child),
leaf nodes get iKey=keyCount-1 (last key) — matches Harbour
NTX B-tree build (build.go):
- CreateIndex: proper B-tree insertion (insert keys one by one)
- insertKeyBTree: search → insert at leaf → propagate splits up
- pageInsertKey: Harbour-style offset swapping (not data moving)
- pageSplit: collect all entries, split at midpoint, promote separator
- Proper offset table initialization for all pages
Unit tests: all 5 RDD packages PASS
Stress test: partial progress (Seek issues with split pages)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
