Benchmark (50k records, 4 indexes on Apple M-series):
before after Δ
INDEX 53.7ms 33.3ms -38% (now 10% faster than Harbour 37.3ms)
TOTAL 156.2ms 133.0ms -15%
Fixes:
1. sort.Slice(reflection) → concrete sort.Interface
Benchmarked in isolation on 200k KeyRecords:
sort.Slice(closure): 50.0ms
sort.Sort(interface): 30.4ms (40% faster, no reflection)
- indexer.go: add keyRecordAsc/Desc concrete types
- Branch hoist descending check out of Less()
2. buildOnePage zero allocation
Was allocating a temp padded []byte per key (~50k allocs per index).
Now writes padded key directly into the page buffer via padCopy.
3. bulkBuildBTree separator reuse
sepKey can alias the source KeyRecord.Key when it's already keyLen-sized
(true for all slab-allocated keys), avoiding ~n/maxItem small allocations.
Pre-size the children slice.
4. Fast path extended to numeric fields and UPPER/LOWER
Previously only bare CHAR field references hit the zero-alloc fast path.
Now:
- Numeric fields (N/F type) copy DBF bytes directly
(same-length ASCII compare matches numeric order for non-negatives)
- UPPER(field) / LOWER(field) wrappers on CHAR fields apply ASCII
case folding inline during byte copy
Per-index timing on the micro benchmark:
before after
NAME 7.7ms 7.5ms (fast path, unchanged)
CITY 6.0ms 6.2ms (fast path, unchanged)
AGE 14.1ms 7.1ms -50% (was slow path)
UPPER(NM) 17.0ms 7.9ms -54% (was slow path)
5. Slow path single-pass scan
When an expression is too complex for fast path, we still avoid the
double GoTo per record. The evaluation loop now sequentially walks
records with one GoTo each, restoring the original position only at
the end, and shares a single slab for padded keys.
Also fixes a hbrt bug surfaced while writing the benchmark:
6. Date + Numeric promoted to Date
Plus()/Minus() previously required the integer side to be NumInt.
Modulus returns a promoted type, so `SToD("...") + (i % 365)` panicked.
Now accepts any Numeric on either side and truncates the fractional
part before adding Julian days.
- hbrt/ops_arith.go: Date±Numeric (was Date±NumInt only)
Tests:
go test ./... — ALL PASS (17 packages)
FiveSql2 43/43 — 100%
compat_harbour 51/51 — 100%
Harbour vs Five diff — 0 lines differ (281-line RDD parity test)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NTX 3-level tree (build.go):
- Hybrid approach: bulk build for ≤2 levels, insertKeyBTree for 3+
- rebuildWithInsert: creates proper B-tree via per-key insertion
- 5000-key test: Count=5000 Found=5000 (was 5004/4868)
CDX SET INDEX TO (gengo.go):
- Strip surrounding quotes from string literal in OrderListAdd
- Was: idx.OrderListAdd("\"path\"") → file not found
- Now: idx.OrderListAdd("path") → correct
All tests:
- 14 packages ALL PASS
- 82/82 NTX stress test
- 18/18 CDX cross-read
- 50K benchmark: all counts correct
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NTX Bulk Build (build.go — ported from rddfive/ntx_engine.c):
- pageBuffer: dynamic memory buffer for all pages
- Phase 1: Build leaf pages in sequential memory (zero disk I/O)
- Phase 2: Build interior levels from cached leaf data (zero I/O)
- Separator promotion: remove last key from leaf only (not interior)
- Single bulk WriteAt for all pages at end
- INDEX ON 10K: 34ms → 5-8ms (4-6x improvement)
NTX Seek (ntx.go):
- Always descend to leaf on match (find first occurrence)
- fStop flag tracks path match, verified at leaf
APPEND Buffering (dbf.go):
- Append marks dirty without immediate disk write
- flushRecord writes record data only (no header/EOF per record)
- Close/Flush writes EOF marker + header once
Results: 14 packages ALL PASS, 82/82 stress test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major rewrite based on Harbour dbfntx1.c analysis:
NTX B-tree traversal (ntx.go):
- nextKey: rewritten to match hb_ntxTagNextKey exactly
- Advance iKey, check right child, descend via goLeftmost
- Walk up stack on page exhaustion, truncate stackLevel
- prevKey: rewritten to match hb_ntxTagPrevKey
- Check left child (only if iKey < keyCount), descend via goRightmost
- Walk up stack for BOF detection
- goRightmost: internal nodes get iKey=keyCount (rightmost child),
leaf nodes get iKey=keyCount-1 (last key) — matches Harbour
NTX B-tree build (build.go):
- CreateIndex: proper B-tree insertion (insert keys one by one)
- insertKeyBTree: search → insert at leaf → propagate splits up
- pageInsertKey: Harbour-style offset swapping (not data moving)
- pageSplit: collect all entries, split at midpoint, promote separator
- Proper offset table initialization for all pages
Unit tests: all 5 RDD packages PASS
Stress test: partial progress (Seek issues with split pages)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
