50K records benchmark on native ext4 (home directory):
- APPEND 50K: Five 140ms / Harbour 61ms (2.3x)
- INDEX 50K: Five 31ms / Harbour 6ms (5.2x)
- SEEK 50K: Five 142ms / Harbour 23ms (6.2x)
- SCAN 50K: Five 35ms / Harbour 5ms (7x)
- PACK 50K: Five 19ms / Harbour 16ms (1.2x)
All within acceptable Go vs C overhead (2-7x).
PACK nearly identical. APPEND close (2.3x).
Known issue: 3-level NTX bulk build has separator duplication
at interior→root level (count=50083 vs 50000).
Does not affect correctness for <= 2-level trees (100 records OK).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
10,000 records, 3 indexes, 12 benchmarks:
- APPEND: Five 72s vs Harbour 16ms (flush-per-record — optimization needed)
- INDEX: Five 30-36ms vs Harbour 1-2ms (per-key insert vs bulk)
- SEEK: Five 35ms vs Harbour 5ms (7x — acceptable)
- SCAN: Five 8-11ms vs Harbour 1-4ms (3-9x — acceptable)
- PACK: Five 4ms = Harbour 4ms (identical!)
B6 correctness: Five found=10000 (all), Harbour found=1 (hash collision)
All counts match: 10000 records, 8000 after SET DELETED, 8000 after PACK
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes from 77/77 thorough test:
- SOFTSEEK uses CurRecNo() (was requiring recNo>0)
- SEEK reads SET SOFTSEEK at runtime (was compile-time only)
- SkipIndexed skips deleted records when SET DELETED ON
- GoTopIndexed skips deleted at top position
- evalKeyExprInner TrimSpace on fieldName (compound key fix)
- SET INDEX TO uses exprToString (was emitExpr treating as variable)
- hasXBaseCommands scans nested blocks (BEGIN SEQUENCE, IF, FOR, etc.)
77/77 thorough test PASS. Stress test (82 items) in progress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. SOFTSEEK: use idx.CurRecNo() for positioning (was checking recNo > 0)
- SEEK with SET SOFTSEEK ON now positions at next higher key
- SEEK command reads SET SOFTSEEK at runtime (was compile-time only)
- rtlDbSeek defaults to GetSetSoftSeek() when no explicit param
2. SET DELETED ON + INDEX: SkipIndexed skips deleted records
- GoTopIndexed: skip deleted record at top position
- SkipIndexed: inner loop continues past deleted records
3. Compound key (CITY+NAME): field name TrimSpace before lookup
- evalKeyExprInner: TrimSpace on fieldName after FIELD-> strip
- Fixed "CITY " != "CITY" mismatch from + operator splitting
4. SET INDEX TO filename: treated as string, not variable
- gengo uses exprToString for SET INDEX TO (was emitExpr)
- Prevents identifier being resolved as local variable
5. hasXBaseCommands: recursive scan into nested blocks
- BEGIN SEQUENCE, IF, FOR, DO WHILE, SWITCH bodies now scanned
- Fixes missing hbrdd import for DB commands inside blocks
Thorough test: 77 items (14 sections) covering exact/partial/soft seek,
SET DELETED, duplicate keys, numeric keys, compound keys, empty/single
table, state consistency, order switching, full traversal — all identical.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Core change:
- dbf.KeyEvalFunc: global callback set by gengo before OrderCreate
- evalKeyExprInner default case: calls KeyEvalFunc for unknown functions
- Final fallback: any unresolvable expression → KeyEvalFunc → MacroEval
- valueToKeyBytes: converts MacroEval result to index key bytes
- gengo: sets dbf.KeyEvalFunc = t.MacroEval before OrderCreate, clears after
Examples that now work:
INDEX ON MyFunc(FIELD->NAME) TO idx // UDF in key expression
INDEX ON CityKey(FIELD->CITY, NAME) TO idx // multi-param UDF
INDEX ON Left(MyFunc(NAME), 15) TO idx // nested built-in + UDF
Also fixed:
- SET ORDER TO n: int→string via hbrt.NtoS (was empty string)
- CDX compound leaf decoder: proper bit-packed tag name extraction
- CDX compound recNo = direct byte offset (not page number)
All existing tests pass, NTX 47/47 + CDX 20/20 Harbour compat maintained.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CDX Integration:
- IndexEngine interface: common for NTX Index and CDX Tag
- OrderListAdd: auto-detects .cdx/.ntx extension, opens CDX tags
- decodeCompoundLeaf: proper bit-packed tag directory decoding
(was stub falling through to scanCompoundLeaves with wrong names)
- CDX Tag: added KeyLen(), KeyExpr(), ForExpr(), IsDescending(), Close()
- CDX compound recNo = direct byte offset (not page number)
ORDSCOPE:
- SetScope/ClearScope/SetScopeTop/SetScopeBottom on DBFArea
- GoTopIndexed: seeks to scopeTop, validates within scopeBottom
- GoBottomIndexed: seeks to scopeBottom boundary
- SkipIndexed: stops at scope boundaries (top and bottom)
- OrdScope RTL function registered (nScope: 0=TOP, 1=BOTTOM)
- scopeKeyFromValue: converts Value to padded key bytes
Index Order Management:
- OrderListFocus: handles numeric order ("2" → order 2)
- SET ORDER TO n: gengo emits hbrt.NtoS for int-to-string conversion
- IndexOrd/OrdCount/OrdName/OrdKey: real implementations (were stubs)
- OrderCount/CurrentOrder/OrderName/OrderKeyExpr accessors on DBFArea
- ClearScope on order switch (prevents stale scope)
Cross-read test: Harbour-created CDX → Five reads, 20/20 items match:
NAME/CITY/ID seek, ORDSCOPE count, GoTop/GoBottom all identical
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five reads DBF + NTX files created by Harbour:
- NAME index: exact/partial seek, GoTop/Bottom, Skip, SoftSeek
- CITY index: duplicate key seek with correct RecNo order
- ID index: numeric key (Str(ID,6)) seek
17/17 items match Harbour output exactly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug 1: FIELD->NAME in INDEX ON expression
- evalKeyExprInner: strip FIELD->/alias-> prefix before field lookup
- exprToString: handle AliasExpr (FIELD->NAME → "FIELD->NAME")
Bug 2: AsNumInt() on Double returned IEEE 754 raw bits
- Value.AsNumInt(): check tDouble and convert via Float64frombits
- Fixed array index crash when index is result of % modulo
Bug 3: PACK/ZAP crash with open indexes
- OrderListRebuild: fully implemented (was TODO stub)
Saves index info, closes all, sets idxState=nil, recreates
- OrderCreate: set current=-1 during key evaluation (natural GoTo)
- PACK/ZAP: save/restore idxState, rebuild after operation
- Register __DBPACK, __DBZAP, DBRECALL symbol aliases
Harbour vs Five: 45/47 match (96%), 2 diffs are duplicate-key sort order
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
47 test items comparing Harbour and Five output:
- T01-T28: 100% match (CRUD, navigation, SET DELETED)
- T29-T39: 100% match (SEEK exact/partial/softseek)
- T40-T41: Found matches, RecNo differs (duplicate key sort stability)
- T42-T43: 100% match
- T44-T47: Five crashes (PACK with open index)
Known issues found:
- FIELD->NAME syntax not supported in INDEX ON expression
- Modulo % returns Double causing array index hang (Int() workaround)
- PACK crashes when NTX index is open
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- skipFilter: skip deleted records in GoTop/GoBottom/Skip when SET DELETED ON
- hbrdd.IsSetDeleted callback: avoids circular import hbrdd→hbrtl
- Parser: capture ON/OFF for boolean SET commands (DELETED, EXACT, SOFTSEEK, etc.)
- Parser: capture TO expr for SET DATE/DECIMALS/EPOCH
- Gengo: emit proper t.Do() calls for 11 SET toggles + 3 value SETs
- stmtSet: was stub (skipToEOL), now calls parseSet()
- RTL: register 11 SET toggle functions (SETDELETED, SETEXACT, etc.)
- RTL: DBLOCATE/DBCONTINUE for sequential search
- RTL: DBSETFILTER/DBCLEARFILTER/DBFILTER
- PadL/PadR: support 3rd param fill character
- Area interface: added SetFound, SetLocate, LocateBlock, filter methods
- MemRDD: implements new Area interface methods
- Comprehensive PRG test: test_search.prg (7 test suites all pass)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- five-syntax-en/ko: Add Math comparison table (Harbour RTL vs Go math)
- go_math_compare.prg: Detailed English comments explaining each section
- Example lists updated with go_math_compare.prg
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
