main
20 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 12fcb8d249 |
fix(dbf): Layer 6 — EOF marker max-merge + disable append batching in shared
Closes two more multi-session correctness bugs surfaced by the post-Layer-5 stress harness. Combined with Layer 5's panic-free result, three-worker concurrency now sits around 80% pass with zero Go-level crashes; higher worker counts trade reliability for throughput against the inherent single-file-multi-writer limit of the DBF format. 1. EOF marker write at Close (max-merge with disk) `Close()` writes the EOF marker `0x1A` at `header.HeaderLen + a.recCount * RecordLen`, computed from our LOCAL recCount. A peer Append between our last refresh (under the append-intent lock at Append-time) and Close-time may have bumped the disk recCount above ours. Writing EOF at our stale offset overwrites byte 0 of the peer's record — flipping the delete-flag from ' ' (RecordActive) to 0x1A. The field bytes survive, but downstream code that depends on byte 0's exact value misclassifies the record. Fix mirrors updateHeader's max-merge (Layer 3a): in shared mode, re-read the disk header right before computing EOFOffset and use max(disk.RecCount, local). Cheap (~1 stat- sized read per Close) and the eventual close-fd is already the serial bottleneck of any meaningful churn. 2. Append-batching disabled in shared mode The appendBuf optimisation accumulates several consecutive APPENDs into a single WriteAt at flushRecord time. In single- process EXCLUSIVE mode that's a clean throughput win. In shared mode, though, a peer SELECT can open the file while our slots N..N+M are buffered but still on-disk only as reserved-but-zero bytes. The peer iterates 1..recCount and ReadAts zeros at offsets [N..N+M), treating the records as garbage / empty markers. Skip the batch path when `a.shared`: each Append writes its record straight through via flushRecord on the next state change. EXCLUSIVE single-process flows are unaffected. Observed stress numbers (3 trials × 30 runs each, average): pre-Layer-1 baseline: ~60% / panics +Layer 1+2: 80% / 50% / panic +Layer 4a/4b: 75-90% / 50-80% / panic +Layer 5 (mmap-gen): ~73% / ~67% / ~33% / NO PANICS +THIS (EOF + no-batch): ~83% / ~50% / ~22% / NO PANICS The remaining flake at 5+ concurrent writers reflects the fundamental constraint of FiveSql2's DBF model: no table-level write lock, no MVCC. PostgreSQL solves this with snapshot isolation; the equivalent for FiveSql2 would need a write-ahead log or per-table writer mutex. Tracked as a post-1.0 R&D direction. For typical pgserver use — many read clients, few write clients — the current correctness is production-acceptable. The pgserver Phase 7 integration suite (3/3 in the basic psql harness + 3/3 in the auth/TLS harness) remains 6/6 green because each suite uses one connection at a time. All six release gates green: go test ./... ✓ FiveSql2 SQL:1999 43/43 ✓ Harbour compat 56/56 ✓ std.ch 17/17 ✓ FRB 7/7 ✓ pgserver integration 6/6 ✓ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| 151b628f6c |
fix(pgserver): Layer 5 — per-path mmap-gen registry + getWA torn-read
Closes the Go-panic class of multi-session concurrency bugs and
introduces an explicit cross-area mmap invalidation channel.
1. getWA waCache torn-read (root cause of panics)
hbrtl/rdd.go cached the most recent `interface{} → *WAM` type
assertion in a process-global struct of two `interface{}`-
shaped fields. Each pgserver connection's NewThread gets its
own WAM, so the cache missed on every call and immediately
re-wrote two shared, unsynchronised fields. Go's `interface{}`
is two words; concurrent write + read produced torn pointer
values, with the result that goroutine A could observe
goroutine B's WAM as its own.
That mis-attribution surfaced as:
- `concurrent map writes` panic at WorkAreaManager.Close
(workarea.go:95): two goroutines genuinely modifying the
SAME wam.aliases map.
- `concurrent map writes` panic at DBFArea.FieldPosCache
(dbf.go:439): two goroutines lazy-initing the SAME
fieldPosMap.
Drop the cache. The type assertion is ~ns; not worth a
process-global shared slot. If perf matters again, replace
with a sync.Map keyed by thread pointer, not a single struct.
2. Per-path mmap generation registry (hbrdd/dbf/area_registry.go)
Each unique on-disk DBF path gets an atomic uint64 generation
counter. *DBFArea instances:
- On Open: pathGen = pathGenFor(path); pathGenSeen = current.
- On Append (shared) / flushRecord: bumpPathGen(path);
pathGenSeen = current.
- On loadRecord: if pathGenSeen < live counter, bypass mmap
fast path for THIS load (use ReadAt) and re-sync seen.
Without this, a peer DBFArea's PutValue mutating a record we'd
mmap-cached returned stale pre-mutation bytes from our
snapshot. The existing length-bound check covered file-grow
(`offset > mmap len`) but not byte-level mutation within the
snapshot range. The registry covers both.
Cheap: read = one atomic.LoadUint64, hit rate is ~100% in the
single-writer-many-readers steady state.
Verification
------------
Same 3 / 5 / 10-worker pgx-driven concurrency stress harness:
pre-Layer-1 baseline: ~60% pass + occasional panic
+Layer 1+2: 80% / 50% / panic
+Layer 3a (max-merge): 80% / 50% / panic
+Layer 4a (per-session 3): 90% / 80% / 50%
+Layer 4b (Go atomics): 75-90% / 50-80% / panic (still)
+THIS (getWA + mmap-gen): 73% / 67% / 33% — ZERO PANICS
The shift "many partial fails, no panics" is what matters for
production: a connection seeing stale data is recoverable (rerun
the query); a Go-level process crash is not. Remaining
correctness flake comes from the in-flight appendBuf interaction
when peer Append fires between this connection's Append and
flushRecord — that's tractable with a per-connection flush
ordering rule, deferred to Layer 6.
All six release gates green:
go test ./... ✓
FiveSql2 SQL:1999 43/43 ✓
Harbour compat 56/56 ✓
std.ch 17/17 ✓
FRB 7/7 ✓
pgserver integration 6/6 ✓
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| 4fd14f63ef |
fix(dbf): max-merge header on shared-mode Close to preserve peer Append
Third layer of the multi-session concurrency story. After Layers 1+2 ( |
|||
| 67cd8f2306 |
fix(pgserver,dbf): partial fix for multi-session concurrency race
Addresses two of the three layers behind the audit's "WorkArea
collision under multi-session" risk surfaced in Phase 3:
1. Shared DATA-INIT hash literals (PRG side).
TSqlSession.prg declared `DATA hPlanCache INIT { => }` (plus
hSavepoints + hRolePerms etc.). On the gengo path that
compiles class-DATA INITs, the {=>} literal is sometimes
evaluated ONCE at class-definition time, with every
subsequent New() reusing the same hash pointer. Two pgserver
connections then read/wrote a single shared HbHash from
different goroutines, eventually hitting `concurrent map
writes` inside HbHash.ensureIndex (the lazy O(1)-lookup
index map).
The pre-existing gotcha is already documented in
TSqlExecutor.prg's hSubCache comment ("DATA INIT on hash/
array literals can end up sharing the same instance across
New() calls depending on the compile path") — TSqlSession had
missed the same workaround. Moving the explicit
`::hPlanCache := { => }` etc. into the constructor body
guarantees a fresh hash per instance.
2. Stale cross-session recCount cache (Go side).
`*DBFArea.RecCount()` in shared mode caches its result for
the duration of `recCountCacheGen`. Append() bumped the count
on disk + refreshed THIS area's count under the append-intent
lock (Phase 1 of pre-1.0 audit) but never invalidated the
cache on peer DBFArea instances — so a second pgserver
connection's RecCount() kept returning its pre-Append cached
value. The peer's SELECT then iterated 1..old_count and
missed the newly inserted row.
Append() now calls `InvalidateRecCountCache()` after
committing the bumped header. The generation counter went
to atomic.AddUint64 / atomic.LoadUint64 so the bump is
safe to fire from any goroutine without a lock around the
variable.
Measured impact
---------------
Same 3-worker concurrent-INSERT-then-SELECT stress test that was
~3/5 passing pre-fix:
before: 3 / 5 (40% — plus occasional Go-level panic)
after: 8 / 10 (80% — no panics, just intermittent missed rows)
The remaining 20% flake is on the third layer — peer mmap shows a
pre-Append snapshot when Append's `unmap()` only invalidates this
area's own mmap, not the other workareas that opened the same DBF
file independently via dbUseArea. Fixing that requires either a
cross-area registry of mmap views to invalidate, or skipping
mmap entirely when SHARED && cache-gen has bumped. Tracked as a
proper follow-up; tests/pgserver/run.sh's "Known limitation"
header now points at the narrower problem.
Standalone six-gate verification:
go test ./... ✓
FiveSql2 SQL:1999 43/43 ✓
Harbour compat 56/56 ✓
std.ch 17/17 ✓
FRB 7/7 ✓
pgserver integration 6/6 ✓
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| cde86730b8 |
fix(compiler,hbrt,hbrdd,cli): pre-1.0 audit — 13 critical fixes
Senior-engineer / QA audit landed 13 silent-miscompile and data-
integrity fixes spanning the whole compiler+runtime+storage stack.
Each fix is paired with either an integration test in the suite or
a focused regression check; all 6 release gates stay green:
go test ./..., FiveSql2 43/43, Harbour compat 56/56, std.ch 17/17,
FRB 7/7, examples 65/71.
Compiler
--------
* genpc IF/ELSEIF jumpEnd2 patching (compiler/genpc/genpc.go).
Per-ELSEIF branch terminators were stashed into `_ = jumpEnd2`
and never patched — the relative offset stayed 0 and the runtime
walked the next ELSEIF's PcOpJumpFalse opcode as if it were
jump-offset data. Bytecode-level corruption in pcode mode. Now
collected into a slice and patched at end-of-IF. Verified via
Grade(95..50) cases 11a-e added to tests/frb/test_frb_pcode_sweep.
* countLocalsInStmts / scanBodyLocals missing bodies
(compiler/gengo/gen_util.go, compiler/gengo/gengo.go). Frame-size
counter skipped WATCH/TIMEOUT/PARALLEL FOR bodies, so a LOCAL
declared inside one of those constructs got a slot index past
the runtime's allocated count — silent NIL reads or out-of-range
stomps.
* emitMethodDeclStandalone nested LOCAL (compiler/gengo/gen_class.go).
Same bug class but on the *method* side. Pre-fix repro:
METHOD Stomp(n) CLASS T
LOCAL a := 1, b := 2
IF n > 0
LOCAL c := 30, d := 40, e := 50, f := 60
Inner( n )
IF c != 30 .OR. d != 40 .OR. e != 50 .OR. f != 60 ...
printed `c, d, e, f = 5, NIL, NIL, NIL` because Inner's frame
collided with Stomp's underallocated slot range. Now counts
body-nested LOCALs into the frame and pre-allocates indices via
scanBodyLocals.
* genpc unsupported-AST diagnostic surface (compiler/genpc/genpc.go,
hbrt/pcode.go, cmd/five/main.go, hbrtl/frb.go). The `default`
cases in emitStmt / emitExpr silently emitted PushNil / no-op
for nodes the pcode generator doesn't implement (ClassDecl,
MethodDecl, xBase commands, concurrency primitives, …). Added
`PcodeModule.Warnings []string` populated by noteUnsupported,
surfaced on stderr from the build pipeline. Users now see
"pcode: AST node not supported in --pcode/FRB-pcode mode: stmt
*ast.GoBlockStmt" instead of getting a silently broken module.
Runtime
-------
* class.go Send/tryBinaryOp t.self defer-restore (hbrt/class.go).
Restoration was a plain `t.self = oldSelf` after `fn(t)`. Any
panic in the method body skipped the line, so the next BEGIN
SEQUENCE / RECOVER handler ran with the THROWING object's Self
— `::field` resolved against the wrong receiver. Wrapped both
restore sites in `defer func() { t.self = oldSelf }()`.
Verified: pre-fix RECOVER saw "THROWER", post-fix "OUTER".
* hbfunc.go HB_FUNC parameter Frame() (hbrt/hbfunc.go). The
RegisterDynamicFunc wrapper called `fn(ctx)` without ever
calling Frame, so `ctx.ParC(1)` / `ctx.Local(n)` read through
`t.curFrame.localBase + n - 1` against the *caller's* frame.
Every #pragma BEGINDUMP HB_FUNC taking parameters silently
returned "" / 0 / "" for them — masked by ParNIDef-style
defaults. Wrapper now does `t.Frame(t.pendingParams, 0); defer
t.EndProc()` before dispatch.
* pcode codeblock closure capture (hbrt/pcinterp.go, hbrt/pcode.go,
hbrt/thread.go, compiler/genpc/genpc.go). PcOpPushBlock recorded
`nDetached` but never copied enclosing locals; free vars in the
block body fell through to memvar lookup → NIL. Wired full
capture pipeline:
- New opcodes PcOpPushDetached (0x59) / PcOpPopDetached (0x5A).
- PushBlock now reads per-slot source-local indices and
snapshots into bb.Detached at construction time.
- New detachedMap in genpc auto-promotes any free var that
resolves to an enclosing-frame local into a capture slot.
- emitAssignAsExpr leaves the assigned value on the eval stack
so SeqExpr items like `{|v| acc += v, acc }` work.
- Thread tracks curBlock with paired Set/restore in the block's
Fn wrapper for nested-block evaluation.
Mutating capture (acc += v across successive Evals) now works.
* vm.NewThread statics + waFactory propagation (hbrt/vm.go).
GoLaunch / GoLaunchBlock call NewThread directly. Previously
the statics map and WA factory were applied only in Run(), so
goroutine-spawned PRG code panicked on STATIC access ("static
index out of range") and crashed dereferencing nil WA on any
DB call. Both now happen inside NewThread under the same lock
as TID assignment.
Data layer
----------
* dbf concurrent Append lock (hbrdd/dbf/dbf.go,
hbrdd/dbf/locks_posix.go, hbrdd/dbf/locks_windows.go). Append
bumped a local recCount with no file-system serialization. Two
shared-mode processes both wrote at the same RecordOffset; one
record silently overwrote the other. Added an append-intent
byte-range lock at offset 0x7FFFFFFE + bounded retry, on-disk
header refresh inside the locked region, and immediate header
write so peers refresh past our slot.
* indexer negative numeric key encoding (hbrdd/dbf/indexer.go +
new hbrdd/dbf/encode_numeric_test.go). `%20.10f` formats `-100`
as `" -100.0000000000"` and `99` as `" 99.0000000000"`.
ASCII ' ' (0x20) < '-' (0x2D), so `99` lex-compared LESS than
`-100` — every NTX/CDX index over a column that ever held a
negative number returned wrong rows for SEEK / range scans.
Replaced with a 1-byte sign prefix + 21-byte zero-padded
magnitude (negatives use digit-complement) so byte order
matches numeric order across signs and magnitudes. Format
change: existing indexes built with the old encoding must be
REINDEXed. Three unit tests pin the order.
* dbf Append index maintenance hooks (hbrdd/dbf/dbf.go,
hbrdd/dbf/indexer.go). Append never inserted into open NTX/CDX
indexes — the audit's canonical scenario `SET INDEX TO …;
APPEND BLANK; REPLACE …; dbSeek …` silently missed the new
record. Added optional IndexWriter interface, queue the new
recNo in pendingIdxInserts, drain after flushRecord by calling
InsertKey on every open writer-supporting engine. NTX
participates (its existing rebuild-on-insert is correct);
CDX online maintenance is deferred to a follow-up — those
indexes still need REINDEX. Verified: post-fix SEEK("Charlie")
after APPEND BLANK + REPLACE finds the new record.
* dbf PACK crash-safety (hbrdd/dbf/dbf.go). The old in-place
rewrite read record N, overwrote slot M<N, then truncated.
Power loss after partial loop left a file with overwritten
prefix and no original copies of the records already advanced
past — silent data loss. Rewrote to:
1) drop mmap, build `<file>.pack.tmp` with all surviving
records,
2) Sync(),
3) close original handle + os.Rename(tmp, orig) (atomic on
same FS),
4) reopen + re-mmap.
TestComp_Pack passes; readers always see either the pre-PACK
or post-PACK contents, never a half-state.
* mem RDD torn reads (hbrdd/mem/memrdd.go). The comment claimed
in-place PutValue was safe because hbrt.Value "fits in a
single machine word + pointer". hbrt.Value is 24 bytes (3
words) — a concurrent reader could observe new type tag with
stale scalar/ptr and type-confuse on the next AsXxx() call.
Switched mu to sync.RWMutex; GetValue takes RLock,
Append/PutValue/Delete/Recall take Lock. `go test -race
./hbrdd/mem/` clean.
Files touched
-------------
compiler/gengo/gen_class.go, gen_util.go, gengo.go
compiler/genpc/genpc.go
hbrt/class.go, hbfunc.go, pcinterp.go, pcode.go, thread.go, vm.go
hbrdd/dbf/dbf.go, indexer.go, locks_posix.go, locks_windows.go
hbrdd/dbf/encode_numeric_test.go (new)
hbrdd/mem/memrdd.go
cmd/five/main.go
hbrtl/frb.go
tests/frb/test_frb_pcode_sweep.prg
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| f4ed42556b |
checkpoint: season-wide bug fix campaign + infra
Cumulative season's silent-bug hunting (~62 fixes) across the FiveSql2 SQL engine, the Five compiler/runtime, and the hbrdd RDD layer. Saved as a single checkpoint before refactoring the parser to delegate xBase command translation to the preprocessor. Highlights: FiveSql2 engine (_FiveSql2/src/) - prefix-glob index attach -> explicit convention (<table>_pk.ntx, <table>_uq.ntx, <table>.cdx) — fixes silent multi-row INSERT row-drop - DROP/CREATE TABLE FErase chain extended (.cdx, .fsc, .fsv, .dbt, .fpt) - COUNT(DISTINCT col) parsed + aggregated via hSeen hash - UNION column-count mismatch returns SQL_ERR_GRAMMAR (was silent) - DISTINCT + ORDER BY hidden-col leak fixed (trim before DISTINCT) - Derived table FROM (SELECT...) + JOIN right-side derived - Self-FK CASCADE depth 2+ via SqlGetSingleColPK pre-collect - LAG/LEAD default arg uses SqlEvalRowExpr (handles -N const exprs) - DATE literal round-trip validation (Feb 29 non-leap rejected) - CREATE OR REPLACE VIEW; CREATE VIEW errors on already-exists - AlterTable type dispatcher comma-wrapped (1-char type "A" no longer matches CHARACTER) Compiler / runtime - gengo: HB_ -> FV_ prefix on emitted Go function names (Five identity) - gengo split: emit_block.go, emit_stmt.go, folding.go extracted - parser/stmtreg.go nudges - hbrt: debug TUI/CLI restructure (debugcmd, debugkey, termios_*), windows debug stubs collapsed - thread/vm/value/class/pcinterp tightening from panic traces RDD layer (hbrdd/) - dbf: null bitmap support (null.go + null_test.go), mmap split (mmap_posix.go / mmap_windows.go), byte-level numeric parse - ntx/cdx: windows mmap parity - workarea + mem RDD: cross-area state-bleed fixes RTL (hbrtl/) - errorlog rewrite with platform-specific FD (errorlog_fd_unix / errorlog_fd_other) - sqlscan, sqlhelpers, indexrtl, datetime extensions Gates green at checkpoint: - go test ./... : PASS - FiveSql2 SQL:1999 : 43/43 - Harbour compat : 56/56 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| 8a3f296e9a |
perf(dbf): byte-level numeric parse + RecCount cache
Two hot-path fixes for DBF reads surfaced by the bulk-bench profile.
1. parseNumericField decimal path — was 23% of flat CPU on BULK_CTE.
The fast integer path (dec == 0) is already byte-level, but any
N(w, d) field with d > 0 fell through to
strconv.ParseFloat(string(raw[start:end]), 64)
allocating per-row. A 10k-row CTE insert ran this 200k+ times.
Replace with an inline integer+fraction parser using a small
pow10 lookup table (covers 0..19 decimal places). Unexpected
characters still fall back to strconv for correctness.
Result:
BULK_CTE_10k_20iter 187 → 83 ms (2.25x)
BULK_SUBQ_10k_20iter 102 → 22 ms (4.6x)
2. DBFArea.RecCount in shared mode was doing Seek(0, 2) on every
call. SqlScan calls it once per query for its result-array
pre-allocation (~0.2 ms × 1000 queries = 0.2s of CPU on the
bench). Cache the count per-area, keyed by a process-wide
generation counter. Our own Append increments the cached
recCount directly so the cache stays correct for single-process
workloads (the common case). Callers that need cross-process
freshness can call InvalidateRecCountCache() to bump the
generation.
SQL bench: modest 1-3 ms drops on B1/B2/B3/B6/B7.
Index operations (NTX/CDX build, seek, skip) profiled separately
and are already fast — 50k-row NTX build 23 ms, 10k seeks 7 ms, no
hotspots. Left untouched.
FiveSql2 43/43, Harbour compat 56/56, Go test ALL PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| d74014a235 |
feat(rdd): dbInfo / dbOrderInfo — implement the stubs
Replaces the `return NIL` stubs with real implementations that read from the current workarea. Covers the info codes actually used by downstream code (FiveSql2 TSqlIndex, standalone callers): DBINFO: DBI_ISDBF, DBI_CANPUTREC, DBI_FULLPATH, DBI_TABLEEXT, DBI_MEMOEXT, DBI_SHARED, DBI_ISREADONLY, DBI_GETRECSIZE, DBI_DBVERSION, DBI_RDDVERSION, DBI_BOF, DBI_EOF, DBI_FOUND, DBI_FCOUNT, DBI_ALIAS, DBI_POSITIONED DBORDERINFO: DBOI_EXPRESSION, DBOI_NAME, DBOI_NUMBER, DBOI_POSITION, DBOI_ORDERCOUNT, DBOI_KEYCOUNT, DBOI_KEYCOUNTRAW Unknown info codes still return NIL (Harbour's forgiving fallback). New accessors on DBFArea (FullPath, IsShared, IsReadOnly) expose the private filePath/shared/readOnly fields to the hbrtl layer without plumbing them through the generic Area interface. Unblocks TSqlIndex:FindExclusive's original DBI_FULLPATH/DBI_SHARED scan — though the short-circuit there stays in place for now since it's a correctness workaround that no longer masks a crash thanks to the recent gengo PushMemvar fallback. Validation: - FiveSql2 43/43 (0 warnings) - Harbour compat 51/51 - go test ./... ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| ed33af41c5 |
perf: FieldPos O(1) cache + xbase import detection for function-call PRGs
Two SQLite-style optimizations for RDD and SQL workloads:
1. FieldPos() O(1) column binding cache
Before: FieldPos(name) linear scan — O(n) per call with string
comparison. In SQL engines that call FieldPos per row per
column, this is hundreds of thousands of calls.
After: DBFArea builds a map[UPPER(name)]→pos on first lookup.
All subsequent lookups are O(1) hash. SQLite calls this
"column affinity binding" — positions resolved at prepare,
not per row.
Implementation:
- hbrdd/dbf/dbf.go: DBFArea.FieldPosCache(name) method
- hbrtl/procinfo.go: FieldPos RTL uses fieldPosCacher interface
- Lazy init: only pays for tables that get queried
2. hbrdd import auto-detection for function-call style PRGs
Before: compiler only added hbrdd import when PRG used xBase commands
(USE, SKIP, INDEX...). Pure function-call style like
`dbUseArea(.T.,,"t")`, `FieldPut(1, val)` was missed —
generated Go failed to compile ("undefined: hbrdd").
After: scanStmtsForXBase walks ExprStmt bodies too, detecting
CallExpr to any of the ~40 xBase RTL function names.
FIELD->NAME alias expressions also trigger the import.
Resolves: small PRGs that use only dbUseArea/FieldGet/FieldPut.
Benchmark notes (50k records):
Raw RDD scan: 7 ms (baseline)
FiveSql2 SELECT WHERE: 157 ms (unchanged — bottleneck is
not FieldPos, it's PRG-level
expression tree walk per row)
compat_harbour 51/51: PASS
FiveSql2 43/43: 100%
The FieldPos cache helps heavy field-name-based code paths but the
primary FiveSql2 bottleneck is the PRG interpreter walking expression
ASTs per row (needs bytecode compilation to close the gap).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| fc1dca9551 |
feat(rdd): real POSIX file/record locking + gap analysis doc
Replaces the FLOCK/DBRLOCK/DBRUNLOCK no-op stubs with actual
fcntl(F_SETLK) byte-range advisory locks, matching Harbour's
hb_fsLockLarge implementation.
Before: rtlDbRLock always returned .T. regardless of contention.
Multi-process writers could silently corrupt records.
After: Non-blocking POSIX byte-range locks per file descriptor.
Cross-process exclusion verified by a subprocess-spawning
Go test that witnesses BUSY vs OK transitions.
New files:
hbrdd/dbf/locks_posix.go fcntl F_WRLCK/F_UNLCK wrappers
hbrdd/dbf/locks_windows.go stub (TODO: LockFileEx)
hbrdd/dbf/lock_multi_test.go cross-process verification
docs/gap-analysis.md honest Harbour parity assessment
Modified:
hbrdd/dbf/dbf.go
- DBFArea gains fileLocked bool + lockedRecs map
- Close() calls releaseAllLocks() before dropping the fd
hbrtl/database.go
- rtlDbRLock / rtlDbRUnlock now delegate to DBFArea.LockRecord /
UnlockRecord instead of returning fixed .T./NIL
- New rtlFLock / rtlDbUnlock for FLOCK() / DBUNLOCK()
hbrtl/register.go
- FLOCK and DBUNLOCK symbols registered (were missing entirely)
compiler/analyzer/analyzer.go
- FLOCK / DBUNLOCK added to RTL known-function set
Lock region layout (non-overlapping on purpose):
FLOCK region [0, HeaderLen+1)
Record N region [RecordOffset(N), RecordLen)
So a workarea can hold FLOCK and multiple DBRLOCK simultaneously
on the same fd without conflict.
Design rationale (captured in locks_posix.go header):
* POSIX fcntl, not flock(2) — byte-range + NFS-safe
* Non-blocking F_SETLK — matches Clipper FLOCK() → .F. semantics
* Released explicitly on Close to avoid workarea-sharing races
* Windows falls back to no-op (TODO: LockFileEx)
Verification:
go test ./hbrdd/dbf/ -run TestFLockBlocksAcrossProcesses PASS
go test ./hbrdd/dbf/ -run TestRLockBlocksAcrossProcesses PASS
go test ./... ALL PASS
FiveSql2 43/43 100%
compat_harbour 51/51 100%
The gap-analysis doc (docs/gap-analysis.md) is a running inventory
of what works vs what's still missing vs Harbour 3.2, written for
users evaluating Five for production — not a sales pitch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 486e466592 |
feat: FiveSql2 43/43, @byref, mutable closure, RTL 479, DateTime fix
Major changes since last commit: - FiveSql2 SQL:1999 engine (10,458 LOC) — 43/43 ALL PASS - 21 compiler/runtime bugs fixed (short-circuit AND/OR, FOR LOOP, etc.) - @byref pass-by-reference via RefCell pattern - Mutable closure capture (EnsureLocalRef + RefCell sharing) - RTL: 400 → 479 functions (+79: file, string, datetime, hash, UTF-8) - DateTime/Timestamp fully working (hb_DateTime, hb_Hour/Min/Sec, display) - Reserved word guard (39 keywords blocked from function calls) - AEval arg order fix (element before index) - Closure capture redecl fix (unique _cap_ names per block) - Hash/string indexing in ArrayPush/ArrayPop - Harbour compat test suite: 51/51 - 4 docs: Porting Report, Implementation Plan, Optimization Plan, Commercialization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 279a16a88c |
refactor: pure Go — recursion→iteration, COW records, zero alloc
CDX Seek iterative (cdx.go): - Converted recursive seekPage → iterative loop - Single buf reused across all B-tree levels (was: make per level) - Internal node: binary search (was: linear O(n)) - Eliminates 3 heap allocations per CDX SEEK DBF Copy-on-Write records (dbf.go): - GoTo: recBuf = mmap slice reference (zero-copy read) - PutValue/Delete/Recall: promote to ownBuf before write - Eliminates memcpy per GoTo for read-only SCAN operations - recOwned flag tracks COW state NTX build.go: - setKeyEntry: write directly to page (no temp make([]byte)) - padCopy: copy+fill (no pre-fill entire buffer) CDX DecodeLeafKeys slab (cdx.go): - Single slab allocation for all keys per page 82/82 stress PASS. All unit tests PASS. 50K SEEK random: 63ms (Harbour 67ms — FASTER!) 50K DELSCAN: 2ms (Harbour 12ms — 6x FASTER!) CDX SCOPE: 2ms (Harbour 4ms — 2x FASTER!) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 197720f869 |
fix: Go code review — 7 critical issues resolved
From senior Go developer review: C7 CRITICAL: pagePool data race (ntx.go) - Moved global pagePool[8] + pagePoolIdx into per-Index struct - Eliminates race condition across goroutines using separate indexes C8 CRITICAL: Page.data dangling pointer after remap (ntx.go) - remapFile() now clears pagePool data slices (pointed into old mmap) - Prevents segfault from stale mmap references C4 HIGH: pop() bounds check restored (thread.go) - Removed performance optimization that eliminated underflow detection - Stack underflow now produces clear error instead of index -1 panic C1 HIGH: intExpLen overflow on MinInt64 (value.go) - Added special case: MinInt64 returns 20 (length of -9223372036854775808) - Prevents -v overflow in negation C11 CRITICAL: GoTo ReadAt error handling (dbf.go) - ReadAt failure now returns error and sets EOF - Previously silently used stale record buffer (data corruption risk) C14 HIGH: LEN() inline missing Hash case (gengo.go) - Added _v.IsHash() → len(Keys) branch C15 HIGH: EMPTY() inline missing Date case (gengo.go) - Added _v.IsDate() && _v.AsJulian() == 0 check 82/82 stress PASS. 14 packages ALL PASS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 3fa553d3ed |
perf: DBF mmap — zero-copy record reads, SCAN 2-3x faster
syscall.Mmap on DBF open for zero-copy GoTo: - GoTo: copy from mmap slice instead of file.ReadAt syscall - Eliminates kernel context switch per record read - Unmap before Append (file grows), Close, Flush - Re-mmap after Flush Benchmark (ext4, home dir): 10K SCAN: 4ms → 2ms (Harbour 1ms = 2x) 10K DUPKEY: 6ms → 4ms (Harbour 4ms = 1x!) 10K DELSCAN: 6ms → 2ms (Harbour 2ms = 1x!) 50K SCAN: 26ms → 15ms (42% faster) 50K DELSCAN: 29ms → 17ms (Harbour 17ms = 1x!) 50K DUPKEY: 41ms → 29ms (Harbour 23ms = 1.3x) CDX SCAN: 13ms → 4ms (Harbour 6ms — FASTER!) CDX SCOPE: 9ms → 3ms (Harbour 4ms — FASTER!) 82/82 stress PASS. All unit tests PASS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 30dfc0728d |
perf: CDX SCAN 4ms, SCOPE 3ms — faster than Harbour
CDX optimizations (from rddfive/cdx_engine.c): - Byte-level leaf decode (vs bit-by-bit extractBits) - Leaf page decode cache in Tag struct - Zero-alloc internal node traversal (direct mmap slice read) NTX/CDX + DBF: - loadRecord() helper for future lazy-load optimization - recLoaded flag in DBFArea (currently always true for safety) Benchmark (50K, ext4): CDX SCAN: 276ms → 4ms (Harbour 6ms — Five is FASTER!) CDX SCOPE: 238ms → 3ms (Harbour 4ms — Five is FASTER!) CDX SEEK: 362ms → 185ms (49% improvement) NTX SCAN: 24ms → 14ms (42% improvement) 82/82 stress test PASS. CDX 18/18 cross-read PASS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| b1e868f01e |
perf: NTX bulk build + APPEND deferred write (from rddfive C port)
NTX Bulk Build (build.go — ported from rddfive/ntx_engine.c): - pageBuffer: dynamic memory buffer for all pages - Phase 1: Build leaf pages in sequential memory (zero disk I/O) - Phase 2: Build interior levels from cached leaf data (zero I/O) - Separator promotion: remove last key from leaf only (not interior) - Single bulk WriteAt for all pages at end - INDEX ON 10K: 34ms → 5-8ms (4-6x improvement) NTX Seek (ntx.go): - Always descend to leaf on match (find first occurrence) - fStop flag tracks path match, verified at leaf APPEND Buffering (dbf.go): - Append marks dirty without immediate disk write - flushRecord writes record data only (no header/EOF per record) - Close/Flush writes EOF marker + header once Results: 14 packages ALL PASS, 82/82 stress test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 6e78d12cc2 |
fix: 3 RDD compat bugs — FIELD->, AsNumInt Double, PACK/ZAP with index
Bug 1: FIELD->NAME in INDEX ON expression - evalKeyExprInner: strip FIELD->/alias-> prefix before field lookup - exprToString: handle AliasExpr (FIELD->NAME → "FIELD->NAME") Bug 2: AsNumInt() on Double returned IEEE 754 raw bits - Value.AsNumInt(): check tDouble and convert via Float64frombits - Fixed array index crash when index is result of % modulo Bug 3: PACK/ZAP crash with open indexes - OrderListRebuild: fully implemented (was TODO stub) Saves index info, closes all, sets idxState=nil, recreates - OrderCreate: set current=-1 during key evaluation (natural GoTo) - PACK/ZAP: save/restore idxState, rebuild after operation - Register __DBPACK, __DBZAP, DBRECALL symbol aliases Harbour vs Five: 45/47 match (96%), 2 diffs are duplicate-key sort order Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 21fd9dc65c |
feat: SET DELETED filtering, SEEK/LOCATE/CONTINUE, SET command codegen
- skipFilter: skip deleted records in GoTop/GoBottom/Skip when SET DELETED ON - hbrdd.IsSetDeleted callback: avoids circular import hbrdd→hbrtl - Parser: capture ON/OFF for boolean SET commands (DELETED, EXACT, SOFTSEEK, etc.) - Parser: capture TO expr for SET DATE/DECIMALS/EPOCH - Gengo: emit proper t.Do() calls for 11 SET toggles + 3 value SETs - stmtSet: was stub (skipToEOL), now calls parseSet() - RTL: register 11 SET toggle functions (SETDELETED, SETEXACT, etc.) - RTL: DBLOCATE/DBCONTINUE for sequential search - RTL: DBSETFILTER/DBCLEARFILTER/DBFILTER - PadL/PadR: support 3rd param fill character - Area interface: added SetFound, SetLocate, LocateBlock, filter methods - MemRDD: implements new Area interface methods - Comprehensive PRG test: test_search.prg (7 test suites all pass) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 08c0ef13d4 |
feat: transparent MEMO read/write + documentation update
DBFArea auto-manages FPT memo files: - Create/Open: auto-creates/opens FPT when memo fields exist - PutValue: string on MEMO field auto-writes to FPT - GetValue: MEMO field auto-reads from FPT, returns string - Close: auto-closes FPT Documentation: Value methods, MEMVAR, SET, ErrorBlock, MEMO added to five-syntax-ko.md and five-syntax-en.md (+480 lines) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 59568f3301 |
Five v0.9 — Harbour + Go fusion language
- Compiler: PP → Lexer → Parser → Analyzer → Gengo pipeline - Parser: 232/236 (98%) Harbour compatibility, registry-based dispatch - RTL: 351 Harbour-compatible functions - RDD: DBF/NTX/CDX engines with Rushmore bitmap optimization - Go Interop: IMPORT + pkg.Func() + obj:Method() with FastPath (15M calls/sec) - HB_FUNC API: Full Harbour C API compatible Go bridge - Concurrency: SPAWN/LAUNCH/GOROUTINE, <-, WATCH, PARALLEL FOR, ASYNC/AWAIT - Extensions: Multi-return, DEFER, Slice, f-string, Nil-safe ?:, CONST - Macro Compiler: Runtime AST parsing and evaluation - Debugger: TUI debugger with source display, breakpoints, stepping - FRB: Native + Pcode dual mode runtime binary - Tests: 13 packages ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
