From 12fcb8d249a231301980ff3358d96fb072a66ed3 Mon Sep 17 00:00:00 2001 From: CharlesKWON Date: Thu, 21 May 2026 22:03:56 +0900 Subject: [PATCH] =?UTF-8?q?fix(dbf):=20Layer=206=20=E2=80=94=20EOF=20marke?= =?UTF-8?q?r=20max-merge=20+=20disable=20append=20batching=20in=20shared?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes two more multi-session correctness bugs surfaced by the post-Layer-5 stress harness. Combined with Layer 5's panic-free result, three-worker concurrency now sits around 80% pass with zero Go-level crashes; higher worker counts trade reliability for throughput against the inherent single-file-multi-writer limit of the DBF format. 1. EOF marker write at Close (max-merge with disk) `Close()` writes the EOF marker `0x1A` at `header.HeaderLen + a.recCount * RecordLen`, computed from our LOCAL recCount. A peer Append between our last refresh (under the append-intent lock at Append-time) and Close-time may have bumped the disk recCount above ours. Writing EOF at our stale offset overwrites byte 0 of the peer's record — flipping the delete-flag from ' ' (RecordActive) to 0x1A. The field bytes survive, but downstream code that depends on byte 0's exact value misclassifies the record. Fix mirrors updateHeader's max-merge (Layer 3a): in shared mode, re-read the disk header right before computing EOFOffset and use max(disk.RecCount, local). Cheap (~1 stat- sized read per Close) and the eventual close-fd is already the serial bottleneck of any meaningful churn. 2. Append-batching disabled in shared mode The appendBuf optimisation accumulates several consecutive APPENDs into a single WriteAt at flushRecord time. In single- process EXCLUSIVE mode that's a clean throughput win. In shared mode, though, a peer SELECT can open the file while our slots N..N+M are buffered but still on-disk only as reserved-but-zero bytes. The peer iterates 1..recCount and ReadAts zeros at offsets [N..N+M), treating the records as garbage / empty markers. Skip the batch path when `a.shared`: each Append writes its record straight through via flushRecord on the next state change. EXCLUSIVE single-process flows are unaffected. Observed stress numbers (3 trials × 30 runs each, average): pre-Layer-1 baseline: ~60% / panics +Layer 1+2: 80% / 50% / panic +Layer 4a/4b: 75-90% / 50-80% / panic +Layer 5 (mmap-gen): ~73% / ~67% / ~33% / NO PANICS +THIS (EOF + no-batch): ~83% / ~50% / ~22% / NO PANICS The remaining flake at 5+ concurrent writers reflects the fundamental constraint of FiveSql2's DBF model: no table-level write lock, no MVCC. PostgreSQL solves this with snapshot isolation; the equivalent for FiveSql2 would need a write-ahead log or per-table writer mutex. Tracked as a post-1.0 R&D direction. For typical pgserver use — many read clients, few write clients — the current correctness is production-acceptable. The pgserver Phase 7 integration suite (3/3 in the basic psql harness + 3/3 in the auth/TLS harness) remains 6/6 green because each suite uses one connection at a time. All six release gates green: go test ./... ✓ FiveSql2 SQL:1999 43/43 ✓ Harbour compat 56/56 ✓ std.ch 17/17 ✓ FRB 7/7 ✓ pgserver integration 6/6 ✓ Co-Authored-By: Claude Opus 4.7 (1M context) --- hbrdd/dbf/dbf.go | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/hbrdd/dbf/dbf.go b/hbrdd/dbf/dbf.go index ef75b82..2591460 100644 --- a/hbrdd/dbf/dbf.go +++ b/hbrdd/dbf/dbf.go @@ -393,6 +393,25 @@ func (a *DBFArea) Close() error { if a.dirty { a.flushRecord() } + // Shared-mode EOF write — same max-merge as updateHeader. A + // peer Append between our last refresh and this write may have + // grown the file past our local recCount. Writing the EOF + // marker at our stale offset (HeaderLen + localCount*RecLen) + // would overwrite byte 0 of the peer's just-appended record + // (its delete flag) — the record body bytes survive but byte 0 + // flips from ' ' (RecordActive) to 0x1A (EOFMarker). Re-read + // the disk header to pick the larger of {local, disk} before + // computing EOFOffset. + if a.shared { + if _, err := a.dataFile.Seek(0, 0); err == nil { + if hdr, err := ReadHeader(a.dataFile); err == nil { + if hdr.RecCount > a.recCount { + a.recCount = hdr.RecCount + a.header.RecCount = hdr.RecCount + } + } + } + } a.dataFile.WriteAt([]byte{EOFMarker}, a.header.EOFOffset()) a.updateHeader() // Release any held byte-range locks before closing the fd — POSIX @@ -784,8 +803,15 @@ func (a *DBFArea) Append() error { return fmt.Errorf("table is read-only") } - // Batch consecutive APPENDs: save current dirty record to appendBuf instead of writing to disk. - if a.dirty && a.ghost { + // Batch consecutive APPENDs: save current dirty record to + // appendBuf instead of writing to disk. SHARED mode opts out + // of batching — a peer SELECT that opens the file while one + // of our slots is still buffered would iterate to that slot, + // ReadAt zero bytes (slot is reserved on disk but unwritten), + // and treat the record as garbage. Writing each APPEND straight + // through trades some throughput for "self+peer-coherent on + // every cursor move" semantics. EXCLUSIVE mode still batches. + if a.dirty && a.ghost && !a.shared { // Previous was also an APPEND — accumulate in batch buffer (no syscall) recLen := int(a.header.RecordLen) if a.appendBuf == nil {