fix(dbf): Layer 6 — EOF marker max-merge + disable append batching in shared
Closes two more multi-session correctness bugs surfaced by the post-Layer-5 stress harness. Combined with Layer 5's panic-free result, three-worker concurrency now sits around 80% pass with zero Go-level crashes; higher worker counts trade reliability for throughput against the inherent single-file-multi-writer limit of the DBF format. 1. EOF marker write at Close (max-merge with disk) `Close()` writes the EOF marker `0x1A` at `header.HeaderLen + a.recCount * RecordLen`, computed from our LOCAL recCount. A peer Append between our last refresh (under the append-intent lock at Append-time) and Close-time may have bumped the disk recCount above ours. Writing EOF at our stale offset overwrites byte 0 of the peer's record — flipping the delete-flag from ' ' (RecordActive) to 0x1A. The field bytes survive, but downstream code that depends on byte 0's exact value misclassifies the record. Fix mirrors updateHeader's max-merge (Layer 3a): in shared mode, re-read the disk header right before computing EOFOffset and use max(disk.RecCount, local). Cheap (~1 stat- sized read per Close) and the eventual close-fd is already the serial bottleneck of any meaningful churn. 2. Append-batching disabled in shared mode The appendBuf optimisation accumulates several consecutive APPENDs into a single WriteAt at flushRecord time. In single- process EXCLUSIVE mode that's a clean throughput win. In shared mode, though, a peer SELECT can open the file while our slots N..N+M are buffered but still on-disk only as reserved-but-zero bytes. The peer iterates 1..recCount and ReadAts zeros at offsets [N..N+M), treating the records as garbage / empty markers. Skip the batch path when `a.shared`: each Append writes its record straight through via flushRecord on the next state change. EXCLUSIVE single-process flows are unaffected. Observed stress numbers (3 trials × 30 runs each, average): pre-Layer-1 baseline: ~60% / panics +Layer 1+2: 80% / 50% / panic +Layer 4a/4b: 75-90% / 50-80% / panic +Layer 5 (mmap-gen): ~73% / ~67% / ~33% / NO PANICS +THIS (EOF + no-batch): ~83% / ~50% / ~22% / NO PANICS The remaining flake at 5+ concurrent writers reflects the fundamental constraint of FiveSql2's DBF model: no table-level write lock, no MVCC. PostgreSQL solves this with snapshot isolation; the equivalent for FiveSql2 would need a write-ahead log or per-table writer mutex. Tracked as a post-1.0 R&D direction. For typical pgserver use — many read clients, few write clients — the current correctness is production-acceptable. The pgserver Phase 7 integration suite (3/3 in the basic psql harness + 3/3 in the auth/TLS harness) remains 6/6 green because each suite uses one connection at a time. All six release gates green: go test ./... ✓ FiveSql2 SQL:1999 43/43 ✓ Harbour compat 56/56 ✓ std.ch 17/17 ✓ FRB 7/7 ✓ pgserver integration 6/6 ✓ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -393,6 +393,25 @@ func (a *DBFArea) Close() error {
|
||||
if a.dirty {
|
||||
a.flushRecord()
|
||||
}
|
||||
// Shared-mode EOF write — same max-merge as updateHeader. A
|
||||
// peer Append between our last refresh and this write may have
|
||||
// grown the file past our local recCount. Writing the EOF
|
||||
// marker at our stale offset (HeaderLen + localCount*RecLen)
|
||||
// would overwrite byte 0 of the peer's just-appended record
|
||||
// (its delete flag) — the record body bytes survive but byte 0
|
||||
// flips from ' ' (RecordActive) to 0x1A (EOFMarker). Re-read
|
||||
// the disk header to pick the larger of {local, disk} before
|
||||
// computing EOFOffset.
|
||||
if a.shared {
|
||||
if _, err := a.dataFile.Seek(0, 0); err == nil {
|
||||
if hdr, err := ReadHeader(a.dataFile); err == nil {
|
||||
if hdr.RecCount > a.recCount {
|
||||
a.recCount = hdr.RecCount
|
||||
a.header.RecCount = hdr.RecCount
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
a.dataFile.WriteAt([]byte{EOFMarker}, a.header.EOFOffset())
|
||||
a.updateHeader()
|
||||
// Release any held byte-range locks before closing the fd — POSIX
|
||||
@@ -784,8 +803,15 @@ func (a *DBFArea) Append() error {
|
||||
return fmt.Errorf("table is read-only")
|
||||
}
|
||||
|
||||
// Batch consecutive APPENDs: save current dirty record to appendBuf instead of writing to disk.
|
||||
if a.dirty && a.ghost {
|
||||
// Batch consecutive APPENDs: save current dirty record to
|
||||
// appendBuf instead of writing to disk. SHARED mode opts out
|
||||
// of batching — a peer SELECT that opens the file while one
|
||||
// of our slots is still buffered would iterate to that slot,
|
||||
// ReadAt zero bytes (slot is reserved on disk but unwritten),
|
||||
// and treat the record as garbage. Writing each APPEND straight
|
||||
// through trades some throughput for "self+peer-coherent on
|
||||
// every cursor move" semantics. EXCLUSIVE mode still batches.
|
||||
if a.dirty && a.ghost && !a.shared {
|
||||
// Previous was also an APPEND — accumulate in batch buffer (no syscall)
|
||||
recLen := int(a.header.RecordLen)
|
||||
if a.appendBuf == nil {
|
||||
|
||||
Reference in New Issue
Block a user