fix(compiler,hbrt,hbrdd,cli): pre-1.0 audit — 13 critical fixes
Senior-engineer / QA audit landed 13 silent-miscompile and data-
integrity fixes spanning the whole compiler+runtime+storage stack.
Each fix is paired with either an integration test in the suite or
a focused regression check; all 6 release gates stay green:
go test ./..., FiveSql2 43/43, Harbour compat 56/56, std.ch 17/17,
FRB 7/7, examples 65/71.
Compiler
--------
* genpc IF/ELSEIF jumpEnd2 patching (compiler/genpc/genpc.go).
Per-ELSEIF branch terminators were stashed into `_ = jumpEnd2`
and never patched — the relative offset stayed 0 and the runtime
walked the next ELSEIF's PcOpJumpFalse opcode as if it were
jump-offset data. Bytecode-level corruption in pcode mode. Now
collected into a slice and patched at end-of-IF. Verified via
Grade(95..50) cases 11a-e added to tests/frb/test_frb_pcode_sweep.
* countLocalsInStmts / scanBodyLocals missing bodies
(compiler/gengo/gen_util.go, compiler/gengo/gengo.go). Frame-size
counter skipped WATCH/TIMEOUT/PARALLEL FOR bodies, so a LOCAL
declared inside one of those constructs got a slot index past
the runtime's allocated count — silent NIL reads or out-of-range
stomps.
* emitMethodDeclStandalone nested LOCAL (compiler/gengo/gen_class.go).
Same bug class but on the *method* side. Pre-fix repro:
METHOD Stomp(n) CLASS T
LOCAL a := 1, b := 2
IF n > 0
LOCAL c := 30, d := 40, e := 50, f := 60
Inner( n )
IF c != 30 .OR. d != 40 .OR. e != 50 .OR. f != 60 ...
printed `c, d, e, f = 5, NIL, NIL, NIL` because Inner's frame
collided with Stomp's underallocated slot range. Now counts
body-nested LOCALs into the frame and pre-allocates indices via
scanBodyLocals.
* genpc unsupported-AST diagnostic surface (compiler/genpc/genpc.go,
hbrt/pcode.go, cmd/five/main.go, hbrtl/frb.go). The `default`
cases in emitStmt / emitExpr silently emitted PushNil / no-op
for nodes the pcode generator doesn't implement (ClassDecl,
MethodDecl, xBase commands, concurrency primitives, …). Added
`PcodeModule.Warnings []string` populated by noteUnsupported,
surfaced on stderr from the build pipeline. Users now see
"pcode: AST node not supported in --pcode/FRB-pcode mode: stmt
*ast.GoBlockStmt" instead of getting a silently broken module.
Runtime
-------
* class.go Send/tryBinaryOp t.self defer-restore (hbrt/class.go).
Restoration was a plain `t.self = oldSelf` after `fn(t)`. Any
panic in the method body skipped the line, so the next BEGIN
SEQUENCE / RECOVER handler ran with the THROWING object's Self
— `::field` resolved against the wrong receiver. Wrapped both
restore sites in `defer func() { t.self = oldSelf }()`.
Verified: pre-fix RECOVER saw "THROWER", post-fix "OUTER".
* hbfunc.go HB_FUNC parameter Frame() (hbrt/hbfunc.go). The
RegisterDynamicFunc wrapper called `fn(ctx)` without ever
calling Frame, so `ctx.ParC(1)` / `ctx.Local(n)` read through
`t.curFrame.localBase + n - 1` against the *caller's* frame.
Every #pragma BEGINDUMP HB_FUNC taking parameters silently
returned "" / 0 / "" for them — masked by ParNIDef-style
defaults. Wrapper now does `t.Frame(t.pendingParams, 0); defer
t.EndProc()` before dispatch.
* pcode codeblock closure capture (hbrt/pcinterp.go, hbrt/pcode.go,
hbrt/thread.go, compiler/genpc/genpc.go). PcOpPushBlock recorded
`nDetached` but never copied enclosing locals; free vars in the
block body fell through to memvar lookup → NIL. Wired full
capture pipeline:
- New opcodes PcOpPushDetached (0x59) / PcOpPopDetached (0x5A).
- PushBlock now reads per-slot source-local indices and
snapshots into bb.Detached at construction time.
- New detachedMap in genpc auto-promotes any free var that
resolves to an enclosing-frame local into a capture slot.
- emitAssignAsExpr leaves the assigned value on the eval stack
so SeqExpr items like `{|v| acc += v, acc }` work.
- Thread tracks curBlock with paired Set/restore in the block's
Fn wrapper for nested-block evaluation.
Mutating capture (acc += v across successive Evals) now works.
* vm.NewThread statics + waFactory propagation (hbrt/vm.go).
GoLaunch / GoLaunchBlock call NewThread directly. Previously
the statics map and WA factory were applied only in Run(), so
goroutine-spawned PRG code panicked on STATIC access ("static
index out of range") and crashed dereferencing nil WA on any
DB call. Both now happen inside NewThread under the same lock
as TID assignment.
Data layer
----------
* dbf concurrent Append lock (hbrdd/dbf/dbf.go,
hbrdd/dbf/locks_posix.go, hbrdd/dbf/locks_windows.go). Append
bumped a local recCount with no file-system serialization. Two
shared-mode processes both wrote at the same RecordOffset; one
record silently overwrote the other. Added an append-intent
byte-range lock at offset 0x7FFFFFFE + bounded retry, on-disk
header refresh inside the locked region, and immediate header
write so peers refresh past our slot.
* indexer negative numeric key encoding (hbrdd/dbf/indexer.go +
new hbrdd/dbf/encode_numeric_test.go). `%20.10f` formats `-100`
as `" -100.0000000000"` and `99` as `" 99.0000000000"`.
ASCII ' ' (0x20) < '-' (0x2D), so `99` lex-compared LESS than
`-100` — every NTX/CDX index over a column that ever held a
negative number returned wrong rows for SEEK / range scans.
Replaced with a 1-byte sign prefix + 21-byte zero-padded
magnitude (negatives use digit-complement) so byte order
matches numeric order across signs and magnitudes. Format
change: existing indexes built with the old encoding must be
REINDEXed. Three unit tests pin the order.
* dbf Append index maintenance hooks (hbrdd/dbf/dbf.go,
hbrdd/dbf/indexer.go). Append never inserted into open NTX/CDX
indexes — the audit's canonical scenario `SET INDEX TO …;
APPEND BLANK; REPLACE …; dbSeek …` silently missed the new
record. Added optional IndexWriter interface, queue the new
recNo in pendingIdxInserts, drain after flushRecord by calling
InsertKey on every open writer-supporting engine. NTX
participates (its existing rebuild-on-insert is correct);
CDX online maintenance is deferred to a follow-up — those
indexes still need REINDEX. Verified: post-fix SEEK("Charlie")
after APPEND BLANK + REPLACE finds the new record.
* dbf PACK crash-safety (hbrdd/dbf/dbf.go). The old in-place
rewrite read record N, overwrote slot M<N, then truncated.
Power loss after partial loop left a file with overwritten
prefix and no original copies of the records already advanced
past — silent data loss. Rewrote to:
1) drop mmap, build `<file>.pack.tmp` with all surviving
records,
2) Sync(),
3) close original handle + os.Rename(tmp, orig) (atomic on
same FS),
4) reopen + re-mmap.
TestComp_Pack passes; readers always see either the pre-PACK
or post-PACK contents, never a half-state.
* mem RDD torn reads (hbrdd/mem/memrdd.go). The comment claimed
in-place PutValue was safe because hbrt.Value "fits in a
single machine word + pointer". hbrt.Value is 24 bytes (3
words) — a concurrent reader could observe new type tag with
stale scalar/ptr and type-confuse on the next AsXxx() call.
Switched mu to sync.RWMutex; GetValue takes RLock,
Append/PutValue/Delete/Recall take Lock. `go test -race
./hbrdd/mem/` clean.
Files touched
-------------
compiler/gengo/gen_class.go, gen_util.go, gengo.go
compiler/genpc/genpc.go
hbrt/class.go, hbfunc.go, pcinterp.go, pcode.go, thread.go, vm.go
hbrdd/dbf/dbf.go, indexer.go, locks_posix.go, locks_windows.go
hbrdd/dbf/encode_numeric_test.go (new)
hbrdd/mem/memrdd.go
cmd/five/main.go
hbrtl/frb.go
tests/frb/test_frb_pcode_sweep.prg
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -884,6 +884,13 @@ func buildFRBPcode(prgFile, outputFile string) {
|
||||
|
||||
// Phase 3: Generate pcode
|
||||
pcMod := genpc.Generate(file)
|
||||
// Surface any genpc compile-time diagnostics (unsupported AST
|
||||
// nodes etc.) so users learn their PRG isn't fully pcode-
|
||||
// compilable instead of getting silent wrong results at run
|
||||
// time. These are warnings, not errors — emission continues.
|
||||
for _, w := range pcMod.Warnings {
|
||||
fmt.Fprintln(os.Stderr, w)
|
||||
}
|
||||
|
||||
// Phase 4: Serialize
|
||||
pcData := hbrt.SerializePcodeModule(pcMod)
|
||||
|
||||
@@ -156,6 +156,15 @@ func (g *Generator) emitMethodDeclStandalone(md *ast.MethodDecl) {
|
||||
nLocals += len(vd.Vars)
|
||||
}
|
||||
}
|
||||
// Mid-method and nested LOCAL declarations (inside IF / FOR /
|
||||
// WHILE / DO CASE / SEQUENCE / WATCH / TIMEOUT / PARALLEL FOR)
|
||||
// must also be counted into the runtime frame size. The
|
||||
// FuncDecl emitter already walks the body via
|
||||
// countLocalsInStmts; methods used to short-circuit this and
|
||||
// only count top-level Decls, so `METHOD Foo(): … IF cond …
|
||||
// LOCAL x …` underallocated the frame and `x` either read NIL
|
||||
// or stomped a sibling local.
|
||||
nLocals += countLocalsInStmts(md.Body)
|
||||
|
||||
g.writeln(fmt.Sprintf("func %s(t *hbrt.Thread) {", goFuncName))
|
||||
g.indent++
|
||||
@@ -182,6 +191,9 @@ func (g *Generator) emitMethodDeclStandalone(md *ast.MethodDecl) {
|
||||
}
|
||||
}
|
||||
}
|
||||
// Pre-allocate slots for body-nested LOCALs so emitStmt's
|
||||
// mid-function VarDecl handler stores into the right index.
|
||||
scanBodyLocals(md.Body, localMap, &idx)
|
||||
|
||||
g.curLocals = localMap
|
||||
// Bind defining class for ::super: resolution in emitSendExpr.
|
||||
|
||||
@@ -105,6 +105,19 @@ func countLocalsInStmts(stmts []ast.Stmt) int {
|
||||
n += countLocalsInStmts(c.Body)
|
||||
}
|
||||
n += countLocalsInStmts(v.Otherwise)
|
||||
case *ast.WatchStmt:
|
||||
// `WATCH ... CASE ... CASE ... OTHERWISE ... END WATCH`
|
||||
// — each CASE body and OTHERWISE may declare LOCALs.
|
||||
for _, c := range v.Cases {
|
||||
n += countLocalsInStmts(c.Body)
|
||||
}
|
||||
n += countLocalsInStmts(v.Otherwise)
|
||||
case *ast.TimeoutStmt:
|
||||
// `WITH TIMEOUT n SECONDS ... END` — body may declare LOCALs.
|
||||
n += countLocalsInStmts(v.Body)
|
||||
case *ast.ParallelForStmt:
|
||||
// `PARALLEL FOR i := ... NEXT` — body may declare LOCALs.
|
||||
n += countLocalsInStmts(v.Body)
|
||||
}
|
||||
}
|
||||
return n
|
||||
|
||||
@@ -726,6 +726,18 @@ func scanBodyLocals(stmts []ast.Stmt, m localMap, idx *int) {
|
||||
case *ast.SeqStmt:
|
||||
scanBodyLocals(st.Body, m, idx)
|
||||
scanBodyLocals(st.RecoverBody, m, idx)
|
||||
case *ast.WatchStmt:
|
||||
// Each WATCH CASE / OTHERWISE body may host LOCALs that
|
||||
// need pre-assigned slot indices to match the frame size
|
||||
// counted by countLocalsInStmts.
|
||||
for _, c := range st.Cases {
|
||||
scanBodyLocals(c.Body, m, idx)
|
||||
}
|
||||
scanBodyLocals(st.Otherwise, m, idx)
|
||||
case *ast.TimeoutStmt:
|
||||
scanBodyLocals(st.Body, m, idx)
|
||||
case *ast.ParallelForStmt:
|
||||
scanBodyLocals(st.Body, m, idx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,7 +11,9 @@ import (
|
||||
"five/compiler/ast"
|
||||
"five/compiler/token"
|
||||
"five/hbrt"
|
||||
"fmt"
|
||||
"math"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
@@ -29,9 +31,16 @@ func Generate(file *ast.File) *hbrt.PcodeModule {
|
||||
switch decl := d.(type) {
|
||||
case *ast.FuncDecl:
|
||||
g.emitFunc(decl)
|
||||
default:
|
||||
// ClassDecl, MethodDecl, top-level VarDecl, etc. are not
|
||||
// expressible in pcode form today — record the kind so the
|
||||
// caller can surface a clear "rebuild without --pcode"
|
||||
// diagnostic instead of silently dropping the declaration.
|
||||
g.noteUnsupported(fmt.Sprintf("%T", decl))
|
||||
}
|
||||
}
|
||||
|
||||
g.mod.Warnings = g.Warnings()
|
||||
return g.mod
|
||||
}
|
||||
|
||||
@@ -64,6 +73,89 @@ type generator struct {
|
||||
mod *hbrt.PcodeModule
|
||||
code []byte
|
||||
locals map[string]int
|
||||
// detached, when non-nil, intercepts IdentExpr/AssignExpr lookups
|
||||
// inside a block body. Names found in the *enclosing* locals are
|
||||
// auto-promoted to capture slots in declaration order so the body
|
||||
// emits PcOpPushDetached / PcOpPopDetached against the slot index
|
||||
// rather than falling through to the runtime memvar table.
|
||||
detached *detachedMap
|
||||
// Unsupported AST node kinds encountered during emit, recorded
|
||||
// once per kind. Exposed via Module().Warnings so the build
|
||||
// pipeline can surface a clear "node X not supported in pcode
|
||||
// mode" diagnostic instead of silently emitting PushNil/no-op
|
||||
// (the previous behavior, which masked bugs as wrong results).
|
||||
unsupported map[string]bool
|
||||
}
|
||||
|
||||
// noteUnsupported records that an AST node kind was hit by the
|
||||
// silent-fallback path. Caller emits PushNil/Pop to keep the stack
|
||||
// shape valid; the diagnostic itself is collected and reported once
|
||||
// per kind at module-level after Generate completes.
|
||||
func (g *generator) noteUnsupported(kind string) {
|
||||
if g.unsupported == nil {
|
||||
g.unsupported = map[string]bool{}
|
||||
}
|
||||
g.unsupported[kind] = true
|
||||
}
|
||||
|
||||
// Warnings returns the accumulated unsupported-node diagnostics in
|
||||
// stable (sorted) order so build output is deterministic.
|
||||
func (g *generator) Warnings() []string {
|
||||
if len(g.unsupported) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(g.unsupported))
|
||||
for k := range g.unsupported {
|
||||
out = append(out, "pcode: AST node not supported in --pcode/FRB-pcode mode: "+k+
|
||||
" (emitted as no-op; rebuild without --pcode to keep this construct)")
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// detachedMap accumulates the closure captures requested by a block
|
||||
// body in encounter order. The enclosing scope's locals map is read
|
||||
// to translate a free-variable name into the source-local index that
|
||||
// the PushBlock op must snapshot.
|
||||
type detachedMap struct {
|
||||
enclosing map[string]int // outer scope's local name -> 1-based local index
|
||||
slot map[string]int // captured name -> 0-based detached slot
|
||||
srcOrder []int // 0-based slot -> source local index (1-based)
|
||||
}
|
||||
|
||||
func newDetachedMap(enclosing map[string]int) *detachedMap {
|
||||
return &detachedMap{
|
||||
enclosing: enclosing,
|
||||
slot: map[string]int{},
|
||||
}
|
||||
}
|
||||
|
||||
// resolve returns (slot, true) if `name` resolves through the
|
||||
// enclosing scope and reserves a capture slot for it on first use.
|
||||
// Returns (0, false) for names not in the enclosing scope — caller
|
||||
// falls back to the memvar lookup or another resolution path.
|
||||
func (d *detachedMap) resolve(name string) (int, bool) {
|
||||
if d == nil {
|
||||
return 0, false
|
||||
}
|
||||
if s, ok := d.slot[name]; ok {
|
||||
return s, true
|
||||
}
|
||||
src, ok := d.enclosing[name]
|
||||
if !ok {
|
||||
return 0, false
|
||||
}
|
||||
s := len(d.srcOrder)
|
||||
d.slot[name] = s
|
||||
d.srcOrder = append(d.srcOrder, src)
|
||||
return s, true
|
||||
}
|
||||
|
||||
func (d *detachedMap) sources() []int {
|
||||
if d == nil {
|
||||
return nil
|
||||
}
|
||||
return d.srcOrder
|
||||
}
|
||||
|
||||
func (g *generator) emit(b ...byte) {
|
||||
@@ -239,7 +331,10 @@ func (g *generator) emitStmt(stmt ast.Stmt) {
|
||||
}
|
||||
|
||||
default:
|
||||
// Unsupported statement — skip
|
||||
// Unsupported statement — record once per kind so the build
|
||||
// pipeline can surface a clear "AST node not supported in
|
||||
// pcode mode" warning instead of silently dropping the stmt.
|
||||
g.noteUnsupported(fmt.Sprintf("stmt %T", stmt))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -252,7 +347,14 @@ func (g *generator) emitIf(s *ast.IfStmt) {
|
||||
}
|
||||
|
||||
if len(s.ElseIfs) > 0 || len(s.ElseBody) > 0 {
|
||||
jumpEnd := g.emitJumpPlaceholder(hbrt.PcOpJump)
|
||||
// `jumpEnds` collects every "branch-taken → skip rest of IF"
|
||||
// jump that has to be patched once the entire IF chain ends.
|
||||
// Original code only stashed each ELSEIF's terminator in `_ =
|
||||
// jumpEnd2` and never patched it, so the offset stayed 0 and
|
||||
// the runtime kept walking into the next ELSEIF's
|
||||
// PcOpJumpFalse opcode as if it were data — silent bytecode
|
||||
// corruption in pcode mode.
|
||||
jumpEnds := []int{g.emitJumpPlaceholder(hbrt.PcOpJump)}
|
||||
g.patchJump(jumpFalse)
|
||||
|
||||
for _, elif := range s.ElseIfs {
|
||||
@@ -261,15 +363,16 @@ func (g *generator) emitIf(s *ast.IfStmt) {
|
||||
for _, stmt := range elif.Body {
|
||||
g.emitStmt(stmt)
|
||||
}
|
||||
jumpEnd2 := g.emitJumpPlaceholder(hbrt.PcOpJump)
|
||||
jumpEnds = append(jumpEnds, g.emitJumpPlaceholder(hbrt.PcOpJump))
|
||||
g.patchJump(nextJump)
|
||||
_ = jumpEnd2 // will be patched by end
|
||||
}
|
||||
|
||||
for _, stmt := range s.ElseBody {
|
||||
g.emitStmt(stmt)
|
||||
}
|
||||
g.patchJump(jumpEnd)
|
||||
for _, j := range jumpEnds {
|
||||
g.patchJump(j)
|
||||
}
|
||||
} else {
|
||||
g.patchJump(jumpFalse)
|
||||
}
|
||||
@@ -409,6 +512,12 @@ func (g *generator) emitExpr(expr ast.Expr) {
|
||||
if idx, ok := g.locals[upper]; ok {
|
||||
g.emit(hbrt.PcOpPushLocal)
|
||||
g.emitU16(uint16(idx))
|
||||
} else if slot, ok := g.detached.resolve(upper); ok {
|
||||
// Free variable that resolves to an enclosing-frame
|
||||
// local — promote to a closure capture slot and read it
|
||||
// from this block's Detached at runtime.
|
||||
g.emit(hbrt.PcOpPushDetached)
|
||||
g.emitU16(uint16(slot))
|
||||
} else {
|
||||
// Unknown at compile time → runtime memvar lookup. This
|
||||
// makes `&(expr)` and the debugger's `p` see PRIVATEs
|
||||
@@ -471,33 +580,43 @@ func (g *generator) emitExpr(expr ast.Expr) {
|
||||
|
||||
case *ast.BlockExpr:
|
||||
// `{|p| body }` — compile body to its own pcode buffer with
|
||||
// the block's params occupying locals 1..len(Params), then
|
||||
// emit PcOpPushBlock + length + body bytes + nDetached (zero
|
||||
// — closure capture isn't wired up in pcode mode yet, so
|
||||
// blocks see their declared params and any module-local
|
||||
// symbol but no caller locals).
|
||||
// Without this case, BlockExpr fell through to the generic
|
||||
// PushNil and Eval(NIL, ...) returned NIL — silently
|
||||
// breaking every higher-order function (Eval / AEval /
|
||||
// SqlScan predicate compile / etc.) inside a pcode body.
|
||||
// the block's params occupying locals 1..len(Params). Free
|
||||
// variables in the body that resolve to an enclosing-frame
|
||||
// local are routed through Detached[i]: PcOpPushDetached /
|
||||
// PcOpPopDetached. The block creator (PcOpPushBlock) records
|
||||
// each captured slot's source-local index so the interpreter
|
||||
// snapshots the enclosing value into Detached[i] at block
|
||||
// construction time.
|
||||
//
|
||||
// Without this, every closure that referenced a caller local
|
||||
// fell through to the runtime memvar lookup and silently
|
||||
// returned NIL — silently breaking AEval/Eval/SqlScan
|
||||
// predicates in --pcode / FRB-pcode mode.
|
||||
savedCode := g.code
|
||||
savedLocals := g.locals
|
||||
savedDet := g.detached
|
||||
g.code = nil
|
||||
g.locals = make(map[string]int, len(e.Params))
|
||||
g.detached = newDetachedMap(savedLocals) // capture-on-demand
|
||||
for i, p := range e.Params {
|
||||
g.locals[strings.ToUpper(p)] = i + 1
|
||||
}
|
||||
g.emitExpr(e.Body)
|
||||
g.emit(hbrt.PcOpRetValue)
|
||||
body := g.code
|
||||
captureIdx := g.detached.sources() // src indices in capture order
|
||||
g.code = savedCode
|
||||
g.locals = savedLocals
|
||||
g.detached = savedDet
|
||||
|
||||
g.emit(hbrt.PcOpPushBlock)
|
||||
g.emitI32(int32(len(body)))
|
||||
g.code = append(g.code, body...)
|
||||
g.emitU16(uint16(len(e.Params))) // nParams
|
||||
g.emitU16(0) // nDetached — no closure capture yet
|
||||
g.emitU16(uint16(len(e.Params))) // nParams
|
||||
g.emitU16(uint16(len(captureIdx))) // nDetached
|
||||
for _, srcIdx := range captureIdx {
|
||||
g.emitU16(uint16(srcIdx))
|
||||
}
|
||||
|
||||
case *ast.SeqExpr:
|
||||
// Comma-separated expression list inside a code block:
|
||||
@@ -580,11 +699,91 @@ func (g *generator) emitExpr(expr ast.Expr) {
|
||||
}
|
||||
g.emit(hbrt.PcOpPushNil)
|
||||
|
||||
case *ast.AssignExpr:
|
||||
// Assignment as an expression — perform the store and leave
|
||||
// the assigned value on the eval stack so a containing
|
||||
// expression (e.g. SeqExpr inside `{|| acc += v, acc }`) can
|
||||
// consume it. emitAssign by itself is statement-shaped and
|
||||
// pops the value; we route through it then push the final
|
||||
// value with a load matching the destination.
|
||||
g.emitAssignAsExpr(e)
|
||||
|
||||
default:
|
||||
g.emit(hbrt.PcOpPushNil) // fallback
|
||||
// Record the unsupported kind and emit PushNil so the stack
|
||||
// shape stays valid — callers can keep compiling but the
|
||||
// build pipeline raises a clear pcode-mode-incompat warning.
|
||||
g.noteUnsupported(fmt.Sprintf("expr %T", expr))
|
||||
g.emit(hbrt.PcOpPushNil)
|
||||
}
|
||||
}
|
||||
|
||||
// emitAssignAsExpr emits an assignment whose value remains on the
|
||||
// eval stack (expression context). Mirrors emitAssign's storage
|
||||
// paths but appends a value-producing load so callers — typically
|
||||
// SeqExpr items inside a code block body — can chain.
|
||||
func (g *generator) emitAssignAsExpr(a *ast.AssignExpr) {
|
||||
// Local / detached compound op.
|
||||
if a.Op != token.ASSIGN {
|
||||
if op, ok := compoundBinOp(a.Op); ok {
|
||||
if ident, isIdent := a.Left.(*ast.IdentExpr); isIdent {
|
||||
up := strings.ToUpper(ident.Name)
|
||||
if idx, found := g.locals[up]; found {
|
||||
g.emit(hbrt.PcOpPushLocal)
|
||||
g.emitU16(uint16(idx))
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(op)
|
||||
g.emit(hbrt.PcOpDup) // keep value as expression result
|
||||
g.emit(hbrt.PcOpPopLocal)
|
||||
g.emitU16(uint16(idx))
|
||||
return
|
||||
}
|
||||
if slot, ok := g.detached.resolve(up); ok {
|
||||
g.emit(hbrt.PcOpPushDetached)
|
||||
g.emitU16(uint16(slot))
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(op)
|
||||
g.emit(hbrt.PcOpDup)
|
||||
g.emit(hbrt.PcOpPopDetached)
|
||||
g.emitU16(uint16(slot))
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Plain assignment.
|
||||
if ident, ok := a.Left.(*ast.IdentExpr); ok {
|
||||
up := strings.ToUpper(ident.Name)
|
||||
if idx, found := g.locals[up]; found {
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(hbrt.PcOpDup)
|
||||
g.emit(hbrt.PcOpPopLocal)
|
||||
g.emitU16(uint16(idx))
|
||||
return
|
||||
}
|
||||
if slot, ok := g.detached.resolve(up); ok {
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(hbrt.PcOpDup)
|
||||
g.emit(hbrt.PcOpPopDetached)
|
||||
g.emitU16(uint16(slot))
|
||||
return
|
||||
}
|
||||
}
|
||||
// Self field setter — :=. PcOpSetSelfField consumes the value
|
||||
// and pushes nothing; re-emit Right after to leave the value.
|
||||
if send, ok := a.Left.(*ast.SendExpr); ok {
|
||||
if _, isSelf := send.Object.(*ast.SelfExpr); isSelf {
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(hbrt.PcOpDup)
|
||||
g.emitString(hbrt.PcOpSetSelfField, strings.ToUpper(send.Method))
|
||||
return
|
||||
}
|
||||
}
|
||||
// Fallback: evaluate Right and leave it as the expression value
|
||||
// (no destination wired). Mirrors the statement-form fallback
|
||||
// minus the trailing Pop.
|
||||
g.emitExpr(a.Right)
|
||||
}
|
||||
|
||||
func (g *generator) emitBinaryOp(op token.Kind) {
|
||||
switch op {
|
||||
case token.PLUS:
|
||||
@@ -701,7 +900,8 @@ func (g *generator) emitAssign(a *ast.AssignExpr) {
|
||||
op, ok := compoundBinOp(a.Op)
|
||||
if ok {
|
||||
if ident, isIdent := a.Left.(*ast.IdentExpr); isIdent {
|
||||
if idx, found := g.locals[strings.ToUpper(ident.Name)]; found {
|
||||
up := strings.ToUpper(ident.Name)
|
||||
if idx, found := g.locals[up]; found {
|
||||
g.emit(hbrt.PcOpPushLocal)
|
||||
g.emitU16(uint16(idx))
|
||||
g.emitExpr(a.Right)
|
||||
@@ -710,16 +910,35 @@ func (g *generator) emitAssign(a *ast.AssignExpr) {
|
||||
g.emitU16(uint16(idx))
|
||||
return
|
||||
}
|
||||
if slot, ok := g.detached.resolve(up); ok {
|
||||
// Compound on a captured outer local — read/
|
||||
// write through Detached so the closure mutates
|
||||
// the captured snapshot.
|
||||
g.emit(hbrt.PcOpPushDetached)
|
||||
g.emitU16(uint16(slot))
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(op)
|
||||
g.emit(hbrt.PcOpPopDetached)
|
||||
g.emitU16(uint16(slot))
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if ident, ok := a.Left.(*ast.IdentExpr); ok {
|
||||
if idx, found := g.locals[strings.ToUpper(ident.Name)]; found {
|
||||
up := strings.ToUpper(ident.Name)
|
||||
if idx, found := g.locals[up]; found {
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(hbrt.PcOpPopLocal)
|
||||
g.emitU16(uint16(idx))
|
||||
return
|
||||
}
|
||||
if slot, ok := g.detached.resolve(up); ok {
|
||||
g.emitExpr(a.Right)
|
||||
g.emit(hbrt.PcOpPopDetached)
|
||||
g.emitU16(uint16(slot))
|
||||
return
|
||||
}
|
||||
}
|
||||
// Self field assignment
|
||||
if send, ok := a.Left.(*ast.SendExpr); ok {
|
||||
|
||||
175
hbrdd/dbf/dbf.go
175
hbrdd/dbf/dbf.go
@@ -58,6 +58,15 @@ type DBFArea struct {
|
||||
appendBuf []byte // buffered appended records (not yet written to disk)
|
||||
appendStart uint32 // first recNo in appendBuf (1-based)
|
||||
|
||||
// Pending index inserts for records that were appended but not
|
||||
// yet indexed. flushRecord walks this set after the body bytes
|
||||
// reach disk and calls InsertKey on every open index that
|
||||
// implements IndexWriter — so a follow-up dbSeek finds the new
|
||||
// record without a manual REINDEX. The set is populated by
|
||||
// Append() and drained by flushRecord(); typed as a slice (not
|
||||
// map) because append order matches the desired index walk.
|
||||
pendingIdxInserts []uint32
|
||||
|
||||
// mmap for zero-copy record reads
|
||||
mmapData []byte
|
||||
|
||||
@@ -761,10 +770,46 @@ func (a *DBFArea) Append() error {
|
||||
a.unmapDBF()
|
||||
}
|
||||
|
||||
// Shared mode: serialize concurrent appends across processes and
|
||||
// re-read the header so we increment past whatever any peer has
|
||||
// already committed. Without this, two processes racing
|
||||
// `dbAppend` both bumped their *local* recCount from the stale
|
||||
// value they cached at Open time and wrote to the same byte
|
||||
// range — one record silently overwrote the other.
|
||||
//
|
||||
// We also immediately write the updated header to disk *inside*
|
||||
// the locked region so the next peer to acquire the lock sees
|
||||
// our reservation. The record body itself is still flushed
|
||||
// lazily by flushRecord; subsequent peers will pick up the
|
||||
// reserved slot via the bumped RecCount before our record body
|
||||
// reaches disk.
|
||||
if a.shared {
|
||||
if err := a.lockAppendIntent(); err != nil {
|
||||
return err
|
||||
}
|
||||
defer a.unlockAppendIntent()
|
||||
if _, err := a.dataFile.Seek(0, 0); err == nil {
|
||||
if hdr, err := ReadHeader(a.dataFile); err == nil {
|
||||
if hdr.RecCount > a.recCount {
|
||||
a.recCount = hdr.RecCount
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
a.recCount++
|
||||
a.recNo = a.recCount
|
||||
a.header.RecCount = a.recCount
|
||||
|
||||
if a.shared {
|
||||
// Persist the bumped RecCount immediately so a concurrent
|
||||
// appender refreshes past our slot in its own locked region.
|
||||
a.header.UpdateDate()
|
||||
if _, err := a.dataFile.Seek(0, 0); err == nil {
|
||||
_ = WriteHeader(a.dataFile, &a.header)
|
||||
}
|
||||
}
|
||||
|
||||
// Promote to owned buffer for writing
|
||||
a.recBuf = a.ownBuf
|
||||
a.recOwned = true
|
||||
@@ -777,6 +822,15 @@ func (a *DBFArea) Append() error {
|
||||
a.dirty = true
|
||||
a.ghost = true
|
||||
a.recLoaded = true
|
||||
|
||||
// Queue the new record for index maintenance. flushRecord drains
|
||||
// the queue after the body bytes are durable — by that point the
|
||||
// user has had a chance to PutValue into the appended record
|
||||
// (the common APPEND BLANK → REPLACE → dbSeek pattern), and the
|
||||
// key expression resolves against the final field values.
|
||||
if a.idxState != nil && len(a.idxState.indexes) > 0 {
|
||||
a.pendingIdxInserts = append(a.pendingIdxInserts, a.recNo)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -812,6 +866,13 @@ func (a *DBFArea) Recall() error {
|
||||
|
||||
// Pack removes all deleted records.
|
||||
// Harbour: hb_dbfPack — requires exclusive access.
|
||||
//
|
||||
// Crash-safe: writes the surviving records into `<file>.pack.tmp`,
|
||||
// fsyncs, then atomic-renames over the original. Power loss /
|
||||
// kill -9 between Reads and Writes of the old in-place rewrite
|
||||
// could leave the file with overwritten prefix and a tail that
|
||||
// looked legitimate but contained no original copies of the
|
||||
// records we'd already advanced past — silent data loss.
|
||||
func (a *DBFArea) Pack() error {
|
||||
if a.readOnly {
|
||||
return fmt.Errorf("table is read-only")
|
||||
@@ -831,18 +892,52 @@ func (a *DBFArea) Pack() error {
|
||||
a.idxState = nil
|
||||
}
|
||||
|
||||
// Drop mmap so we can do clean reads + so the source file can
|
||||
// be replaced atomically below.
|
||||
if a.mmapData != nil {
|
||||
a.unmapDBF()
|
||||
}
|
||||
|
||||
origPath := a.dataFile.Name()
|
||||
tmpPath := origPath + ".pack.tmp"
|
||||
|
||||
// Best-effort cleanup of any leftover tmp from a prior crash.
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
tmpFile, err := os.OpenFile(tmpPath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("PACK: create temp: %w", err)
|
||||
}
|
||||
|
||||
// Write the same header to the temp; RecCount is patched at the
|
||||
// end once we know the survivor count.
|
||||
if _, err := tmpFile.Seek(0, 0); err == nil {
|
||||
_ = WriteHeader(tmpFile, &a.header)
|
||||
}
|
||||
// Pad up to HeaderLen with the original header bytes (field
|
||||
// descriptors etc.) so the new file has a structurally
|
||||
// identical envelope. We replay from the source.
|
||||
hdrBytes := make([]byte, a.header.HeaderLen)
|
||||
if _, err := a.dataFile.ReadAt(hdrBytes, 0); err == nil {
|
||||
_, _ = tmpFile.WriteAt(hdrBytes, 0)
|
||||
}
|
||||
|
||||
outRec := uint32(0)
|
||||
buf := make([]byte, a.header.RecordLen)
|
||||
|
||||
for recNo := uint32(1); recNo <= a.recCount; recNo++ {
|
||||
offset := a.header.RecordOffset(recNo)
|
||||
if _, err := a.dataFile.ReadAt(buf, offset); err != nil {
|
||||
tmpFile.Close()
|
||||
os.Remove(tmpPath)
|
||||
return err
|
||||
}
|
||||
if buf[0] != RecordDeleted {
|
||||
outRec++
|
||||
outOffset := a.header.RecordOffset(outRec)
|
||||
if _, err := a.dataFile.WriteAt(buf, outOffset); err != nil {
|
||||
if _, err := tmpFile.WriteAt(buf, outOffset); err != nil {
|
||||
tmpFile.Close()
|
||||
os.Remove(tmpPath)
|
||||
return err
|
||||
}
|
||||
}
|
||||
@@ -851,15 +946,49 @@ func (a *DBFArea) Pack() error {
|
||||
a.recCount = outRec
|
||||
a.header.RecCount = outRec
|
||||
|
||||
// Truncate file
|
||||
newSize := a.header.EOFOffset() + 1 // +1 for EOF marker
|
||||
a.dataFile.Truncate(newSize)
|
||||
// Write EOF marker on the temp.
|
||||
eofOff := int64(a.header.HeaderLen) + int64(outRec)*int64(a.header.RecordLen)
|
||||
if _, err := tmpFile.WriteAt([]byte{EOFMarker}, eofOff); err != nil {
|
||||
tmpFile.Close()
|
||||
os.Remove(tmpPath)
|
||||
return err
|
||||
}
|
||||
|
||||
// Write EOF
|
||||
a.dataFile.WriteAt([]byte{EOFMarker}, a.header.EOFOffset())
|
||||
// Patch the survivor count + date into the temp header.
|
||||
a.header.UpdateDate()
|
||||
if _, err := tmpFile.Seek(0, 0); err == nil {
|
||||
_ = WriteHeader(tmpFile, &a.header)
|
||||
}
|
||||
|
||||
// Update header
|
||||
a.updateHeader()
|
||||
// Make the survivor durable before the rename so a crash between
|
||||
// rename and process exit doesn't leave a truncated temp.
|
||||
if err := tmpFile.Sync(); err != nil {
|
||||
tmpFile.Close()
|
||||
os.Remove(tmpPath)
|
||||
return err
|
||||
}
|
||||
tmpFile.Close()
|
||||
|
||||
// Close the current handle, then atomic-rename. POSIX rename
|
||||
// guarantees readers/writers observe either the old or the
|
||||
// new contents — never an in-progress half-state.
|
||||
a.dataFile.Close()
|
||||
if err := os.Rename(tmpPath, origPath); err != nil {
|
||||
os.Remove(tmpPath)
|
||||
return fmt.Errorf("PACK: rename temp: %w", err)
|
||||
}
|
||||
|
||||
// Reopen the (now repacked) file in the same mode and re-mmap.
|
||||
flag := os.O_RDWR
|
||||
if a.readOnly {
|
||||
flag = os.O_RDONLY
|
||||
}
|
||||
newF, err := os.OpenFile(origPath, flag, 0644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("PACK: reopen: %w", err)
|
||||
}
|
||||
a.dataFile = newF
|
||||
a.mmapDBF()
|
||||
|
||||
// Reposition (natural order, no index yet)
|
||||
if a.recCount > 0 {
|
||||
@@ -930,6 +1059,7 @@ func (a *DBFArea) flushRecord() error {
|
||||
a.appendStart = 0
|
||||
if err == nil {
|
||||
a.dirty = false
|
||||
a.drainPendingIndexInserts()
|
||||
}
|
||||
return err
|
||||
}
|
||||
@@ -937,10 +1067,39 @@ func (a *DBFArea) flushRecord() error {
|
||||
_, err := a.dataFile.WriteAt(a.recBuf, offset)
|
||||
if err == nil {
|
||||
a.dirty = false
|
||||
a.drainPendingIndexInserts()
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// drainPendingIndexInserts pushes any queued APPENDed records into
|
||||
// every open IndexWriter so a follow-up dbSeek finds them. Engines
|
||||
// that don't implement IndexWriter (CDX today) are skipped — those
|
||||
// indexes remain stale and must be REINDEXed manually. Errors from
|
||||
// InsertKey are swallowed because a partial index update is the
|
||||
// strict-best worst case; reporting them up the flush stack would
|
||||
// abort the user's transaction over a recoverable index issue.
|
||||
func (a *DBFArea) drainPendingIndexInserts() {
|
||||
if len(a.pendingIdxInserts) == 0 || a.idxState == nil {
|
||||
a.pendingIdxInserts = nil
|
||||
return
|
||||
}
|
||||
for _, recNo := range a.pendingIdxInserts {
|
||||
for i, idx := range a.idxState.indexes {
|
||||
writer, ok := idx.(IndexWriter)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if i >= len(a.idxState.keyExprs) {
|
||||
continue
|
||||
}
|
||||
key := a.evalKeyExpr(a.idxState.keyExprs[i], recNo)
|
||||
_ = writer.InsertKey(key, recNo)
|
||||
}
|
||||
}
|
||||
a.pendingIdxInserts = nil
|
||||
}
|
||||
|
||||
func (a *DBFArea) updateHeader() {
|
||||
a.header.RecCount = a.recCount
|
||||
a.header.UpdateDate()
|
||||
|
||||
54
hbrdd/dbf/encode_numeric_test.go
Normal file
54
hbrdd/dbf/encode_numeric_test.go
Normal file
@@ -0,0 +1,54 @@
|
||||
// Copyright (c) 2026 Charles KWON OhJun (charleskwonohjun@gmail.com)
|
||||
// All rights reserved.
|
||||
|
||||
package dbf
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestEncodeNumericKey_SortOrder pins the sortable-numeric encoding
|
||||
// used by NTX/CDX indexes. Lexicographic byte order of the encoded
|
||||
// keys must match numeric order across signs and magnitudes — the
|
||||
// previous `%20.10f` encoding sorted -100 AFTER 99 because '-'
|
||||
// (0x2D) > ' ' (0x20).
|
||||
func TestEncodeNumericKey_SortOrder(t *testing.T) {
|
||||
// Strictly ascending input values.
|
||||
values := []float64{-1e9, -1e6, -1000, -100, -99, -1, -0.5, 0, 0.5, 1, 99, 100, 1000, 1e6, 1e9}
|
||||
encoded := make([][]byte, len(values))
|
||||
for i, v := range values {
|
||||
encoded[i] = encodeNumericKey(v)
|
||||
}
|
||||
for i := 1; i < len(encoded); i++ {
|
||||
if bytes.Compare(encoded[i-1], encoded[i]) >= 0 {
|
||||
t.Errorf("encoding[%v]=%q !< encoding[%v]=%q",
|
||||
values[i-1], encoded[i-1], values[i], encoded[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEncodeNumericKey_NegativeBeforePositive guards the specific
|
||||
// inversion called out in the bug report: -100 must encode to a key
|
||||
// that sorts before 99.
|
||||
func TestEncodeNumericKey_NegativeBeforePositive(t *testing.T) {
|
||||
neg100 := encodeNumericKey(-100)
|
||||
pos99 := encodeNumericKey(99)
|
||||
if bytes.Compare(neg100, pos99) >= 0 {
|
||||
t.Errorf("expected -100 < 99 lexicographically; got %q !< %q", neg100, pos99)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEncodeNumericKey_NegativeMagnitudeOrder ensures that within
|
||||
// the negative range, more-negative values sort earlier.
|
||||
func TestEncodeNumericKey_NegativeMagnitudeOrder(t *testing.T) {
|
||||
neg200 := encodeNumericKey(-200)
|
||||
neg100 := encodeNumericKey(-100)
|
||||
neg1 := encodeNumericKey(-1)
|
||||
if bytes.Compare(neg200, neg100) >= 0 {
|
||||
t.Errorf("-200 should sort before -100: %q vs %q", neg200, neg100)
|
||||
}
|
||||
if bytes.Compare(neg100, neg1) >= 0 {
|
||||
t.Errorf("-100 should sort before -1: %q vs %q", neg100, neg1)
|
||||
}
|
||||
}
|
||||
@@ -34,6 +34,17 @@ type IndexEngine interface {
|
||||
Close() error
|
||||
}
|
||||
|
||||
// IndexWriter is optional: engines that support online maintenance
|
||||
// implement this so APPEND / REPLACE / DELETE can patch the index
|
||||
// in place. Engines that don't implement it become stale on any
|
||||
// write — callers must REINDEX manually. NTX implements via
|
||||
// rebuild (correct but O(N)); CDX online maintenance is not yet
|
||||
// wired up so it falls back to the stale-index path for now.
|
||||
type IndexWriter interface {
|
||||
InsertKey(key []byte, recNo uint32) error
|
||||
DeleteKey(recNo uint32) error
|
||||
}
|
||||
|
||||
// indexState holds active index state for a DBFArea.
|
||||
type indexState struct {
|
||||
indexes []IndexEngine // open NTX/CDX index engines
|
||||
@@ -1332,12 +1343,37 @@ func (a *DBFArea) evalForInner(expr string) bool {
|
||||
}
|
||||
|
||||
// valueToKeyBytes converts a hbrt.Value to index key bytes.
|
||||
//
|
||||
// Numerics are encoded so that lexicographic byte order matches
|
||||
// numeric order *including negatives*. The previous "%20.10f"
|
||||
// format was wrong for any negative key: " 99" (space-padded) and
|
||||
// "-100" both fit in 20 chars, but ' ' (0x20) sorts BEFORE '-'
|
||||
// (0x2D), so -100 was indexed as GREATER than 99. Any NTX/CDX
|
||||
// built over a column that ever held a negative number returned
|
||||
// wrong rows for SEEK / range scans.
|
||||
//
|
||||
// Encoding: 1-byte sign prefix + 21-byte zero-padded magnitude.
|
||||
// * positive / zero → '1' + "%021.10f" of value
|
||||
// * negative → '0' + digitComplement("%021.10f" of |value|)
|
||||
//
|
||||
// digitComplement maps '0'..'9' → '9'..'0' (other bytes unchanged).
|
||||
// Properties:
|
||||
// * Positives all sort after negatives (prefix '1' > '0').
|
||||
// * Within positives, magnitude order is preserved (zero-padded).
|
||||
// * Within negatives, larger magnitude → smaller complement →
|
||||
// sorts EARLIER, which is correct (-200 < -100).
|
||||
//
|
||||
// Format change is intentional. Indexes built with the old "%20.10f"
|
||||
// scheme over numeric columns must be REINDEXed before use; this is
|
||||
// a one-line `dbReindex()` per affected index. The previous output
|
||||
// is silently wrong, so there is no safe migration that doesn't
|
||||
// require rebuilding.
|
||||
func valueToKeyBytes(v hbrt.Value) []byte {
|
||||
switch {
|
||||
case v.IsString():
|
||||
return []byte(v.AsString())
|
||||
case v.IsNumeric():
|
||||
return []byte(fmt.Sprintf("%20.10f", v.AsNumDouble()))
|
||||
return encodeNumericKey(v.AsNumDouble())
|
||||
case v.IsDate(), v.IsTimestamp():
|
||||
y, m, d := julianToDate(v.AsJulian())
|
||||
return []byte(fmt.Sprintf("%04d%02d%02d", y, m, d))
|
||||
@@ -1351,6 +1387,31 @@ func valueToKeyBytes(v hbrt.Value) []byte {
|
||||
}
|
||||
}
|
||||
|
||||
func encodeNumericKey(d float64) []byte {
|
||||
if d < 0 {
|
||||
// Magnitude with sign-aware Sprintf would interleave the
|
||||
// '-' inside the zero padding (e.g. "-0000000100.000000"),
|
||||
// which breaks lexicographic comparison. Format |value|
|
||||
// then prepend the sign + complement.
|
||||
mag := fmt.Sprintf("%021.10f", -d) // 21-byte unsigned magnitude
|
||||
b := make([]byte, 0, 22)
|
||||
b = append(b, '0')
|
||||
for i := 0; i < len(mag); i++ {
|
||||
c := mag[i]
|
||||
if c >= '0' && c <= '9' {
|
||||
b = append(b, '9'-(c-'0'))
|
||||
} else {
|
||||
b = append(b, c)
|
||||
}
|
||||
}
|
||||
return b
|
||||
}
|
||||
b := make([]byte, 0, 22)
|
||||
b = append(b, '1')
|
||||
b = append(b, fmt.Sprintf("%021.10f", d)...)
|
||||
return b
|
||||
}
|
||||
|
||||
// Helper: find matching close parenthesis
|
||||
func findMatchingParen(s string, openPos int) int {
|
||||
depth := 1
|
||||
|
||||
@@ -24,6 +24,7 @@ package dbf
|
||||
import (
|
||||
"fmt"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// flockAll acquires/releases an exclusive lock on the DBF header area.
|
||||
@@ -32,10 +33,53 @@ import (
|
||||
// - Record locks (at RecordOffset(n), past the header) don't conflict
|
||||
const flockOffset = int64(0)
|
||||
|
||||
// appendLockOffset is the byte range used to serialize concurrent
|
||||
// APPEND across processes. We park it well above any realistic 32-bit
|
||||
// DBF record offset so it doesn't collide with FLOCK (header) or
|
||||
// record locks. POSIX fcntl allows locking byte ranges past EOF.
|
||||
// Harbour's protocol uses an equivalent special offset for append.
|
||||
const appendLockOffset = int64(0x7FFFFFFE)
|
||||
const appendLockLen = int64(1)
|
||||
|
||||
func flockLen(a *DBFArea) int64 {
|
||||
return int64(a.header.HeaderLen) + 1
|
||||
}
|
||||
|
||||
// lockAppendIntent acquires the append-intent lock with bounded
|
||||
// retries. Without serialization, two processes simultaneously
|
||||
// running APPEND would both read the same `recCount` from the
|
||||
// header, write at the same RecordOffset, and one record would
|
||||
// silently overwrite the other.
|
||||
//
|
||||
// The retry budget is intentionally short — append contention is
|
||||
// usually a few hundred microseconds, so a long block here is the
|
||||
// signal of a misbehaving peer rather than legitimate competition.
|
||||
func (a *DBFArea) lockAppendIntent() error {
|
||||
if a.dataFile == nil {
|
||||
return fmt.Errorf("file not open")
|
||||
}
|
||||
fd := int(a.dataFile.Fd())
|
||||
deadline := 100 // ~100 ms total (1 ms × 100); cheap bound vs blocking F_SETLKW
|
||||
for i := 0; i < deadline; i++ {
|
||||
ok, err := tryLock(fd, appendLockOffset, appendLockLen)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if ok {
|
||||
return nil
|
||||
}
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
return fmt.Errorf("APPEND: could not acquire append-intent lock (peer holding too long)")
|
||||
}
|
||||
|
||||
func (a *DBFArea) unlockAppendIntent() error {
|
||||
if a.dataFile == nil {
|
||||
return nil
|
||||
}
|
||||
return unlockRange(int(a.dataFile.Fd()), appendLockOffset, appendLockLen)
|
||||
}
|
||||
|
||||
// tryLock acquires an exclusive byte-range lock. Non-blocking: returns
|
||||
// (false, nil) if another process already holds a conflicting lock,
|
||||
// (false, err) on system error, (true, nil) on success.
|
||||
|
||||
@@ -19,15 +19,47 @@ package dbf
|
||||
import (
|
||||
"fmt"
|
||||
"syscall"
|
||||
"time"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
const flockOffset = int64(0)
|
||||
|
||||
// appendLockOffset / Len — see locks_posix.go for the rationale.
|
||||
// Windows LockFileEx accepts arbitrary byte ranges past EOF.
|
||||
const appendLockOffset = int64(0x7FFFFFFE)
|
||||
const appendLockLen = int64(1)
|
||||
|
||||
func flockLen(a *DBFArea) int64 {
|
||||
return int64(a.header.HeaderLen) + 1
|
||||
}
|
||||
|
||||
// lockAppendIntent — see locks_posix.go for the rationale.
|
||||
func (a *DBFArea) lockAppendIntent() error {
|
||||
if a.dataFile == nil {
|
||||
return fmt.Errorf("file not open")
|
||||
}
|
||||
h := a.winHandle()
|
||||
for i := 0; i < 100; i++ {
|
||||
ok, err := tryLock(h, appendLockOffset, appendLockLen)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if ok {
|
||||
return nil
|
||||
}
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
return fmt.Errorf("APPEND: could not acquire append-intent lock")
|
||||
}
|
||||
|
||||
func (a *DBFArea) unlockAppendIntent() error {
|
||||
if a.dataFile == nil {
|
||||
return nil
|
||||
}
|
||||
return unlockRange(a.winHandle(), appendLockOffset, appendLockLen)
|
||||
}
|
||||
|
||||
var (
|
||||
modkernel32 = syscall.NewLazyDLL("kernel32.dll")
|
||||
procLockFileEx = modkernel32.NewProc("LockFileEx")
|
||||
|
||||
@@ -104,13 +104,19 @@ func normalizeName(s string) string {
|
||||
// --- Table (shared data) ---
|
||||
|
||||
type memTable struct {
|
||||
// mu serializes WRITERS only (Append/Delete/Recall/PutValue/Pack).
|
||||
// Readers use records() — a lock-free atomic load of the current
|
||||
// snapshot. Matches Harbour SHARED semantics: readers see a
|
||||
// point-in-time view of the record slice; in-place field mutations
|
||||
// are last-writer-wins (callers that need row consistency take an
|
||||
// explicit RLock via the runtime's record-lock RTL).
|
||||
mu sync.Mutex
|
||||
// mu was previously a sync.Mutex with the comment that readers
|
||||
// could safely race against in-place PutValue because hbrt.Value
|
||||
// "fits in a single machine word + pointer". That assumption was
|
||||
// wrong: hbrt.Value is 24 bytes (3 words). A concurrent reader
|
||||
// could observe a half-written struct — type tag from the new
|
||||
// value, scalar/ptr from the old — and crash on subsequent type
|
||||
// dispatch.
|
||||
//
|
||||
// Switched to sync.RWMutex: PutValue/Append/Delete/Recall take
|
||||
// Lock; GetValue/Skip/Seek/scan take RLock. Adds ~30ns per read
|
||||
// to make field reads consistent. Still matches Harbour SHARED
|
||||
// semantics — readers never see a partial write.
|
||||
mu sync.RWMutex
|
||||
// recordsP holds the current []memRecord snapshot. Stored as
|
||||
// *[]memRecord to work with atomic.Pointer's typed API. Writers
|
||||
// publish new slices via setRecords() after mutation; readers Load
|
||||
@@ -411,10 +417,13 @@ func (a *memArea) GetFieldInfo(index int) hbrdd.FieldInfo {
|
||||
}
|
||||
|
||||
func (a *memArea) GetValue(fieldIndex int) (hbrt.Value, error) {
|
||||
// Hot path — lock-free read. The atomic load gives us a
|
||||
// point-in-time snapshot; a concurrent PutValue mutating the same
|
||||
// rec.data[fieldIndex] in place is tolerated (last-writer-wins,
|
||||
// matches Harbour SHARED semantics).
|
||||
// Read under RLock so a concurrent PutValue (which holds Lock)
|
||||
// completes its 24-byte hbrt.Value store before this read
|
||||
// observes the field. Without the RLock the reader could see a
|
||||
// half-written value — new type tag with stale scalar/ptr —
|
||||
// which type-confuses on the next AsXxx() call.
|
||||
a.tbl.mu.RLock()
|
||||
defer a.tbl.mu.RUnlock()
|
||||
recs := a.tbl.records()
|
||||
i := int(a.recNo) - 1
|
||||
if i < 0 || i >= len(recs) {
|
||||
|
||||
@@ -279,9 +279,16 @@ func (t *Thread) Send(methodName string, nArgs int) {
|
||||
panic(t.runtimeError(fmt.Sprintf("unknown method %s in class %s", methodName, cls.Name)))
|
||||
}
|
||||
|
||||
// Set up Self context
|
||||
// Set up Self context. Restore via defer so a panic in the
|
||||
// method body (HbError, BreakValue from `Break(…)`, runtime
|
||||
// panic) unwinds with `t.self` pointing at the caller's
|
||||
// receiver — not at this method's. Without defer, a RECOVER
|
||||
// USING handler that runs after the panic still saw the stale
|
||||
// `t.self`, so `::field` / `::method()` resolved against the
|
||||
// wrong object — silent data corruption in the recovery path.
|
||||
oldSelf := t.self
|
||||
t.self = objVal
|
||||
defer func() { t.self = oldSelf }()
|
||||
|
||||
// Push args for Frame
|
||||
for _, arg := range args {
|
||||
@@ -291,9 +298,6 @@ func (t *Thread) Send(methodName string, nArgs int) {
|
||||
t.pendingParams = nArgs
|
||||
fn(t)
|
||||
|
||||
// Restore Self
|
||||
t.self = oldSelf
|
||||
|
||||
// Push return value
|
||||
t.push(t.retVal)
|
||||
}
|
||||
@@ -330,10 +334,11 @@ func (t *Thread) tryBinaryOp(op int) bool {
|
||||
t.pop() // discard a (Self takes over)
|
||||
oldSelf := t.self
|
||||
t.self = a
|
||||
// defer restore — see comment in Send.
|
||||
defer func() { t.self = oldSelf }()
|
||||
t.push(b)
|
||||
t.pendingParams = 1
|
||||
fn(t)
|
||||
t.self = oldSelf
|
||||
t.push(t.retVal)
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -42,8 +42,22 @@ type HBContext struct {
|
||||
|
||||
// HB_FUNC registers a Go function as a Harbour-callable function.
|
||||
// Equivalent to Harbour's HB_FUNC(name) macro.
|
||||
//
|
||||
// The wrapper sets up a runtime Frame so `ctx.ParC(n)` / `ctx.Local(n)`
|
||||
// resolves through *this* function's locals instead of leaking into
|
||||
// the caller's frame. Without Frame the body's `t.Local(n)` indexed
|
||||
// `t.curFrame.localBase + n - 1` against whatever frame the caller
|
||||
// happened to be on — silently reading the caller's locals, and
|
||||
// any default-value path (`ParNIDef`) hid the bug by masking NIL.
|
||||
func HB_FUNC(name string, fn func(ctx *HBContext)) {
|
||||
RegisterDynamicFunc(strings.ToUpper(name), func(t *Thread) {
|
||||
// Snapshot pendingParams before Frame consumes it; the body
|
||||
// expects to declare nParams=actual so each positional arg
|
||||
// lands in a fresh slot, but EndProc/EndProcFast cleanup
|
||||
// must run regardless of how the body returns.
|
||||
nArgs := t.pendingParams
|
||||
t.Frame(nArgs, 0)
|
||||
defer t.EndProc() // retains panic propagation for RECOVER
|
||||
ctx := &HBContext{T: t}
|
||||
fn(ctx)
|
||||
})
|
||||
|
||||
@@ -298,6 +298,20 @@ func execPcodeBody(t *Thread, fn *PcodeFunc, mod *PcodeModule) {
|
||||
nDetached := int(binary.LittleEndian.Uint16(code[pc:]))
|
||||
pc += 2
|
||||
|
||||
// Snapshot closure-captured locals from the *current*
|
||||
// frame into the new block's Detached slice. The body
|
||||
// reads/writes them via PcOpPushDetached / PcOpPopDetached
|
||||
// at the indices the compiler reserved. Without this,
|
||||
// `{|x| x + outer }` saw `outer` as NIL because the
|
||||
// block fn ran with its own frame and the body's lookup
|
||||
// for `outer` fell through to the memvar table.
|
||||
captured := make([]Value, nDetached)
|
||||
for i := 0; i < nDetached; i++ {
|
||||
srcIdx := int(binary.LittleEndian.Uint16(code[pc:]))
|
||||
pc += 2
|
||||
captured[i] = t.Local(srcIdx)
|
||||
}
|
||||
|
||||
// Create a Go function that interprets the block's pcode.
|
||||
// Params count must be threaded through so ExecPcode's
|
||||
// Frame() pulls Eval()'s args off the stack into the
|
||||
@@ -305,9 +319,42 @@ func execPcodeBody(t *Thread, fn *PcodeFunc, mod *PcodeModule) {
|
||||
// and `x * x` panicked on the multiplication.
|
||||
blockFn := &PcodeFunc{Code: blockCode, Params: nParams}
|
||||
modCopy := mod
|
||||
t.PushBlock(func(t2 *Thread) {
|
||||
blockVal := MakeBlock(nil, nDetached) // Fn patched below
|
||||
bb := (*HbBlock)(blockVal.ptr)
|
||||
if nDetached > 0 {
|
||||
copy(bb.Detached, captured)
|
||||
}
|
||||
bb.Fn = func(t2 *Thread) {
|
||||
// Install this block as the currently-executing
|
||||
// block so PcOpPushDetached / PcOpPopDetached can
|
||||
// resolve their slots. Restore the previous one on
|
||||
// exit so nested-block evaluation (`{|| eval(b2) }`)
|
||||
// pops back to the outer block.
|
||||
prev := t2.CurBlock()
|
||||
t2.SetCurBlock(bb)
|
||||
defer t2.SetCurBlock(prev)
|
||||
ExecPcode(t2, blockFn, modCopy)
|
||||
}, nDetached)
|
||||
}
|
||||
t.PushValue(blockVal)
|
||||
|
||||
case PcOpPushDetached:
|
||||
slot := int(binary.LittleEndian.Uint16(code[pc:]))
|
||||
pc += 2
|
||||
bb := t.CurBlock()
|
||||
if bb != nil && slot < len(bb.Detached) {
|
||||
t.PushValue(bb.Detached[slot])
|
||||
} else {
|
||||
t.PushNil()
|
||||
}
|
||||
|
||||
case PcOpPopDetached:
|
||||
slot := int(binary.LittleEndian.Uint16(code[pc:]))
|
||||
pc += 2
|
||||
val := t.pop()
|
||||
bb := t.CurBlock()
|
||||
if bb != nil && slot < len(bb.Detached) {
|
||||
bb.Detached[slot] = val
|
||||
}
|
||||
|
||||
// --- Local ops ---
|
||||
case PcOpLocalAddInt:
|
||||
|
||||
@@ -92,8 +92,17 @@ const (
|
||||
PcOpArrayPush byte = 0x52
|
||||
PcOpArrayPop byte = 0x53
|
||||
|
||||
// Block
|
||||
PcOpPushBlock byte = 0x58 // + uint32 codeLen + pcode bytes + uint16 nDetached
|
||||
// Block — operand layout:
|
||||
// PcOpPushBlock + uint32 codeLen + body bytes
|
||||
// + uint16 nParams + uint16 nDetached
|
||||
// + nDetached × uint16 (source-local index per slot)
|
||||
// Each captured slot snapshots the current frame's Local(idx)
|
||||
// into the block's Detached[i] at creation time. Body accesses
|
||||
// captured values via PcOpPushDetached / PcOpPopDetached with the
|
||||
// 0-based slot index.
|
||||
PcOpPushBlock byte = 0x58
|
||||
PcOpPushDetached byte = 0x59 // + uint16 0-based detached slot
|
||||
PcOpPopDetached byte = 0x5A // + uint16 0-based detached slot
|
||||
|
||||
// Local operations
|
||||
PcOpLocalAddInt byte = 0x60 // + uint16 index + int32 value
|
||||
@@ -129,4 +138,10 @@ type PcodeModule struct {
|
||||
Name string
|
||||
Funcs map[string]*PcodeFunc
|
||||
Strings []string // string constant pool
|
||||
// Warnings captures compile-time diagnostics from genpc — most
|
||||
// commonly "AST node X not supported in pcode mode". Surfaced
|
||||
// by the build pipeline so users learn their PRG isn't fully
|
||||
// pcode-compilable instead of seeing silent wrong results from
|
||||
// no-op fallbacks. Empty slice = clean compile.
|
||||
Warnings []string
|
||||
}
|
||||
|
||||
@@ -33,6 +33,16 @@ type CallFrame struct {
|
||||
// CurFrame returns the current call frame (for closure capture).
|
||||
func (t *Thread) CurFrame() *CallFrame { return t.curFrame }
|
||||
|
||||
// CurBlock returns the *HbBlock for the codeblock currently executing
|
||||
// (or nil outside a block body). Used by pcode dispatch to resolve
|
||||
// detached-local opcodes against the running block's capture slice.
|
||||
func (t *Thread) CurBlock() *HbBlock { return t.curBlock }
|
||||
|
||||
// SetCurBlock installs the executing block. The pcode block wrapper
|
||||
// pairs Set(block) with a deferred Set(prev) to nest correctly across
|
||||
// blocks that call other blocks (`{|| eval(b1) + eval(b2) }`).
|
||||
func (t *Thread) SetCurBlock(b *HbBlock) { t.curBlock = b }
|
||||
|
||||
// LocalsSlice returns the underlying locals array (for closure capture).
|
||||
func (t *Thread) LocalsSlice() []Value { return t.locals }
|
||||
|
||||
@@ -111,6 +121,13 @@ type Thread struct {
|
||||
// OOP: current Self object (set during method dispatch)
|
||||
self Value
|
||||
|
||||
// Current code block (set while a block body is executing).
|
||||
// Pcode opcodes PcOpPushDetached / PcOpPopDetached read/write
|
||||
// Detached[i] through this pointer. The block's wrapper Fn
|
||||
// sets it before ExecPcode and restores on exit. Stays nil
|
||||
// outside block bodies; nil-checking opcodes fall back to NIL.
|
||||
curBlock *HbBlock
|
||||
|
||||
// Error handling: last error from BEGIN SEQUENCE
|
||||
lastError *HbError
|
||||
|
||||
|
||||
16
hbrt/vm.go
16
hbrt/vm.go
@@ -154,13 +154,29 @@ func (t *Thread) GetSym(cache **Symbol, name string) *Symbol {
|
||||
}
|
||||
|
||||
// NewThread creates a new Thread attached to this VM.
|
||||
//
|
||||
// Statics + WA are initialized here (not just in Run) so threads
|
||||
// spawned via GoLaunch / GoLaunchBlock — which call NewThread
|
||||
// directly — see the same module-static map and have a workarea
|
||||
// manager available. Without this, PRG code running in a goroutine
|
||||
// that touched a STATIC panicked with "static index out of range",
|
||||
// and any DB/RDD call crashed dereferencing nil WA.
|
||||
func (vm *VM) NewThread() *Thread {
|
||||
t := NewThread(vm)
|
||||
vm.mu.Lock()
|
||||
vm.nextTID++
|
||||
t.tid = vm.nextTID
|
||||
vm.threads = append(vm.threads, t)
|
||||
// Snapshot the statics map under the same lock — late
|
||||
// goroutines see whatever was registered up to this point.
|
||||
for k, v := range vm.statics {
|
||||
t.statics[k] = v
|
||||
}
|
||||
wf := vm.waFactory
|
||||
vm.mu.Unlock()
|
||||
if t.WA == nil && wf != nil {
|
||||
t.WA = wf()
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
|
||||
@@ -46,6 +46,12 @@ func frbCompileInProc(vm *hbrt.VM, prgSource string) (*hbrt.FrbModule, error) {
|
||||
return nil, fmt.Errorf("parse: %s", strings.Join(msgs, "; "))
|
||||
}
|
||||
pcMod := genpc.Generate(file)
|
||||
// Surface unsupported-AST-node warnings on stderr so dynamic
|
||||
// FrbCompile callers see "pcode: AST node X not supported …"
|
||||
// instead of getting a silently-broken module back.
|
||||
for _, w := range pcMod.Warnings {
|
||||
fmt.Fprintln(os.Stderr, w)
|
||||
}
|
||||
|
||||
// Build a FrbModule from the pcode functions. Mirrors what
|
||||
// hbrt/frb.go's frbLoadPcode does, but without the disk hop.
|
||||
|
||||
@@ -54,7 +54,20 @@ PROCEDURE Main()
|
||||
' n++' + Chr(10) + ;
|
||||
' ENDIF' + Chr(10) + ;
|
||||
' NEXT' + Chr(10) + ;
|
||||
' RETURN n' + Chr(10)
|
||||
' RETURN n' + Chr(10) + ;
|
||||
'FUNCTION Grade(n)' + Chr(10) + ;
|
||||
' IF n >= 90' + Chr(10) + ;
|
||||
' RETURN "A"' + Chr(10) + ;
|
||||
' ELSEIF n >= 80' + Chr(10) + ;
|
||||
' RETURN "B"' + Chr(10) + ;
|
||||
' ELSEIF n >= 70' + Chr(10) + ;
|
||||
' RETURN "C"' + Chr(10) + ;
|
||||
' ELSEIF n >= 60' + Chr(10) + ;
|
||||
' RETURN "D"' + Chr(10) + ;
|
||||
' ELSE' + Chr(10) + ;
|
||||
' RETURN "F"' + Chr(10) + ;
|
||||
' ENDIF' + Chr(10) + ;
|
||||
' RETURN "?"' + Chr(10)
|
||||
|
||||
pMod := FrbCompile(src)
|
||||
IF pMod == NIL
|
||||
@@ -219,6 +232,34 @@ PROCEDURE Main()
|
||||
fail++
|
||||
ENDIF
|
||||
|
||||
/* 11a-e. Multi-ELSEIF chain — regression test for genpc jumpEnd
|
||||
patching (every taken branch needs to jump past the rest of
|
||||
the IF, not fall through into the next ELSEIF's bytecode). */
|
||||
label := "11a. Grade(95)"
|
||||
expect := "A"
|
||||
got := FrbDo(pMod, "GRADE", 95)
|
||||
IF got == expect ; ? "PASS", label, "=", got ; pass++ ; ELSE ; ? "FAIL", label, "expect", expect, "got", got ; fail++ ; ENDIF
|
||||
|
||||
label := "11b. Grade(85)"
|
||||
expect := "B"
|
||||
got := FrbDo(pMod, "GRADE", 85)
|
||||
IF got == expect ; ? "PASS", label, "=", got ; pass++ ; ELSE ; ? "FAIL", label, "expect", expect, "got", got ; fail++ ; ENDIF
|
||||
|
||||
label := "11c. Grade(75)"
|
||||
expect := "C"
|
||||
got := FrbDo(pMod, "GRADE", 75)
|
||||
IF got == expect ; ? "PASS", label, "=", got ; pass++ ; ELSE ; ? "FAIL", label, "expect", expect, "got", got ; fail++ ; ENDIF
|
||||
|
||||
label := "11d. Grade(65)"
|
||||
expect := "D"
|
||||
got := FrbDo(pMod, "GRADE", 65)
|
||||
IF got == expect ; ? "PASS", label, "=", got ; pass++ ; ELSE ; ? "FAIL", label, "expect", expect, "got", got ; fail++ ; ENDIF
|
||||
|
||||
label := "11e. Grade(50)"
|
||||
expect := "F"
|
||||
got := FrbDo(pMod, "GRADE", 50)
|
||||
IF got == expect ; ? "PASS", label, "=", got ; pass++ ; ELSE ; ? "FAIL", label, "expect", expect, "got", got ; fail++ ; ENDIF
|
||||
|
||||
FrbUnload(pMod)
|
||||
|
||||
? ""
|
||||
|
||||
Reference in New Issue
Block a user