Files
five/cmd/five/main.go
CharlesKWON cde86730b8 fix(compiler,hbrt,hbrdd,cli): pre-1.0 audit — 13 critical fixes
Senior-engineer / QA audit landed 13 silent-miscompile and data-
integrity fixes spanning the whole compiler+runtime+storage stack.
Each fix is paired with either an integration test in the suite or
a focused regression check; all 6 release gates stay green:
go test ./..., FiveSql2 43/43, Harbour compat 56/56, std.ch 17/17,
FRB 7/7, examples 65/71.

Compiler
--------

* genpc IF/ELSEIF jumpEnd2 patching (compiler/genpc/genpc.go).
  Per-ELSEIF branch terminators were stashed into `_ = jumpEnd2`
  and never patched — the relative offset stayed 0 and the runtime
  walked the next ELSEIF's PcOpJumpFalse opcode as if it were
  jump-offset data. Bytecode-level corruption in pcode mode. Now
  collected into a slice and patched at end-of-IF. Verified via
  Grade(95..50) cases 11a-e added to tests/frb/test_frb_pcode_sweep.

* countLocalsInStmts / scanBodyLocals missing bodies
  (compiler/gengo/gen_util.go, compiler/gengo/gengo.go). Frame-size
  counter skipped WATCH/TIMEOUT/PARALLEL FOR bodies, so a LOCAL
  declared inside one of those constructs got a slot index past
  the runtime's allocated count — silent NIL reads or out-of-range
  stomps.

* emitMethodDeclStandalone nested LOCAL (compiler/gengo/gen_class.go).
  Same bug class but on the *method* side. Pre-fix repro:

      METHOD Stomp(n) CLASS T
         LOCAL a := 1, b := 2
         IF n > 0
            LOCAL c := 30, d := 40, e := 50, f := 60
            Inner( n )
            IF c != 30 .OR. d != 40 .OR. e != 50 .OR. f != 60 ...

  printed `c, d, e, f = 5, NIL, NIL, NIL` because Inner's frame
  collided with Stomp's underallocated slot range. Now counts
  body-nested LOCALs into the frame and pre-allocates indices via
  scanBodyLocals.

* genpc unsupported-AST diagnostic surface (compiler/genpc/genpc.go,
  hbrt/pcode.go, cmd/five/main.go, hbrtl/frb.go). The `default`
  cases in emitStmt / emitExpr silently emitted PushNil / no-op
  for nodes the pcode generator doesn't implement (ClassDecl,
  MethodDecl, xBase commands, concurrency primitives, …). Added
  `PcodeModule.Warnings []string` populated by noteUnsupported,
  surfaced on stderr from the build pipeline. Users now see
  "pcode: AST node not supported in --pcode/FRB-pcode mode: stmt
  *ast.GoBlockStmt" instead of getting a silently broken module.

Runtime
-------

* class.go Send/tryBinaryOp t.self defer-restore (hbrt/class.go).
  Restoration was a plain `t.self = oldSelf` after `fn(t)`. Any
  panic in the method body skipped the line, so the next BEGIN
  SEQUENCE / RECOVER handler ran with the THROWING object's Self
  — `::field` resolved against the wrong receiver. Wrapped both
  restore sites in `defer func() { t.self = oldSelf }()`.
  Verified: pre-fix RECOVER saw "THROWER", post-fix "OUTER".

* hbfunc.go HB_FUNC parameter Frame() (hbrt/hbfunc.go). The
  RegisterDynamicFunc wrapper called `fn(ctx)` without ever
  calling Frame, so `ctx.ParC(1)` / `ctx.Local(n)` read through
  `t.curFrame.localBase + n - 1` against the *caller's* frame.
  Every #pragma BEGINDUMP HB_FUNC taking parameters silently
  returned "" / 0 / "" for them — masked by ParNIDef-style
  defaults. Wrapper now does `t.Frame(t.pendingParams, 0); defer
  t.EndProc()` before dispatch.

* pcode codeblock closure capture (hbrt/pcinterp.go, hbrt/pcode.go,
  hbrt/thread.go, compiler/genpc/genpc.go). PcOpPushBlock recorded
  `nDetached` but never copied enclosing locals; free vars in the
  block body fell through to memvar lookup → NIL. Wired full
  capture pipeline:
  - New opcodes PcOpPushDetached (0x59) / PcOpPopDetached (0x5A).
  - PushBlock now reads per-slot source-local indices and
    snapshots into bb.Detached at construction time.
  - New detachedMap in genpc auto-promotes any free var that
    resolves to an enclosing-frame local into a capture slot.
  - emitAssignAsExpr leaves the assigned value on the eval stack
    so SeqExpr items like `{|v| acc += v, acc }` work.
  - Thread tracks curBlock with paired Set/restore in the block's
    Fn wrapper for nested-block evaluation.
  Mutating capture (acc += v across successive Evals) now works.

* vm.NewThread statics + waFactory propagation (hbrt/vm.go).
  GoLaunch / GoLaunchBlock call NewThread directly. Previously
  the statics map and WA factory were applied only in Run(), so
  goroutine-spawned PRG code panicked on STATIC access ("static
  index out of range") and crashed dereferencing nil WA on any
  DB call. Both now happen inside NewThread under the same lock
  as TID assignment.

Data layer
----------

* dbf concurrent Append lock (hbrdd/dbf/dbf.go,
  hbrdd/dbf/locks_posix.go, hbrdd/dbf/locks_windows.go). Append
  bumped a local recCount with no file-system serialization. Two
  shared-mode processes both wrote at the same RecordOffset; one
  record silently overwrote the other. Added an append-intent
  byte-range lock at offset 0x7FFFFFFE + bounded retry, on-disk
  header refresh inside the locked region, and immediate header
  write so peers refresh past our slot.

* indexer negative numeric key encoding (hbrdd/dbf/indexer.go +
  new hbrdd/dbf/encode_numeric_test.go). `%20.10f` formats `-100`
  as `"     -100.0000000000"` and `99` as `"        99.0000000000"`.
  ASCII ' ' (0x20) < '-' (0x2D), so `99` lex-compared LESS than
  `-100` — every NTX/CDX index over a column that ever held a
  negative number returned wrong rows for SEEK / range scans.
  Replaced with a 1-byte sign prefix + 21-byte zero-padded
  magnitude (negatives use digit-complement) so byte order
  matches numeric order across signs and magnitudes. Format
  change: existing indexes built with the old encoding must be
  REINDEXed. Three unit tests pin the order.

* dbf Append index maintenance hooks (hbrdd/dbf/dbf.go,
  hbrdd/dbf/indexer.go). Append never inserted into open NTX/CDX
  indexes — the audit's canonical scenario `SET INDEX TO …;
  APPEND BLANK; REPLACE …; dbSeek …` silently missed the new
  record. Added optional IndexWriter interface, queue the new
  recNo in pendingIdxInserts, drain after flushRecord by calling
  InsertKey on every open writer-supporting engine. NTX
  participates (its existing rebuild-on-insert is correct);
  CDX online maintenance is deferred to a follow-up — those
  indexes still need REINDEX. Verified: post-fix SEEK("Charlie")
  after APPEND BLANK + REPLACE finds the new record.

* dbf PACK crash-safety (hbrdd/dbf/dbf.go). The old in-place
  rewrite read record N, overwrote slot M<N, then truncated.
  Power loss after partial loop left a file with overwritten
  prefix and no original copies of the records already advanced
  past — silent data loss. Rewrote to:
    1) drop mmap, build `<file>.pack.tmp` with all surviving
       records,
    2) Sync(),
    3) close original handle + os.Rename(tmp, orig) (atomic on
       same FS),
    4) reopen + re-mmap.
  TestComp_Pack passes; readers always see either the pre-PACK
  or post-PACK contents, never a half-state.

* mem RDD torn reads (hbrdd/mem/memrdd.go). The comment claimed
  in-place PutValue was safe because hbrt.Value "fits in a
  single machine word + pointer". hbrt.Value is 24 bytes (3
  words) — a concurrent reader could observe new type tag with
  stale scalar/ptr and type-confuse on the next AsXxx() call.
  Switched mu to sync.RWMutex; GetValue takes RLock,
  Append/PutValue/Delete/Recall take Lock. `go test -race
  ./hbrdd/mem/` clean.

Files touched
-------------

  compiler/gengo/gen_class.go, gen_util.go, gengo.go
  compiler/genpc/genpc.go
  hbrt/class.go, hbfunc.go, pcinterp.go, pcode.go, thread.go, vm.go
  hbrdd/dbf/dbf.go, indexer.go, locks_posix.go, locks_windows.go
  hbrdd/dbf/encode_numeric_test.go  (new)
  hbrdd/mem/memrdd.go
  cmd/five/main.go
  hbrtl/frb.go
  tests/frb/test_frb_pcode_sweep.prg

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 05:29:56 +09:00

949 lines
27 KiB
Go

// Copyright (c) 2026 Charles KWON OhJun (charleskwonohjun@gmail.com)
// All rights reserved.
// Five CLI — the Harbour + Go fusion language tool.
//
// Usage:
// five run <file.prg> Compile and run a PRG file
// five build <file.prg> [-o out] Compile to native binary
// five gen <file.prg> Generate Go source (for debugging)
package main
import (
"five/compiler/analyzer"
"five/compiler/ast"
"five/compiler/gengo"
"five/compiler/genpc"
"five/compiler/parser"
"five/compiler/pp"
"five/hbrt"
"fmt"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
)
func main() {
if len(os.Args) < 2 {
printUsage()
os.Exit(1)
}
cmd := os.Args[1]
switch cmd {
case "run":
if len(os.Args) < 3 {
fatal("usage: five run <file.prg>")
}
runPRG(os.Args[2])
case "build":
if len(os.Args) < 3 {
fatal("usage: five build <file.prg> [file2.prg ...] [-o output] [-I includedir]")
}
output := ""
var prgFiles []string
var includeDirs []string
for i := 2; i < len(os.Args); i++ {
if os.Args[i] == "-o" && i+1 < len(os.Args) {
output = os.Args[i+1]
i++
} else if os.Args[i] == "-I" && i+1 < len(os.Args) {
includeDirs = append(includeDirs, os.Args[i+1])
i++
} else if strings.HasPrefix(os.Args[i], "-I") {
includeDirs = append(includeDirs, os.Args[i][2:])
} else {
prgFiles = append(prgFiles, os.Args[i])
}
}
if len(prgFiles) == 1 {
buildPRGWithIncludes(prgFiles[0], output, includeDirs)
} else {
buildMultiPRGWithIncludes(prgFiles, output, includeDirs)
}
case "gen":
if len(os.Args) < 3 {
fatal("usage: five gen <file.prg>")
}
genPRG(os.Args[2])
case "debug":
if len(os.Args) < 3 {
fatal("usage: five debug <file.prg> [-b [module:]line ...] [--cli]")
}
prg := ""
var breakpoints []string
var watches []string
useCLI := false
for i := 2; i < len(os.Args); i++ {
a := os.Args[i]
switch a {
case "-b", "--break":
if i+1 >= len(os.Args) {
fatal("-b requires an argument: [module:]line")
}
breakpoints = append(breakpoints, os.Args[i+1])
i++
case "-w", "--watch":
if i+1 >= len(os.Args) {
fatal("-w requires an argument: <expr>")
}
watches = append(watches, os.Args[i+1])
i++
case "--cli":
useCLI = true
default:
if prg == "" {
prg = a
} else {
fatal("unexpected argument: " + a)
}
}
}
if prg == "" {
fatal("usage: five debug <file.prg> [-b [module:]line ...] [-w <expr> ...] [--cli]")
}
debugPRGWithOpts(prg, breakpoints, watches, useCLI)
case "frb":
if len(os.Args) < 3 {
fatal("usage: five frb <file.prg> [-o output.frb] [--pcode]")
}
output := ""
prgFile := os.Args[2]
pcodeMode := false
for i := 3; i < len(os.Args); i++ {
if os.Args[i] == "-o" && i+1 < len(os.Args) {
output = os.Args[i+1]
i++
} else if os.Args[i] == "--pcode" {
pcodeMode = true
}
}
if output == "" {
base := strings.TrimSuffix(filepath.Base(prgFile), filepath.Ext(prgFile))
output = base + ".frb"
}
if pcodeMode {
buildFRBPcode(prgFile, output)
} else {
buildFRB(prgFile, output)
}
case "version":
fmt.Println("Five 0.1.0 — Harbour + Go fusion language")
fmt.Println("Copyright (c) 2026 Charles KWON OhJun")
default:
fmt.Fprintf(os.Stderr, "unknown command: %s\n", cmd)
printUsage()
os.Exit(1)
}
}
func printUsage() {
fmt.Fprintln(os.Stderr, "Five — Harbour + Go fusion language")
fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "Usage:")
fmt.Fprintln(os.Stderr, " five run <file.prg> Compile and run")
fmt.Fprintln(os.Stderr, " five build <file.prg> [-o out] Compile to binary")
fmt.Fprintln(os.Stderr, " five gen <file.prg> Show generated Go code")
fmt.Fprintln(os.Stderr, " five debug <file.prg> Debug with interactive debugger")
fmt.Fprintln(os.Stderr, " five frb <file.prg> [-o out] Compile to FRB module")
fmt.Fprintln(os.Stderr, " five version Show version")
}
// runPRG compiles a PRG file and runs it immediately.
func runPRG(prgFile string) {
goCode := compilePRG(prgFile)
// Create temp build directory
tmpDir, err := os.MkdirTemp("", "five-build-*")
if err != nil {
fatal("failed to create temp dir: " + err.Error())
}
if os.Getenv("FIVE_KEEP_BUILD") == "" {
defer os.RemoveAll(tmpDir)
} else {
fmt.Fprintln(os.Stderr, "[FIVE_KEEP_BUILD] keeping:", tmpDir)
}
writeGoProject(tmpDir, prgFile, goCode)
// go run
cmd := exec.Command(goPath(), "run", ".")
cmd.Dir = tmpDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
os.Exit(exitErr.ExitCode())
}
fatal("run failed: " + err.Error())
}
}
// buildPRG compiles a PRG file to a native binary.
func buildPRG(prgFile, output string) {
goCode := compilePRG(prgFile)
if output == "" {
base := strings.TrimSuffix(filepath.Base(prgFile), filepath.Ext(prgFile))
output = base
}
// Create temp build directory
tmpDir, err := os.MkdirTemp("", "five-build-*")
if err != nil {
fatal("failed to create temp dir: " + err.Error())
}
if os.Getenv("FIVE_KEEP_BUILD") == "" {
defer os.RemoveAll(tmpDir)
} else {
fmt.Fprintln(os.Stderr, "[FIVE_KEEP_BUILD] keeping:", tmpDir)
}
writeGoProject(tmpDir, prgFile, goCode)
// Resolve absolute path for output
absOutput, err := filepath.Abs(output)
if err != nil {
fatal("cannot resolve output path: " + err.Error())
}
// go build — pgoArgs() adds -pgo=default.pgo when available.
cmd := exec.Command(goPath(), buildArgs(absOutput)...)
cmd.Dir = tmpDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
fatal("build failed: " + err.Error())
}
fmt.Fprintf(os.Stderr, "Built: %s\n", absOutput)
}
// genPRG generates Go source and prints it to stdout.
func genPRG(prgFile string) {
goCode := compilePRG(prgFile)
fmt.Print(goCode)
}
// buildMultiPRG compiles multiple PRG files into one binary.
func buildMultiPRG(prgFiles []string, output string) {
if output == "" {
base := strings.TrimSuffix(filepath.Base(prgFiles[0]), filepath.Ext(prgFiles[0]))
output = base
}
tmpDir, err := os.MkdirTemp("", "five-build-*")
if err != nil {
fatal("failed to create temp dir: " + err.Error())
}
if os.Getenv("FIVE_KEEP_BUILD") == "" {
defer os.RemoveAll(tmpDir)
} else {
fmt.Fprintln(os.Stderr, "[FIVE_KEEP_BUILD] keeping:", tmpDir)
}
// Phase 1: Parse all files and collect cross-file function names
type parsedFile struct {
file *ast.File
prgFile string
}
var parsed []parsedFile
crossFileFuncs := make(map[string]bool)
for _, prgFile := range prgFiles {
f := parsePRGFile(prgFile)
parsed = append(parsed, parsedFile{file: f, prgFile: prgFile})
for _, d := range f.Decls {
switch decl := d.(type) {
case *ast.FuncDecl:
crossFileFuncs[strings.ToUpper(decl.Name)] = true
case *ast.ClassDecl:
crossFileFuncs[strings.ToUpper(decl.Name)] = true
}
}
}
// Phase 2: Analyze and generate each file with cross-file function awareness
for i, pf := range parsed {
diags := analyzer.Analyze(pf.file, crossFileFuncs)
for _, d := range diags {
if d.Severity <= analyzer.SevWarning {
fmt.Fprintf(os.Stderr, "%s\n", d)
}
}
var goCode string
if i > 0 {
goCode = gengo.GenerateLibrary(pf.file)
} else {
goCode = gengo.Generate(pf.file)
}
goFile := fmt.Sprintf("prg_%d.go", i)
writeFile(filepath.Join(tmpDir, goFile), goCode)
}
// Write go.mod
fiveRoot := findFiveRoot()
goMod := fmt.Sprintf("module five-generated\n\ngo 1.21\n\nrequire five v0.0.0\n\nreplace five => %s\n", fiveRoot)
writeFile(filepath.Join(tmpDir, "go.mod"), goMod)
// go mod tidy
tidy := exec.Command(goPath(), "mod", "tidy")
tidy.Dir = tmpDir
tidy.Stderr = os.Stderr
if err := tidy.Run(); err != nil {
fmt.Fprintf(os.Stderr, "warning: go mod tidy: %v\n", err)
}
// go build
absOutput, err := filepath.Abs(output)
if err != nil {
fatal("cannot resolve output path: " + err.Error())
}
cmd := exec.Command(goPath(), buildArgs(absOutput)...)
cmd.Dir = tmpDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
fatal("build failed: " + err.Error())
}
fmt.Fprintf(os.Stderr, "Built: %s\n", absOutput)
}
// Global user include dirs (set by build command, used by compilePRGMode)
var userIncludeDirs []string
// buildPRGWithIncludes is buildPRG with -I support.
func buildPRGWithIncludes(prgFile, output string, includes []string) {
userIncludeDirs = includes
buildPRG(prgFile, output)
}
// buildMultiPRGWithIncludes is buildMultiPRG with -I support.
func buildMultiPRGWithIncludes(prgFiles []string, output string, includes []string) {
// Multi-file builds bring along sibling PRGs whose own #include
// references a .ch file living next to them (e.g. FiveSqlDef.ch
// in _FiveSql2/src/). Each file's PP only adds its OWN dir by
// default, so a test under _FiveSql2/test/ couldn't find a .ch
// kept in _FiveSql2/src/. Promote every input file's dir into
// the shared user-include list so siblings can resolve each
// other's headers.
seen := map[string]bool{}
for _, dir := range includes {
seen[dir] = true
}
for _, f := range prgFiles {
dir := filepath.Dir(f)
if dir != "" && !seen[dir] {
seen[dir] = true
includes = append(includes, dir)
}
}
userIncludeDirs = includes
buildMultiPRG(prgFiles, output)
}
// parsePRGFile preprocesses and parses a PRG file, returning the AST.
func parsePRGFile(prgFile string) *ast.File {
source, err := os.ReadFile(prgFile)
if err != nil {
fatal("cannot read file: " + err.Error())
}
pre := pp.New()
pre.AddIncludeDir(filepath.Dir(prgFile))
pre.AddIncludeDir(filepath.Join(filepath.Dir(prgFile), "include"))
fiveRoot := findFiveRoot()
pre.AddIncludeDir(filepath.Join(fiveRoot, "include"))
if exePath, err := os.Executable(); err == nil {
pre.AddIncludeDir(filepath.Join(filepath.Dir(exePath), "include"))
}
for _, dir := range userIncludeDirs {
pre.AddIncludeDir(dir)
}
if hbInc := os.Getenv("HB_INC"); hbInc != "" {
pre.AddIncludeDir(hbInc)
}
for _, p := range []string{"/usr/local/include/harbour", "/usr/include/harbour"} {
if _, err := os.Stat(p); err == nil {
pre.AddIncludeDir(p)
}
}
processed, ppErrors := pre.Process(prgFile, string(source))
for _, e := range ppErrors {
fmt.Fprintf(os.Stderr, "pp: %s\n", e)
}
if len(ppErrors) > 0 {
fatal(fmt.Sprintf("%d preprocessor error(s) in %s", len(ppErrors), prgFile))
}
file, errs := parser.ParseWithGoDumps(prgFile, processed, pre.GoDumps)
if len(errs) > 0 {
for _, e := range errs {
fmt.Fprintf(os.Stderr, "%s\n", e)
}
fatal(fmt.Sprintf("%d parse error(s) in %s", len(errs), prgFile))
}
return file
}
// compilePRGMode compiles with library flag support.
func compilePRGMode(prgFile string, isLibrary bool) string {
source, err := os.ReadFile(prgFile)
if err != nil {
fatal("cannot read file: " + err.Error())
}
pre := pp.New()
pre.AddIncludeDir(filepath.Dir(prgFile))
pre.AddIncludeDir(filepath.Join(filepath.Dir(prgFile), "include"))
// Five's own include directory (from project root) — must come BEFORE Harbour system includes
fiveRoot := findFiveRoot()
pre.AddIncludeDir(filepath.Join(fiveRoot, "include"))
if exePath, err := os.Executable(); err == nil {
pre.AddIncludeDir(filepath.Join(filepath.Dir(exePath), "include"))
}
// User-specified -I directories — before Harbour system includes
for _, dir := range userIncludeDirs {
pre.AddIncludeDir(dir)
}
// Harbour include paths from environment or standard locations (LAST)
if hbInc := os.Getenv("HB_INC"); hbInc != "" {
pre.AddIncludeDir(hbInc)
}
for _, p := range []string{"/usr/local/include/harbour", "/usr/include/harbour"} {
if _, err := os.Stat(p); err == nil {
pre.AddIncludeDir(p)
}
}
processed, ppErrors := pre.Process(prgFile, string(source))
for _, e := range ppErrors {
fmt.Fprintf(os.Stderr, "pp: %s\n", e)
}
if len(ppErrors) > 0 {
fatal(fmt.Sprintf("%d preprocessor error(s) in %s", len(ppErrors), prgFile))
}
file, errs := parser.ParseWithGoDumps(prgFile, processed, pre.GoDumps)
if len(errs) > 0 {
for _, e := range errs {
fmt.Fprintf(os.Stderr, "%s\n", e)
}
fatal(fmt.Sprintf("%d parse error(s) in %s", len(errs), prgFile))
}
// Semantic analysis (warnings — non-fatal)
diags := analyzer.Analyze(file)
for _, d := range diags {
if d.Severity <= analyzer.SevWarning {
fmt.Fprintf(os.Stderr, "%s\n", d)
}
}
if isLibrary {
return gengo.GenerateLibrary(file)
}
return gengo.Generate(file)
}
// compilePRG preprocesses, parses, and generates Go source.
func compilePRG(prgFile string) string {
source, err := os.ReadFile(prgFile)
if err != nil {
fatal("cannot read file: " + err.Error())
}
// Phase 1: Preprocessor (#include, #define, #ifdef)
pre := pp.New()
pre.AddIncludeDir(filepath.Dir(prgFile))
pre.AddIncludeDir(filepath.Join(filepath.Dir(prgFile), "include"))
// Five's own include directory — MUST come before Harbour system includes
fiveRoot := findFiveRoot()
pre.AddIncludeDir(filepath.Join(fiveRoot, "include"))
if exePath, err := os.Executable(); err == nil {
pre.AddIncludeDir(filepath.Join(filepath.Dir(exePath), "include"))
}
// User-specified -I directories
for _, dir := range userIncludeDirs {
pre.AddIncludeDir(dir)
}
// Harbour include paths (LAST — Five's own headers take priority)
if harbourInc := os.Getenv("HB_INC"); harbourInc != "" {
pre.AddIncludeDir(harbourInc)
}
for _, p := range []string{"/usr/local/include/harbour", "/usr/include/harbour"} {
if _, err := os.Stat(p); err == nil {
pre.AddIncludeDir(p)
}
}
processed, ppErrors := pre.Process(prgFile, string(source))
for _, e := range ppErrors {
fmt.Fprintf(os.Stderr, "pp: %s\n", e)
}
if len(ppErrors) > 0 {
fatal(fmt.Sprintf("%d preprocessor error(s) in %s", len(ppErrors), prgFile))
}
// Phase 2: Parse
file, errs := parser.ParseWithGoDumps(prgFile, processed, pre.GoDumps)
if len(errs) > 0 {
for _, e := range errs {
fmt.Fprintf(os.Stderr, "%s\n", e)
}
fatal(fmt.Sprintf("%d parse error(s)", len(errs)))
}
// Phase 2.5: Analyze (warnings — non-fatal)
diags := analyzer.Analyze(file)
for _, d := range diags {
if d.Severity <= analyzer.SevWarning {
fmt.Fprintf(os.Stderr, "%s\n", d)
}
}
// Phase 3: Generate
return gengo.Generate(file)
}
// writeGoProject creates a temporary Go project with the generated code.
func writeGoProject(dir, prgFile, goCode string) {
// Find the Five module path (where go.mod is)
fiveRoot := findFiveRoot()
// go.mod with replace directive pointing to Five source
goMod := fmt.Sprintf(`module five-generated
go 1.21
require five v0.0.0
replace five => %s
`, fiveRoot)
writeFile(filepath.Join(dir, "go.mod"), goMod)
writeFile(filepath.Join(dir, "main.go"), goCode)
// Run go mod tidy to resolve dependencies
tidy := exec.Command(goPath(), "mod", "tidy")
tidy.Dir = dir
tidy.Stderr = os.Stderr
if err := tidy.Run(); err != nil {
// Non-fatal — some environments may not need it
_ = err
}
}
// findFiveRoot finds the root directory of the Five project (where go.mod is).
// findFiveRoot finds the Five project root (directory containing go.mod with "module five").
// Tries: 1) walk up from cwd, 2) walk up from executable. Fallback: cwd.
func findFiveRoot() string {
if root := walkUpForGoMod(""); root != "" {
return root
}
if exe, err := os.Executable(); err == nil {
if root := walkUpForGoMod(filepath.Dir(exe)); root != "" {
return root
}
}
abs, _ := filepath.Abs(".")
return abs
}
// walkUpForGoMod walks up from startDir (or cwd if empty) looking for go.mod.
func walkUpForGoMod(startDir string) string {
dir := startDir
if dir == "" {
dir, _ = os.Getwd()
}
for {
modPath := filepath.Join(dir, "go.mod")
if _, err := os.Stat(modPath); err == nil {
return dir
}
parent := filepath.Dir(dir)
if parent == dir {
break
}
dir = parent
}
return ""
}
// goPath is an alias for findGoBin (deduplicated).
func goPath() string { return findGoBin() }
// pgoArgs returns ["-pgo=<path>"] when the Five project root contains a
// default.pgo file — profile-guided compilation. Empty otherwise, so
// builds proceed without PGO when the profile hasn't been collected.
// The FIVE_NO_PGO env var forces it off (useful when collecting a new
// profile or A/B benchmarking).
func pgoArgs() []string {
if os.Getenv("FIVE_NO_PGO") != "" {
return nil
}
root := findFiveRoot()
p := filepath.Join(root, "default.pgo")
if fi, err := os.Stat(p); err == nil && !fi.IsDir() && fi.Size() > 0 {
return []string{"-pgo=" + p}
}
return nil
}
// buildArgs composes the full args for a `go build` invocation,
// inserting -pgo when a profile is available.
func buildArgs(output string) []string {
args := []string{"build"}
args = append(args, pgoArgs()...)
return append(args, "-o", output, ".")
}
func writeFile(path, content string) {
if err := os.WriteFile(path, []byte(content), 0644); err != nil {
fatal("cannot write " + path + ": " + err.Error())
}
}
// buildFRB compiles a PRG to Five Runtime Binary (.frb).
// FRB = Go plugin (.so) wrapped in FRB header for runtime loading.
func buildFRB(prgFile, outputFile string) {
source, err := os.ReadFile(prgFile)
if err != nil {
fatal("cannot read file: " + err.Error())
}
// Phase 1: Preprocess
pre := pp.New()
pre.AddIncludeDir(filepath.Dir(prgFile))
pre.AddIncludeDir(filepath.Join(filepath.Dir(prgFile), "include"))
if exePath, err := os.Executable(); err == nil {
pre.AddIncludeDir(filepath.Join(filepath.Dir(exePath), "include"))
}
processed, ppErrors := pre.Process(prgFile, string(source))
for _, e := range ppErrors {
fmt.Fprintf(os.Stderr, "pp: %s\n", e)
}
if len(ppErrors) > 0 {
fatal(fmt.Sprintf("%d preprocessor error(s) in %s", len(ppErrors), prgFile))
}
// Phase 2: Parse
file, parseErrors := parser.ParseWithGoDumps(prgFile, processed, pre.GoDumps)
if len(parseErrors) > 0 {
for _, e := range parseErrors {
fmt.Fprintf(os.Stderr, "%s\n", e)
}
fatal(fmt.Sprintf("%d parse error(s)", len(parseErrors)))
}
// Phase 3: Generate Go source as library
goSrc := gengo.GenerateLibrary(file)
// Phase 4: Build as Go plugin
tmpDir, err := os.MkdirTemp("", "five-frb-*")
if err != nil {
fatal("cannot create temp dir: " + err.Error())
}
if os.Getenv("FIVE_KEEP_BUILD") == "" {
defer os.RemoveAll(tmpDir)
} else {
fmt.Fprintln(os.Stderr, "[FIVE_KEEP_BUILD] keeping:", tmpDir)
}
// Write go.mod — point to Five's module root
fiveRoot := mustAbs(".")
if exePath, err := os.Executable(); err == nil {
fiveRoot = filepath.Dir(exePath)
}
// Find go.mod in fiveRoot or parent directories
for d := fiveRoot; ; d = filepath.Dir(d) {
if _, err := os.Stat(filepath.Join(d, "go.mod")); err == nil {
fiveRoot = d
break
}
if d == filepath.Dir(d) {
break
}
}
modName := fmt.Sprintf("frbmod_%d_%d", os.Getpid(), time.Now().UnixNano())
goModContent := fmt.Sprintf("module %s\n\ngo 1.21.13\n\nrequire five v0.0.0\n\nreplace five => %s\n", modName, fiveRoot)
writeFile(filepath.Join(tmpDir, "go.mod"), goModContent)
// Write Go source (no additional exports needed — init() handles registration)
writeFile(filepath.Join(tmpDir, "module.go"), goSrc)
// Find go binary
goBin := findGoBin()
// go mod tidy
tidyCmd := exec.Command(goBin, "mod", "tidy")
tidyCmd.Dir = tmpDir
if output, err := tidyCmd.CombinedOutput(); err != nil {
fmt.Fprintf(os.Stderr, "tidy error: %s\n%s\n", err, string(output))
fatal("go mod tidy failed")
}
// Build plugin
soFile := filepath.Join(tmpDir, "module.so")
buildCmd := exec.Command(goBin, "build", "-buildmode=plugin", "-o", soFile, ".")
buildCmd.Dir = tmpDir
if output, err := buildCmd.CombinedOutput(); err != nil {
fmt.Fprintf(os.Stderr, "%s\n", string(output))
fatal("plugin build failed")
}
// Phase 5: Package as FRB
soData, err := os.ReadFile(soFile)
if err != nil {
fatal("cannot read .so: " + err.Error())
}
frbData := make([]byte, hbrt.FrbHeaderSize+len(soData))
frbData[0] = hbrt.FrbMagic0
frbData[1] = hbrt.FrbMagic1
frbData[2] = hbrt.FrbMagic2
frbData[3] = hbrt.FrbMagic3
frbData[4] = hbrt.FrbVersion1
frbData[5] = 0 // flags
// Flags: 0
frbData[6] = 0
frbData[7] = 0
// SymCount: 0 (symbols are self-registering via init())
frbData[8] = 0
frbData[9] = 0
frbData[10] = 0
frbData[11] = 0
copy(frbData[12:], soData)
if err := os.WriteFile(outputFile, frbData, 0755); err != nil {
fatal("cannot write FRB: " + err.Error())
}
fmt.Fprintf(os.Stderr, "FRB: %s (%d bytes)\n", outputFile, len(frbData))
}
// debugPRG is kept as a thin wrapper for backward-compatibility — no
// pre-launch breakpoints, TUI frontend.
func debugPRG(prgFile string) { debugPRGWithOpts(prgFile, nil, nil, false) }
// parseBPSpec parses "[module:]line" into (module, line, ok). A bare
// "42" uses defaultMod. Colons inside module (Windows paths) aren't
// supported — use forward slashes.
func parseBPSpec(spec, defaultMod string) (string, int, bool) {
mod := defaultMod
lineStr := spec
if i := strings.LastIndex(spec, ":"); i > 0 {
mod = spec[:i]
lineStr = spec[i+1:]
}
n, err := strconv.Atoi(strings.TrimSpace(lineStr))
if err != nil || n <= 0 {
return "", 0, false
}
return mod, n, true
}
// debugPRGWithOpts compiles PRG with debug info and runs with interactive
// debugger. breakpoints is a list of "[module:]line" strings (module
// defaults to the PRG's basename). watches is a list of PRG expressions
// auto-evaluated at each stop. useCLI picks the gdb-style CLI frontend
// instead of the full-screen TUI.
func debugPRGWithOpts(prgFile string, breakpoints, watches []string, useCLI bool) {
source, err := os.ReadFile(prgFile)
if err != nil {
fatal("cannot read file: " + err.Error())
}
// Phase 1: Preprocess
pre := pp.New()
pre.AddIncludeDir(filepath.Dir(prgFile))
pre.AddIncludeDir(filepath.Join(filepath.Dir(prgFile), "include"))
if exePath, err := os.Executable(); err == nil {
pre.AddIncludeDir(filepath.Join(filepath.Dir(exePath), "include"))
}
processed, ppErrors := pre.Process(prgFile, string(source))
for _, e := range ppErrors {
fmt.Fprintf(os.Stderr, "pp: %s\n", e)
}
if len(ppErrors) > 0 {
fatal(fmt.Sprintf("%d preprocessor error(s) in %s", len(ppErrors), prgFile))
}
// Phase 2: Parse
file, parseErrors := parser.ParseWithGoDumps(prgFile, processed, pre.GoDumps)
if len(parseErrors) > 0 {
for _, e := range parseErrors {
fmt.Fprintf(os.Stderr, "%s\n", e)
}
fatal(fmt.Sprintf("%d parse error(s)", len(parseErrors)))
}
// Phase 3: Generate Go with debug info
goSrc := gengo.GenerateWithDebug(file)
// Phase 4: Build and run (same as runPRG but with debug setup)
tmpDir, err := os.MkdirTemp("", "five-debug-*")
if err != nil {
fatal("cannot create temp dir: " + err.Error())
}
if os.Getenv("FIVE_KEEP_BUILD") == "" {
defer os.RemoveAll(tmpDir)
} else {
fmt.Fprintln(os.Stderr, "[FIVE_KEEP_BUILD] keeping:", tmpDir)
}
fiveRoot := findProjectRoot()
goMod := fmt.Sprintf("module five-generated\n\ngo 1.21.13\n\nrequire five v0.0.0\n\nreplace five => %s\n", fiveRoot)
writeFile(filepath.Join(tmpDir, "go.mod"), goMod)
// Build the debug setup: create Debugger, register any pre-launch
// breakpoints + watches, choose frontend, then run.
callback := "hbrt.TUIDebugger()"
if useCLI {
callback = "hbrt.CLIDebugger()"
}
prgBase := filepath.Base(prgFile)
var setupLines []string
for _, spec := range breakpoints {
mod, line, ok := parseBPSpec(spec, prgBase)
if !ok {
fatal(fmt.Sprintf("invalid -b %q — expected [module:]line", spec))
}
setupLines = append(setupLines,
fmt.Sprintf("vm.Debugger.AddBreakpoint(%q, %d)", mod, line))
}
for _, w := range watches {
setupLines = append(setupLines,
fmt.Sprintf("vm.Debugger.Watches = append(vm.Debugger.Watches, %q)", w))
}
// If any breakpoints were set, start in Continue mode so the program
// runs until it hits one. Otherwise keep step-line (legacy behavior).
startMode := "hbrt.DbgStepLine"
if len(breakpoints) > 0 {
startMode = "hbrt.DbgContinue"
}
debugSetup := fmt.Sprintf(
"vm.Debugger = hbrt.NewDebugger()\n\tvm.Debugger.SourceDir = %s\n\tvm.Debugger.Mode = %s\n\tvm.Debugger.Callback = %s\n\t%s\n\tvm.Run(\"MAIN\")",
fmt.Sprintf("%q", mustAbs(".")),
startMode,
callback,
strings.Join(setupLines, "\n\t"))
goSrc = strings.Replace(goSrc, "vm.Run(\"MAIN\")", debugSetup, 1)
// Remove unused fmt import if added
// (no longer needed since we don't use fmt.Println in generated code)
writeFile(filepath.Join(tmpDir, "main.go"), goSrc)
goBin := findGoBin()
tidyCmd := exec.Command(goBin, "mod", "tidy")
tidyCmd.Dir = tmpDir
if out, err := tidyCmd.CombinedOutput(); err != nil {
fmt.Fprintf(os.Stderr, "warning: go mod tidy: %v\n%s", err, out)
}
runCmd := exec.Command(goBin, "run", ".")
runCmd.Dir = tmpDir
runCmd.Stdin = os.Stdin
runCmd.Stdout = os.Stdout
runCmd.Stderr = os.Stderr
runCmd.Run()
}
// findProjectRoot is an alias for findFiveRoot (deduplicated).
func findProjectRoot() string { return findFiveRoot() }
// buildFRBPcode compiles PRG to pcode FRB (no Go compiler needed to run).
func buildFRBPcode(prgFile, outputFile string) {
source, err := os.ReadFile(prgFile)
if err != nil {
fatal("cannot read file: " + err.Error())
}
// Phase 1: Preprocess
pre := pp.New()
pre.AddIncludeDir(filepath.Dir(prgFile))
pre.AddIncludeDir(filepath.Join(filepath.Dir(prgFile), "include"))
if exePath, err := os.Executable(); err == nil {
pre.AddIncludeDir(filepath.Join(filepath.Dir(exePath), "include"))
}
processed, _ := pre.Process(prgFile, string(source))
// Phase 2: Parse
file, parseErrors := parser.ParseWithGoDumps(prgFile, processed, pre.GoDumps)
if len(parseErrors) > 0 {
for _, e := range parseErrors {
fmt.Fprintf(os.Stderr, "%s\n", e)
}
fatal(fmt.Sprintf("%d parse error(s)", len(parseErrors)))
}
// Phase 3: Generate pcode
pcMod := genpc.Generate(file)
// Surface any genpc compile-time diagnostics (unsupported AST
// nodes etc.) so users learn their PRG isn't fully pcode-
// compilable instead of getting silent wrong results at run
// time. These are warnings, not errors — emission continues.
for _, w := range pcMod.Warnings {
fmt.Fprintln(os.Stderr, w)
}
// Phase 4: Serialize
pcData := hbrt.SerializePcodeModule(pcMod)
// Phase 5: Write FRB
frbData := make([]byte, hbrt.FrbHeaderSize+len(pcData))
frbData[0] = hbrt.FrbMagic0
frbData[1] = hbrt.FrbMagic1
frbData[2] = hbrt.FrbMagic2
frbData[3] = hbrt.FrbMagic3
frbData[4] = hbrt.FrbVersion1
frbData[5] = 0
frbData[6] = hbrt.FrbModePcode
frbData[7] = 0
// func count
frbData[8] = byte(len(pcMod.Funcs))
frbData[9] = byte(len(pcMod.Funcs) >> 8)
frbData[10] = 0
frbData[11] = 0
frbData = append(frbData[:12], pcData...)
if err := os.WriteFile(outputFile, frbData, 0755); err != nil {
fatal("cannot write FRB: " + err.Error())
}
fmt.Fprintf(os.Stderr, "FRB (pcode): %s (%d bytes, %d functions)\n", outputFile, len(frbData), len(pcMod.Funcs))
}
func findGoBin() string {
// Try PATH first
if p, err := exec.LookPath("go"); err == nil {
return p
}
// Common locations
for _, p := range []string{"/usr/local/go/bin/go", "/usr/bin/go", "/snap/bin/go"} {
if _, err := os.Stat(p); err == nil {
return p
}
}
return "go"
}
func mustAbs(path string) string {
abs, err := filepath.Abs(path)
if err != nil {
return path
}
return abs
}
func fatal(msg string) {
fmt.Fprintf(os.Stderr, "five: %s\n", msg)
os.Exit(1)
}