e83787750aaa01f7e8b9c82f5d1b97a02d4edf61
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 5bba0c2dae |
refactor(FiveSql2): per-session aOuterStack/hDmlPcodeCache/lCteDiskSeen
Continues the multi-session concurrency cleanup. Phase 1 moved
the visible txn + plan-cache state onto TSqlSession; this pass
takes the next batch of "shared by accident" STATICs that
surfaced as Go-level `concurrent map writes` panics under
5-worker pgserver load:
s_aOuterStack — subquery-nesting stack
s_hDmlPcodeCache — DML pcode cache (schema-version keyed)
s_lCteDiskSeen — CTE-materialised-to-DBF flag
Each is now a DATA field on TSqlSession, initialised in New().
TSqlExecutor's 25 access sites (sed-rewritten under inspection)
now route through `::oSession:fieldname`. The standalone
`SqlDmlPcodeCacheReset()` helper keeps a backward-compatible
signature: callers may pass an explicit oSession, otherwise it
falls back to SqlDefaultSession() (preserves embedded-mode
ergonomics).
Remaining STATICs in TSqlExecutor.prg (s_nSchemaVer, s_nRCJSeq,
s_hAutoInc) are cross-session-shared by design — schema-version
bumps must invalidate every peer's plan cache, RCJ alias
sequence needs cross-connection uniqueness, and IDENTITY columns
must hand out monotonically increasing values across all writers
into the same table. Those need atomic / mutex guards rather
than per-session ownership; tracked as a follow-up.
Measured impact on the pgserver stress harness (20 runs each):
3-worker 5-worker
Layer 1+2: 16/20 (80%) 10/20 (50%)
+3a: 16/20 (80%) 10/20 (50%)
+THIS: 18/20 (90%) 16/20 (80%)
The remaining flake comes from s_hAutoInc's lazy map init under
concurrent IDENTITY-table writers and a few interleavings of the
header max-merge path. Both are tractable with the planned
atomic / mutex shims and the multi-area mmap-gen registry; both
deferred to the follow-up commit to keep this diff focused on
the move-to-session pattern.
All six release gates green:
go test ./... ✓
FiveSql2 SQL:1999 43/43 ✓
Harbour compat 56/56 ✓
std.ch 17/17 ✓
FRB 7/7 ✓
pgserver integration 6/6 ✓
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| 93cf5c8bfa |
refactor(FiveSql2): per-session state — TSqlSession isolates txn + plan cache
Foundation for the upcoming PostgreSQL-wire server. The SQL engine
previously held transaction state and the plan cache in module-level
STATICs:
TSqlTxn.prg:16-18
STATIC s_aTxnLog := {}
STATIC s_lInTxn := .F.
STATIC s_hSavepoints := NIL
TFiveSQL.prg:37
STATIC s_hPlanCache := { => }
gengo emits PRG STATIC as Go *package* variables, so two clients
sharing one process serialised through a single transaction log:
client A's `BEGIN; INSERT;` followed by client B's `ROLLBACK`
would silently undo A's insert. Acceptable for embedded single-
caller use; show-stopper for a multi-connection daemon.
Moved each of those into instance fields on a new TSqlSession class.
Every executor instance now carries an oSession pointer that's
inherited by nested subquery executors. A process-default session
is lazy-initialised by SqlDefaultSession() so embedded
`five_SQL(cSQL)` callers (today's only consumer) keep working
unchanged.
Changes
-------
* `_FiveSql2/src/TSqlSession.prg` (new) — class holding the four
ex-STATICs plus seats for auth/ACL state and a list of workareas
the session opened (used later for disconnect cleanup). Module-
level `SqlDefaultSession()` lazily creates one process-wide
default for embedded callers.
* `_FiveSql2/src/TSqlTxn.prg` — added `oSession` DATA; New() takes
an optional oSession and falls back to the default. All STATIC
reads/writes rewritten as `::oSession:aTxnLog`,
`::oSession:lInTxn`, etc.
* `_FiveSql2/src/TFiveSQL.prg` — added `oSession` DATA; New() takes
an optional second arg. Plan-cache reads/writes route through
`::oSession:hPlanCache`. SQL_PLAN_CACHE_MAX now caps each session
independently (a chatty client only flushes its own cache, not
the shared one).
* `_FiveSql2/src/TSqlExecutor.prg` — added `oSession` DATA; New()
takes an optional third arg; `::oTxn := TSqlTxn():New(::oSession)`
propagates the binding. Every in-class `TSqlExecutor():New(...)`
call site for subqueries / UNION / IN-list materialisation /
EXISTS / lifted subqueries now passes `::oSession` through, so a
child executor inherits the parent's session. Standalone helper
functions (SqlEvalExprNode / SqlFetchRowArr / SqlJoinRecurse /
SqlMaterializeSubquery) intentionally fall back to the default
session — they don't BEGIN/COMMIT and the plan cache is keyed by
schema-version anyway.
* `_FiveSql2/src/FiveSqlCls.prg` — `five_SQL()` gains an optional
fourth arg `oSession`. Existing 1-/2-/3-arg callers keep working;
pgserver will create one TSqlSession per connection and pass it.
Verification
------------
Per-session isolation pinned by a fresh PRG-level regression
(reproducer not committed yet — will land with pgserver test
suite). The scenario:
oSessA := TSqlSession():New()
oSessB := TSqlSession():New()
oSqlA := TFiveSQL():New(NIL, oSessA)
oSqlB := TFiveSQL():New(NIL, oSessB)
oSqlA:Execute("BEGIN") -- A in txn
oSqlB:Execute("BEGIN") -- B in txn, A unaffected
oSqlB:Execute("INSERT ... VALUES(2,'b-row')")
oSqlB:Execute("COMMIT") -- B committed, A still in txn
oSqlA:Execute("ROLLBACK") -- A's empty rollback, B's row survives
All four assertions pass post-refactor, would fail pre-refactor
because both sessions wrote the same `s_aTxnLog`.
All six release gates green:
go test ./... ✓
FiveSql2 SQL:1999 43/43 ✓
Harbour compat 56/56 ✓
std.ch 17/17 ✓
FRB 7/7 ✓
examples 65/71 ✓ (unchanged baseline)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| f4ed42556b |
checkpoint: season-wide bug fix campaign + infra
Cumulative season's silent-bug hunting (~62 fixes) across the FiveSql2 SQL engine, the Five compiler/runtime, and the hbrdd RDD layer. Saved as a single checkpoint before refactoring the parser to delegate xBase command translation to the preprocessor. Highlights: FiveSql2 engine (_FiveSql2/src/) - prefix-glob index attach -> explicit convention (<table>_pk.ntx, <table>_uq.ntx, <table>.cdx) — fixes silent multi-row INSERT row-drop - DROP/CREATE TABLE FErase chain extended (.cdx, .fsc, .fsv, .dbt, .fpt) - COUNT(DISTINCT col) parsed + aggregated via hSeen hash - UNION column-count mismatch returns SQL_ERR_GRAMMAR (was silent) - DISTINCT + ORDER BY hidden-col leak fixed (trim before DISTINCT) - Derived table FROM (SELECT...) + JOIN right-side derived - Self-FK CASCADE depth 2+ via SqlGetSingleColPK pre-collect - LAG/LEAD default arg uses SqlEvalRowExpr (handles -N const exprs) - DATE literal round-trip validation (Feb 29 non-leap rejected) - CREATE OR REPLACE VIEW; CREATE VIEW errors on already-exists - AlterTable type dispatcher comma-wrapped (1-char type "A" no longer matches CHARACTER) Compiler / runtime - gengo: HB_ -> FV_ prefix on emitted Go function names (Five identity) - gengo split: emit_block.go, emit_stmt.go, folding.go extracted - parser/stmtreg.go nudges - hbrt: debug TUI/CLI restructure (debugcmd, debugkey, termios_*), windows debug stubs collapsed - thread/vm/value/class/pcinterp tightening from panic traces RDD layer (hbrdd/) - dbf: null bitmap support (null.go + null_test.go), mmap split (mmap_posix.go / mmap_windows.go), byte-level numeric parse - ntx/cdx: windows mmap parity - workarea + mem RDD: cross-area state-bleed fixes RTL (hbrtl/) - errorlog rewrite with platform-specific FD (errorlog_fd_unix / errorlog_fd_other) - sqlscan, sqlhelpers, indexrtl, datetime extensions Gates green at checkpoint: - go test ./... : PASS - FiveSql2 SQL:1999 : 43/43 - Harbour compat : 56/56 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| dd270d5d9d |
perf: RTL Go-native migration — 27 optimizations, DML up to 70-90x
Systematic pass through PRG hot paths, promoting them to Go RTL while
preserving Harbour/FiveSql2 semantics. Full log in
docs/RTL-Go-Native-Migration.md.
Bench (bench_sql) vs 2026-04-08 baseline
- B1 SELECT * 2,192 → 114 µs (19x)
- B6 INNER JOIN 9,291 → 233 µs (40x)
- B7 CTE simple 8,037 → 129 µs (62x)
- B9 ROW_NUMBER 3,705 → 265 µs (14x)
- B10 RANK PARTITION 4,748 → 309 µs (15x)
- B12 INSERT (WA cache) 4,319 → 63 µs (69x)
- B13 UPDATE (WA cache) 6,144 → 68 µs (90x)
- B15 CTE+WIN+JOIN 18,395 → 1,873 µs (10x)
Infrastructure
- HbHash O(1) Index preserving insertion order (Harbour KEEPORDER)
- HbDeepClone Go RTL (scalar-sharing, immutable hash keys)
- MEMRDD auto-imported via gengo; all Five programs get mem:name driver
- SQL plan + pcode caches (s_hPlanCache, s_hDmlPcodeCache)
- Opt-in SqlWACacheEnable — dbUseArea/Close/Commit batched for DML
SQL engine
- FiveSql2 lexer ported to Go (byte FSM) with combined automatic
template parameterization (literals → ?, concat queries share plan)
- Go RTL: SqlDistinct, SqlGroupRows, SqlWindowPartitions,
SqlWindowSortPartition, SqlWindowAssignRank, SqlComputeAggSimple,
SqlBulkInsert, SqlBulkUpdate, SqlExprHasAgg, SqlEvalHaving
- CTE / subquery / driving-table materialize paths use MEMRDD
- SqlCoerce/SqlCmp/SqlIsTrue helpers moved from PRG to Go
- SqlBulkUpdate defers Flush when WA cache active (APFS fsync was
dominant B13 cost — 1.6ms/call → gone)
Correctness fixes uncovered during migration
- ASort default path now sorts dates/logicals/timestamps (was no-op)
- ORDER BY default NULL placement matches PRG SqlRowCompare across
Go fast path; explicit NULLS FIRST/LAST honored by both paths
- SqlBulkUpdate respects EXCLUSIVE vs SHARED mode record locks
- SqlCmp/SqlCmpEq normalize NumInt vs Double (caught by test 6b)
Verification
- go test ./... ALL PASS
- FiveSql2 test_sql1999 43/43
- tests/compat_harbour 56/56 (+5 new: ASort dates/logicals,
AScan int cross-type)
- Regression test test_null_order.prg for ORDER BY NULL ordering
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| e75167c2e9 |
feat(FiveSql2): five_SQL block-callback integration — SQL beats raw PRG
Wires the new SqlEach RTL into FiveSql2's front-end so users write
the SQL they know and opt into streaming with a familiar Harbour
code block — no manual RTL plumbing.
API:
/* Existing array form — unchanged, 43-test still green */
aR := five_SQL( "SELECT name FROM t" )
/* New block form — zero intermediate rows, 2x raw PRG */
five_SQL( "SELECT id, name FROM t WHERE salary > 50000", NIL,
{|nID, cName| Process(nID, cName)} )
Parameter order (cSQL, aParams, bBlock) keeps backward compatibility
with every existing call site. Passing NIL for aParams when only a
block is needed is standard Harbour idiom.
Routing:
* TFiveSQL:Execute now takes an optional bBlock parameter and
stores it on TSqlExecutor as ::bRowBlock.
* TSqlExecutor:RunSelect's existing Go fast path (same guards as
before: single table, no JOIN/GROUP/aggregate, plain column
projections, WHERE compilable via SqlExprToPrg) branches on
::bRowBlock:
- block present → SqlEach streams rows through the block
- block absent → SqlScan materializes into aRows (current path)
* Post-processing (GROUP BY / ORDER BY / window / DISTINCT / LIMIT)
runs on empty aRows when block mode fires — all are no-ops on
empty input, so the sequence stays harmless.
* RunSelect returns NIL (not {fields, rows}) when ::bRowBlock was
used — signals "streaming semantics, all work done in the block".
Complex queries (JOIN, GROUP BY, subquery, window, ORDER BY not
matchable by an index, LIMIT/OFFSET, etc.) still fall back to the
array path even when a block is supplied — those genuinely require
materialization. Block mode is a fast-path opt-in, not a semantic
change.
End-to-end bench (50k rows, steady state — includes the user-side
loop/block for every row):
Path Time Speedup vs raw
──────────────────────────────────────────────────────────────
Raw PRG DO WHILE !Eof() + WHERE sum 7.6ms 1.00x
five_SQL array + FOR 7.7ms ~same
five_SQL + block (new) 3.7ms 2.05x ← beats raw
──────────────────────────────────────────────────────────────
Raw PRG no WHERE 6.1ms 1.00x
five_SQL + block, no WHERE 2.9ms 2.10x ← beats raw
SQL now pays for itself on end-to-end timing — not just competitive
with hand-rolled RDD loops, but faster than them. The layered cost
of FieldGet's Frame+RTL-dispatch that hand-written loops incur per
call is gone; the block-callback path captures *dbf.DBFArea directly
via FastFieldGetter and uses PcOpFieldGet to bypass dispatch in the
compiled WHERE predicate.
Validation:
- FiveSql2 43/43 (array API unchanged)
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 486e466592 |
feat: FiveSql2 43/43, @byref, mutable closure, RTL 479, DateTime fix
Major changes since last commit: - FiveSql2 SQL:1999 engine (10,458 LOC) — 43/43 ALL PASS - 21 compiler/runtime bugs fixed (short-circuit AND/OR, FOR LOOP, etc.) - @byref pass-by-reference via RefCell pattern - Mutable closure capture (EnsureLocalRef + RefCell sharing) - RTL: 400 → 479 functions (+79: file, string, datetime, hash, UTF-8) - DateTime/Timestamp fully working (hb_DateTime, hb_Hour/Min/Sec, display) - Reserved word guard (39 keywords blocked from function calls) - AEval arg order fix (element before index) - Closure capture redecl fix (unique _cap_ names per block) - Hash/string indexing in ArrayPush/ArrayPop - Harbour compat test suite: 51/51 - 4 docs: Porting Report, Implementation Plan, Optimization Plan, Commercialization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
