5bba0c2daecd4bd5b65d5a4a45c4bd20ec5e1398
35 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 5bba0c2dae |
refactor(FiveSql2): per-session aOuterStack/hDmlPcodeCache/lCteDiskSeen
Continues the multi-session concurrency cleanup. Phase 1 moved
the visible txn + plan-cache state onto TSqlSession; this pass
takes the next batch of "shared by accident" STATICs that
surfaced as Go-level `concurrent map writes` panics under
5-worker pgserver load:
s_aOuterStack — subquery-nesting stack
s_hDmlPcodeCache — DML pcode cache (schema-version keyed)
s_lCteDiskSeen — CTE-materialised-to-DBF flag
Each is now a DATA field on TSqlSession, initialised in New().
TSqlExecutor's 25 access sites (sed-rewritten under inspection)
now route through `::oSession:fieldname`. The standalone
`SqlDmlPcodeCacheReset()` helper keeps a backward-compatible
signature: callers may pass an explicit oSession, otherwise it
falls back to SqlDefaultSession() (preserves embedded-mode
ergonomics).
Remaining STATICs in TSqlExecutor.prg (s_nSchemaVer, s_nRCJSeq,
s_hAutoInc) are cross-session-shared by design — schema-version
bumps must invalidate every peer's plan cache, RCJ alias
sequence needs cross-connection uniqueness, and IDENTITY columns
must hand out monotonically increasing values across all writers
into the same table. Those need atomic / mutex guards rather
than per-session ownership; tracked as a follow-up.
Measured impact on the pgserver stress harness (20 runs each):
3-worker 5-worker
Layer 1+2: 16/20 (80%) 10/20 (50%)
+3a: 16/20 (80%) 10/20 (50%)
+THIS: 18/20 (90%) 16/20 (80%)
The remaining flake comes from s_hAutoInc's lazy map init under
concurrent IDENTITY-table writers and a few interleavings of the
header max-merge path. Both are tractable with the planned
atomic / mutex shims and the multi-area mmap-gen registry; both
deferred to the follow-up commit to keep this diff focused on
the move-to-session pattern.
All six release gates green:
go test ./... ✓
FiveSql2 SQL:1999 43/43 ✓
Harbour compat 56/56 ✓
std.ch 17/17 ✓
FRB 7/7 ✓
pgserver integration 6/6 ✓
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| 67cd8f2306 |
fix(pgserver,dbf): partial fix for multi-session concurrency race
Addresses two of the three layers behind the audit's "WorkArea
collision under multi-session" risk surfaced in Phase 3:
1. Shared DATA-INIT hash literals (PRG side).
TSqlSession.prg declared `DATA hPlanCache INIT { => }` (plus
hSavepoints + hRolePerms etc.). On the gengo path that
compiles class-DATA INITs, the {=>} literal is sometimes
evaluated ONCE at class-definition time, with every
subsequent New() reusing the same hash pointer. Two pgserver
connections then read/wrote a single shared HbHash from
different goroutines, eventually hitting `concurrent map
writes` inside HbHash.ensureIndex (the lazy O(1)-lookup
index map).
The pre-existing gotcha is already documented in
TSqlExecutor.prg's hSubCache comment ("DATA INIT on hash/
array literals can end up sharing the same instance across
New() calls depending on the compile path") — TSqlSession had
missed the same workaround. Moving the explicit
`::hPlanCache := { => }` etc. into the constructor body
guarantees a fresh hash per instance.
2. Stale cross-session recCount cache (Go side).
`*DBFArea.RecCount()` in shared mode caches its result for
the duration of `recCountCacheGen`. Append() bumped the count
on disk + refreshed THIS area's count under the append-intent
lock (Phase 1 of pre-1.0 audit) but never invalidated the
cache on peer DBFArea instances — so a second pgserver
connection's RecCount() kept returning its pre-Append cached
value. The peer's SELECT then iterated 1..old_count and
missed the newly inserted row.
Append() now calls `InvalidateRecCountCache()` after
committing the bumped header. The generation counter went
to atomic.AddUint64 / atomic.LoadUint64 so the bump is
safe to fire from any goroutine without a lock around the
variable.
Measured impact
---------------
Same 3-worker concurrent-INSERT-then-SELECT stress test that was
~3/5 passing pre-fix:
before: 3 / 5 (40% — plus occasional Go-level panic)
after: 8 / 10 (80% — no panics, just intermittent missed rows)
The remaining 20% flake is on the third layer — peer mmap shows a
pre-Append snapshot when Append's `unmap()` only invalidates this
area's own mmap, not the other workareas that opened the same DBF
file independently via dbUseArea. Fixing that requires either a
cross-area registry of mmap views to invalidate, or skipping
mmap entirely when SHARED && cache-gen has bumped. Tracked as a
proper follow-up; tests/pgserver/run.sh's "Known limitation"
header now points at the narrower problem.
Standalone six-gate verification:
go test ./... ✓
FiveSql2 SQL:1999 43/43 ✓
Harbour compat 56/56 ✓
std.ch 17/17 ✓
FRB 7/7 ✓
pgserver integration 6/6 ✓
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| d98f5e1767 |
feat(pgserver): PostgreSQL-wire MVP — psql can SELECT from FiveSql2
First end-to-end working version of the PostgreSQL-wire-compatible
TCP server frontend. A standard `psql` client now connects, runs
`SELECT * FROM employees`, and gets back a properly typed result
set rendered by psql with the right column alignment:
ID | NAME | SALARY
----+----------------------+----------
1 | Alice | 50000.00
2 | Bob | 42000.50
3 | Cho | 77500.00
This is the Phase 2 deliverable from the approved plan at
/Users/charleskwon/.claude/plans/compiled-launching-shore.md.
Builds on the session-state refactor in
|
|||
| 93cf5c8bfa |
refactor(FiveSql2): per-session state — TSqlSession isolates txn + plan cache
Foundation for the upcoming PostgreSQL-wire server. The SQL engine
previously held transaction state and the plan cache in module-level
STATICs:
TSqlTxn.prg:16-18
STATIC s_aTxnLog := {}
STATIC s_lInTxn := .F.
STATIC s_hSavepoints := NIL
TFiveSQL.prg:37
STATIC s_hPlanCache := { => }
gengo emits PRG STATIC as Go *package* variables, so two clients
sharing one process serialised through a single transaction log:
client A's `BEGIN; INSERT;` followed by client B's `ROLLBACK`
would silently undo A's insert. Acceptable for embedded single-
caller use; show-stopper for a multi-connection daemon.
Moved each of those into instance fields on a new TSqlSession class.
Every executor instance now carries an oSession pointer that's
inherited by nested subquery executors. A process-default session
is lazy-initialised by SqlDefaultSession() so embedded
`five_SQL(cSQL)` callers (today's only consumer) keep working
unchanged.
Changes
-------
* `_FiveSql2/src/TSqlSession.prg` (new) — class holding the four
ex-STATICs plus seats for auth/ACL state and a list of workareas
the session opened (used later for disconnect cleanup). Module-
level `SqlDefaultSession()` lazily creates one process-wide
default for embedded callers.
* `_FiveSql2/src/TSqlTxn.prg` — added `oSession` DATA; New() takes
an optional oSession and falls back to the default. All STATIC
reads/writes rewritten as `::oSession:aTxnLog`,
`::oSession:lInTxn`, etc.
* `_FiveSql2/src/TFiveSQL.prg` — added `oSession` DATA; New() takes
an optional second arg. Plan-cache reads/writes route through
`::oSession:hPlanCache`. SQL_PLAN_CACHE_MAX now caps each session
independently (a chatty client only flushes its own cache, not
the shared one).
* `_FiveSql2/src/TSqlExecutor.prg` — added `oSession` DATA; New()
takes an optional third arg; `::oTxn := TSqlTxn():New(::oSession)`
propagates the binding. Every in-class `TSqlExecutor():New(...)`
call site for subqueries / UNION / IN-list materialisation /
EXISTS / lifted subqueries now passes `::oSession` through, so a
child executor inherits the parent's session. Standalone helper
functions (SqlEvalExprNode / SqlFetchRowArr / SqlJoinRecurse /
SqlMaterializeSubquery) intentionally fall back to the default
session — they don't BEGIN/COMMIT and the plan cache is keyed by
schema-version anyway.
* `_FiveSql2/src/FiveSqlCls.prg` — `five_SQL()` gains an optional
fourth arg `oSession`. Existing 1-/2-/3-arg callers keep working;
pgserver will create one TSqlSession per connection and pass it.
Verification
------------
Per-session isolation pinned by a fresh PRG-level regression
(reproducer not committed yet — will land with pgserver test
suite). The scenario:
oSessA := TSqlSession():New()
oSessB := TSqlSession():New()
oSqlA := TFiveSQL():New(NIL, oSessA)
oSqlB := TFiveSQL():New(NIL, oSessB)
oSqlA:Execute("BEGIN") -- A in txn
oSqlB:Execute("BEGIN") -- B in txn, A unaffected
oSqlB:Execute("INSERT ... VALUES(2,'b-row')")
oSqlB:Execute("COMMIT") -- B committed, A still in txn
oSqlA:Execute("ROLLBACK") -- A's empty rollback, B's row survives
All four assertions pass post-refactor, would fail pre-refactor
because both sessions wrote the same `s_aTxnLog`.
All six release gates green:
go test ./... ✓
FiveSql2 SQL:1999 43/43 ✓
Harbour compat 56/56 ✓
std.ch 17/17 ✓
FRB 7/7 ✓
examples 65/71 ✓ (unchanged baseline)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| f4ed42556b |
checkpoint: season-wide bug fix campaign + infra
Cumulative season's silent-bug hunting (~62 fixes) across the FiveSql2 SQL engine, the Five compiler/runtime, and the hbrdd RDD layer. Saved as a single checkpoint before refactoring the parser to delegate xBase command translation to the preprocessor. Highlights: FiveSql2 engine (_FiveSql2/src/) - prefix-glob index attach -> explicit convention (<table>_pk.ntx, <table>_uq.ntx, <table>.cdx) — fixes silent multi-row INSERT row-drop - DROP/CREATE TABLE FErase chain extended (.cdx, .fsc, .fsv, .dbt, .fpt) - COUNT(DISTINCT col) parsed + aggregated via hSeen hash - UNION column-count mismatch returns SQL_ERR_GRAMMAR (was silent) - DISTINCT + ORDER BY hidden-col leak fixed (trim before DISTINCT) - Derived table FROM (SELECT...) + JOIN right-side derived - Self-FK CASCADE depth 2+ via SqlGetSingleColPK pre-collect - LAG/LEAD default arg uses SqlEvalRowExpr (handles -N const exprs) - DATE literal round-trip validation (Feb 29 non-leap rejected) - CREATE OR REPLACE VIEW; CREATE VIEW errors on already-exists - AlterTable type dispatcher comma-wrapped (1-char type "A" no longer matches CHARACTER) Compiler / runtime - gengo: HB_ -> FV_ prefix on emitted Go function names (Five identity) - gengo split: emit_block.go, emit_stmt.go, folding.go extracted - parser/stmtreg.go nudges - hbrt: debug TUI/CLI restructure (debugcmd, debugkey, termios_*), windows debug stubs collapsed - thread/vm/value/class/pcinterp tightening from panic traces RDD layer (hbrdd/) - dbf: null bitmap support (null.go + null_test.go), mmap split (mmap_posix.go / mmap_windows.go), byte-level numeric parse - ntx/cdx: windows mmap parity - workarea + mem RDD: cross-area state-bleed fixes RTL (hbrtl/) - errorlog rewrite with platform-specific FD (errorlog_fd_unix / errorlog_fd_other) - sqlscan, sqlhelpers, indexrtl, datetime extensions Gates green at checkpoint: - go test ./... : PASS - FiveSql2 SQL:1999 : 43/43 - Harbour compat : 56/56 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| 325fe51656 |
fix(fivesql2): DML transaction + constraint ordering
Three correctness bugs in the DML executor that the 4.7 audit
surfaced:
1. RunInsert logged the transaction BEFORE dbAppend() and validation.
LogRecord captured the PREVIOUS row's RecNo, and a CHECK/FK
violation that rolled back via dbDelete() still left a spurious
INSERT entry in the log pointing at the wrong record. Move
LogRecord to after all field puts and all validators pass, so
the log only records committed INSERTs at the correct RecNo.
2. RunUpdate (fallback path) skipped CHECK and FK validation entirely
— only RunInsert validated. An UPDATE could violate the same
constraints INSERT protects against. Add the same validator calls
after FieldPut, with a captured aPrevVals snapshot so the in-
memory record can roll back cleanly on failure. Gated by
SqlLoadConstraints to skip the validator (and its recursive
five_SQL) for tables without SQL-level metadata — tables created
via plain dbCreate see no change.
3. RunDelete had no transaction logging at all — a BEGIN / DELETE /
ROLLBACK cycle silently lost the row. Add LogRecord("DELETE")
before dbDelete so undo can re-surface it. (A full FK-cascade
check on delete would require parent→child scanning; deferred.)
The fast-path SqlBulkUpdate branch still bypasses per-record
validation by design (documented) — it's gated by
`! ::oTxn:IsActive()`, so txn-active queries always take the
validated fallback.
FiveSql2 43/43 (including SAVEPOINT + ROLLBACK TO and all four CHECK/
FK tests), Harbour compat 56/56, Go test ALL PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| e368402682 |
chore: audit cleanup — remove orphan parser + dead TSqlIndex methods
Opus 4.7 audit of the codebase surfaced several items that Opus 4.6
sessions left behind. This pass removes what's definitively dead and
fixes one trivial defensive bug; the real logic bugs (transaction
ordering, missing RunUpdate/RunDelete validation) come in a separate
commit.
Deletions:
- `_FiveSql2/src/TSqlParser_orig.prg` (1173 lines) — superseded by
`TSqlParser2.prg` (Pratt). Production never instantiates the old
parser; the only callers were the comparison/benchmark test files
also being removed.
- `_FiveSql2/test/test_parser_cmp.prg` — compared orig vs Pratt AST,
useless now that orig is gone.
- `_FiveSql2/test/bench_parser.prg` — benched both, same reason.
- `_FiveSql2/Makefile` `test_cmp:` and `bench:` targets referenced
the removed files.
- `TSqlIndex.prg` methods `ApplyScope`, `ClearScope`, `ApplySeek`,
`IndexInfo`, `CreateTempIndex`, `DropTempIndex` — each declared in
the class header and implemented (~165 lines total) but zero
callers anywhere in `_FiveSql2/` or `hbrtl/`. Class declarations
removed alongside the bodies.
Small fixes:
- `TSqlDDL.prg:179-180` stale comment claiming Five doesn't support
`@byref` — false since commit
|
|||
| c4ae88e76e |
perf(fivesql2): gate CTE __cte_*.dbf cleanup on legacy disk fallback
CTE tables now materialise via MEMRDD (no file on disk), yet the RunSelect cleanup loop was still stat-ing __cte_<name>.dbf for every CTE in every CTE query. Profile after the FetchRow rewrite pinned HbFileExists at 20.28% of total CPU — pure waste when MEMRDD is the common path. Add s_lCteDiskSeen flag, set only when the legacy DBFNTX fallback in RunSelect actually opens a pre-existing __cte_<name>.dbf (line 1247 path — rare, only for sub-executors referencing a CTE by name on a crashed-prior-run .dbf). Cleanup runs only when the flag is set. pprof delta (full bench with cache enabled): rawsyscalln: 25.56% → 8.50% (~17 points removed) HbFileExists: 20.28% → 0% (dropped out of top) Wall-clock unchanged (ENOENT stats are kernel-cached on Darwin), but this removes the last visible avoidable syscall. What's left in the profile (kevent, madvise, pthread_cond_*) is Go runtime + scheduler overhead that application code can't touch. FiveSql2 43/43, Harbour compat 56/56. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| 935883bb88 |
perf(fivesql2): Go-native FetchRow fast path — 1.3-1.7x on agg/window
TSqlExecutor:FetchRow was the per-row workhorse for aggregation, HAVING, and window queries. Even with the pre-built aFetchCache binding columns to (nWA, nFPos), the PRG FOR loop paid one method dispatch per column per row (dbSelectArea, FieldGet, AllTrim, AAdd) — profile pinned it at ~30% of B4 CPU. SqlFetchRowFast collapses the cache-path loop into a single Go call: - bound entry: SelectByNum + area.GetValue directly - unbound (aggregate/expression): self:EvalExpr via Send - character values: TrimSpace inline The PRG FetchRow keeps its original cache-miss fallback path unchanged for rare queries where aFetchCache isn't built. Bench deltas (median of 3 steady runs, 1000 iters): B4_GROUP_HAVING 418 → 327 us -22% (1.28x) B9_ROW_NUMBER 191 → 120 us -37% (1.59x) B10_RANK_PART 228 → 135 us -41% (1.69x) B11_SUM_OVER 249 → 156 us -37% (1.60x) B14_COUNT 235 → 219 us -7% B15_CTE_WIN_JOIN 1577 → 1452 us -8% Single-table SELECT (B1-B3, B5-B7, B8) stays flat — those already hit the column-binding fast path and don't need aggregate dispatch. FiveSql2 43/43, Harbour compat 56/56. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| c84cde6175 |
perf(fivesql2): Go-native SqlIsAggName — drop per-row substring scan
B4 GROUP+HAVING profile showed SqlIsAggName at ~9% of CPU —
SqlEvalFunc checks it for every function in every row, and the
PRG body was two string allocations + a substring scan:
RETURN ("," + c + ",") $ ("," + AGG_FUNCTIONS + ",")
Replace with a hash lookup against the existing aggFuncSet map
in hbrtl/sqlexpr.go (already populated for SqlExprHasAgg, same
AGG_FUNCTIONS list). Upper-casing skips the allocation when the
input is already upper, which it almost always is in practice.
Bench deltas (median of 3 steady runs, 1000 iters):
B4_GROUP_HAVING 447 → 418 us -6.5%
B14_COUNT 252 → 235 us -7%
B15_CTE_WIN_JOIN 1595 → 1577 us -1%
Other benches unchanged (no aggregate calls per row).
FiveSql2 43/43, Harbour compat 56/56.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| 6746ae4cee |
perf(fivesql2): stable temp alias — 1.67x on JOIN bench
AcquireTemp now returns the purpose string (upper-cased table name) as the alias when available, and falls back to FA_#### only when the same purpose is already in-flight this query — i.e., self-joins. Previously every call returned a fresh FA_####, so the WA cache (keyed by alias) could never hit on JOIN queries and the file got reopened every iteration. Bench deltas vs prior HEAD: B6_INNER_JOIN 217 → 130 us -40% (1.67x) B15_CTE_WIN_JOIN 1678 → 1595 us -5% Single-table benches unchanged — they were already hitting the cache via the table-name alias path. B8 recursive CTE stays flat: its sub-executors at nDepth>1 still cycle through fresh purposes that don't stabilise across queries. FiveSql2 43/43, Harbour compat 56/56. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| 9bb361b2e0 |
perf(fivesql2): gate VIEW temp cleanup on actual view usage
After the SELECT WA cache landed, pprof showed HbFileExists → os.Stat at 28% of remaining CPU — the RunSelect cleanup loop was stat-ing __view_<table>.dbf for every table in every query, even on the common view-free path. Track view materialisation with a TSqlIndex.lViewUsed flag set in OpenTable when CheckView produces a temp. The cleanup loop now runs only when the flag is set, then resets it. View-using queries are unaffected. pprof delta: rawsyscalln: 2.14s → 1.41s (48% → 32% of total CPU) os.Stat: 1.24s → 0.49s (28% → 11%) Wall-clock bench numbers stayed within plus-or-minus 3% noise (stats are cheap when the target file does not exist, so CPU savings do not translate directly to end-to-end time) but this removes the next biggest syscall waste visible in the profile. FiveSql2 43/43, Harbour compat 56/56. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| f27c96c7f0 |
perf(fivesql2): extend WA cache to SELECT path — 2x faster single-table
TSqlExecutor:OpenTable now hands lifetime to the WA cache for stable aliases (user-supplied or table-named). CloseOpened skips those entries, so the DBF mmap stays alive across queries instead of being unmapped + re-opened 1000 times a bench. Previously the WA cache only covered DML (INSERT/UPDATE/DELETE) — SELECT was still paying the full dbUseArea/dbCloseArea syscall bill every query (profile showed rtlDbCloseArea + munmap at ~30% of total CPU). AcquireTemp-generated aliases (FA_####) are excluded — they change every query (self-joins, nested depth), so caching them would just leak entries for no reuse. JOIN / recursive CTE regressions from an earlier unrestricted version are gone. Bench deltas vs prior HEAD (median of 3 steady runs, 1000 iters): B1_SELECT_STAR 82 → 41 us -50% (2.0x) B2_WHERE_FILTER 78 → 35 us -55% (2.2x) B3_ORDER_BY 90 → 48 us -47% (1.88x) B5_DISTINCT 75 → 32 us -57% (2.34x) B7_CTE_SIMPLE 120 → 77 us -36% (1.56x) B9_ROW_NUMBER 239 → 194 us -19% B10_RANK_PART 276 → 233 us -16% B11_SUM_OVER 296 → 252 us -15% B4_GROUP_HAVING 498 → 450 us -10% Others flat (JOIN / recursive CTE / DML already covered). FiveSql2 43/43, Harbour compat 56/56. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
|||
| dd270d5d9d |
perf: RTL Go-native migration — 27 optimizations, DML up to 70-90x
Systematic pass through PRG hot paths, promoting them to Go RTL while
preserving Harbour/FiveSql2 semantics. Full log in
docs/RTL-Go-Native-Migration.md.
Bench (bench_sql) vs 2026-04-08 baseline
- B1 SELECT * 2,192 → 114 µs (19x)
- B6 INNER JOIN 9,291 → 233 µs (40x)
- B7 CTE simple 8,037 → 129 µs (62x)
- B9 ROW_NUMBER 3,705 → 265 µs (14x)
- B10 RANK PARTITION 4,748 → 309 µs (15x)
- B12 INSERT (WA cache) 4,319 → 63 µs (69x)
- B13 UPDATE (WA cache) 6,144 → 68 µs (90x)
- B15 CTE+WIN+JOIN 18,395 → 1,873 µs (10x)
Infrastructure
- HbHash O(1) Index preserving insertion order (Harbour KEEPORDER)
- HbDeepClone Go RTL (scalar-sharing, immutable hash keys)
- MEMRDD auto-imported via gengo; all Five programs get mem:name driver
- SQL plan + pcode caches (s_hPlanCache, s_hDmlPcodeCache)
- Opt-in SqlWACacheEnable — dbUseArea/Close/Commit batched for DML
SQL engine
- FiveSql2 lexer ported to Go (byte FSM) with combined automatic
template parameterization (literals → ?, concat queries share plan)
- Go RTL: SqlDistinct, SqlGroupRows, SqlWindowPartitions,
SqlWindowSortPartition, SqlWindowAssignRank, SqlComputeAggSimple,
SqlBulkInsert, SqlBulkUpdate, SqlExprHasAgg, SqlEvalHaving
- CTE / subquery / driving-table materialize paths use MEMRDD
- SqlCoerce/SqlCmp/SqlIsTrue helpers moved from PRG to Go
- SqlBulkUpdate defers Flush when WA cache active (APFS fsync was
dominant B13 cost — 1.6ms/call → gone)
Correctness fixes uncovered during migration
- ASort default path now sorts dates/logicals/timestamps (was no-op)
- ORDER BY default NULL placement matches PRG SqlRowCompare across
Go fast path; explicit NULLS FIRST/LAST honored by both paths
- SqlBulkUpdate respects EXCLUSIVE vs SHARED mode record locks
- SqlCmp/SqlCmpEq normalize NumInt vs Double (caught by test 6b)
Verification
- go test ./... ALL PASS
- FiveSql2 test_sql1999 43/43
- tests/compat_harbour 56/56 (+5 new: ASort dates/logicals,
AScan int cross-type)
- Regression test test_null_order.prg for ORDER BY NULL ordering
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|||
| 3caadb23b9 |
perf: SqlOrderBy + SqlGroupBy Go RTL — native sort and aggregation
SqlOrderBy: Go sort.Slice for ORDER BY, 10-50x faster than PRG ASort. SqlGroupBy: Go map-based GROUP BY accumulation (ready for integration). TryBuildSortSpec detects simple ORDER BY columns and routes to Go. Fallback to PRG for complex ORDER BY expressions. 43/43 + 41/41 verify + 51/51 compat + go test ALL PASS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 54bf6f5bb4 |
fix: ComputeAgg qualified column lookup for Go SqlHashJoin path
FindColIdx2 searched for bare column name (e.g. 'AMOUNT') but
aFieldNames now contains qualified names ('o.amount') from the
Go join fast path. Added fallback: try xArg[2] (the full AST name)
when the bare name misses. Fixes SUM/AVG/MIN/MAX aggregation after
Go-native hash join.
Verified: 41/41 correctness tests pass (verify_correctness.prg).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 5fc9c3bbea |
perf: SqlHashJoin Go RTL — 3-way JOIN 4.2s→61ms (69x)
Go-native multi-table hash join bypasses per-row PRG overhead. TryGoJoin detects equi-join + plain-col SELECT, aggregate cols get placeholder. 2-way 73→3ms, 3-way 3.9s→61ms. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 53aaa4b69a |
perf: qualify hidden aggregate columns for JOIN FetchRow cache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 79e812a24e |
perf(FiveSql2): fix O(N²) window-function regression for default frame
Q2 Running total regressed 100ms→6.7s from the frame-aware rewrite. Default frame (UNBOUNDED PRECEDING to CURRENT ROW) now uses O(N) incremental path; general per-row-frame loop only for custom frames. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| c869a08365 |
fix(FiveSql2): last 3 — RIGHT JOIN O(N), counter wrap, implicit alias
--- #15 RIGHT JOIN O(N*M) → O(N+M) via matched RecNo set --- --- #19 s_nRCJSeq modular counter (% 100000) --- --- #20 Implicit column alias without AS keyword --- Validation: 43/43 + 51/51 + go test ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| e754aaac3f |
feat+fix(FiveSql2): window frame spec execution + EXISTS LIMIT safety
--- #12 Window frame spec now honoured --- Parser parsed ROWS BETWEEN ... AND ... but discarded the result. Now stores hFrame in a 6th slot on ND_WINDOW nodes via AAdd. ApplyWindowFunctions reads it and computes per-row frame boundaries via SqlFrameOffset helper. Unified SUM/AVG/COUNT/MIN/MAX into one frame-aware CASE branch. --- #6 EXISTS LIMIT mutation removed --- Removed direct parse-tree mutation (hQuery["limit"] := 1) that would corrupt reuse. Semi-join lift handles the fast case. Validation: 43/43 + 51/51 + go test ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 63f75bf2bc |
fix(FiveSql2): 5 more latent bugs — Resolve NULL, LEFT JOIN, UNION order, DATEADD, VIEW cleanup
Continues the static-analysis sweep from
|
|||
| 7babfb7281 |
fix(FiveSql2): 9 latent bugs from static analysis sweep
Systematic bug-hunt driven by an automated analysis of all FiveSql2 source files. Each fix is targeted — no speculative refactoring. --- #1 CLASSDATA hSubCache leaked across queries (CRITICAL) --- CLASSDATA hSubCache INIT { => } SHARED shared one hash across ALL TSqlExecutor instances. A non-correlated subquery cached in query A was silently returned for an unrelated query B if the subquery text happened to produce the same cache key. Converted to instance DATA initialized in New(). --- #5+#21 IS NULL / COALESCE treated empty string as NULL (HIGH) --- RETURN xL == NIL .OR. ( ValType(xL) == "C" .AND. Empty(AllTrim(xL)) ) SQL standard: '' is a valid non-NULL value. Removed the empty-string check from both IS NULL evaluation and COALESCE skip logic. --- #4 Multiple ? parameters all returned first value (HIGH) --- ND_PAR nodes had no index — EvalExpr always returned ::aParams[1]. Parser now stamps each ? with a sequential 1-based index in xNode[2]. EvalExpr uses it to return the correct ::aParams[n]. --- #10+#11 SqlEvalRowExpr missing / and || operators, single-arg function eval (MEDIUM) --- Division and string concatenation fell through to RETURN NIL in the row-expression evaluator used by recursive CTEs and aggregate ComputeAgg. Also, multi-argument functions like SUBSTR(x,2,3) only received the first argument. Both fixed. --- #9 SUM/AVG/MIN/MAX of all NULLs returned 0 instead of NULL (MEDIUM) --- SQL standard requires NULL. Changed the aggregate return path to return NIL when nCount == 0 (SUM/AVG) or when xMin/xMax == NIL. --- #8 MIN/MAX used SqlCoerceNum for comparison (MEDIUM) --- Strings and dates were coerced to numbers (Val()) before comparing, making MIN('banana') == MIN('apple') == 0. Switched to SqlCmpLt which handles type-appropriate comparison. --- #7 SqlExprHasAgg only checked top-level node (MEDIUM) --- Expressions like `salary + COUNT(*)` were not detected as containing an aggregate because the top node was ND_BIN, not ND_FN. Made the function recursive — walks ND_BIN, ND_UNI, ND_FN args, ND_CASE branches. --- #13 SELECT * only expanded first table in JOINs (MEDIUM) --- `SELECT * FROM orders o JOIN customers c ON ...` only included fields from orders. Changed the expansion loop to iterate ALL entries in ::aTables. --- #2 s_aOuterStack not unwound on subquery error (HIGH) --- SubqueryCached's PushOuter/PopOuter pair was not protected by BEGIN SEQUENCE. A runtime error inside the subquery left a stale entry on the module-level outer stack, corrupting all subsequent queries' correlated column resolution. Wrapped in SEQUENCE/RECOVER. Validation: - FiveSql2 43/43 - Harbour compat 51/51 - go test ./... ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 6c8d5f8b3b |
fix(FiveSql2): correlated scalar subquery with JOIN — 3 interacting bugs
A scalar correlated subquery with a JOIN inside:
SELECT e.name,
(SELECT SUM(o.qty * p.price)
FROM ord o INNER JOIN prod p ON o.prod_id = p.id
WHERE o.emp_id = e.id) AS revenue
FROM emp e WHERE e.dept = 'SALES'
returned wrong values (equal to SUM(qty) instead of SUM(qty*price))
or zero for all but the first outer row. Root cause was a triple
interaction between three independent bugs.
--- Bug 1: Subquery cache leaked across five_SQL invocations ---
hSubCorrCache, aSubCacheSlots, aSemiJoinSlots, nSubCacheSeq were
declared as DATA ... INIT { => } / {} / 0. In Five's compiled output,
hash/array INIT literals may share the same backing instance across
New() calls, so the cache from query A (SUM qty, no join) was still
there when query B ran, providing a hit on the same key — returning
A's cached (wrong) value instead of re-executing B's subquery.
Fix: explicit initialization in New().
--- Bug 2: aJoins alias mutation across subquery invocations ---
RunSelect's join-alias sync loop mutated aJoins[i][3] from the
user alias ("p") to the depth-suffixed temp alias ("FA_0003").
aJoins was a direct reference into hQuery["joins"], so the mutation
persisted across re-executions of the same hQuery. On the 2nd call,
the sync loop couldn't find a matching aTables entry because the
stale temp alias ("FA_0003") didn't match the new one ("FA_0005").
The join table's workarea was positioned wrong → empty join result.
Fix: deep-clone both ::aTables and aJoins at the start of RunSelect
so each invocation starts from the parsed originals.
--- Bug 3: SqlCollectCols stripped alias prefixes ---
When adding hidden columns for complex aggregate arguments (e.g.
SUM(o.qty * p.price)), SqlCollectCols returned bare names like
"qty" and "price" instead of qualified "o.qty" / "p.price". In a
JOIN context, unqualified "price" routed FetchRow to the first
table (ord) instead of prod — FieldPos returned 0, the column was
silently NIL, and the multiplication collapsed to qty*1 = qty.
Fix: new SqlCollectColExprs returns the original ND_COL AST nodes
with qualified names preserved. The hidden-column loop now inserts
these directly so FetchRow's dot-qualified path resolves to the
correct workarea via FindWA.
--- Verification ---
Deterministic 5-emp / 6-order / 3-product test:
Expected revenues per emp:
Emp 1: 2*10 + 3*20 = 80 → got 80.00 ✓
Emp 2: 1*10 + 4*30 = 130 → got 130.00 ✓
Emp 3: 5*20 = 100 → got 100.00 ✓
Emp 4: no orders = 0 → got 0 ✓
Emp 5: 7*10 = 70 → got 70.00 ✓
Also verified SUM(qty*2) and SUM(p.price) variants.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 99f3ca5687 |
perf(FiveSql2): EXISTS semi-join lift — H3 correlated EXISTS ~2000x faster
Correlated EXISTS with high-cardinality keys was stuck at O(outer × inner)
because memoization couldn't amortize across unique correlation values.
H3 in the subquery stress bench:
SELECT e.name FROM emp e
WHERE EXISTS (SELECT 1 FROM ord WHERE ord.emp_id = e.id AND ord.qty > 15)
500 outer rows × 500 distinct e.id values × 5000-row ord scan = 10s,
with no path to improvement from caching the subquery result.
Fix: detect the semi-join shape on the subquery and rewrite it at
runtime into a non-correlated DISTINCT scan whose result is cached
as a hash set. Each outer row then becomes an O(1) hash probe.
--- What we lift ---
SELECT ... FROM inner_table
WHERE inner.col = outer.col [AND other_non_correlated_preds]
Shape constraints (all must hold):
- single table, no JOIN
- no GROUP BY, no HAVING, no UNION
- WHERE is an AND tree containing an equi-term where one side is
a column with an alias prefix from the subquery's own FROM
and the other is a column from an outer alias
- the remaining AND terms (non-correlated residue) have no
outer references of their own — rules out patterns like
`WHERE e2.dept = e.dept AND e2.salary > e.salary` where the
second term can't live without the outer context
--- How the lift works ---
1. Walk the WHERE as a flat AND-term list
2. Find and remove the first correlated equi-term, remember the
inner column name and outer column reference
3. Verify residue is non-correlated via a recursive AST walker
(SemiJoinHasOuterRef) — bail to fallback if not
4. Clone hQuery with:
columns = {DISTINCT inner.col}
where = residue (or NIL)
distinct = .T.
limit / top / order_by / group_by / having cleared
5. Run the cloned subquery once via a nested TSqlExecutor — no
PushOuter because it's now non-correlated
6. Build a hash set keyed on SqlValToStr(each distinct inner value)
7. Per EXISTS probe: Resolve the outer column reference, look up
in the hash set
Cached in ::aSemiJoinSlots indexed by xSubNode identity so the
analysis + lifted scan runs exactly once per subquery expression.
Subqueries that don't match the shape store the sentinel "NO" so
subsequent probes skip re-analysis and fall through to the existing
SubqueryCached + LIMIT 1 path.
NOT EXISTS works through the same path — lNegate flag just flips
the final hash-lookup result.
--- Bench (emp=500, prod=100, ord=5k) ---
Pattern Before After Speedup
────────────────────────────────────────────────────────────
H3 EXISTS correlated 10.0s 4.5ms ~2200x
H8 NOT EXISTS self-join 900ms 890ms same (can't lift:
remainder
`e2.salary > e.salary`
is correlated)
H11 Scalar + EXISTS + derived 3.2s 1.0s 3.2x
H8 correctly falls through to the non-lifted path because the
remainder outer-reference check (SemiJoinHasOuterRef) rejects the
`e2.salary > e.salary` term. The 5-row answer is still correct.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
- H3 returns 125 rows (matches pre-change correct result)
- H8 returns 5 rows (matches pre-change correct result)
Known pre-existing bug, unrelated: H7 (scalar correlated subquery
with inner INNER JOIN) returns zero for rows 2..N — workarea state
leaks between consecutive subquery invocations. Not touched here,
filed for follow-up.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| ce7593c50f |
perf(FiveSql2): EXISTS → LIMIT 1 early exit, subquery identity via AScan
Extreme subquery stress bench (12 patterns spanning scalar-in-SELECT,
nested correlation, EXISTS, NOT IN, derived tables, self-joins, and
mixed combinations) exposed three weaknesses in the post-ROLLUP state:
1. EXISTS / NOT EXISTS evaluated the full subquery result per outer
row, even though it only needs to know whether any row matches.
2. EXISTS was routed through a separate code path that bypassed the
correlated-memoization cache from
|
|||
| 2d9023622c |
feat(FiveSql2): ROLLUP/CUBE/GROUPING SETS + correlated subquery memoization
Two SQL:2013 features that were stubs or bugs. Both ship together
because they share testing infrastructure (the SQL:2013 analytics
bench).
--- 1. ROLLUP / CUBE / GROUPING SETS (TSqlAgg) ---
The parser has recognized these for a while, storing them as
`ND_FN "ROLLUP"` / "CUBE" / "GROUPING SETS" nodes inside the
GROUP BY list. GroupBy never actually expanded them — it treated
the ND_FN as an opaque group term, which meant every row hashed
into the empty bucket and the query returned a single row.
New TSqlAgg:ExpandGroupingSets walks the aGroupBy array and
expands each ROLLUP / CUBE / GSETS modifier into a list of flat
grouping sets by cross-product with the surrounding plain terms:
GROUP BY ROLLUP(a, b, c) → {(a,b,c), (a,b), (a), ()}
GROUP BY CUBE(a, b) → {(a,b), (a), (b), ()}
GROUP BY GROUPING SETS((a,b),()) → as-is
GROUP BY x, ROLLUP(a, b) → {(x,a,b), (x,a), (x)}
When the expansion produces more than one set, GroupBy recurses
once per set (passing the plain flat set) and NILs out SELECT
columns that aren't in the current set — the standard subtotal
placeholder. Fast path (no ROLLUP/CUBE/GSETS node) short-circuits
to the original single-pass logic.
Correctness check: `SELECT region, SUM(amount) FROM sales GROUP BY
ROLLUP(region)` on a 5-region dataset now returns 6 rows (5
per-region subtotals + 1 grand total row with region=NIL). Was 1.
--- 2. Correlated subquery memoization (TSqlExecutor) ---
Committed
|
|||
| 9e0f82c5a8 |
perf+fix(FiveSql2): recursive-CTE hash join + correct correlated subqueries
Two fixes uncovered by a SQL:2013 analytics benchmark covering the
query patterns people actually run on DBF data (OLAP, BI, hierarchy
traversal).
--- Fix 1: correlated subquery was silently wrong ---
EvalExpr's ND_SUB handler only pushed the outer context when
`s_aOuterStack` was already non-empty — otherwise it routed the
subquery through CacheSubquery, which stores the first result under
a key derived from the subquery's syntax tokens. For a correlated
subquery in a top-level WHERE:
SELECT name, dept, salary FROM emp e1
WHERE salary > (SELECT AVG(salary) FROM emp e2 WHERE e2.dept = e1.dept)
the first outer row saw an empty stack, cached the result, and
every subsequent outer row got the same cached value regardless of
e1.dept. The query returned all 1000 employees instead of the 505
who actually beat their department's average.
Fix: always PushOuter + Run, no cache. Correctness over caching.
Trade-off: non-correlated scalar subqueries now re-execute per
outer row. A proper per-outer-key memoization is deferred — it
requires walking the subquery AST to collect free variables.
--- Fix 2: WITH RECURSIVE hierarchy join was O(m*n) ---
RecCteJoin (the in-memory join used when a recursive CTE's step
references both a real table and the CTE frontier) ran a flat
nested loop: for each DBF row × each prev-iteration row, build a
combined row buffer and run SqlEvalRowExpr on the ON condition.
For a 4-level 1000-employee hierarchy that's ~1M ON evaluations,
~4.6 seconds.
Fix: detect the shape `dbfAlias.col = cteAlias.col` at join-setup
time, build a PRG hash on the CTE frontier keyed by its join column
(aPrevRows is always small — at most the last iteration's emitted
rows), then scan the DBF side once and probe the hash. Complex ON
predicates fall through to the original nested loop.
--- Bench (SQL:2013 analytics, emp=1k, sales=20k, evt=30k) ---
Query Before After Speedup
──────────────────────────────────────────────────────────────
RECURSIVE hierarchy 4-level 4603ms 30ms ~150x
Correlated subquery (all emp) 10ms ❌ 4933ms ✓ (correct)
Other SQL:2013 queries (ROW_NUMBER top-N, running total, moving
average, DENSE_RANK, LAG, NTILE, gaps-and-islands) are all in the
expected 10–230ms range for these dataset sizes, unchanged by
this commit.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Known follow-ups (not in this commit):
- Q7 ROLLUP(col) parses but isn't expanded in GroupBy — returns
a single grand-total row instead of per-value + total. Grouping
sets implementation is a separate feature.
- Correlated subquery memoization by outer free-variable key
would bring Q8 from 4.9s back to ~50ms for small cardinality
correlations — requires AST free-var analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 64b7cf6676 |
perf(FiveSql2): compound-AND equi-join picks up hash path — CTE+JOIN 22x
FiveSql2's HashJoin only recognized bare equi-terms (xOnCond[1]=ND_BIN,
xOnCond[2]="="), so a compound ON predicate like
ON e.dept_id = t.dept_id AND e.salary = t.max_sal
fell through to the nested-loop ELSE branch:
dbSelectArea(nInnerWA)
dbGoTop()
WHILE !Eof()
IF SqlIsTrue(EvalExpr(xOnCond))
JoinRecurse(...)
ENDIF
dbSkip()
ENDDO
That's O(outer × inner) per outer row, re-evaluating the full AND tree
every probe. Query Q7 in the complex benchmark (CTE top_emp joined back
to emp on compound key) ran at 4.6 seconds for 100 inner × 10k outer.
Fix has two pieces:
1. **Probe-term extraction in JoinRecurse**: when xOnCond is an AND,
walk the left-associative chain looking for the first equi-term
(`a.x = b.x`). Use that as the hash-probe key, drive the normal
hash-join code path through it.
2. **Post-filter in HashJoin**: after a hash match, if the *original*
xOnCond was compound, re-evaluate the full predicate with
EvalExpr to drop matches that satisfied the hash key but not the
rest of the AND (e.g. same dept but different salary). Bare equi-
joins still skip the re-eval — the hash match is conclusive.
Bench (10k × 100 × compound ON predicate):
Query Before After Speedup
─────────────────────────────────────────────────────────
Q7 CTE + JOIN compound ON 4573ms 209ms 21.9x
Still works for the existing bare equi case (43-test unchanged) and
the 3-way JOIN case (no regression). Falls back to the generic nested
loop only when no probe-term can be extracted at all.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
- Q7 result: 100 rows (correct)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| c6799a599e |
fix(FiveSql2): GROUP BY with aliased SELECT collapses all rows into one
Surfaced by complex-query benchmarking. Query like:
SELECT d.name AS dept, COUNT(*) AS n, SUM(o.amount) AS total
FROM dept d INNER JOIN emp e ON ... INNER JOIN ord o ON ...
GROUP BY d.name
returned exactly 1 row instead of 100. Removing the AS aliases made
it work correctly. Semantic bug, not a performance issue.
Root cause: TSqlAgg:GroupBy resolved each GROUP BY column by calling
FindColIdx against aFN — the output alias list. For GROUP BY d.name
with d.name AS dept, the group expression's column name was looked
up in {"dept","n","total"} and missed. FindColIdx returned 0, every
row got an empty group key, and the hash collapsed everything into
one bucket.
Fix: new FindGroupIdx walks aCols (SELECT list expressions) instead,
matching the GROUP BY column against each SELECT item's source
expression ND_COL name. Handles qualified refs (d.name -> NAME) and
falls back to FindColIdx for cases where GROUP BY uses a column not
in the SELECT list.
Also hoisted the resolution out of the per-row loop — GROUP BY
columns resolve once into aGroupIdx[] so each row just indexes.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- Complex bench Q4: 1 row -> 100 rows (correct)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| bfc6ded8cb |
perf(FiveSql2): SqlHashBuild + FetchRow column binding — 3-way JOIN 3x
Complex-query benchmarking turned up two hot paths that the earlier
SqlScan/SqlEach work didn't touch: multi-table JOIN and nested-scan
row fetching. This commit hits both.
--- Part 1: SqlHashBuild — Go-native hash-join build ---
FiveSql2's HashJoin previously built the inner-side hash in PRG:
WHILE !Eof()
xVal := FieldGet(nFPos)
cKey := SqlValToStr(xVal)
IF !hb_HHasKey(hHash, cKey) ; hHash[cKey] := {} ; ENDIF
AAdd(hHash[cKey], RecNo())
dbSkip()
ENDDO
That loop runs at ~40μs per row from class dispatch + hb_HHasKey
lookups + AAdd growth + SqlValToStr formatting. On a 50k-row inner
table that's ~2 seconds wasted on what should be a sub-50ms
housekeeping op.
New hbrtl.SqlHashBuild does the same thing in one Go-native pass:
- Direct *dbf.DBFArea loop (no interface dispatch, same devirt as
SqlScan)
- Go `map[string][]int64` accumulates RecNos by key — one
allocation per distinct key
- Inline ASCII-only digit formatter for numeric keys (strconv.Itoa
is allocation-heavy for small ints)
- CHAR keys are right-trimmed to match SqlCmpEq semantics so the
hash probe matches what EvalExpr would compute
- Final Five hash is built once from Keys/Values/Order slices
directly, skipping the per-key hb_HSet path
HashJoin now calls `SqlHashBuild(nFPos)` instead of running the
PRG loop.
--- Part 2: TSqlExecutor:BuildFetchCache ---
The JOIN fallback loop calls FetchRow per row. FetchRow was already
column-ref-aware but did the string parse (`At + SubStr + Upper`)
and `::FindWA` linear scan every single invocation. For a 50k-row
join emitting 50k result rows, that's ~200k redundant resolutions.
New BuildFetchCache walks the SELECT list once before the scan and
pre-binds each plain-column expression to `{nWA, nFPos}`. FetchRow's
new fast path checks ::aFetchCache and jumps straight to
`dbSelectArea + FieldGet` when bound. Complex exprs (functions,
CASE, subqueries) still fall through to EvalExpr.
::aFetchCache is set right before the join WHILE loop and cleared
after — no cross-query bleed.
--- Bench (50k ord × 10k emp × 100 dept, 3-run steady state) ---
Query Before After Speedup
────────────────────────────────────────────────────────────
2-way INNER JOIN, 10k rows 91ms 68ms 1.34x
2-way JOIN + GROUP BY 110ms 94ms 1.17x
3-way INNER JOIN COUNT 2610ms 610ms 4.28x
3-way JOIN + GROUP BY 2860ms 830ms 3.45x
The 3-way speedup is almost entirely SqlHashBuild. The 2-way case
benefits from the fetch cache because its per-row cost is dominated
by FetchRow (no second hash build to amortize).
--- Limits still standing ---
CTE + JOIN queries (Q7 in bench_complex: ~4.5s) aren't affected by
either optimization — CTE materialization goes through a different
path that writes/reads a temp DBF. Follow-up target.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| e75167c2e9 |
feat(FiveSql2): five_SQL block-callback integration — SQL beats raw PRG
Wires the new SqlEach RTL into FiveSql2's front-end so users write
the SQL they know and opt into streaming with a familiar Harbour
code block — no manual RTL plumbing.
API:
/* Existing array form — unchanged, 43-test still green */
aR := five_SQL( "SELECT name FROM t" )
/* New block form — zero intermediate rows, 2x raw PRG */
five_SQL( "SELECT id, name FROM t WHERE salary > 50000", NIL,
{|nID, cName| Process(nID, cName)} )
Parameter order (cSQL, aParams, bBlock) keeps backward compatibility
with every existing call site. Passing NIL for aParams when only a
block is needed is standard Harbour idiom.
Routing:
* TFiveSQL:Execute now takes an optional bBlock parameter and
stores it on TSqlExecutor as ::bRowBlock.
* TSqlExecutor:RunSelect's existing Go fast path (same guards as
before: single table, no JOIN/GROUP/aggregate, plain column
projections, WHERE compilable via SqlExprToPrg) branches on
::bRowBlock:
- block present → SqlEach streams rows through the block
- block absent → SqlScan materializes into aRows (current path)
* Post-processing (GROUP BY / ORDER BY / window / DISTINCT / LIMIT)
runs on empty aRows when block mode fires — all are no-ops on
empty input, so the sequence stays harmless.
* RunSelect returns NIL (not {fields, rows}) when ::bRowBlock was
used — signals "streaming semantics, all work done in the block".
Complex queries (JOIN, GROUP BY, subquery, window, ORDER BY not
matchable by an index, LIMIT/OFFSET, etc.) still fall back to the
array path even when a block is supplied — those genuinely require
materialization. Block mode is a fast-path opt-in, not a semantic
change.
End-to-end bench (50k rows, steady state — includes the user-side
loop/block for every row):
Path Time Speedup vs raw
──────────────────────────────────────────────────────────────
Raw PRG DO WHILE !Eof() + WHERE sum 7.6ms 1.00x
five_SQL array + FOR 7.7ms ~same
five_SQL + block (new) 3.7ms 2.05x ← beats raw
──────────────────────────────────────────────────────────────
Raw PRG no WHERE 6.1ms 1.00x
five_SQL + block, no WHERE 2.9ms 2.10x ← beats raw
SQL now pays for itself on end-to-end timing — not just competitive
with hand-rolled RDD loops, but faster than them. The layered cost
of FieldGet's Frame+RTL-dispatch that hand-written loops incur per
call is gone; the block-callback path captures *dbf.DBFArea directly
via FastFieldGetter and uses PcOpFieldGet to bypass dispatch in the
compiled WHERE predicate.
Validation:
- FiveSql2 43/43 (array API unchanged)
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| ad69221136 |
revert(FiveSql2): restore TSqlIndex:FindExclusive scan
Previous short-circuit (return 0 unconditionally) was a workaround
for two bugs that are both fixed now:
1. gengo PushLocal(0) panic on unresolved identifiers
→ fixed by
|
|||
| 8aaed994f4 |
perf(FiveSql2): hybrid fast path — 11x speedup on string WHERE scans
Implements hybrid execution model: keep AST tree-walk for SQL:2013+
features (Window, Recursive CTE, JOIN, aggregates) while compiling
simple SELECT hot paths to Go + pcode. See docs/FiveSql2-Hybrid-Plan.md
for the full architecture rationale (why not SQLite-style VDBE).
Hot path (single table, no joins/groups/aggregates):
- TryBuildFieldPositions: resolves SELECT column list to FieldPos
array once per query (bails to PRG loop on any complex expr).
- TryCompileWhere + SqlExprToPrg: walks WHERE AST, emits equivalent
PRG source, runs it through PcCompile to get a PcodeFunc.
- SqlScan RTL: Go-native scan loop — GoTop/EOF/Skip/GetValue
direct, ExecPcode per row for WHERE, result array pre-alloc.
WHERE compiler scope:
- ND_LIT numeric/logical/string (string literals AllTrim'd to match
SqlCmpEq CHAR-padding semantics; rejects embedded quotes/newlines)
- ND_COL: CHAR fields auto-wrapped with AllTrim(FieldGet(n)) based
on dbStruct() lookup cached once per query in aCompileStruct
- ND_BIN: = <> != < <= > >= AND OR + - * /
- ND_UNI: NOT -
- Anything else (ND_FN, ND_CASE, ND_SUB, ND_PAR, LIKE, IN, IS NULL,
BETWEEN, dates) returns NIL → falls back to PRG tree-walk.
Bench (50k rows, ~/tmp ext4):
Before After Speedup
Numeric WHERE ~150ms 11.7ms ~13x
String WHERE 119.3ms 10.5ms 11.4x
No WHERE - 14.6ms -
Raw RDD baseline 6.8ms 6.8ms 1.0x
Remaining gap to raw RDD (~1.5x) is structural: Value boxing, result
array construction, per-row ExecPcode frame overhead. Would need a
Value-pool or SoA refactor to close further.
Side fixes bundled:
- TSqlIndex:FindExclusive short-circuited. Originally called
dbInfo(DBI_FULLPATH)/DBI_SHARED which are unresolved symbols in
Five (dbInfo is a stub, DBI_* never defined). Panic'd with
"local variable index out of range: 0" whenever a standalone PRG
had a workarea Used before calling five_SQL. 43-test masked the
bug because it only reached FindExclusive with no open workareas.
Restore the scan once dbInfo lands in hbrtl.
- cmd/five/main.go: FIVE_KEEP_BUILD=1 env var keeps the temp Go
project around for debugging gengo output.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 486e466592 |
feat: FiveSql2 43/43, @byref, mutable closure, RTL 479, DateTime fix
Major changes since last commit: - FiveSql2 SQL:1999 engine (10,458 LOC) — 43/43 ALL PASS - 21 compiler/runtime bugs fixed (short-circuit AND/OR, FOR LOOP, etc.) - @byref pass-by-reference via RefCell pattern - Mutable closure capture (EnsureLocalRef + RefCell sharing) - RTL: 400 → 479 functions (+79: file, string, datetime, hash, UTF-8) - DateTime/Timestamp fully working (hb_DateTime, hb_Hour/Min/Sec, display) - Reserved word guard (39 keywords blocked from function calls) - AEval arg order fix (element before index) - Closure capture redecl fix (unique _cap_ names per block) - Hash/string indexing in ArrayPush/ArrayPop - Harbour compat test suite: 51/51 - 4 docs: Porting Report, Implementation Plan, Optimization Plan, Commercialization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
