3caadb23b962e64d041f7875cbe33b89e26a5972
18 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 3caadb23b9 |
perf: SqlOrderBy + SqlGroupBy Go RTL — native sort and aggregation
SqlOrderBy: Go sort.Slice for ORDER BY, 10-50x faster than PRG ASort. SqlGroupBy: Go map-based GROUP BY accumulation (ready for integration). TryBuildSortSpec detects simple ORDER BY columns and routes to Go. Fallback to PRG for complex ORDER BY expressions. 43/43 + 41/41 verify + 51/51 compat + go test ALL PASS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 5fc9c3bbea |
perf: SqlHashJoin Go RTL — 3-way JOIN 4.2s→61ms (69x)
Go-native multi-table hash join bypasses per-row PRG overhead. TryGoJoin detects equi-join + plain-col SELECT, aggregate cols get placeholder. 2-way 73→3ms, 3-way 3.9s→61ms. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 53aaa4b69a |
perf: qualify hidden aggregate columns for JOIN FetchRow cache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 79e812a24e |
perf(FiveSql2): fix O(N²) window-function regression for default frame
Q2 Running total regressed 100ms→6.7s from the frame-aware rewrite. Default frame (UNBOUNDED PRECEDING to CURRENT ROW) now uses O(N) incremental path; general per-row-frame loop only for custom frames. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| c869a08365 |
fix(FiveSql2): last 3 — RIGHT JOIN O(N), counter wrap, implicit alias
--- #15 RIGHT JOIN O(N*M) → O(N+M) via matched RecNo set --- --- #19 s_nRCJSeq modular counter (% 100000) --- --- #20 Implicit column alias without AS keyword --- Validation: 43/43 + 51/51 + go test ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| e754aaac3f |
feat+fix(FiveSql2): window frame spec execution + EXISTS LIMIT safety
--- #12 Window frame spec now honoured --- Parser parsed ROWS BETWEEN ... AND ... but discarded the result. Now stores hFrame in a 6th slot on ND_WINDOW nodes via AAdd. ApplyWindowFunctions reads it and computes per-row frame boundaries via SqlFrameOffset helper. Unified SUM/AVG/COUNT/MIN/MAX into one frame-aware CASE branch. --- #6 EXISTS LIMIT mutation removed --- Removed direct parse-tree mutation (hQuery["limit"] := 1) that would corrupt reuse. Semi-join lift handles the fast case. Validation: 43/43 + 51/51 + go test ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 63f75bf2bc |
fix(FiveSql2): 5 more latent bugs — Resolve NULL, LEFT JOIN, UNION order, DATEADD, VIEW cleanup
Continues the static-analysis sweep from
|
|||
| 7babfb7281 |
fix(FiveSql2): 9 latent bugs from static analysis sweep
Systematic bug-hunt driven by an automated analysis of all FiveSql2 source files. Each fix is targeted — no speculative refactoring. --- #1 CLASSDATA hSubCache leaked across queries (CRITICAL) --- CLASSDATA hSubCache INIT { => } SHARED shared one hash across ALL TSqlExecutor instances. A non-correlated subquery cached in query A was silently returned for an unrelated query B if the subquery text happened to produce the same cache key. Converted to instance DATA initialized in New(). --- #5+#21 IS NULL / COALESCE treated empty string as NULL (HIGH) --- RETURN xL == NIL .OR. ( ValType(xL) == "C" .AND. Empty(AllTrim(xL)) ) SQL standard: '' is a valid non-NULL value. Removed the empty-string check from both IS NULL evaluation and COALESCE skip logic. --- #4 Multiple ? parameters all returned first value (HIGH) --- ND_PAR nodes had no index — EvalExpr always returned ::aParams[1]. Parser now stamps each ? with a sequential 1-based index in xNode[2]. EvalExpr uses it to return the correct ::aParams[n]. --- #10+#11 SqlEvalRowExpr missing / and || operators, single-arg function eval (MEDIUM) --- Division and string concatenation fell through to RETURN NIL in the row-expression evaluator used by recursive CTEs and aggregate ComputeAgg. Also, multi-argument functions like SUBSTR(x,2,3) only received the first argument. Both fixed. --- #9 SUM/AVG/MIN/MAX of all NULLs returned 0 instead of NULL (MEDIUM) --- SQL standard requires NULL. Changed the aggregate return path to return NIL when nCount == 0 (SUM/AVG) or when xMin/xMax == NIL. --- #8 MIN/MAX used SqlCoerceNum for comparison (MEDIUM) --- Strings and dates were coerced to numbers (Val()) before comparing, making MIN('banana') == MIN('apple') == 0. Switched to SqlCmpLt which handles type-appropriate comparison. --- #7 SqlExprHasAgg only checked top-level node (MEDIUM) --- Expressions like `salary + COUNT(*)` were not detected as containing an aggregate because the top node was ND_BIN, not ND_FN. Made the function recursive — walks ND_BIN, ND_UNI, ND_FN args, ND_CASE branches. --- #13 SELECT * only expanded first table in JOINs (MEDIUM) --- `SELECT * FROM orders o JOIN customers c ON ...` only included fields from orders. Changed the expansion loop to iterate ALL entries in ::aTables. --- #2 s_aOuterStack not unwound on subquery error (HIGH) --- SubqueryCached's PushOuter/PopOuter pair was not protected by BEGIN SEQUENCE. A runtime error inside the subquery left a stale entry on the module-level outer stack, corrupting all subsequent queries' correlated column resolution. Wrapped in SEQUENCE/RECOVER. Validation: - FiveSql2 43/43 - Harbour compat 51/51 - go test ./... ALL PASS Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 6c8d5f8b3b |
fix(FiveSql2): correlated scalar subquery with JOIN — 3 interacting bugs
A scalar correlated subquery with a JOIN inside:
SELECT e.name,
(SELECT SUM(o.qty * p.price)
FROM ord o INNER JOIN prod p ON o.prod_id = p.id
WHERE o.emp_id = e.id) AS revenue
FROM emp e WHERE e.dept = 'SALES'
returned wrong values (equal to SUM(qty) instead of SUM(qty*price))
or zero for all but the first outer row. Root cause was a triple
interaction between three independent bugs.
--- Bug 1: Subquery cache leaked across five_SQL invocations ---
hSubCorrCache, aSubCacheSlots, aSemiJoinSlots, nSubCacheSeq were
declared as DATA ... INIT { => } / {} / 0. In Five's compiled output,
hash/array INIT literals may share the same backing instance across
New() calls, so the cache from query A (SUM qty, no join) was still
there when query B ran, providing a hit on the same key — returning
A's cached (wrong) value instead of re-executing B's subquery.
Fix: explicit initialization in New().
--- Bug 2: aJoins alias mutation across subquery invocations ---
RunSelect's join-alias sync loop mutated aJoins[i][3] from the
user alias ("p") to the depth-suffixed temp alias ("FA_0003").
aJoins was a direct reference into hQuery["joins"], so the mutation
persisted across re-executions of the same hQuery. On the 2nd call,
the sync loop couldn't find a matching aTables entry because the
stale temp alias ("FA_0003") didn't match the new one ("FA_0005").
The join table's workarea was positioned wrong → empty join result.
Fix: deep-clone both ::aTables and aJoins at the start of RunSelect
so each invocation starts from the parsed originals.
--- Bug 3: SqlCollectCols stripped alias prefixes ---
When adding hidden columns for complex aggregate arguments (e.g.
SUM(o.qty * p.price)), SqlCollectCols returned bare names like
"qty" and "price" instead of qualified "o.qty" / "p.price". In a
JOIN context, unqualified "price" routed FetchRow to the first
table (ord) instead of prod — FieldPos returned 0, the column was
silently NIL, and the multiplication collapsed to qty*1 = qty.
Fix: new SqlCollectColExprs returns the original ND_COL AST nodes
with qualified names preserved. The hidden-column loop now inserts
these directly so FetchRow's dot-qualified path resolves to the
correct workarea via FindWA.
--- Verification ---
Deterministic 5-emp / 6-order / 3-product test:
Expected revenues per emp:
Emp 1: 2*10 + 3*20 = 80 → got 80.00 ✓
Emp 2: 1*10 + 4*30 = 130 → got 130.00 ✓
Emp 3: 5*20 = 100 → got 100.00 ✓
Emp 4: no orders = 0 → got 0 ✓
Emp 5: 7*10 = 70 → got 70.00 ✓
Also verified SUM(qty*2) and SUM(p.price) variants.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 99f3ca5687 |
perf(FiveSql2): EXISTS semi-join lift — H3 correlated EXISTS ~2000x faster
Correlated EXISTS with high-cardinality keys was stuck at O(outer × inner)
because memoization couldn't amortize across unique correlation values.
H3 in the subquery stress bench:
SELECT e.name FROM emp e
WHERE EXISTS (SELECT 1 FROM ord WHERE ord.emp_id = e.id AND ord.qty > 15)
500 outer rows × 500 distinct e.id values × 5000-row ord scan = 10s,
with no path to improvement from caching the subquery result.
Fix: detect the semi-join shape on the subquery and rewrite it at
runtime into a non-correlated DISTINCT scan whose result is cached
as a hash set. Each outer row then becomes an O(1) hash probe.
--- What we lift ---
SELECT ... FROM inner_table
WHERE inner.col = outer.col [AND other_non_correlated_preds]
Shape constraints (all must hold):
- single table, no JOIN
- no GROUP BY, no HAVING, no UNION
- WHERE is an AND tree containing an equi-term where one side is
a column with an alias prefix from the subquery's own FROM
and the other is a column from an outer alias
- the remaining AND terms (non-correlated residue) have no
outer references of their own — rules out patterns like
`WHERE e2.dept = e.dept AND e2.salary > e.salary` where the
second term can't live without the outer context
--- How the lift works ---
1. Walk the WHERE as a flat AND-term list
2. Find and remove the first correlated equi-term, remember the
inner column name and outer column reference
3. Verify residue is non-correlated via a recursive AST walker
(SemiJoinHasOuterRef) — bail to fallback if not
4. Clone hQuery with:
columns = {DISTINCT inner.col}
where = residue (or NIL)
distinct = .T.
limit / top / order_by / group_by / having cleared
5. Run the cloned subquery once via a nested TSqlExecutor — no
PushOuter because it's now non-correlated
6. Build a hash set keyed on SqlValToStr(each distinct inner value)
7. Per EXISTS probe: Resolve the outer column reference, look up
in the hash set
Cached in ::aSemiJoinSlots indexed by xSubNode identity so the
analysis + lifted scan runs exactly once per subquery expression.
Subqueries that don't match the shape store the sentinel "NO" so
subsequent probes skip re-analysis and fall through to the existing
SubqueryCached + LIMIT 1 path.
NOT EXISTS works through the same path — lNegate flag just flips
the final hash-lookup result.
--- Bench (emp=500, prod=100, ord=5k) ---
Pattern Before After Speedup
────────────────────────────────────────────────────────────
H3 EXISTS correlated 10.0s 4.5ms ~2200x
H8 NOT EXISTS self-join 900ms 890ms same (can't lift:
remainder
`e2.salary > e.salary`
is correlated)
H11 Scalar + EXISTS + derived 3.2s 1.0s 3.2x
H8 correctly falls through to the non-lifted path because the
remainder outer-reference check (SemiJoinHasOuterRef) rejects the
`e2.salary > e.salary` term. The 5-row answer is still correct.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
- H3 returns 125 rows (matches pre-change correct result)
- H8 returns 5 rows (matches pre-change correct result)
Known pre-existing bug, unrelated: H7 (scalar correlated subquery
with inner INNER JOIN) returns zero for rows 2..N — workarea state
leaks between consecutive subquery invocations. Not touched here,
filed for follow-up.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| ce7593c50f |
perf(FiveSql2): EXISTS → LIMIT 1 early exit, subquery identity via AScan
Extreme subquery stress bench (12 patterns spanning scalar-in-SELECT,
nested correlation, EXISTS, NOT IN, derived tables, self-joins, and
mixed combinations) exposed three weaknesses in the post-ROLLUP state:
1. EXISTS / NOT EXISTS evaluated the full subquery result per outer
row, even though it only needs to know whether any row matches.
2. EXISTS was routed through a separate code path that bypassed the
correlated-memoization cache from
|
|||
| 2d9023622c |
feat(FiveSql2): ROLLUP/CUBE/GROUPING SETS + correlated subquery memoization
Two SQL:2013 features that were stubs or bugs. Both ship together
because they share testing infrastructure (the SQL:2013 analytics
bench).
--- 1. ROLLUP / CUBE / GROUPING SETS (TSqlAgg) ---
The parser has recognized these for a while, storing them as
`ND_FN "ROLLUP"` / "CUBE" / "GROUPING SETS" nodes inside the
GROUP BY list. GroupBy never actually expanded them — it treated
the ND_FN as an opaque group term, which meant every row hashed
into the empty bucket and the query returned a single row.
New TSqlAgg:ExpandGroupingSets walks the aGroupBy array and
expands each ROLLUP / CUBE / GSETS modifier into a list of flat
grouping sets by cross-product with the surrounding plain terms:
GROUP BY ROLLUP(a, b, c) → {(a,b,c), (a,b), (a), ()}
GROUP BY CUBE(a, b) → {(a,b), (a), (b), ()}
GROUP BY GROUPING SETS((a,b),()) → as-is
GROUP BY x, ROLLUP(a, b) → {(x,a,b), (x,a), (x)}
When the expansion produces more than one set, GroupBy recurses
once per set (passing the plain flat set) and NILs out SELECT
columns that aren't in the current set — the standard subtotal
placeholder. Fast path (no ROLLUP/CUBE/GSETS node) short-circuits
to the original single-pass logic.
Correctness check: `SELECT region, SUM(amount) FROM sales GROUP BY
ROLLUP(region)` on a 5-region dataset now returns 6 rows (5
per-region subtotals + 1 grand total row with region=NIL). Was 1.
--- 2. Correlated subquery memoization (TSqlExecutor) ---
Committed
|
|||
| 9e0f82c5a8 |
perf+fix(FiveSql2): recursive-CTE hash join + correct correlated subqueries
Two fixes uncovered by a SQL:2013 analytics benchmark covering the
query patterns people actually run on DBF data (OLAP, BI, hierarchy
traversal).
--- Fix 1: correlated subquery was silently wrong ---
EvalExpr's ND_SUB handler only pushed the outer context when
`s_aOuterStack` was already non-empty — otherwise it routed the
subquery through CacheSubquery, which stores the first result under
a key derived from the subquery's syntax tokens. For a correlated
subquery in a top-level WHERE:
SELECT name, dept, salary FROM emp e1
WHERE salary > (SELECT AVG(salary) FROM emp e2 WHERE e2.dept = e1.dept)
the first outer row saw an empty stack, cached the result, and
every subsequent outer row got the same cached value regardless of
e1.dept. The query returned all 1000 employees instead of the 505
who actually beat their department's average.
Fix: always PushOuter + Run, no cache. Correctness over caching.
Trade-off: non-correlated scalar subqueries now re-execute per
outer row. A proper per-outer-key memoization is deferred — it
requires walking the subquery AST to collect free variables.
--- Fix 2: WITH RECURSIVE hierarchy join was O(m*n) ---
RecCteJoin (the in-memory join used when a recursive CTE's step
references both a real table and the CTE frontier) ran a flat
nested loop: for each DBF row × each prev-iteration row, build a
combined row buffer and run SqlEvalRowExpr on the ON condition.
For a 4-level 1000-employee hierarchy that's ~1M ON evaluations,
~4.6 seconds.
Fix: detect the shape `dbfAlias.col = cteAlias.col` at join-setup
time, build a PRG hash on the CTE frontier keyed by its join column
(aPrevRows is always small — at most the last iteration's emitted
rows), then scan the DBF side once and probe the hash. Complex ON
predicates fall through to the original nested loop.
--- Bench (SQL:2013 analytics, emp=1k, sales=20k, evt=30k) ---
Query Before After Speedup
──────────────────────────────────────────────────────────────
RECURSIVE hierarchy 4-level 4603ms 30ms ~150x
Correlated subquery (all emp) 10ms ❌ 4933ms ✓ (correct)
Other SQL:2013 queries (ROW_NUMBER top-N, running total, moving
average, DENSE_RANK, LAG, NTILE, gaps-and-islands) are all in the
expected 10–230ms range for these dataset sizes, unchanged by
this commit.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Known follow-ups (not in this commit):
- Q7 ROLLUP(col) parses but isn't expanded in GroupBy — returns
a single grand-total row instead of per-value + total. Grouping
sets implementation is a separate feature.
- Correlated subquery memoization by outer free-variable key
would bring Q8 from 4.9s back to ~50ms for small cardinality
correlations — requires AST free-var analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 64b7cf6676 |
perf(FiveSql2): compound-AND equi-join picks up hash path — CTE+JOIN 22x
FiveSql2's HashJoin only recognized bare equi-terms (xOnCond[1]=ND_BIN,
xOnCond[2]="="), so a compound ON predicate like
ON e.dept_id = t.dept_id AND e.salary = t.max_sal
fell through to the nested-loop ELSE branch:
dbSelectArea(nInnerWA)
dbGoTop()
WHILE !Eof()
IF SqlIsTrue(EvalExpr(xOnCond))
JoinRecurse(...)
ENDIF
dbSkip()
ENDDO
That's O(outer × inner) per outer row, re-evaluating the full AND tree
every probe. Query Q7 in the complex benchmark (CTE top_emp joined back
to emp on compound key) ran at 4.6 seconds for 100 inner × 10k outer.
Fix has two pieces:
1. **Probe-term extraction in JoinRecurse**: when xOnCond is an AND,
walk the left-associative chain looking for the first equi-term
(`a.x = b.x`). Use that as the hash-probe key, drive the normal
hash-join code path through it.
2. **Post-filter in HashJoin**: after a hash match, if the *original*
xOnCond was compound, re-evaluate the full predicate with
EvalExpr to drop matches that satisfied the hash key but not the
rest of the AND (e.g. same dept but different salary). Bare equi-
joins still skip the re-eval — the hash match is conclusive.
Bench (10k × 100 × compound ON predicate):
Query Before After Speedup
─────────────────────────────────────────────────────────
Q7 CTE + JOIN compound ON 4573ms 209ms 21.9x
Still works for the existing bare equi case (43-test unchanged) and
the 3-way JOIN case (no regression). Falls back to the generic nested
loop only when no probe-term can be extracted at all.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
- Q7 result: 100 rows (correct)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| bfc6ded8cb |
perf(FiveSql2): SqlHashBuild + FetchRow column binding — 3-way JOIN 3x
Complex-query benchmarking turned up two hot paths that the earlier
SqlScan/SqlEach work didn't touch: multi-table JOIN and nested-scan
row fetching. This commit hits both.
--- Part 1: SqlHashBuild — Go-native hash-join build ---
FiveSql2's HashJoin previously built the inner-side hash in PRG:
WHILE !Eof()
xVal := FieldGet(nFPos)
cKey := SqlValToStr(xVal)
IF !hb_HHasKey(hHash, cKey) ; hHash[cKey] := {} ; ENDIF
AAdd(hHash[cKey], RecNo())
dbSkip()
ENDDO
That loop runs at ~40μs per row from class dispatch + hb_HHasKey
lookups + AAdd growth + SqlValToStr formatting. On a 50k-row inner
table that's ~2 seconds wasted on what should be a sub-50ms
housekeeping op.
New hbrtl.SqlHashBuild does the same thing in one Go-native pass:
- Direct *dbf.DBFArea loop (no interface dispatch, same devirt as
SqlScan)
- Go `map[string][]int64` accumulates RecNos by key — one
allocation per distinct key
- Inline ASCII-only digit formatter for numeric keys (strconv.Itoa
is allocation-heavy for small ints)
- CHAR keys are right-trimmed to match SqlCmpEq semantics so the
hash probe matches what EvalExpr would compute
- Final Five hash is built once from Keys/Values/Order slices
directly, skipping the per-key hb_HSet path
HashJoin now calls `SqlHashBuild(nFPos)` instead of running the
PRG loop.
--- Part 2: TSqlExecutor:BuildFetchCache ---
The JOIN fallback loop calls FetchRow per row. FetchRow was already
column-ref-aware but did the string parse (`At + SubStr + Upper`)
and `::FindWA` linear scan every single invocation. For a 50k-row
join emitting 50k result rows, that's ~200k redundant resolutions.
New BuildFetchCache walks the SELECT list once before the scan and
pre-binds each plain-column expression to `{nWA, nFPos}`. FetchRow's
new fast path checks ::aFetchCache and jumps straight to
`dbSelectArea + FieldGet` when bound. Complex exprs (functions,
CASE, subqueries) still fall through to EvalExpr.
::aFetchCache is set right before the join WHILE loop and cleared
after — no cross-query bleed.
--- Bench (50k ord × 10k emp × 100 dept, 3-run steady state) ---
Query Before After Speedup
────────────────────────────────────────────────────────────
2-way INNER JOIN, 10k rows 91ms 68ms 1.34x
2-way JOIN + GROUP BY 110ms 94ms 1.17x
3-way INNER JOIN COUNT 2610ms 610ms 4.28x
3-way JOIN + GROUP BY 2860ms 830ms 3.45x
The 3-way speedup is almost entirely SqlHashBuild. The 2-way case
benefits from the fetch cache because its per-row cost is dominated
by FetchRow (no second hash build to amortize).
--- Limits still standing ---
CTE + JOIN queries (Q7 in bench_complex: ~4.5s) aren't affected by
either optimization — CTE materialization goes through a different
path that writes/reads a temp DBF. Follow-up target.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| e75167c2e9 |
feat(FiveSql2): five_SQL block-callback integration — SQL beats raw PRG
Wires the new SqlEach RTL into FiveSql2's front-end so users write
the SQL they know and opt into streaming with a familiar Harbour
code block — no manual RTL plumbing.
API:
/* Existing array form — unchanged, 43-test still green */
aR := five_SQL( "SELECT name FROM t" )
/* New block form — zero intermediate rows, 2x raw PRG */
five_SQL( "SELECT id, name FROM t WHERE salary > 50000", NIL,
{|nID, cName| Process(nID, cName)} )
Parameter order (cSQL, aParams, bBlock) keeps backward compatibility
with every existing call site. Passing NIL for aParams when only a
block is needed is standard Harbour idiom.
Routing:
* TFiveSQL:Execute now takes an optional bBlock parameter and
stores it on TSqlExecutor as ::bRowBlock.
* TSqlExecutor:RunSelect's existing Go fast path (same guards as
before: single table, no JOIN/GROUP/aggregate, plain column
projections, WHERE compilable via SqlExprToPrg) branches on
::bRowBlock:
- block present → SqlEach streams rows through the block
- block absent → SqlScan materializes into aRows (current path)
* Post-processing (GROUP BY / ORDER BY / window / DISTINCT / LIMIT)
runs on empty aRows when block mode fires — all are no-ops on
empty input, so the sequence stays harmless.
* RunSelect returns NIL (not {fields, rows}) when ::bRowBlock was
used — signals "streaming semantics, all work done in the block".
Complex queries (JOIN, GROUP BY, subquery, window, ORDER BY not
matchable by an index, LIMIT/OFFSET, etc.) still fall back to the
array path even when a block is supplied — those genuinely require
materialization. Block mode is a fast-path opt-in, not a semantic
change.
End-to-end bench (50k rows, steady state — includes the user-side
loop/block for every row):
Path Time Speedup vs raw
──────────────────────────────────────────────────────────────
Raw PRG DO WHILE !Eof() + WHERE sum 7.6ms 1.00x
five_SQL array + FOR 7.7ms ~same
five_SQL + block (new) 3.7ms 2.05x ← beats raw
──────────────────────────────────────────────────────────────
Raw PRG no WHERE 6.1ms 1.00x
five_SQL + block, no WHERE 2.9ms 2.10x ← beats raw
SQL now pays for itself on end-to-end timing — not just competitive
with hand-rolled RDD loops, but faster than them. The layered cost
of FieldGet's Frame+RTL-dispatch that hand-written loops incur per
call is gone; the block-callback path captures *dbf.DBFArea directly
via FastFieldGetter and uses PcOpFieldGet to bypass dispatch in the
compiled WHERE predicate.
Validation:
- FiveSql2 43/43 (array API unchanged)
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 8aaed994f4 |
perf(FiveSql2): hybrid fast path — 11x speedup on string WHERE scans
Implements hybrid execution model: keep AST tree-walk for SQL:2013+
features (Window, Recursive CTE, JOIN, aggregates) while compiling
simple SELECT hot paths to Go + pcode. See docs/FiveSql2-Hybrid-Plan.md
for the full architecture rationale (why not SQLite-style VDBE).
Hot path (single table, no joins/groups/aggregates):
- TryBuildFieldPositions: resolves SELECT column list to FieldPos
array once per query (bails to PRG loop on any complex expr).
- TryCompileWhere + SqlExprToPrg: walks WHERE AST, emits equivalent
PRG source, runs it through PcCompile to get a PcodeFunc.
- SqlScan RTL: Go-native scan loop — GoTop/EOF/Skip/GetValue
direct, ExecPcode per row for WHERE, result array pre-alloc.
WHERE compiler scope:
- ND_LIT numeric/logical/string (string literals AllTrim'd to match
SqlCmpEq CHAR-padding semantics; rejects embedded quotes/newlines)
- ND_COL: CHAR fields auto-wrapped with AllTrim(FieldGet(n)) based
on dbStruct() lookup cached once per query in aCompileStruct
- ND_BIN: = <> != < <= > >= AND OR + - * /
- ND_UNI: NOT -
- Anything else (ND_FN, ND_CASE, ND_SUB, ND_PAR, LIKE, IN, IS NULL,
BETWEEN, dates) returns NIL → falls back to PRG tree-walk.
Bench (50k rows, ~/tmp ext4):
Before After Speedup
Numeric WHERE ~150ms 11.7ms ~13x
String WHERE 119.3ms 10.5ms 11.4x
No WHERE - 14.6ms -
Raw RDD baseline 6.8ms 6.8ms 1.0x
Remaining gap to raw RDD (~1.5x) is structural: Value boxing, result
array construction, per-row ExecPcode frame overhead. Would need a
Value-pool or SoA refactor to close further.
Side fixes bundled:
- TSqlIndex:FindExclusive short-circuited. Originally called
dbInfo(DBI_FULLPATH)/DBI_SHARED which are unresolved symbols in
Five (dbInfo is a stub, DBI_* never defined). Panic'd with
"local variable index out of range: 0" whenever a standalone PRG
had a workarea Used before calling five_SQL. 43-test masked the
bug because it only reached FindExclusive with no open workareas.
Restore the scan once dbInfo lands in hbrtl.
- cmd/five/main.go: FIVE_KEEP_BUILD=1 env var keeps the temp Go
project around for debugging gengo output.
Validation:
- FiveSql2 43/43
- Harbour compat 51/51
- go test ./... ALL PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|||
| 486e466592 |
feat: FiveSql2 43/43, @byref, mutable closure, RTL 479, DateTime fix
Major changes since last commit: - FiveSql2 SQL:1999 engine (10,458 LOC) — 43/43 ALL PASS - 21 compiler/runtime bugs fixed (short-circuit AND/OR, FOR LOOP, etc.) - @byref pass-by-reference via RefCell pattern - Mutable closure capture (EnsureLocalRef + RefCell sharing) - RTL: 400 → 479 functions (+79: file, string, datetime, hash, UTF-8) - DateTime/Timestamp fully working (hb_DateTime, hb_Hour/Min/Sec, display) - Reserved word guard (39 keywords blocked from function calls) - AEval arg order fix (element before index) - Closure capture redecl fix (unique _cap_ names per block) - Hash/string indexing in ArrayPush/ArrayPop - Harbour compat test suite: 51/51 - 4 docs: Porting Report, Implementation Plan, Optimization Plan, Commercialization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
