Commit Graph

4 Commits

Author SHA1 Message Date
dd270d5d9d perf: RTL Go-native migration — 27 optimizations, DML up to 70-90x
Systematic pass through PRG hot paths, promoting them to Go RTL while
preserving Harbour/FiveSql2 semantics. Full log in
docs/RTL-Go-Native-Migration.md.

Bench (bench_sql) vs 2026-04-08 baseline
 - B1  SELECT *             2,192 → 114   µs   (19x)
 - B6  INNER JOIN           9,291 → 233   µs   (40x)
 - B7  CTE simple           8,037 → 129   µs   (62x)
 - B9  ROW_NUMBER           3,705 → 265   µs   (14x)
 - B10 RANK PARTITION       4,748 → 309   µs   (15x)
 - B12 INSERT (WA cache)    4,319 →  63   µs   (69x)
 - B13 UPDATE (WA cache)    6,144 →  68   µs   (90x)
 - B15 CTE+WIN+JOIN        18,395 → 1,873 µs   (10x)

Infrastructure
 - HbHash O(1) Index preserving insertion order (Harbour KEEPORDER)
 - HbDeepClone Go RTL (scalar-sharing, immutable hash keys)
 - MEMRDD auto-imported via gengo; all Five programs get mem:name driver
 - SQL plan + pcode caches (s_hPlanCache, s_hDmlPcodeCache)
 - Opt-in SqlWACacheEnable — dbUseArea/Close/Commit batched for DML

SQL engine
 - FiveSql2 lexer ported to Go (byte FSM) with combined automatic
   template parameterization (literals → ?, concat queries share plan)
 - Go RTL: SqlDistinct, SqlGroupRows, SqlWindowPartitions,
   SqlWindowSortPartition, SqlWindowAssignRank, SqlComputeAggSimple,
   SqlBulkInsert, SqlBulkUpdate, SqlExprHasAgg, SqlEvalHaving
 - CTE / subquery / driving-table materialize paths use MEMRDD
 - SqlCoerce/SqlCmp/SqlIsTrue helpers moved from PRG to Go
 - SqlBulkUpdate defers Flush when WA cache active (APFS fsync was
   dominant B13 cost — 1.6ms/call → gone)

Correctness fixes uncovered during migration
 - ASort default path now sorts dates/logicals/timestamps (was no-op)
 - ORDER BY default NULL placement matches PRG SqlRowCompare across
   Go fast path; explicit NULLS FIRST/LAST honored by both paths
 - SqlBulkUpdate respects EXCLUSIVE vs SHARED mode record locks
 - SqlCmp/SqlCmpEq normalize NumInt vs Double (caught by test 6b)

Verification
 - go test ./...              ALL PASS
 - FiveSql2 test_sql1999      43/43
 - tests/compat_harbour       56/56 (+5 new: ASort dates/logicals,
                              AScan int cross-type)
 - Regression test test_null_order.prg for ORDER BY NULL ordering

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 20:20:14 +09:00
63f75bf2bc fix(FiveSql2): 5 more latent bugs — Resolve NULL, LEFT JOIN, UNION order, DATEADD, VIEW cleanup
Continues the static-analysis sweep from 7babfb7.

--- #3 Resolve NIL ambiguity (HIGH) ---

ResolveFromOuter returned NIL for both "column not found" and
"column value is NULL". Callers tested `xVal != NIL` to decide
success, which silently dropped legitimate NULL outer-row values
in correlated subqueries. Added a by-reference lFound flag so
callers distinguish the two cases.

--- #14 Multi-level LEFT JOIN null-fill (MEDIUM) ---

LEFT JOIN null-fill only fired at the last join level
(`nIdx >= Len(aJoins)`). For `a LEFT JOIN b ON ... JOIN c ON ...`
where b had no match, the null-fill for b was skipped and the
outer row was dropped entirely. Now recurses into subsequent joins
when the match fails, so the base case can still emit a row with
NULLs for b's columns.

--- #18 UNION/INTERSECT/EXCEPT applied after LIMIT (MEDIUM) ---

SQL standard requires set operations before ORDER BY / DISTINCT /
OFFSET / LIMIT. Reordered to:
  RIGHT JOIN pass → UNION/INTERSECT/EXCEPT → DISTINCT → ORDER BY
  → OFFSET → LIMIT.
Previously LIMIT clipped the first SELECT before UNION merged the
second's rows, producing more rows than intended.

--- #22 DATEADD month overflow (LOW) ---

`DATEADD('MONTH', 1, '2024-01-31')` produced `SToD("20240231")`
(Feb 31) → empty date. Now normalizes month overflow/underflow
into year rollover and clamps the day to the target month's last
day. Year addition also handles Feb 29 → Feb 28 on non-leap years.

--- #23 VIEW temp file leak (LOW) ---

TSqlIndex:CheckView creates `__view_<table>.dbf` temp files that
were never cleaned up. Added post-scan cleanup in RunSelect's
close section (after CTE cleanup) that erases matching temp files.

Validation:
  - FiveSql2 43/43
  - Harbour compat 51/51
  - go test ./... ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 20:34:42 +09:00
7babfb7281 fix(FiveSql2): 9 latent bugs from static analysis sweep
Systematic bug-hunt driven by an automated analysis of all FiveSql2
source files. Each fix is targeted — no speculative refactoring.

--- #1 CLASSDATA hSubCache leaked across queries (CRITICAL) ---

  CLASSDATA hSubCache INIT { => } SHARED

shared one hash across ALL TSqlExecutor instances. A non-correlated
subquery cached in query A was silently returned for an unrelated
query B if the subquery text happened to produce the same cache key.
Converted to instance DATA initialized in New().

--- #5+#21 IS NULL / COALESCE treated empty string as NULL (HIGH) ---

  RETURN xL == NIL .OR. ( ValType(xL) == "C" .AND. Empty(AllTrim(xL)) )

SQL standard: '' is a valid non-NULL value. Removed the empty-string
check from both IS NULL evaluation and COALESCE skip logic.

--- #4 Multiple ? parameters all returned first value (HIGH) ---

ND_PAR nodes had no index — EvalExpr always returned ::aParams[1].
Parser now stamps each ? with a sequential 1-based index in xNode[2].
EvalExpr uses it to return the correct ::aParams[n].

--- #10+#11 SqlEvalRowExpr missing / and || operators, single-arg
    function eval (MEDIUM) ---

Division and string concatenation fell through to RETURN NIL in the
row-expression evaluator used by recursive CTEs and aggregate
ComputeAgg. Also, multi-argument functions like SUBSTR(x,2,3) only
received the first argument. Both fixed.

--- #9 SUM/AVG/MIN/MAX of all NULLs returned 0 instead of NULL
    (MEDIUM) ---

SQL standard requires NULL. Changed the aggregate return path to
return NIL when nCount == 0 (SUM/AVG) or when xMin/xMax == NIL.

--- #8 MIN/MAX used SqlCoerceNum for comparison (MEDIUM) ---

Strings and dates were coerced to numbers (Val()) before comparing,
making MIN('banana') == MIN('apple') == 0. Switched to SqlCmpLt
which handles type-appropriate comparison.

--- #7 SqlExprHasAgg only checked top-level node (MEDIUM) ---

Expressions like `salary + COUNT(*)` were not detected as containing
an aggregate because the top node was ND_BIN, not ND_FN. Made the
function recursive — walks ND_BIN, ND_UNI, ND_FN args, ND_CASE
branches.

--- #13 SELECT * only expanded first table in JOINs (MEDIUM) ---

`SELECT * FROM orders o JOIN customers c ON ...` only included
fields from orders. Changed the expansion loop to iterate ALL
entries in ::aTables.

--- #2 s_aOuterStack not unwound on subquery error (HIGH) ---

SubqueryCached's PushOuter/PopOuter pair was not protected by
BEGIN SEQUENCE. A runtime error inside the subquery left a stale
entry on the module-level outer stack, corrupting all subsequent
queries' correlated column resolution. Wrapped in SEQUENCE/RECOVER.

Validation:
  - FiveSql2 43/43
  - Harbour compat 51/51
  - go test ./... ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 17:26:05 +09:00
486e466592 feat: FiveSql2 43/43, @byref, mutable closure, RTL 479, DateTime fix
Major changes since last commit:
- FiveSql2 SQL:1999 engine (10,458 LOC) — 43/43 ALL PASS
- 21 compiler/runtime bugs fixed (short-circuit AND/OR, FOR LOOP, etc.)
- @byref pass-by-reference via RefCell pattern
- Mutable closure capture (EnsureLocalRef + RefCell sharing)
- RTL: 400 → 479 functions (+79: file, string, datetime, hash, UTF-8)
- DateTime/Timestamp fully working (hb_DateTime, hb_Hour/Min/Sec, display)
- Reserved word guard (39 keywords blocked from function calls)
- AEval arg order fix (element before index)
- Closure capture redecl fix (unique _cap_ names per block)
- Hash/string indexing in ArrayPush/ArrayPop
- Harbour compat test suite: 51/51
- 4 docs: Porting Report, Implementation Plan, Optimization Plan, Commercialization

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 11:35:37 +09:00