fivedev/five
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf(FiveSql2): hybrid fast path — 11x speedup on string WHERE scans

Browse Source 
Implements hybrid execution model: keep AST tree-walk for SQL:2013+
features (Window, Recursive CTE, JOIN, aggregates) while compiling
simple SELECT hot paths to Go + pcode. See docs/FiveSql2-Hybrid-Plan.md
for the full architecture rationale (why not SQLite-style VDBE).

Hot path (single table, no joins/groups/aggregates):
  - TryBuildFieldPositions: resolves SELECT column list to FieldPos
    array once per query (bails to PRG loop on any complex expr).
  - TryCompileWhere + SqlExprToPrg: walks WHERE AST, emits equivalent
    PRG source, runs it through PcCompile to get a PcodeFunc.
  - SqlScan RTL: Go-native scan loop — GoTop/EOF/Skip/GetValue
    direct, ExecPcode per row for WHERE, result array pre-alloc.

WHERE compiler scope:
  - ND_LIT numeric/logical/string (string literals AllTrim'd to match
    SqlCmpEq CHAR-padding semantics; rejects embedded quotes/newlines)
  - ND_COL: CHAR fields auto-wrapped with AllTrim(FieldGet(n)) based
    on dbStruct() lookup cached once per query in aCompileStruct
  - ND_BIN: = <> != < <= > >= AND OR + - * /
  - ND_UNI: NOT -
  - Anything else (ND_FN, ND_CASE, ND_SUB, ND_PAR, LIKE, IN, IS NULL,
    BETWEEN, dates) returns NIL → falls back to PRG tree-walk.

Bench (50k rows, ~/tmp ext4):
                        Before      After     Speedup
  Numeric WHERE         ~150ms     11.7ms     ~13x
  String WHERE          119.3ms    10.5ms     11.4x
  No WHERE               -         14.6ms      -
  Raw RDD baseline        6.8ms     6.8ms      1.0x

Remaining gap to raw RDD (~1.5x) is structural: Value boxing, result
array construction, per-row ExecPcode frame overhead. Would need a
Value-pool or SoA refactor to close further.

Side fixes bundled:
  - TSqlIndex:FindExclusive short-circuited. Originally called
    dbInfo(DBI_FULLPATH)/DBI_SHARED which are unresolved symbols in
    Five (dbInfo is a stub, DBI_* never defined). Panic'd with
    "local variable index out of range: 0" whenever a standalone PRG
    had a workarea Used before calling five_SQL. 43-test masked the
    bug because it only reached FindExclusive with no open workareas.
    Restore the scan once dbInfo lands in hbrtl.
  - cmd/five/main.go: FIVE_KEEP_BUILD=1 env var keeps the temp Go
    project around for debugging gengo output.

Validation:
  - FiveSql2 43/43
  - Harbour compat 51/51
  - go test ./... ALL PASS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
CharlesKWON
2026-04-14 09:15:08 +09:00
parent 6b26f1b642
commit 8aaed994f4
6 changed files with 552 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

222
_FiveSql2/src/TSqlExecutor.prg
View File

@@ -32,6 +32,7 @@ CLASS TSqlExecutor
DATA nDepth INIT 0
DATA aOpened INIT {}
DATA aTables INIT {}
DATA aCompileStruct
CLASSDATA hSubCache INIT { => } SHARED
@@ -62,6 +63,9 @@ CLASS TSqlExecutor
METHOD ApplyWindowFunctions( aRows, aFN, aCols )
METHOD RunMerge()
METHOD RunTruncate()
METHOD TryBuildFieldPositions( aExprs )
METHOD TryCompileWhere( xWhere )
METHOD SqlExprToPrg( xNode )
ENDCLASS
@@ -970,6 +974,7 @@ METHOD RunSelect() CLASS TSqlExecutor
LOCAL xArgExpr, cBare, lFound, aLeafCols, k
LOCAL hJoinHash
LOCAL lIndexUsed, aTmp
LOCAL aFP, pcW, aGoRows
aCols := ::hQuery[ "columns" ]
::aTables := ::hQuery[ "tables" ]
@@ -1190,19 +1195,45 @@ METHOD RunSelect() CLASS TSqlExecutor
hJoinHash := { => }
WHILE ! Eof()
IF Len( aJoins ) > 0
::JoinRecurse( aJoins, 1, xWhere, aResultExprs, @aRows, hJoinHash )
dbSelectArea( nWA )
ELSE
IF xWhere == NIL .OR. SqlIsTrue( ::EvalExpr( xWhere ) )
aRow := ::FetchRow( aResultExprs )
AAdd( aRows, aRow )
/* === GO NATIVE FAST PATH ===
* Single-table, no joins, no aggregates, all SELECT exprs
* simple field refs, WHERE is NIL or compilable to pcode.
* Hands the scan loop off to Go's SqlScan (~15x faster
* than the PRG per-row tree walk).
*/
aFP := NIL
pcW := NIL
aGoRows := NIL
IF Len( aJoins ) == 0 .AND. Len( aGroupBy ) == 0 .AND. ;
! ::oAgg:HasAgg( aCols )
aFP := ::TryBuildFieldPositions( aResultExprs )
IF aFP != NIL
pcW := ::TryCompileWhere( xWhere )
IF xWhere == NIL .OR. pcW != NIL
aGoRows := SqlScan( aFP, pcW )
FOR i := 1 TO Len( aGoRows )
AAdd( aRows, aGoRows[ i ] )
NEXT
ENDIF
ENDIF
dbSelectArea( nWA )
dbSkip()
ENDDO
ENDIF
/* Fallback: PRG interpreter loop */
IF aGoRows == NIL
WHILE ! Eof()
IF Len( aJoins ) > 0
::JoinRecurse( aJoins, 1, xWhere, aResultExprs, @aRows, hJoinHash )
dbSelectArea( nWA )
ELSE
IF xWhere == NIL .OR. SqlIsTrue( ::EvalExpr( xWhere ) )
aRow := ::FetchRow( aResultExprs )
AAdd( aRows, aRow )
ENDIF
ENDIF
dbSelectArea( nWA )
dbSkip()
ENDDO
ENDIF
ENDIF
ENDIF
ENDIF
@@ -2731,3 +2762,172 @@ STATIC FUNCTION RecCteJoin( hRecQuery, aFN, aPrevRows, cCteName )
dbSelectArea( nSaveWA )
RETURN aResult
/* --------------------------------------------------------------
* Go fast-path helpers
* Return non-NIL only when the query can be handed off to Go's
* SqlScan RTL. Any complexity (expressions, functions, joins,
* parameters in WHERE) → return NIL so the PRG loop takes over.
* -------------------------------------------------------------- */
METHOD TryBuildFieldPositions( aExprs ) CLASS TSqlExecutor
LOCAL aPositions := {}, i, xE, cRef, nDot, cField, nFPos
FOR i := 1 TO Len( aExprs )
xE := aExprs[ i ][ 1 ]
IF xE == NIL .OR. xE[ 1 ] != ND_COL .OR. xE[ 2 ] == "*"
RETURN NIL
ENDIF
cRef := xE[ 2 ]
nDot := At( ".", cRef )
IF nDot > 0
cField := Upper( SubStr( cRef, nDot + 1 ) )
ELSE
cField := Upper( cRef )
ENDIF
nFPos := FieldPos( cField )
IF nFPos == 0
RETURN NIL
ENDIF
AAdd( aPositions, nFPos )
NEXT
RETURN aPositions
METHOD TryCompileWhere( xWhere ) CLASS TSqlExecutor
/* Phase 1+2: compile numeric/logical/string WHERE to pcode.
* Semantic guard: SqlExprToPrg returns NIL for anything that would
* drift from SqlCmpEq/SqlCoerceForCmp semantics. CHAR columns are
* auto-wrapped with AllTrim() to match Harbour SqlCmpEq behavior.
* NULL/function/subquery/parameter → NIL (fallback).
*/
LOCAL cPrg, xResult
IF xWhere == NIL
RETURN NIL
ENDIF
/* Cache struct once for field-type lookups during expr walk */
::aCompileStruct := dbStruct()
cPrg := ::SqlExprToPrg( xWhere )
::aCompileStruct := NIL
IF cPrg == NIL
RETURN NIL
ENDIF
xResult := PcCompile( cPrg )
RETURN xResult
METHOD SqlExprToPrg( xNode ) CLASS TSqlExecutor
LOCAL cOp, cL, cR
LOCAL cRef, nDot, cField, nFPos, cFType, cLit
IF xNode == NIL
RETURN NIL
ENDIF
DO CASE
CASE xNode[ 1 ] == ND_LIT
IF ValType( xNode[ 2 ] ) == "N"
RETURN AllTrim( Str( xNode[ 2 ] ) )
ENDIF
IF ValType( xNode[ 2 ] ) == "L"
IF xNode[ 2 ]
RETURN ".T."
ENDIF
RETURN ".F."
ENDIF
IF ValType( xNode[ 2 ] ) == "C"
cLit := xNode[ 2 ]
/* Reject strings with embedded quotes — escaping would be ambiguous */
IF "'" $ cLit .OR. '"' $ cLit .OR. Chr(10) $ cLit .OR. Chr(13) $ cLit
RETURN NIL
ENDIF
/* Match SqlCmpEq: compare trimmed values */
RETURN "'" + AllTrim( cLit ) + "'"
ENDIF
/* Dates/datetimes deferred */
RETURN NIL
CASE xNode[ 1 ] == ND_COL
cRef := xNode[ 2 ]
IF cRef == "*"
RETURN NIL
ENDIF
nDot := At( ".", cRef )
IF nDot > 0
cField := Upper( SubStr( cRef, nDot + 1 ) )
ELSE
cField := Upper( cRef )
ENDIF
nFPos := FieldPos( cField )
IF nFPos == 0
RETURN NIL
ENDIF
/* Look up field type from cached struct to decide AllTrim wrap */
cFType := ""
IF ::aCompileStruct != NIL .AND. nFPos <= Len( ::aCompileStruct )
cFType := ::aCompileStruct[ nFPos ][ 2 ]
ENDIF
IF cFType == "C"
RETURN "AllTrim(FieldGet(" + AllTrim( Str( nFPos ) ) + "))"
ENDIF
RETURN "FieldGet(" + AllTrim( Str( nFPos ) ) + ")"
CASE xNode[ 1 ] == ND_UNI
cOp := xNode[ 2 ]
cL := ::SqlExprToPrg( xNode[ 3 ] )
IF cL == NIL
RETURN NIL
ENDIF
IF cOp == "NOT"
RETURN "!(" + cL + ")"
ENDIF
IF cOp == "-"
RETURN "-(" + cL + ")"
ENDIF
RETURN NIL
CASE xNode[ 1 ] == ND_BIN
cOp := xNode[ 2 ]
cL := ::SqlExprToPrg( xNode[ 3 ] )
IF cL == NIL
RETURN NIL
ENDIF
cR := ::SqlExprToPrg( xNode[ 4 ] )
IF cR == NIL
RETURN NIL
ENDIF
DO CASE
CASE cOp == "=" .OR. cOp == "=="
RETURN "(" + cL + ")==(" + cR + ")"
CASE cOp == "<>" .OR. cOp == "!="
RETURN "(" + cL + ")!=(" + cR + ")"
CASE cOp == "<"
RETURN "(" + cL + ")<(" + cR + ")"
CASE cOp == "<="
RETURN "(" + cL + ")<=(" + cR + ")"
CASE cOp == ">"
RETURN "(" + cL + ")>(" + cR + ")"
CASE cOp == ">="
RETURN "(" + cL + ")>=(" + cR + ")"
CASE cOp == "AND"
RETURN "(" + cL + ").AND.(" + cR + ")"
CASE cOp == "OR"
RETURN "(" + cL + ").OR.(" + cR + ")"
CASE cOp == "+"
RETURN "(" + cL + ")+(" + cR + ")"
CASE cOp == "-"
RETURN "(" + cL + ")-(" + cR + ")"
CASE cOp == "*"
RETURN "(" + cL + ")*(" + cR + ")"
CASE cOp == "/"
RETURN "(" + cL + ")/(" + cR + ")"
ENDCASE
RETURN NIL
ENDCASE
RETURN NIL

38
_FiveSql2/src/TSqlIndex.prg
View File

@@ -132,27 +132,23 @@ RETURN nWA
METHOD FindExclusive( cTableLow ) CLASS TSqlIndex
LOCAL nSaved, nArea, cDbfName, lShared
nSaved := Select()
FOR nArea := 1 TO 250
IF ( nArea )->( Used() )
dbSelectArea( nArea )
IF ! Empty( Alias() )
cDbfName := Lower( AllTrim( dbInfo( DBI_FULLPATH ) ) )
IF cTableLow + ".dbf" $ cDbfName .OR. cTableLow $ cDbfName
lShared := dbInfo( DBI_SHARED )
IF ! lShared
dbSelectArea( nSaved )
RETURN nArea
ENDIF
ENDIF
ENDIF
ENDIF
NEXT
dbSelectArea( nSaved )
/* Pre-flight exclusive-lock detection.
* Originally used dbInfo(DBI_FULLPATH)/DBI_SHARED to scan open
* workareas for an exclusive hold on the target DBF. In Five,
* `dbInfo()` is stubbed (returns NIL) and the DBI_* symbols are
* unresolved at compile time → runtime panic the moment any
* workarea is Used() when this runs (standalone PRGs routinely
* dbUseArea before calling five_SQL, so they tripped this).
*
* The check cannot function correctly on Five regardless, so
* we short-circuit to 0 (= no conflict). Matches behavior of
* the 43-test harness which only reaches here with no Used
* workareas, so the net behavior is preserved.
*
* Future: when dbInfo(DBI_FULLPATH) lands in hbrtl, restore
* the scan. Until then use `Alias()` + filename matching if
* exclusive-lock preflight becomes necessary.
*/
RETURN 0