five/docs/gap-analysis.md

# Five vs Harbour — Gap Analysis

_Last updated: 2026-04-11_

This document is an **honest** account of what works in Five today and
what is still missing versus Harbour 3.2.0dev. It is written for users
who are evaluating Five for production, not for marketing.

The scope is deliberately narrow: we compare **language runtime, RDD,
RTL, tooling, and ecosystem**. FiveSql2 (SQL:1999 on DBF) is out of
scope — it has no Harbour counterpart and is documented separately.

---

## Summary

| Area | Status | One-line takeaway |
|------|--------|-------------------|
| Core language (LOCAL, IF, FOR, DO WHILE, CLASS, BEGIN SEQUENCE, code blocks, `@byref`, mutable closures) | ✅ | Works |
| DBF engine (open/append/update/delete/pack/zap) | ✅ | Works |
| NTX single-tag indexes (read + write) | ✅ | Works |
| CDX compound indexes (read) | ✅ | Byte-compatible with Harbour |
| CDX compound indexes (write) | ❌ | Not implemented — `INDEX ON ... TAG` currently produces NTX, not CDX |
| Memo fields (FPT) | ✅ | Works |
| Memo fields (DBT) | ❌ | Not implemented |
| Shared mode + file/record locking | ✅ **NEW** | POSIX byte-range locks via fcntl (verified cross-process) |
| Harbour parity test (281-line RDD diff) | ✅ | 0 lines differ |
| FLOCK / DBRLOCK / DBRUNLOCK / DBUNLOCK | ✅ **NEW** | Real locks (not stubs) |
| Preprocessor (`#include`, `#define`, `#ifdef`, `#command`, `#translate`) | ✅ | Works for most Harbour headers |
| Class system basics (DATA, METHOD, INHERIT, ACCESS/ASSIGN, SELF/SUPER) | ✅ | Works |
| Class advanced (`CLASSDATA`, `HIDDEN`/`PROTECTED`/`EXPORTED`, `PROPERTY`, operator overload) | ❌ | Not implemented |
| Threading exposed to PRG (`hb_threadStart`, `hb_mutexCreate`, …) | ❌ | Not registered — goroutines available internally only |
| Networking (sockets, HTTP, TCP/UDP, SSL/TLS) | ❌ | No RTL functions registered |
| Serialization (`hb_Serialize` / `hb_Deserialize`) | ❌ | Not implemented |
| Compression (`hb_ZCompress` / `hb_ZUncompress`) | ❌ | Not implemented |
| JSON (`hb_JSONEncode` / `hb_JSONDecode`) | ❌ | Not registered |
| `hbmk2` equivalent | ⚠️ | `five build` is a thin wrapper around `go build` |
| Contrib libraries (curl/cairo/gd/qt/ssl/mysql/pgsql/…) | ❌ | None |
| GUI frameworks (HWGUI/HMG/MiniGUI) | ❌ | None |
| GET / @...SAY / READ forms | ⚠️ | Minimal implementation |
| FRM report form system | ❌ | Not implemented |
| TBrowse | ⚠️ | Basic scrolling only |
| CLI debugger | ✅ | Line-level TUI debugger works |
| Remote/DAP debugging | ❌ | Not implemented |

Legend: ✅ works · ⚠️ partial · ❌ missing

---

## RDD — Shared Mode & Locking (Fixed 2026-04-11)

### Before

`FLOCK()`, `DBRLOCK()`, `DBRUNLOCK()` were registered as **no-op stubs**
that always returned `.T.`. The `OpenParams.Shared` flag was accepted
but no byte-range locks were ever acquired. Multi-process writers could
corrupt records silently.

```go
// hbrtl/database.go (before)
func rtlDbRLock(t *hbrt.Thread) {
    ...
    t.RetBool(true) // always succeeds
}
```

### After

Real POSIX byte-range advisory locks via `fcntl(F_SETLK)`.

| RTL function | Harbour behavior | Five status |
|---|---|---|
| `FLOCK()` | Exclusive file lock | ✅ fcntl F_WRLCK on header region |
| `DBRLOCK([n])` | Exclusive record lock | ✅ fcntl F_WRLCK on `RecordOffset(n) + RecordLen` |
| `DBRUNLOCK([n])` | Release one / all record locks | ✅ fcntl F_UNLCK |
| `DBUNLOCK()` | Release all locks | ✅ fcntl F_UNLCK both ranges |

**Design choices** (aligned with Harbour's `hb_fsLockLarge` layout):

1. **POSIX byte-range locks, not `flock(2)`** — byte-range is required for
   record-level semantics, and fcntl locks work across NFS while `flock`
   does not. Harbour/Clipper both use this exact mechanism.
2. **Non-blocking** (`F_SETLK`, not `F_SETLKW`) — matches Clipper
   `FLOCK() → .F.` / `DBRLOCK() → .F.` semantics when another process
   holds a conflicting lock.
3. **Non-overlapping regions** — `FLOCK` covers `[0, HeaderLen+1)`, record
   locks cover `[RecordOffset(n), RecordLen)`. A process can hold both
   simultaneously on the same fd.
4. **Released on `Close()`** — explicit `releaseAllLocks()` before fd
   close avoids races when multiple workareas share the same file.
5. **Windows stub retained** — `//go:build windows` falls back to the
   old always-true behavior. `LockFileEx` port is listed below.

### Verified cross-process

[`hbrdd/dbf/lock_multi_test.go`](../hbrdd/dbf/lock_multi_test.go) spawns
a separate OS process via `go build` + `exec.Command` and verifies:

- Process A holds `FLOCK()` → process B's `FLOCK()` returns **BUSY**
- A releases → B can acquire
- A holds `DBRLOCK(2)` → B's `DBRLOCK(2)` returns **BUSY**
- B's `DBRLOCK(1)` on a different record returns **OK** (non-overlap)
- A releases record 2 → B can acquire it

These tests exercise real `fcntl(F_SETLK)` behavior in separate processes,
not goroutines within the same process (POSIX locks are per-process, not
per-fd, so same-process tests would be meaningless).

### Limitations

- **Windows**: real `LockFileEx`/`UnlockFileEx` implementation in
  `locks_windows.go` using `kernel32.dll` direct syscalls (no external
  dependency). Note: full Windows cross-compile requires separate fixes
  for mmap (`hbrdd/cdx`, `hbrdd/ntx`) and termios (`hbrt/debugcli.go`)
  which are unrelated to locking.
- **Advisory only**: processes that don't call `FLOCK`/`DBRLOCK` bypass
  protection. Same as Harbour and Clipper — this is expected behavior.
- **No timeout**: Harbour's `HB_SET_LOCKRETRY` is not honored. Callers
  must implement their own retry loop.
- **No deadlock detection**: POSIX `F_SETLK` is non-blocking, so
  deadlocks are not possible, but starvation is.

---

## What's Still Missing (Ranked by Impact)

### 🔴 Blocker for enterprise adoption

1. **CDX write support** — Five reads Harbour CDX byte-for-byte but
   cannot create or modify them. `INDEX ON field TAG name TO bag`
   currently produces an NTX file. Estimated: 2 weeks
   ([`hbrdd/cdx/cdx.go`](../hbrdd/cdx/cdx.go) needs `CreateIndex`,
   `InsertKey`, structural root rebuilder).

2. **SQL RDDs** — No MySQL, PostgreSQL, ODBC, or Advantage. Makes Five
   unsuitable for apps that mix DBF with modern RDBMS.
   Estimated: 4-8 weeks per driver via `database/sql`.

3. **Threading exposed to PRG** — `hb_threadStart`, `hb_mutexCreate`,
   `hb_condNew`, `hb_threadSelf` are all missing. Goroutines exist
   inside the runtime but aren't callable from PRG. Harbour ports use
   `hb_thread*` heavily for background workers.
   Estimated: 1 week (wrap Go primitives).

4. **Windows file locking** — no-op stub, see above. Any Windows
   deployment is currently single-writer only.
   Estimated: 1 day.

### 🟡 Blocker for specific domains

5. **Networking** — no `hb_socketConnect`, `hb_inetConnect`, HTTP
   client, SSL. Harbour apps that talk to web APIs or use hbnetio for
   remote DBF won't run.
   Estimated: 2 weeks (`net` + `net/http` wrappers).

6. **JSON** — `hb_JSONEncode`/`hb_JSONDecode` not registered.
   Trivial to add via `encoding/json`.
   Estimated: 1 day.

7. **Compression** — `hb_ZCompress`/`hb_ZUncompress` missing. Required
   for FPT BLOB compression, network protocols.
   Estimated: 1 day (`compress/zlib`).

8. **Serialization** — `hb_Serialize`/`hb_Deserialize` missing. Any
   code that persists Harbour objects or sends them over sockets
   breaks.
   Estimated: 1 week (needs VM type bridge).

9. **DBT memo** — dBASE III memo format not supported. Old Clipper
   codebases that haven't migrated to FPT will fail on open.
   Estimated: 3 days.

### 🟢 Polish / ecosystem

10. **Class advanced features** — `CLASSDATA`, `HIDDEN`/`PROTECTED`/
    `EXPORTED`, `PROPERTY`, operator overload. Framework-style code
    that uses these won't compile.
    Estimated: 2 weeks (analyzer + gengo + runtime).

11. **GET/SAY form system** — Harbour's `@ row, col GET var` /
    `READ` system is barely implemented. Classic text-mode data
    entry apps don't work.
    Estimated: 2 weeks (GTCGI integration + validation).

12. **FRM report forms** — no support for `.frm` report files.
    Clipper reports are dead.
    Estimated: 2 weeks.

13. **Advanced TBrowse** — only basic scrolling. No custom headers,
    column blocks, keyboard handlers.
    Estimated: 1 week.

14. **`hbmk2` equivalent** — no `.hbp` project files, no library
    linking, no resource files, no parallel builds. `five build` is
    just `go build` with PRG → Go transpilation.
    Estimated: 3 weeks for basic `.hbp` parsing.

15. **Contrib libraries** — Harbour has 100+ contrib libs (hbcurl,
    hbssl, hbmysql, hbpgsql, hbqt, hbfship, hbgd, hbcairo, …). Five
    has none.
    Estimated: each contrib is 1-4 weeks.

---

## Language/Compiler Corner Cases

From [`CLAUDE.md`](../CLAUDE.md):

| Issue | Workaround |
|---|---|
| `STATIC inside FUNCTION` triggers `local index 0` error | Use module-level `STATIC` |
| Semicolon-inline `IF ... ENDIF` on one line | Split into multiple lines |
| `LOCAL` inside `IF`/`FOR` block | Declare all `LOCAL`s at function top |
| `FIELD->NAME` (fixed 2026-04-11) | Now works — see commit `e95afad` |
| `OrdSetFocus(n)` numeric (fixed 2026-04-11) | Now works — same commit |
| `OutStd()`/`OutErr()` (fixed 2026-04-11) | Now registered |
| `Date + (i % 365)` panic (fixed 2026-04-11) | Fixed in `6c53747` |
| `INDEX ON ... TAG name TO bag` emits NTX, not CDX | Open issue — gengo drops TAG parameter |

---

## Quantitative Comparison

| Metric | Five | Harbour |
|---|---:|---:|
| RTL functions registered | **483** | ~700+ |
| Core LOC | ~47,000 Go | ~500,000 C + PRG |
| RDD drivers | 4 (DBF, NTX, CDX-read, MEM) | 15+ |
| Compiler platforms | darwin/linux/windows/freebsd (any Go target) | 30+ including AIX, HP-UX, OS/2, DOS, Win CE |
| Built-in SQL engine | ✅ FiveSql2 (43/43 tests) | ❌ |
| Contrib libraries | 0 | 100+ |
| GUI frameworks | 0 | 4+ (HWGUI, HMG, MiniGUI, Qt) |
| Test suites in tree | compat_harbour (51), FiveSql2 (43), Go units | Thousands |

---

## Performance (50k records, Apple M-series, same DBFNTX workload)

| Operation | Harbour 3.2 | **Five 0.1** | Winner |
|---|---:|---:|:---|
| APPEND 50k | 90.7 ms | **86.6 ms** | ⚡ Five 5% faster |
| INDEX 4 keys | 38.0 ms | **33.3 ms** | ⚡ Five 12% faster |
| SEEK 10k | 12.3 ms | **6.8 ms** | ⚡ Five 45% faster |
| SKIP forward scan | 4.0 ms | **3.4 ms** | ⚡ Five 15% faster |
| SKIP reverse scan | 4.0 ms | 4.1 ms | ≈ tie |
| GOTO 10k random | 5.0 ms | **1.5 ms** | ⚡ Five 3.3x faster |
| **Total** | **154.0 ms** | **132.7 ms** | ⚡ Five 14% faster |

Source: [`docs/benchmarks.md`](benchmarks.md) (if/when published).
Reproducible via `bench_core.prg` in the session archive.

---

## Honest Positioning

**Five is a Harbour-compatible runtime optimized for modern cloud-native
deployments of classic dBASE/Clipper business logic.**

### When to use Five

- ✅ Migrating a CLI data-processing app from Harbour to a single
  cross-compiled binary (no CGo, no dynamic linker, no glibc matching)
- ✅ Running legacy DBF-based ETL / reporting pipelines on Linux
  containers or macOS
- ✅ Adding SQL:1999 query capabilities to an existing DBF codebase
  (FiveSql2 — not available in Harbour)
- ✅ Performance-critical RDD operations where Five is measurably
  faster than Harbour
- ✅ Multi-process shared DBF workloads on Unix (now that locks work)

### When NOT to use Five (yet)

- ❌ Harbour apps that use `@...GET`, `READ`, `FRM` reports, or
  TBrowse-based data entry — the UI layer is too minimal
- ❌ Apps that need SQL RDDs (MySQL, PostgreSQL, ODBC, ADS)
- ❌ Apps that use threading, sockets, HTTP, or SSL from PRG
- ❌ Apps that require CDX index creation (CDX read works; write is
  missing)
- ❌ Windows deployments needing multi-writer safety (Windows lock
  stub is still no-op)
- ❌ Anything depending on contrib libraries (hbcurl, hbssl, hbqt, …)

### Roadmap priorities

In order of impact on "can this run real Harbour code":

1. Windows `LockFileEx` (1 day)
2. JSON RTL (1 day)
3. Compression RTL (1 day)
4. Threading RTL (1 week)
5. Networking RTL + SSL (2 weeks)
6. CDX write support (2 weeks)
7. Serialization (1 week)
8. SQL RDD (`database/sql` bridge) (4-8 weeks)

The first 5 items would close the gap for most server-side data workloads
within a month of focused work.

---

_Maintained by: Charles KWON OhJun <charleskwonohjun@gmail.com>_
_Honesty policy: this document MUST be updated when features are added
or limitations are discovered. Do not remove items — mark them ✅ or
move them to a "fixed" section so readers see the history._