# Rushmore Bitmap Index — Five's Query Optimization

> FoxPro Rushmore technology adapted for Five's RDD system

Copyright (c) 2026 Charles KWON OhJun (charleskwonohjun@gmail.com). All rights reserved.

## Overview

Rushmore is a query optimization technology originally developed by Fox Software
(later Microsoft FoxPro). It uses bitmap indexes to dramatically accelerate
filtered record navigation. Five implements this as the BMDBF* family of RDD
drivers and the `BM_*` RTL functions.

## The Problem

Traditional `SET FILTER TO` evaluates the filter condition for **every record**
during each `SKIP` operation:

```harbour
USE customers
SET FILTER TO CITY = "Seoul"
GO TOP          // evaluates CITY="Seoul" for records 1,2,3... until match
SKIP            // evaluates CITY="Seoul" for next records until match
```

For a table with 1,000,000 records where only 1% match, each `SKIP` must
test ~100 records on average. Navigation is O(N) per skip.

## The Rushmore Solution

Rushmore pre-computes a **bitmap** — one bit per record — before navigation
begins. `SKIP` then only needs to find the next set bit:

```harbour
USE customers VIA "BMDBFNTX"
BM_DbSetFilter({|| CITY = "Seoul"})   // builds bitmap: 1 scan of all records
GO TOP          // finds first set bit: O(1)
SKIP            // finds next set bit: O(1)
```

The bitmap is a compact bit array: 1,000,000 records = 122 KB of memory.

## How It Works

### Bitmap Structure

```
Record:   1  2  3  4  5  6  7  8  9  10 11 12 ...
City:     S  T  N  L  P  S  T  N  L  P  S  T  ...
Filter:   1  0  0  0  0  1  0  0  0  0  1  0  ...
          ^              ^              ^
          Seoul          Seoul          Seoul
```

Each record gets one bit. The bitmap is stored as `[]uint64` — 64 records
per machine word, enabling hardware-accelerated bit operations.

### Compound Filters

When combining multiple conditions, Rushmore uses bitwise operations
instead of evaluating compound expressions:

```harbour
// Traditional: evaluates BOTH conditions per record
SET FILTER TO CITY = "Seoul" .AND. AGE > 30

// Rushmore: builds TWO bitmaps, combines with AND
bitmap_city := BM_DbSetFilter({|| CITY = "Seoul"})    // 1 full scan
bitmap_age  := BM_DbSetFilter({|| AGE > 30})          // 1 full scan
result      := bitmap_city AND bitmap_age              // 50μs for 1M records!
```

Bitwise AND/OR on 1M records takes **50 microseconds** — faster than
evaluating even a single record's compound condition.

## Benchmark Results

Tested on Intel Core Ultra 7 255H, 1,000,000 records:

| Operation | Time | Memory | Description |
|-----------|------|--------|-------------|
| **Bitmap NextSet (1% match)** | **45 μs** | 0 alloc | Traverse all matching records |
| **Sequential Scan** | **102 μs** | 0 alloc | Check every record |
| **Speedup** | **2.2x** | | Bitmap vs sequential |
| **Bitmap AND (1M)** | **50 μs** | 131 KB | Combine two filter bitmaps |
| **Bitmap OR (1M)** | **52 μs** | 131 KB | Union two filter bitmaps |
| **Bitmap NOT (1M)** | instant | 131 KB | Invert filter |
| **NextSet (99% dense)** | **1 ns** | 0 alloc | Nearly all records match |

### When Rushmore Wins

| Scenario | Sequential | Rushmore | Speedup |
|----------|-----------|----------|---------|
| 1% match rate, 1M records | 102 μs | 45 μs | **2.2x** |
| Compound AND (2 conditions) | 200+ μs | 50 μs | **4x+** |
| Compound AND+OR (3 conditions) | 300+ μs | 100 μs | **3x+** |
| Repeated navigation (same filter) | N × 102 μs | N × 45 μs | **2.2x per pass** |
| Dense match (99%) | 102 μs | ~1 ns/call | **100x+** |

### When Sequential Is Fine

- Small tables (< 1000 records): overhead of bitmap creation outweighs benefit
- One-time scan (no repeated navigation): bitmap build cost amortized over zero reuses
- No filter (full table scan): no bitmap needed

## API Reference

### Drivers

| Driver | Base | Description |
|--------|------|-------------|
| `BMDBFNTX` | DBFNTX | Bitmap + NTX index |
| `BMDBFCDX` | DBFCDX | Bitmap + CDX compound index |
| `BMDBFNSX` | DBFNSX | Bitmap + NSX index |

### Functions

| Function | Description |
|----------|-------------|
| `BM_DbSetFilter(bBlock)` | Build bitmap by evaluating block on all records |
| `BM_DbSeekWild(cPattern)` | Wildcard seek (e.g., `"Park*"`) |
| `BM_Turbo(lOnOff)` | Enable/disable turbo mode |
| `BM_DbGetFilterArray()` | Get matching record numbers as array |
| `BM_DbSetFilterArray(aRecNos)` | Set bitmap from record number array |
| `BM_DbSetFilterArrayAdd(aRecNos)` | Add records to bitmap |
| `BM_DbSetFilterArrayDel(aRecNos)` | Remove records from bitmap |

### Internal Operations

| Operation | Function | Description |
|-----------|----------|-------------|
| `AND` | `bitmap.And(other)` | Intersection of two filters |
| `OR` | `bitmap.Or(other)` | Union of two filters |
| `NOT` | `bitmap.Not()` | Invert filter |
| `NextSet(n)` | `bitmap.NextSet(n)` | Find next matching record |
| `PrevSet(n)` | `bitmap.PrevSet(n)` | Find previous matching record |
| `Count()` | `bitmap.Count()` | Number of matching records |

## Implementation Details

### Memory Usage

```
Records      Bitmap Size    Per Record
1,000        128 bytes      1 bit
10,000       1.2 KB         1 bit
100,000      12.2 KB        1 bit
1,000,000    122 KB         1 bit
10,000,000   1.2 MB         1 bit
```

### Go Optimization

Five's bitmap uses Go's `math/bits` package for hardware-accelerated
bit operations:

- `bits.TrailingZeros64()` — find first set bit in O(1) via CPU instruction
- `bits.OnesCount64()` — population count via POPCNT instruction
- 64-bit word operations — process 64 records per CPU cycle

This is significantly faster than Harbour's C implementation because
Go's compiler inlines these as single CPU instructions on modern hardware.

## Comparison with Other Index Types

| Feature | NTX | CDX | Rushmore Bitmap |
|---------|-----|-----|-----------------|
| **Seek** | B-tree O(log n) | B-tree O(log n) | Linear scan (build) |
| **Ordered navigation** | Yes | Yes | No (position only) |
| **Filter optimization** | No | No | **Yes — primary purpose** |
| **Compound conditions** | No | No | **AND/OR/NOT in μs** |
| **Memory** | File-based | File-based | In-memory (122KB/1M) |
| **Build time** | Sort + write | Sort + write | Single sequential scan |
| **Update cost** | O(log n) per change | O(log n) | Rebuild bitmap |

### When to Use Which

| Use Case | Recommended |
|----------|-------------|
| Ordered browsing (A-Z) | NTX or CDX |
| Key lookup (SEEK) | NTX or CDX |
| Complex filtered navigation | **Rushmore bitmap** |
| `CITY="Seoul" .AND. AGE>30` | **Rushmore bitmap** |
| Large table with small result set | **Rushmore bitmap** |
| Compound conditions on multiple fields | **Rushmore bitmap** |
| Static reference tables | CDX (persistent) |
| Frequently updated tables | NTX (simple) |

## Migration from Harbour

| Harbour | Five |
|---------|------|
| `REQUEST BMDBFCDX` | Automatic (driver pre-registered) |
| `USE ... VIA "BMDBFCDX"` | Same syntax |
| `BM_DBSETFILTER(bBlock)` | Same function |
| `BM_DBSEEKWILD(cPattern)` | Same function |
| `BM_TURBO(lOnOff)` | Same function |
| Manual bitmap arrays | Same API |

Five adds Go-native bitmap operations that leverage modern CPU instructions
(POPCNT, TZCNT) for additional performance over Harbour's C implementation.

## Verified Test Results

```
=== Bitmap Unit Tests: 6/6 PASS ===
Basic, NextSet, Full, And, Or, Not

=== Benchmarks (1M records) ===
Bitmap NextSet (sparse):  45,334 ns/op    0 allocs
Sequential Scan:          101,620 ns/op   0 allocs
Bitmap AND:               50,235 ns/op    2 allocs
Bitmap OR:                52,210 ns/op    2 allocs
```