Show HN: I wrote a VectorDB in Go, built for Hackers, not Hyperscalers

A high-performance hybrid vector index written in Go. Comet brings together multiple indexing strategies and search modalities into a unified, hackable package. Hybrid retrieval with reciprocal rank fusion, autocut, pre-filtering, semantic search, full-text search, and multi-KNN searches, and multi-query operations — all in pure Go.

Understand search internals from the inside out. Built for hackers, not hyperscalers. Tiny enough to fit in your head. Decent enough to blow it.

Choose from:

• Flat (exact), HNSW (graph), IVF (clustering), PQ (quantization), or IVFPQ (hybrid) indexes
• Full-Text Search: BM25 ranking algorithm with tokenization and normalization
• Metadata Filtering: Fast filtering using Roaring Bitmaps and Bit-Sliced Indexes
• Ranking Programmability: Reciprocal Rank Fusion, Fixed size result sets, Threshold based result sets, Dynamic result sets etc.
• Hybrid Search: Unified interface combining vector, text, and metadata search

• Overview
• Features
• Installation
• Quick Start
• Architecture
• Core Concepts
• API Reference
• Examples
• Configuration
• Best Practices
• Testing
• Use Cases
• Contributing
• License

Everything you need to understand how vector databases actually work—and build one yourself.

What's inside:

• 5 Vector Index Types: Flat, HNSW, IVF, PQ, IVFPQ
• 3 Distance Metrics: L2, L2 Squared, Cosine
• Full-Text Search: BM25 ranking with Unicode tokenization
• Metadata Filtering: Roaring bitmaps + Bit-Sliced Indexes
• Hybrid Search: Combine vector + text + metadata with Reciprocal Rank Fusion
• Advanced Search: Multi-KNN queries, multi-query operations, autocut result truncation
• Production Features: Thread-safe, serialization, soft deletes, configurable parameters

Everything you need to understand how vector databases actually work—and build one yourself.

• Flat: Brute-force exact search (100% recall baseline)
• HNSW: Hierarchical navigable small world graphs (95-99% recall, O(log n) search)
• IVF: Inverted file index with k-means clustering (85-95% recall, 10-20x speedup)
• PQ: Product quantization for compression (85-95% recall, 10-500x memory reduction)
• IVFPQ: IVF + PQ combined (85-95% recall, 100x speedup + 500x compression)

• Vector Search: L2, L2 Squared, and Cosine distance metrics
• Full-Text Search: BM25 ranking with Unicode-aware tokenization
• Metadata Filtering: Boolean queries on structured attributes
• Hybrid Search: Combine all three with configurable fusion strategies

• Weighted Sum: Linear combination with configurable weights
• Reciprocal Rank Fusion (RRF): Scale-independent rank-based fusion
• Max/Min Score: Simple score aggregation

• HNSW Graphs: Multi-layer skip lists for approximate nearest neighbor search
• Roaring Bitmaps: Compressed bitmaps for metadata filtering (array, bitmap, run-length encoding)
• Bit-Sliced Index (BSI): Efficient numeric range queries without full scans
• Product Quantization Codebooks: Learned k-means centroids for vector compression
• Inverted Indexes: Token-to-document mappings for full-text search

• Quantization: Full precision, half precision, int8 precision
• Soft Deletes: Fast deletion with lazy cleanup
• Serialization: Persist and reload indexes
• Thread-Safe: Concurrent read/write operations
• Autocut: Automatic result truncation based on score gaps

Output:

Manages vector storage and similarity search across multiple index types.

Responsibilities:

• Vector preprocessing (normalization for cosine distance)
• Distance calculations (Euclidean, L2², Cosine)
• K-nearest neighbor search
• Serialization/deserialization
• Soft delete management with flush mechanism

Performance Characteristics:

• Flat: O(n×d) search, 100% recall
• HNSW: O(M×ef×log n) search, 95-99% recall
• IVF: O(nProbes×n/k×d) search, 85-95% recall

Full-text search using BM25 ranking algorithm.

Responsibilities:

• Text tokenization (UAX#29 word segmentation)
• Unicode normalization (NFKC)
• Inverted index maintenance
• BM25 score calculation
• Top-K retrieval with heap

Performance Characteristics:

• Add: O(m) where m = tokens
• Search: O(q×d_avg) where q = query terms, d_avg = avg docs per term
• Memory: Compressed inverted index, no original text stored

Fast filtering using compressed bitmaps.

Responsibilities:

• Bitmap index maintenance (Roaring compression)
• BSI for numeric range queries
• Boolean query evaluation (AND, OR, NOT)
• Existence checks
• Set operations (IN, NOT IN)

Performance Characteristics:

• Add: O(f) where f = fields
• Query: O(f×log n) with bitmap operations
• Memory: Highly compressed, 1-10% of uncompressed

A vector index stores high-dimensional embeddings and enables efficient similarity search. The choice of index type determines the tradeoff between search speed, memory usage, and accuracy.

Example: How vectors are stored and searched

Given these input vectors:

The index computes distances and returns nearest neighbors:

Visual Representation: Index Types Comparison

Benefits of Different Index Types:

• Flat Index: Perfect recall, simple, no training. Use for small datasets (<100K vectors)
• HNSW: Excellent speed/accuracy tradeoff, no training. Use for most production workloads
• IVF: Fast filtering with clusters, scalable. Use for very large datasets (>10M vectors)
• PQ: Massive memory savings. Use when storage cost is a concern
• IVFPQ: Best of IVF + PQ. Use for extremely large, memory-constrained environments

BM25 (Best Matching 25) ranks documents by relevance to a text query using term frequency and inverse document frequency.

Step-by-Step Algorithm Execution:

Why Doc 3 scores higher than Doc 1:

• Both have same term frequencies (1 occurrence each)
• Doc 3 is shorter (4 words vs 9 words)
• BM25's length normalization penalizes longer documents
• Shorter documents with same term frequency are considered more relevant

Hybrid search combines vector similarity, text relevance, and metadata filtering. Different fusion strategies handle score normalization differently.

Fusion Strategy Visualization:

Why RRF over Weighted Sum:

• Scale Independence: Vector distances (0-2) and BM25 scores (0-100+) have different ranges
• Robustness: Ranks are stable even if score distributions change
• No Tuning: Weighted sum requires manual weight calibration
• Industry Standard: Used by Elasticsearch, Vespa, and other search engines

Choose your poison: Flat (exact), HNSW (graph), IVF (clustering), PQ (quantization), or IVFPQ (hybrid). Each trades speed, memory, and accuracy differently.

Full Deep Dive: See INDEX.md for complete algorithms, benchmarks, and implementation details.

The Simplest Approach: Compare query against EVERY vector. 100% recall, zero approximation.

How it works:

• Store vectors as raw float32 arrays
• Search = compute distance to every vector, keep top-k
• No index structure, no preprocessing

Complexity:

• Search: O(n × d) where n = vectors, d = dimensions
• Memory: O(n × d)
• Build: O(1)

When to use: Small datasets (<10k vectors), when 100% recall is mandatory, benchmarking baseline.

The Graph Navigator: Build a multi-layer graph where each node connects to nearby vectors. Search by greedy navigation from layer to layer.

How it works:

• Each vector exists on Layer 0, subset on higher layers (exponentially fewer)
• Each node has M connections to nearest neighbors per layer
• Search: Start at top, greedily move to closest neighbor, descend layers
• Insert: Add to Layer 0, probabilistically add to higher layers, connect to M nearest

Complexity:

• Search: O(M × efSearch × log n) - typically check <1% of vectors
• Insert: O(M × efConstruction × log n)
• Memory: O(n × d + n × M × L) where L ≈ log(n)

Key Parameters:

• M: Connections per layer (16-48, higher = better recall but more memory)
• efConstruction: Build-time quality (200-500, higher = better graph)
• efSearch: Search-time quality (100-500, higher = better recall)

When to use: Large datasets, need 90-99% recall with <1ms latency, willing to trade memory for speed.

The Clustering Approach: Partition vectors into clusters using k-means. Search only the nearest clusters.

How it works:

• Build: k-means clustering to create nClusters partitions
• Each vector stored in nearest cluster
• Search: Find nProbe nearest centroids, search only those clusters

Complexity:

• Search: O(nClusters × d + (n/nClusters) × nProbe × d) - typically check 5-20% of vectors
• Build: O(iterations × n × nClusters × d)
• Memory: O(n × d + nClusters × d)

Key Parameters:

• nClusters: Number of partitions (√n to n/10, more = faster search but slower build)
• nProbe: Clusters to search (1-20, higher = better recall but slower)

When to use: Very large datasets (>100k vectors), can tolerate 80-95% recall, want to reduce search from O(n) to O(n/k).

The Compression Master: Split vectors into subvectors, quantize each subvector to 256 codes. Reduce memory by 16-32×.

How it works:

• Split d-dimensional vectors into m subvectors of d/m dimensions
• Train codebook: k-means on each subspace to create 256 centroids
• Encode: Replace each subvector with nearest centroid index (uint8)
• Search: Precompute distance tables, scan using table lookups

Complexity:

• Search: O(m × k + n × m) where k=256 codes - super fast, cache-friendly
• Memory: O(n × m) bytes vs O(n × d × 4) bytes for float32
• Compression: 16-32× smaller (float32 → uint8)

Key Parameters:

• nSubspaces (m): Subvector count (8-64, more = better accuracy but more memory)
• bitsPerCode: Usually 8 (256 codes per subspace)

When to use: Massive datasets (millions of vectors), memory-constrained, can tolerate 70-85% recall for 30× memory savings.

Best of Both Worlds: IVF clusters to reduce search space, PQ compression to reduce memory. The ultimate scalability index.

How it works:

• Combines IVF partitioning with PQ compression
• IVF reduces vectors to search (search 10% instead of 100%)
• PQ reduces memory per vector (32× compression)
• Search: IVF to find clusters, PQ for fast distance within clusters

Complexity:

• Search: O(nClusters × d + (n/nClusters) × nProbe × m) - fast scan, small memory
• Memory: O(n × m) bytes + O(nClusters × d) for centroids
• Build: O(IVF_train + PQ_train)

Key Parameters:

• nClusters: Number of IVF partitions (√n to n/10)
• nProbe: Clusters to search (1-20)
• nSubspaces (m): PQ subvectors (8-64)

When to use: Billion-scale datasets, need <10ms latency with <1GB memory for millions of vectors, can tolerate 70-90% recall.

Rules of thumb:

• Small data (<10k): Flat - brute force is fast enough
• Medium data (10k-1M): HNSW - best recall/speed tradeoff
• Large data (1M-100M): IVF or PQ - choose speed (IVF) vs memory (PQ)
• Massive data (>100M): IVFPQ - only option that scales to billions

Latency targets:

• Flat: 1-100ms (depends on n)
• HNSW: 0.1-2ms (sub-millisecond possible)
• IVF: 0.5-5ms
• PQ: 1-10ms (fast scan but approximate)
• IVFPQ: 0.5-3ms (fastest for massive datasets)

For detailed API documentation, see API.md.

For practical examples and code samples, see EXAMPLE.md.

M (Connections Per Layer)

• Type: int
• Default: 16
• Range: 4 to 64
• Description: Number of bidirectional connections per node at each layer (except layer 0 which uses 2×M)

Effects of different values:

efConstruction (Build-Time Candidate List Size)

• Type: int
• Default: 200
• Range: 100 to 1000
• Description: Size of candidate list during index construction. Higher = better graph quality

Effects:

efSearch (Search-Time Candidate List Size)

• Type: int
• Default: 200
• Range: k to 1000 (must be >= k)
• Description: Size of candidate list during search. Can be adjusted dynamically.

Effects:

Example:

nClusters (Number of Clusters)

• Type: int
• Default: sqrt(n) where n = dataset size
• Range: 16 to n/100
• Description: Number of k-means clusters (Voronoi cells) to partition the space

nProbes (Search-Time Clusters to Probe)

• Type: int
• Default: 1
• Range: 1 to nClusters
• Description: Number of nearest clusters to search during query

Example:

Explanation: These patterns optimize performance and memory usage.

Explanation: These patterns waste resources or cause errors.

Current coverage statistics (as of latest commit):

Example test structure:

Example benchmark results:

Problem: A documentation platform needs to find relevant documents based on user queries, going beyond simple keyword matching to understand intent.

Solution: Use Comet's vector search with text embeddings from a sentence transformer model.

Implementation:

Problem: Recommend similar products based on browsing history, with filtering by price, category, and availability.

Solution: Hybrid search combining product embeddings with metadata filters.

Implementation:

Problem: Build a QA system that combines semantic understanding (vector search) with keyword precision (BM25) for better answer retrieval.

Solution: Use RRF fusion to combine both modalities.

Implementation:

Contributions are welcome! Please follow these guidelines:

• Follow standard Go conventions (gofmt, go vet)
• Write descriptive comments for exported functions
• Include examples in documentation comments
• Add tests for new features (maintain >95% coverage)
• Keep functions focused and composable
• Use meaningful variable names

Follow conventional commits format:

Good commit messages:

• feat: Add IVF index with k-means clustering
• fix: Handle zero vectors in cosine distance
• perf: Optimize HNSW search with heap pooling
• docs: Update API reference for hybrid search
• test: Add benchmarks for BM25 search

Bad commit messages:

• Update code
• Fix bug
• WIP
• asdf

1. Fork the repository and create a feature branch
2. Write your code following the style guidelines
3. Add tests for all new functionality
4. Run the full test suite and ensure all tests pass
5. Update documentation (README, godoc comments)
6. Commit with descriptive messages using conventional commits
7. Push to your fork and open a Pull Request
8. Respond to review feedback promptly

Before submitting a PR, ensure:

• All tests pass (make test)
• Linter shows no issues (make lint)
• Coverage remains >95% (make test-coverage)
• Documentation is updated
• Benchmarks show no regression (if applicable)
• No breaking changes (unless discussed)

MIT License - see LICENSE file for details

Copyright (c) 2025 wizenheimer

• HNSW Algorithm: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
• Product Quantization: Product quantization for nearest neighbor search
• BM25 Ranking: The Probabilistic Relevance Framework: BM25 and Beyond
• Roaring Bitmaps: Better bitmap performance with Roaring bitmaps
• Reciprocal Rank Fusion: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods