Search Concepts

Architecture Overview

LumoSearch uses a multi-stage retrieval and ranking pipeline:

1. Index — Normalize and tokenize configured fields. Build token, trigram, and prefix inverted indexes.
2. Retrieve — For each query, gather candidates from postings (token 4x, prefix 3x, trigram 1x weight).
3. Prune — Keep only the top candidateLimit candidates based on weighted postings.
4. Score — Rank with BM25F + exact/prefix/phrase/proximity boosts.
5. Return — Top-k results with character-level highlights.

BM25F Ranking

BM25F is an industry-standard ranking algorithm that considers:

• Term Frequency (TF) — How often query terms appear in a document
• Inverse Document Frequency (IDF) — Rarity of terms across all documents
• Field Weights — Custom weights per field (e.g., title 3x, body 1x)
• Length Normalization — Prevents long documents from dominating

The "F" in BM25F means field-weighted: you can boost importance of specific fields like titles, tags, or names.

// Title matches are 3x more important than body matches
const search = new LumoSearch(docs, {
  keys: [
    { name: 'title', weight: 3 },
    { name: 'body', weight: 1 }
  ]
})

Fuzzy Matching

LumoSearch handles typos and misspellings using trigram-based fuzzy matching:

• Each token is split into 3-character sequences (trigrams)
• Example: "javascript" → ["jav", "ava", "vas", "asc", "scr", "cri", "rip", "ipt"]
• Query tokens are matched against indexed trigrams using Jaccard similarity
• Allows "javscrippt" to match "javascript" even with typos

Candidate Pruning

To stay fast, LumoSearch doesn't score every document. Instead:

1. Gather candidate documents from inverted indexes
2. Score candidates using weighted postings (token matches 4x, prefix 3x, trigram 1x)
3. Keep only top candidateLimit candidates (default: 250)
4. Run full BM25F scoring on these candidates
5. Return top limit results (default: 10)

// Higher candidateLimit = better recall, slower performance
const search = new LumoSearch(docs, {
  keys: ['title', 'body'],
  candidateLimit: 500,  // Consider more candidates
  limit: 20             // Return top 20 results
})

Inverted Indexes

LumoSearch builds three inverted indexes at initialization:

• Token Index — Exact token matches (e.g., "javascript")
• Prefix Index — Token prefixes (e.g., "java" matches "javascript")
• Trigram Index — 3-character sequences for fuzzy matching

Each index maps tokens/trigrams to document IDs and field positions. This enables fast candidate retrieval.

Scoring Boosts

On top of BM25F, LumoSearch applies several heuristic boosts:

• Exact Match Boost (2.0x) — Full token matches get double weight
• Prefix Boost (1.25x) — Prefix matches get 25% bonus
• Phrase Proximity — Query terms appearing close together score higher
• Field Weight — User-configured per-field weights

Text Normalization

Before indexing or querying, text is normalized:

1. Convert to lowercase
2. Apply Unicode NFD normalization
3. Remove diacritics (é → e, ñ → n)
4. Tokenize on whitespace and punctuation

This ensures "Café" matches "cafe" and "résumé" matches "resume".

Synonyms

Token-level synonyms expand query terms at search time:

const search = new LumoSearch(docs, {
  keys: ['title', 'body'],
  synonyms: {
    js: ['javascript'],
    auth: ['authentication', 'authorization']
  }
})

// Searching "js auth" will also match:
// "javascript authentication"
// "javascript authorization"
// etc.

Hybrid Reranking

For semantic search, use searchAsync() with a custom reranker:

const results = await search.searchAsync('modern UI frameworks', {
  rerankLimit: 20,
  reranker: {
    async rerank({ query, candidates }) {
      // 1. LumoSearch retrieves top 20 lexical candidates
      // 2. Your reranker scores them semantically
      // 3. Final results combine both scores
      const embeddings = await getEmbeddings([
        query,
        ...candidates.map(c => c.text)
      ])
      return candidates.map((c, i) => ({
        refIndex: c.refIndex,
        score: cosineSimilarity(embeddings[0], embeddings[i + 1])
      }))
    }
  }
})

This two-stage approach combines the speed of lexical search with the semantic understanding of embeddings.

Frequently Asked Questions

Related