Skip to main content
Vector indexes are robust tools in facilitating fast searches across large numeric datasets. LanceDB implements ANN (Approximate Nearest-Neighbor) queries with several techniques that provide benefits across a variety of use cases.

​
Choosing the Right Index

LanceDB vector indexes are combined with several quantization techniques to admit efficient storage. The following table lists provided quantized vector indexes and their common use cases. You can specify index type manually in Lance with index_type.
If your vector search frequently includes metadata filters (where(...)), use IVF_RQ or IVF_PQ. In filtered workloads, HNSW-backed IVF indexes such as IVF_HNSW_FLAT and IVF_HNSW_SQ can show higher latency variance.

​
Understanding Vector Indexes

LanceDB offers two vector indexes, which can be created on any numeric Lance dataset: Inverted File (IVF) and Hierarchical Navigable Small World (HNSW).

​
IVF

The Inverted File Index (IVF) accelerates ANN searches by drastically reducing the search space. The index consists of a small set of centroids corresponding to an approximate solution to the kk-means clustering problem. Each vector remembers its nearest centroid, and each centroid remembers its associated set of vectors, called its partition.

Example partition of a geometric space, with colored marks denoting centroids.

At query time, we can compare the queried vector to the smaller set of centroids (as opposed to the entire dataset) for a closest match, then run a brute-force comparison against its resulting partition. This technique quickly prunes a large search space, giving an approximate ANN result. However, observe that a queried vector may lie near the boundary of 22 or more partitions; thus, the true nearest neighbors are scattered across several different partitions. To address this, LanceDB exposes the nprobes parameter, which specifies the number of partitions searched. A high nprobes parameter will yield more accurate results at slightly higher runtime.

Edge case for single-partition brute force.

​
HNSW

Hierarchical Navigable Small World (HNSW) constructs a layered graph hierarchy on the vector set, with edges representing distances. We can visualize an HNSW index as a vertical stack of graphs, with the top layer having very few edges and each other layer having a multiplicative factor more edges than the layer above it. High layers of a HNSW hierarchy represent sparse, higher-distance networks, and lower layers represent finer, lower-distance networks. To query a vector qq, the index proceeds iteratively through layers, first finding qq‘s nearest neighbor in the graph, then proceeding recursively through the induced subhierarchy until the lowest layer is reached.

HNSW hierarchy: sparse network at the top layer, fine network at the bottom.

To visualize this process, imagine that you must drive your car from San Francisco to a specific house in Boston. Initially, you must first drive thousands of miles on the interstate freeway I-90 E. Eventually, you merge onto the Massachusetts Turnpike, the center of the greater Boston highway system. From there, you use a series of increasingly smaller, narrower roads within the city (Charles River Bridge, St. Paul St, Thatcher St) before finally reaching the house. A key observation is that you must initially travel far distances through long-distance road networks (the interstate freeway system), before proceeding to finer and finer road networks (greater Boston highway system, central Brookline neighborhood connectors) before finally reaching your destination. This iterative series of road networks mimics the layered graph traversals performed by a HNSW query.
IVF + HNSWIn LanceDB, HNSW is not exposed as a top-level vector index. Instead, it’s available as a substructure which further indexes the selected vectors inside each IVF partition. This combines the scalability of IVF with the high recall of HNSW. LanceDB supports IVF-HNSW-based quantized indexes IVF_HNSW_FLAT, IVF_HNSW_PQ, and IVF_HNSW_SQ.

​
Using Vector Indexes

Learn how to configure, build, and search LanceDB vector indexes, including build and search time parameters, asynchronous objects, and several examples.

​
Configuration

​
Build-time Parameters

​
Search-time Parameters

Recommended nprobes behavior by index type:
Filtered queries and adaptive nprobes. When a where(...) filter is active, LanceDB initially scans minimum_nprobes partitions and uses a wider scan if too few rows are found. Set minimum_nprobes == maximum_nprobes or call nprobes(n) to instead fix the partition count.
Here is an example of a vector search exercising several of the above parameters. LanceDB also supports advanced search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints. Thresholding with distance_range: Using bypass_vector_index: Use bypass_vector_index to compute an exact kNN result. Note that exact queries may be prohibitively slow on production scales.
Multivector indexing currently requires distance_type="cosine". Use bypass_vector_index() for non-cosine queries on a multivector column. See Multivector Search for the full rules.

​
Async API and Config Objects

Create vector indexes asynchronously with await table.create_index("vector", config=...). The config object admits the same build-time parameters described above — pass an instance of a Python config class from the table in Choosing the Right Index:

​
IVF Indexes

This example creates and queries an IVF_PQ index for a table of vectors with respect to cosine similarity. Specify vector_column_name if you have multiple vector columns or non-default names.
TypeScript currently doesn’t support IvfSq (IVF with Scalar Quantization).
For a vector field nested inside a struct, pass its full dotted path as vector_column_name (e.g. image.embedding) — the same path is used at query time and is what list_indices() reports under columns. See Selecting the vector column for the full path syntax.

​
IVF-HNSW Indexes

Beyond the general build-time parameters above, two additional parameters are specific to IVF-HNSW indexes: Partition sizing follows the general num_partitions/target_partition_size guidance above. The snippet below builds and queries an IVF_HNSW_SQ index.

​
Managing Vector Indexes

Enterprise-only In LanceDB Enterprise, vector indexes are managed automatically. The system asynchronously updates and optimizes indexes as a background process:
  • Automatically manages indexing parameters and storage
  • Infers vector columns from the schema
Open-Source LanceDB OSS users can manually create vector indexes by calling table.create_index(). See the above sections for guidance on manually tuning index parameters as data changes.
create_index() returns immediately, but the vector index builds asynchronously. To wait until all data is indexed, specify the wait_timeout parameter, or call wait_for_index(...) afterward — it waits for the named index to exist and for index_stats(...) to report num_unindexed_rows == 0.
Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal search still checks those unindexed rows with a slower fallback path; fast_search() skips that fallback and searches only indexed rows.
Operational checksAfter appends or other writes, use optimize() to fold new rows into existing indexes.

​
Check Index Status

Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status through the API or the LanceDB Enterprise UI. In the LanceDB Enterprise UI, navigate to your table page - the “Index” column reflects each column’s index status: it is blank when no index exists, shows an “in progress” label while the index is being built, and shows the index type once the build completes. To check status programmatically, use list_indices() and index_stats(). By default, the index name is formed by appending _idx to the column name (e.g., a keywords_embeddings column produces keywords_embeddings_idx). Note that list_indices() only returns information after the index is fully built. Each entry returned by list_indices() also carries detailed per-index metadata, so you can inspect an index without a follow-up index_stats() call. Node.js exposes the same fields in camelCase (num_indexed_rows → numIndexedRows):
These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as None / undefined until the server response surfaces them.

​
Custom Index Names

The {column}_idx suffix is the default naming convetion. Pass name=... to create_index() to override it. Once set, the custom name will be reflected in list_indices(), index_stats(name), and wait_for_index([name]).