Embeddings

The wippy/embeddings module generates embeddings through wippy/llm, stores them in an application database, and performs vector similarity searches. It supports PostgreSQL with pgvector and SQLite with sqlite-vec.

This page is an API primer with reference snippets, not a standalone tutorial. The snippets assume an existing Wippy project, a configured database, and the embedding model, provider, and credentials described below. Remote embedding calls may incur provider charges. For a complete application that indexes and searches content, follow Build a RAG Pipeline.

Setup

Add the module to your project:

wippy add wippy/embeddings
wippy install

Required Model and Provider

Before calling the embeddings API, register an llm.model whose meta.name is text-embedding-3-small, whose capabilities include embed, and whose provider mapping resolves to an embedding provider. Configure that provider's credentials, such as OPENAI_API_KEY, through the environment storage used by wippy/llm. See LLM model configuration.

Database Dependency

Declare the dependency and set its target_db parameter to the application database:

version: "1.0"
namespace: app

entries:
  - name: app_db
    kind: db.sql.sqlite
    file: ./data/app.db

  - name: dep.embeddings
    kind: ns.dependency
    component: wippy/embeddings
    version: "*"
    parameters:
      - name: target_db
        value: app:app_db

On startup, wippy/migration picks up the 01_create_embeddings_table migration and creates the embeddings_512 table for the configured database driver.

If you use the relative SQLite path shown above, create the data directory before starting the application.

Current Fixed Constants

The module currently defines these private constants; they are not dependency parameters:

Tokens are estimated as ceil(#text / 4). Oversized batches are split between items. An individual item larger than the budget is not split and causes that sub-batch to fail before the LLM call.

Import

entries:
  - name: my_app
    kind: library.lua
    source: file://my_app.lua
    imports:
      embeddings: wippy.embeddings:embeddings

local embeddings = require("embeddings")

High-Level API (wippy.embeddings:embeddings)

add

local result, err = embeddings.add(content, content_type, origin_id, context_id, meta)

Generates an embedding for content and persists it.

Returns { entry_id, origin_id, content_type, context_id } or nil, err.

add_batch

The following uses SQLite-compatible application IDs. For PostgreSQL, replace doc-1 with a UUID because the PostgreSQL schema stores origin_id as UUID.

local result, err = embeddings.add_batch({
    { content = "...", content_type = "chunk", origin_id = "doc-1" },
    { content = "...", content_type = "chunk", origin_id = "doc-1", context_id = "s1" },
})

Embeds and stores multiple items in one call. If the total estimated token count exceeds MAX_TOKENS_PER_REQUEST, the method splits the batch into chunks. Each repository chunk is transactional, but a split high-level batch is not atomic across chunks: earlier chunks remain stored if a later chunk fails. Returns { count, items = { ... } }.

To remove records created while testing, use the repository API's delete_by_origin(origin_id) method for each sample origin.

search

local hits, err = embeddings.search("how do migrations work?", {
    content_type = "document_chunk",
    origin_id    = "doc-1",
    context_id   = "section-2",
    limit        = 10,
})

Embeds the query string and performs a similarity search against stored vectors. All filters are optional; matching records are ordered by similarity.

origin_id may be a string or a non-empty array of strings. Each hit contains entry_id, origin_id, content_type, context_id, content, decoded meta, timestamps, and similarity.

find_by_type

local hits, err = embeddings.find_by_type(
    "how do migrations work?",
    "document_chunk",
    { limit = 10 }
)

Calls search with a single content_type. The default limit is 10.

find_by_origin

local hits, err = embeddings.find_by_origin("how do migrations work?", "doc-1", {
    content_type = "document_chunk",
    context_id   = "section-2",
    limit        = 5,
})

Calls search with a single origin_id and optional content_type and context_id filters. The default limit is 5.

Repository API (wippy.embeddings:embedding_repo)

Use the repository directly when you already have a vector and want to skip embedding generation. Raw embeddings must contain exactly 512 numeric values:

search_by_embedding accepts { content_type, origin_id, context_id, limit }.

Database Support

The migration creates the schema appropriate for the database driver at target_db:

• PostgreSQL — embeddings_512 table with a vector(512) column and an IVFFlat cosine index. The migration attempts to install the vector extension, so the database role must either be allowed to create it or the extension must already exist. PostgreSQL stores origin_id as UUID.
• SQLite — embeddings_512 vec0 virtual table holding the embedding float[512] vector column alongside the metadata and content columns for KNN search.

See Also

• LLM — llm.embed(...) for raw embedding generation
• Migrations — Migration runner that provisions the table
• Framework Overview — Framework module usage