Embeddings

The wippy/embeddings module provides vector embedding storage and similarity search for both PostgreSQL (pgvector) and SQLite (sqlite-vec). It wraps wippy/llm to generate embeddings and persists them to an application database.

Setup

Add the module to your project:

wippy add wippy/embeddings
wippy install

Declare the dependency and point the target_db requirement at your application database:

version: "1.0"
namespace: app

entries:
  - name: app_db
    kind: db.sql.sqlite
    path: ./data/app.db

  - name: dep.embeddings
    kind: ns.dependency
    component: wippy/embeddings
    version: "*"

  - name: target_db
    kind: registry.entry
    meta:
      wippy.embeddings.target_db: app:app_db

On startup, wippy/migration picks up the 01_create_embeddings_table migration and creates the embeddings table with the appropriate vector index for your database driver.

Configuration Constants

The default configuration is embedded in the module:

Constant	Default	Description
`EMBEDDING_MODEL`	`text-embedding-3-small`	LLM model used to generate vectors
`EMBEDDING_DIMENSIONS`	`512`	Vector size passed to the model
`MAX_TOKENS_PER_REQUEST`	`8000`	Per-call token budget; large batches are split
`DEFAULT_SEARCH_LIMIT`	`10`	Default number of hits returned by `search`

Tokens are estimated as #text / 4. Batches that exceed the budget are split automatically.

Import

entries:
  - name: my_app
    kind: library.lua
    source: file://my_app.lua
    imports:
      embeddings: wippy.embeddings:embeddings

local embeddings = require("embeddings")

High-Level API (`wippy.embeddings:embeddings`)

add

local result, err = embeddings.add(content, content_type, origin_id, context_id, meta)

Generates an embedding for content and persists it.

Parameter	Type	Required	Description
`content`	string	yes	Text to embed
`content_type`	string	yes	Free-form label, e.g. `"document_chunk"`, `"question"`
`origin_id`	string	yes	Identifier for the source document or record
`context_id`	string	no	Additional scoping key (section, chat, tenant)
`meta`	table	no	Arbitrary JSON-serialisable metadata

Returns { id, content, content_type, origin_id, context_id, meta } or nil, err.

add_batch

local result, err = embeddings.add_batch({
    { content = "...", content_type = "chunk", origin_id = "doc-1" },
    { content = "...", content_type = "chunk", origin_id = "doc-1", context_id = "s1" },
})

Embeds and stores many items in one call. If the total estimated token count exceeds MAX_TOKENS_PER_REQUEST, the batch is split and processed in chunks. Returns { count, items = { ... } }.

search

local hits, err = embeddings.search("how do migrations work?", {
    content_type = "document_chunk",
    origin_id    = "doc-1",
    context_id   = "section-2",
    limit        = 10,
})

Embeds the query string and performs a similarity search against stored vectors. All filters are optional; matching records are ordered by similarity.

find_by_type

local hits, err = embeddings.find_by_type(query, content_type, { limit = 10 })

Convenience wrapper for search scoped to a single content_type.

find_by_origin

local hits, err = embeddings.find_by_origin(query, origin_id, {
    content_type = "document_chunk",
    context_id   = "section-2",
    limit        = 5,
})

Convenience wrapper scoped to a single origin_id, optionally narrowed further.

Repository API (`wippy.embeddings:embedding_repo`)

Use the repository directly when you already have a vector and want to skip embedding generation:

Function	Description
`embedding_repo.add(content, content_type, origin_id, context_id, meta, embedding)`	Insert a precomputed vector
`embedding_repo.add_batch(batch)`	Insert many precomputed vectors in one statement
`embedding_repo.get_by_origin(origin_id)`	List all records for a given origin
`embedding_repo.delete_by_origin(origin_id)`	Remove all records for a given origin
`embedding_repo.delete_by_entry(entry_id)`	Remove a single record by its row id
`embedding_repo.search_by_embedding(vector, options)`	Similarity search against a raw vector

search_by_embedding accepts { content_type, origin_id, context_id, limit }.

Database Support

The migration creates the schema appropriate for the database driver at target_db:

PostgreSQL - embeddings table with a vector(512) column and an IVFFlat index. Requires the pgvector extension.
SQLite - embeddings table with the vector stored as text plus a companion sqlite-vec virtual table for KNN search.

Vectors are always round-tripped through a plain JSON array at the API layer.