# Wippy - Complete Documentation # What Is Wippy? Concepts and Runtime Overview ## About Wippy Wippy is an open-source actor-model runtime for applications whose behavior changes while they run. It is designed for automation systems, AI agents, plugin architectures, and other applications that need to evolve without rebuilding or redeploying the runtime. The foundation is the actor model. Code runs in isolated processes that communicate through messages, with each process managing its own state. Supervised process services can be restarted according to their lifecycle policy. ```lua local worker, err = process.spawn("app.workers:handler", "app:processes") if not worker then return nil, err end local ok, monitor_err = process.monitor(worker) if not ok then return nil, monitor_err end return process.send(worker, "task", {id = 1, data = payload}) ``` Runtime definitions live in a central registry. For entry kinds dispatched through the event bus, registered handlers reconcile accepted changes without restarting the entire application. Some internal entry kinds intentionally bypass event dispatch. ```lua local db, err = registry.get("app.db:postgres") if not db then return nil, err end local cache, cache_err = registry.get("app.cache:redis") if not cache then return nil, cache_err end ``` For operations that must recover from infrastructure failures, durable workflows persist execution state. This model suits payment flows, multi-step processes, and long-running agent tasks that may need to resume after a restart. The runtime is distributed as a single binary and configured through project files. For the project overview and design notes, see [About Wippy](https://wippy.ai/about). --- # "Installation" ### Install ```bash curl -fsSL https://hub.wippy.ai/install.sh | bash ``` The install script requires a POSIX shell. On Windows, download the runtime from [hub.wippy.ai/releases](https://hub.wippy.ai/releases) and place `wippy.exe` on `PATH`. ### Verify ```bash wippy version ``` ### Initialize Dependency Metadata ```bash ## Create a project directory mkdir myapp cd myapp ## Create or update wippy.lock wippy init ``` `wippy init` writes the dependency lock and its source and module directory settings. It does not scaffold application source files or registry entries. Follow [Hello World](tutorials/hello-world.md) to create a runnable application, then start it with `wippy run`. The runtime includes HTTP, SQL, storage, and process-hosting capabilities. Add framework modules from the Hub when the application needs them: ```bash wippy add wippy/test wippy install ``` ### Commands Overview | Command | Description | | --------- | ------------- | | `wippy init` | Create or update `wippy.lock` | | `wippy run` | Start the runtime | | `wippy test` | Run the test entrypoint | | `wippy lint` | Check code for errors | | `wippy add` | Add a dependency | | `wippy install` | Install dependencies | | `wippy update` | Update dependencies | | `wippy artifacts` | Materialize build-time filesystem artifacts | | `wippy pack` | Create a snapshot | | `wippy publish` | Publish to hub | | `wippy search` | Search for modules | | `wippy readme` | Fetch a module README from the hub | | `wippy registry` | Inspect loaded registry entries | | `wippy auth` | Manage authentication | | `wippy version` | Print version info | See [CLI Reference](guides/cli.md) for full documentation. ### Troubleshooting If the shell cannot find `wippy` after installation, reopen the shell and verify that the installation directory is on `PATH`. ### Next Steps - [Hello World](../tutorials/hello-world.md) — Create your first application - [Project Structure](start/structure.md) — Understand the project layout - [CLI Reference](guides/cli.md) — Review all commands and options --- # "YAML & Project Structure" ### Directory Layout ``` myapp/ ├── .wippy.yaml # Runtime configuration ├── wippy.lock # Source directories and locked modules ├── .wippy/ # Installed modules └── src/ # Application source ├── _index.yaml # Entry definitions ├── api/ │ ├── _index.yaml │ └── *.lua └── workers/ ├── _index.yaml └── *.lua ``` ### YAML Definition Files YAML definitions are loaded into the registry at startup. The registry is the source of truth; YAML files are one way to populate it. Entries can also come from other sources or be created programmatically. #### Definition File Format A definition file contains a `namespace` and either an `entries` array or top-level `name` and `kind` fields. The optional `version` marker is conventionally `"1.0"`; the v0.3.32a loader does not require it. ```yaml version: "1.0" namespace: app.api entries: - name: get_user kind: function.lua meta: comment: Fetches user by ID source: file://get_user.lua method: handler modules: - sql - json - name: get_user.endpoint kind: http.endpoint meta: comment: User API endpoint method: GET path: /users/{id} func: get_user ``` | Field | Required | Description | |-------|----------|-------------| | `version` | No | Manifest version marker (conventionally `"1.0"`) | | `namespace` | Yes | Entry namespace for this file | | `entries` | Conditional | Array of entry definitions; omit only when using top-level `name` and `kind` | #### Naming Convention Use dots (`.`) for semantic separation and underscores (`_`) for words: ```yaml ## Function and its endpoint - name: get_user # The function - name: get_user.endpoint # Its HTTP endpoint ## Multiple endpoints for same function - name: list_orders - name: list_orders.endpoint.get - name: list_orders.endpoint.post ## Routers - name: api.public # Public API router - name: api.admin # Admin API router ``` Pattern: base_name.variant — dots separate semantic parts, while underscores separate words within a part. #### Namespaces Namespaces are dot-separated identifiers: ``` app app.api app.api.v2 app.workers ``` Entry full ID combines namespace and name: `app.api:get_user` #### The Lock File `wippy.lock` records where Wippy loads definitions from and which module versions are selected: ```yaml directories: modules: .wippy src: ./src options: unpack_modules: false modules: - name: acme/http version: v1.2.0 hash: 4ea816fe84ca58a1f0869e5ca6afa93d6ddd72fa09e1162d9e600a7fbf39f0a2 ``` | Field | Description | |-------|-------------| | `directories.src` | Application source directory, scanned recursively for YAML definition files | | `directories.modules` | Base directory for vendored modules; packs land under `/vendor/` | | `options.unpack_modules` | Extract each `.wapp` into a directory beside it instead of loading the pack directly (default `false`) | | `modules[].name` | Module identifier in `org/module` form | | `modules[].version` | Selected version | | `modules[].hash` | Artifact digest the vendored pack must match | | `modules[].root` | Marks the selected deployment root; at most one module may carry it | Vendored packs are kept as `.wapp` files. With `unpack_modules: true`, each module is also extracted into a directory, and the verified `.wapp` stays beside it — installation looks for the pack, so a directory whose pack is missing is downloaded again. A `replacements:` section in `wippy.lock` is deprecated. It still loads, with a warning; declare local module overrides under `workspace.replacements` in a runtime config file instead. See [Dependency Management](guides/dependency-management.md#local-development-with-replacements). ### Entry Definitions Each item in the `entries` array defines one entry. Kind-specific fields can appear beside `name`, `kind`, and `meta`, as in this example: ```yaml entries: - name: hello kind: function.lua meta: comment: Returns hello world source: file://hello.lua method: handler modules: - http - json - name: hello.endpoint kind: http.endpoint meta: comment: Hello endpoint method: GET path: /hello func: hello ``` An explicit `data:` field is also supported. When present, its value is the complete kind-specific payload, so do not mix it with sibling kind-specific fields: ```yaml entries: - name: config kind: registry.entry data: environment: production features: dark_mode: true ``` #### Metadata Use `meta` for UI-friendly information: ```yaml - name: payment_handler kind: function.lua meta: title: Payment Processor comment: Handles Stripe payments source: file://payment.lua ``` Use `meta.title` and `meta.comment` for descriptive information that registry consumers and management interfaces can display. #### Application Entries Use `registry.entry` kind for application-level configuration: ```yaml - name: config kind: registry.entry meta: title: Application Settings type: application environment: production features: dark_mode: true beta_access: false ``` ### Common Entry Kinds | Kind | Purpose | |------|---------| | `registry.entry` | General-purpose data stored without normal event dispatch | | `function.lua` | Callable Lua function | | `process.lua` | Long-running process | | `http.service` | HTTP server | | `http.router` | Route group | | `http.endpoint` | HTTP handler | | `process.host` | Process execution host | See the [Entry Kinds Guide](guides/entry-kinds.md) for the entry-kind reference. #### .wippy.yaml Runtime configuration at project root: ```yaml version: "1.0" logger: encoding: json logmanager: min_level: 0 supervisor: host: worker_count: 16 ``` See the [Configuration Guide](guides/configuration.md) for runtime configuration fields. #### wippy.lock Source directories and the selected module graph — see [The Lock File](#the-lock-file) above. ### Referencing Entries Reference entries by full ID or relative name where the entry kind supports it. HTTP routers and endpoints attach through `meta.server` and `meta.router`, rather than through parent-side child lists: ```yaml ## Router declares itself against a server - name: api kind: http.router meta: server: app:gateway prefix: /api ## Endpoint references router by registry ID (cross-namespace works the same way) - name: get_user.endpoint kind: http.endpoint meta: router: app.api:api method: GET path: /users/{id} func: app.api:get_user ``` ### Example Project ``` myapp/ ├── .wippy.yaml ├── wippy.lock └── src/ ├── _index.yaml # namespace: app ├── api/ │ ├── _index.yaml # namespace: app.api │ ├── users.lua │ └── orders.lua ├── lib/ │ ├── _index.yaml # namespace: app.lib │ └── database.lua └── workers/ ├── _index.yaml # namespace: app.workers └── email_sender.lua ``` ### See Also - [Application Architecture](concepts/architecture.md) — Organize an application into slices and layers - [Entry Kinds Guide](guides/entry-kinds.md) — Review available entry kinds - [Configuration Guide](guides/configuration.md) — Configure runtime options - [Custom Entry Kinds](internals/kinds.md) — Implement handlers (advanced) --- # "CLI Reference" ## CLI Reference Use the Wippy CLI to initialize projects, run the runtime, manage dependencies, inspect registry entries, and publish modules. This is a command reference. The examples assume an existing project or module when the command operates on source, a lock file, registry entries, or publish metadata; they are not a single end-to-end project. ### Global Flags Available on all commands: | Flag | Short | Description | |------|-------|-------------| | `--config` | | Config file, repeatable; later files override earlier ones (default: .wippy.yaml). `wippy publish` defines a different command-local option. | | `--verbose` | `-v` | Enable debug logging | | `--very-verbose` | | Debug with stack traces | | `--console` | `-c` | Colorful console logging | | `--silent` | `-s` | Disable console logging | | `--event-streams` | `-e` | Stream logs to event bus | | `--profiler` | `-p` | Enable pprof on localhost:6060 | | `--memory-limit` | `-m` | Memory limit (e.g., 1G, 512M) | Memory-limit precedence is `--memory-limit`, then `GOMEMLIMIT`, then the 1 GB default. The global `--config` option may be passed multiple times to compose config files. Files merge left to right: later files override matching values and keep everything else. Every explicitly named file must exist; without `--config`, the default `.wippy.yaml` is optional. The first file anchors the directory used to resolve relative paths. Configuration applies in order: file composition, then `--profile` selections, then `--set` overrides. See [Configuration](guides/configuration.md#config-composition). `wippy publish` shadows the global option with a command-local `--config ` option. For that command, the value is the directory containing `wippy.yaml`, not a repeatable runtime configuration file. ### wippy init Create `wippy.lock`, or update its source and module directory settings if it already exists. This command does not scaffold application source files or registry entries. ```bash wippy init wippy init --src-dir ./src --modules-dir .wippy ``` | Flag | Short | Default | Description | |------|-------|---------|-------------| | `--src-dir` | `-d` | ./src | Source directory | | `--modules-dir` | | .wippy | Modules directory | | `--lock-file` | `-l` | wippy.lock | Lock file path | ### wippy run Start the runtime or execute a command. ```bash wippy run # Start runtime wippy run list # List available commands wippy run migrate # Run a named custom command wippy run snapshot.wapp # Run from pack file wippy run acme/http # Run module from hub wippy run acme/http@1.2.3 # Run specific version wippy run --exec app:worker # Start runtime and execute a single process ``` | Flag | Short | Description | |------|-------|-------------| | `--override` | `-o` | Override entry values (`namespace:entry:field=value`); `field` may be `kind` to change the entry kind | | `--set` | | Override a config value (`section.path=value`, repeatable, takes precedence over the config file) | | `--exec` | `-x` | Execute process and exit (`namespace:entry`) | | `--host` | | Terminal host ID for `--exec` (auto-detected if only one `terminal.host` exists) | | `--registry` | | Registry URL for hub modules | | `--profile` | | Apply a runtime profile from `.wippy.yaml` or packed runtime metadata (repeatable, applied in order) | Running a hub module (`wippy run org/module`) resolves it once, records it in `wippy.lock`, and vendors the verified packs locally. Subsequent runs of the same reference start from the lock — no network needed. A version selector that no longer matches the lock is rejected with a hint to run `wippy update`. For a local application, `wippy run` repairs a stale lock before any runtime service starts. It loads the source dependency declarations, and when the lock already satisfies them it re-resolves the graph from local and installed evidence only (verified-offline access, no network). If that offline resolution matches the lock, boot continues unchanged. If it succeeds but differs, it becomes the candidate graph; the hub is asked to resolve only when the offline pass fails or the lock no longer satisfies the source declarations. Packs the candidate graph is missing are downloaded and verified, and only then is `wippy.lock` rewritten. A lock that selects a deployment root is authoritative and is never re-resolved. `--exec` blocks until the launched process produces its result, then propagates the process exit code as the CLI exit code. Ctrl-C during `--exec` cancels the running process and the runtime still shuts down gracefully; a second signal forces exit. `--set` writes any runtime configuration value from the command line, merged over `.wippy.yaml` per leaf: ```bash wippy run --set cluster.enabled=true \ --set cluster.membership.join_addrs=node-2:7946,node-3:7946 \ --set cluster.raft.bootstrap_expect=3 ``` Values are coerced by shape: `true` and `false` become booleans, integers and floats become numbers, and other values remain strings. Fields that expect durations parse values such as `5s`. ### wippy test Run the test entrypoint: the process entry declaring the `test` use case. The runtime boots, executes that entry, and exits. `wippy run` does not auto-run test entrypoints; testing always goes through `wippy test`. ```bash wippy test # Run tests from the local project wippy test snapshot.wapp # Run tests from a pack file wippy test acme/module@1.2.3 # Run tests from a hub module ``` | Flag | Short | Description | |------|-------|-------------| | `--override` | `-o` | Override entry values (`namespace:entry:field=value`) | | `--host` | | Terminal host ID (auto-detected if only one `terminal.host` exists) | | `--registry` | | Registry URL for hub modules | | `--set` | | Override a config value (`section.path=value`, repeatable) | | `--profile` | | Apply a runtime profile (repeatable, applied in order) | ### wippy lint Check Lua code for type errors and warnings. ```bash wippy lint wippy lint --level warning wippy lint --json wippy lint --rules ``` Validates source-bearing `function.lua`, `library.lua`, `process.lua`, and `workflow.lua` entries. Precompiled `.bc` entries do not contain parseable source and are skipped. | Flag | Short | Default | Description | |------|-------|---------|-------------| | `--lock-file` | `-l` | `wippy.lock` | Lock file path | | `--level` | | `warning` | Minimum severity: `error`, `warning`, `hint` | | `--ns` | | | Filter by namespace patterns (e.g. `app`, `lib.*`) | | `--code` | | | Filter by error codes (e.g. `E0001,E0004`) | | `--rules` | | `false` | Enable style/quality lint rules | | `--summary` | | `false` | Group output by error code | | `--limit` | | `0` | Max diagnostics shown (0 = unlimited) | | `--json` | | `false` | JSON output | | `--no-color` | | `false` | Disable colored output | | `--cache-reset` | | `false` | Clear Lua cache before linting | | `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable) | | `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) | ### wippy add Add a module dependency. ```bash wippy add acme/http wippy add acme/http@1.2.3 wippy add acme/http@latest ``` | Flag | Short | Default | Description | |------|-------|---------|-------------| | `--lock-file` | `-l` | wippy.lock | Lock file path | | `--registry` | | | Registry URL | ### wippy install Install dependencies from lock file. ```bash wippy install # Install all wippy install acme/http # Install specific module wippy install --refresh acme/http # Re-fetch a specific module ``` | Flag | Short | Default | Description | |------|-------|---------|-------------| | `--lock-file` | `-l` | wippy.lock | Lock file path | | `--refresh` | | false | Re-fetch the named modules, or every locked module when no names are supplied, bypassing cache | | `--force` | | false | Alias for `--refresh` | | `--repair` | | false | Alias for `--refresh` | | `--registry` | | | Registry URL | | `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable) | | `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) | ### wippy update Update dependencies and regenerate lock file. ```bash wippy update # Update all wippy update acme/http # Update specific module wippy update acme/http demo/sql # Update multiple ``` | Flag | Short | Default | Description | |------|-------|---------|-------------| | `--lock-file` | `-l` | wippy.lock | Lock file path | | `--src-dir` | `-d` | ./src | Source directory | | `--modules-dir` | | .wippy | Modules directory | | `--registry` | | | Registry URL | | `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable) | | `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) | ### wippy artifacts Work with build-time filesystem artifacts. #### wippy artifacts materialize Validate and materialize one artifact filesystem out of an existing pack. ```bash wippy artifacts materialize snapshot.wapp app:package_fs wippy artifacts materialize snapshot.wapp app:package_fs --root build ``` | Flag | Default | Description | |------|---------|-------------| | `--root` | `.wippy` | Materialization root | The resource is addressed by its full `namespace:name`, must declare `meta.artifact.format`, and that format must be registered in the CLI. The command resolves no module dependencies, does not mutate `wippy.lock`, invokes no package managers, and takes no part in runtime composition. See [Build-time artifacts](guides/artifacts.md#materializing-explicitly). ### wippy pack Create a snapshot pack (.wapp file). ```bash wippy pack snapshot.wapp wippy pack release.wapp --description "Release 1.0" wippy pack app.wapp --embed app:assets --bytecode "**" ``` | Flag | Short | Description | |------|-------|-------------| | `--lock-file` | `-l` | Lock file path | | `--description` | `-d` | Pack description | | `--tags` | `-t` | Pack tags (comma-separated) | | `--meta` | | Custom metadata (key=value) | | `--embed` | | Embed fs.directory entries (patterns) | | `--embed-all` | | Embed all fs.directory entries (cannot combine with `--embed`) | | `--list` | | List fs.directory entries (dry-run) | | `--exclude-ns` | | Exclude namespaces (patterns) | | `--exclude` | | Exclude entries (patterns) | | `--bytecode` | | Compile Lua to bytecode (** for all) | | `--profile` | | Apply a runtime profile from `.wippy.yaml` before packing (repeatable, applied in order) | Without `--embed` or `--embed-all`, embed patterns fall back to the `embed:` section of the module manifest `wippy.yaml`. Packing an application also carries embedded resources from its dependency packs, and only the main module's commands are exposed by the resulting pack. The output file is written atomically: the pack is built into a temporary file in the destination directory, synced, verified, and only then renamed over the target, inheriting the existing file's permissions when one is present. A failed pack leaves the previous file untouched. Naming an output that is also one of the pack's inputs — the same path, or a hard link or symlink resolving to the same file — is refused rather than truncating the input mid-read. `--meta` cannot write reserved metadata. The key `registry`, and anything under the `wippy.` or `system.` prefixes, is owned by the pack format and rejected. Resources declaring `meta.artifact.format` are validated while packing, so a malformed artifact fails here rather than in a consumer. See [Build-time artifacts](guides/artifacts.md). ### wippy publish Publish module to the hub. ```bash wippy publish wippy publish --version 1.0.0 wippy publish --dry-run ``` This command reads `wippy.yaml` from the current directory. | Flag | Description | |------|-------------| | `--version` | Version to publish | | `--dry-run` | Validate without publishing | | `--label` | Publish as mutable label instead of version | | `--release-notes` | Release notes | | `--protected` | Mark version as protected | | `--embed` | Embed fs.directory entries by id or name | | `--config` | Path to directory containing wippy.yaml (default: .) | | `--registry` | Registry URL | | `--create` | Create the module on the registry if it does not yet exist | | `--module-visibility` | Visibility for newly created modules (`--create` only): `public` or `private` (default: private) | | `--module-type` | Module type: `library`, `application`, `agent`, or `plugin` (overrides `type:` in wippy.yaml) | | `--module-display-name` | Display name for newly created modules (`--create` only) | The module type is normally declared as `type:` in `wippy.yaml` (see [Publishing](./publishing.md#wippyyaml)); `--module-type` overrides it for a single publish. When neither is set, newly created modules default to `application` with a deprecation warning. ### wippy search Search for modules in the hub. Search uses the stored authentication token for the selected registry, when available. Authenticate with `wippy auth login` to search with your registry identity; `--registry` selects which registry's credentials are used. ```bash wippy search http wippy search "sql driver" --limit 20 wippy search auth --json ``` | Flag | Default | Description | |------|---------|-------------| | `--json` | false | Output as JSON | | `--limit` | 20 | Maximum results | | `--registry` | | Registry URL | ### wippy auth Manage registry authentication. #### wippy auth login ```bash wippy auth login wippy auth login --token YOUR_TOKEN ``` | Flag | Description | |------|-------------| | `--token` | API token | | `--registry` | Registry URL | | `--local` | Store credentials locally | #### wippy auth logout ```bash wippy auth logout ``` | Flag | Description | |------|-------------| | `--registry` | Registry URL | | `--local` | Remove local credentials | #### wippy auth status ```bash wippy auth status wippy auth status --json ``` | Flag | Description | |------|-------------| | `--json` | Output as JSON | ### wippy readme Fetch a module README from the hub. ```bash wippy readme wippy/terminal wippy readme wippy/terminal@1.2.3 wippy readme --json wippy/terminal@latest ``` | Flag | Description | |------|-------------| | `--json` | Output as JSON | | `--registry` | Registry URL (default: from credentials) | ### wippy registry Query and inspect registry entries. Both subcommands accept `--profile` and `--set` to control the merged runtime configuration used to load entries. #### wippy registry list ```bash wippy registry list wippy registry list --kind "function.lua.*" wippy registry list --ns "app.*" --json wippy registry list --meta "type=api" --meta "enabled=true" ``` | Flag | Short | Description | |------|-------|-------------| | `--kind` | `-k` | Filter by kind (glob pattern) | | `--ns` | `-n` | Filter by namespace (glob pattern) | | `--name` | | Filter by name (glob pattern) | | `--meta` | | Filter by metadata (repeatable) | | `--json` | | Output as JSON | | `--yaml` | | Output as YAML | | `--registry-meta` | | Include registry-owned metadata (`owner`, `root`) in JSON or YAML output; requires `--json` or `--yaml` | | `--lock-file` | `-l` | Lock file path | Metadata operators for `--meta`: | Operator | Meaning | |----------|---------| | `field=value` | Exact match | | `field~regex` | Regex match | | `field*substr` | Contains substring | | `field^prefix` | Starts with prefix | | `field$suffix` | Ends with suffix | #### wippy registry show ```bash wippy registry show app:http:handler wippy registry show app:config --yaml ``` | Flag | Short | Description | |------|-------|-------------| | `--field` | `-f` | Show specific field | | `--json` | | Output as JSON | | `--yaml` | | Output as YAML | | `--raw` | | Raw output | | `--lock-file` | `-l` | Lock file path | ### wippy version Print version information. ```bash wippy version wippy version --short ``` ### Custom Commands Any `process.lua` or `process.wasm` entry can be registered as a named command by adding `command` metadata: ```yaml entries: - name: migrate_runner kind: process.lua meta: command: name: migrate short: Run database migrations security: actor: id: app:migrations policies: - app.security:migrations groups: - app.security:operators source: file://runner.lua method: main modules: - io - registry - funcs ``` Run it with: ```bash wippy run migrate ``` List all available commands: ```bash wippy run list ``` `wippy run list` accepts `--profile` and `--set` so the listing reflects the same merged runtime config `wippy run` would use. #### Command Metadata Fields | Field | Required | Description | |-------|----------|-------------| | `name` | Yes | Command name used with `wippy run ` | | `short` | No | Short description shown in `wippy run list` | | `main` | No | Mark this entry as the default entrypoint. When a pack or hub module is run without a command name, the single `main` entry of that use case is executed; a lone entrypoint is picked even without `main`, and several entrypoints with no `main` is an error | | `use_case` | No | Entrypoint category, default `run`. The entry declaring `use_case: test` is what `wippy test` executes | | `security` | No | Security context the command runs under when launched from the CLI | Any process entry kind works (`process.lua`, `process.wasm`). Command names are not checked for uniqueness; when several loaded entries declare the same name, the first match in registry order runs. Arguments after the command name are passed to the process as string payloads. #### Command security A command entry declares the actor and policy scope its CLI launch runs under: ```yaml entries: - name: migrate_runner kind: process.lua meta: command: name: migrate short: Run database migrations security: actor: id: system.migrations meta: role: operator policies: - app.security:migrations_policy groups: - app.security:operators source: file://runner.lua method: main ``` | Field | Description | |-------|-------------| | `actor.id` | Actor identity for the launched process | | `actor.meta` | Actor attributes evaluated by policies | | `policies` | Registry IDs (`namespace:name`) of individual policies added to the scope | | `groups` | Registry IDs of policy groups whose policies are added to the scope | The block lives inside `meta.command` because it applies only to the CLI launch path — the operator started the command on their own deployment, which is the trust anchor. It has no effect on ordinary spawns of the same process entry; those follow the entry's own [`security:` block](guides/entry-kinds.md#process-security). Declaration is fail-closed and validated before the process starts: - Unknown fields inside `security` are rejected. - An empty `security` block (no actor, no policies, no groups) is rejected. - `security` without a `name` is rejected — a command must be nameable to be launched. - A policy or group that cannot be resolved refuses the launch; resolution is atomic, so a partial scope is never installed. When the block omits `actor`, the caller's actor is inherited. When it omits both `policies` and `groups`, the caller's scope is inherited. #### Development Workflow ```bash ## Initialize dependency lock metadata wippy init wippy add wippy/test wippy add wippy/llm wippy install ## Check for errors wippy lint ## Run with debug output wippy run -c -v ## Override config for local dev wippy run -o app:db:host=localhost -o app:db:port=5432 ``` #### Production Deployment ```bash ## Create release pack with bytecode wippy pack release.wapp --bytecode "**" --exclude-ns "test.**" ## Run from pack with memory limit wippy run release.wapp -m 2G ``` #### Debugging ```bash ## Execute single process wippy run --exec app:worker ## With profiler enabled wippy run -p -v ## Then: go tool pprof http://localhost:6060/debug/pprof/heap ``` #### Dependency Management ```bash ## Add new dependency wippy add acme/http@latest ## Force re-download wippy install --force ## Update specific module wippy update acme/http ``` #### Publishing ```bash ## Login to hub wippy auth login ## Validate module wippy publish --dry-run ## Publish wippy publish --version 1.0.0 --release-notes "Initial release" ``` ### Environment Variables | Variable | Effect | |----------|--------| | `WIPPY_TOKEN` | Registry auth token; overrides stored credentials (a token pushed via `hub.auth.authenticate` ranks higher still) | | `WIPPY_REGISTRY` | Default registry URL (overridden by `--registry`) | | `WIPPY_CACHE_DIR` | Cache directory for hub modules run via `wippy run org/module` (default: `~/.wippy/cache`) | | `GOMEMLIMIT` | Memory-limit fallback when `--memory-limit` is not set | Values in `.wippy.yaml` may reference OS environment variables with `${env:NAME}`, resolved at file load; a missing variable fails config loading. Bare `${name}` references resolve from the config's `vars:` section instead. ### Configuration File Create `.wippy.yaml` for persistent settings: ```yaml logger: encoding: console logmanager: stream_to_events: true profiler: enabled: true address: localhost:6060 override: app:gateway:addr: ":9090" app:db:host: "localhost" ``` ### See Also - [Configuration](guides/configuration.md) — Configuration file reference - [Observability](guides/observability.md) — Monitoring and logging --- # "Configuration Reference" ## Configuration Reference Wippy reads runtime configuration from `.wippy.yaml` files. Use the repeatable `wippy run --set section.path=value` option to override the configuration fields below at launch. To override individual registry *entries* rather than configuration sections, use the `override:` section or `-o`; see [Overriding Entries](guides/entry-kinds.md#overriding-entries). ### Config Composition `--config` is repeatable; files compose left to right using the same schema: ```bash wippy run --config .wippy.yaml --config .wippy.local.yaml ``` - Later files override matching values and keep everything else. - Every explicitly named file must exist. Without `--config`, the default `.wippy.yaml` is optional. - The first file anchors the directory used to resolve relative paths. - Filenames carry no reserved meaning; nothing besides the default is auto-discovered. Configuration applies in this order: composed files, selected `--profile` overlays, and then `--set` overrides. For applications run from packs, packed runtime defaults have lower precedence than all three; see [Publishing Runtime Defaults](guides/publishing.md#publishing-runtime-defaults). ### Profiles A configuration file may declare named overlays under `profiles:`. Each profile body mirrors the standard configuration sections. Selecting it with `--profile ` applies those values over the merged base configuration: ```yaml version: "1.0" vars: port: 8085 override: app:db:kind: db.sql.sqlite disable: namespaces: ["legacy.**"] profiles: pg: vars: port: 18085 override: app:db:kind: db.sql.postgres disable: namespaces.add: ["experimental.**"] ``` ```bash wippy run --profile pg ``` - `--profile` is repeatable; profiles compose left to right, after file composition and before `--set`. An unknown name is an error. - Values merge per leaf (last writer wins). The `profiles:` section itself is stripped from the resolved config. - The `disable` section supports list operations inside profiles — `namespaces.add`, `namespaces.remove`, `entries.add`, `entries.remove` — so a profile can adjust the base list instead of replacing it. - `${name}` references interpolate from the merged `vars:` section. OS environment references are not allowed inside profile vars; use `${env:NAME}` in the base config, resolved at file load. `wippy run`, `test`, and `pack` accept `--profile`; `run list`, `install`, `update`, `lint`, and `registry` accept it as well for workspace profiles (together with `--set`). Applications can ship profiles inside packs — see [Publishing Profiles](guides/publishing.md#publishing-profiles). ### Logger Controls the zap logger encoder. CLI flags (`-v`, `-c`, `-s`) override the level and output; encoding is the only YAML-configured option. | Field | Type | Default | Description | |-------|------|---------|-------------| | `encoding` | string | console | Encoder: `console` (humanized) or `json` (structured) | ```yaml logger: encoding: json ``` ### Log Manager Controls runtime log routing. Console output is configured via [CLI flags](guides/cli.md) (`-v`, `-c`, `-s`). | Field | Type | Default | Description | |-------|------|---------|-------------| | `propagate_downstream` | bool | true | Send logs to console/file output | | `stream_to_events` | bool | false | Publish logs to event bus for programmatic access | | `min_level` | int | 0 (`-1` with `-v`) | Minimum level: -1=debug, 0=info, 1=warn, 2=error. The CLI writes this key from its flags after the file is read, so a file value is ignored; change it with `--set logmanager.min_level=` | ```yaml logmanager: propagate_downstream: true stream_to_events: false ``` See: [Logger Module](lua/system/logger.md) ### Profiler Go pprof HTTP server for CPU/memory profiling. Enable with `-p` flag or config. | Field | Type | Default | Description | |-------|------|---------|-------------| | `enabled` | bool | false | Start profiler server | | `address` | string | localhost:6060 | Listen address | | `read_timeout` | duration | 15s | HTTP read timeout | | `write_timeout` | duration | 15s | HTTP write timeout | | `idle_timeout` | duration | 60s | Keep-alive timeout | ```yaml profiler: enabled: true address: "localhost:6060" ``` When enabled with the default address, the profiler is available at `http://localhost:6060/debug/pprof/`. ### Security Global security behavior. Individual policies are defined as [security.policy entries](guides/entry-kinds.md). | Field | Type | Default | Description | |-------|------|---------|-------------| | `strict_mode` | bool | true | Deny access when the security context is incomplete | ```yaml security: strict_mode: false ``` See: [Security System](system/security.md), [Security Module](lua/security/security.md) ### Registry Entry storage and version history. The registry holds all configuration entries. | Field | Type | Default | Description | |-------|------|---------|-------------| | `enable_history` | bool | true | Track entry versions | | `history_type` | string | memory | Storage: `memory`, `sqlite`, `postgres`, `nil` | | `history_path` | string | .wippy/registry.db | SQLite file path (used when `history_type: sqlite`) | | `history_dsn` | string | | Postgres DSN (used when `history_type: postgres`) | | `history_schema` | string | | Postgres schema name (used when `history_type: postgres`) | | `event_wait_timeout` | duration | 30s | Per-operation wait for listener acknowledgement during a registry apply | | `dispatch_internal_kinds` | string[] | `[registry.entry, ns.dependency, ns.requirement, ns.definition]` | Entry kinds handled internally instead of dispatched to component listeners | | `dependency_resolve_timeout` | duration | 0 (none) | Bound on dependency resolution | | `dependency_download_timeout` | duration | 0 (none) | Bound on each module download and download-URL request | | `dependency_lock_path` | string | discovered `wippy.lock` | Lock file the dependency handler reads and writes | | `dependency_vendor_dir` | string | `//vendor` | Directory holding downloaded module packs | ```yaml registry: history_type: sqlite history_path: /var/lib/wippy/registry.db ``` ```yaml registry: history_type: postgres history_dsn: ${env:WIPPY_REGISTRY_HISTORY_DSN} history_schema: wippy_registry ``` See: [Registry Concept](concepts/registry.md), [Registry Module](lua/core/registry.md) ### Artifact Output root for materialized [build-time artifacts](guides/artifacts.md). | Field | Type | Default | Description | |-------|------|---------|-------------| | `materialization_root` | string | parent of the dependency vendor directory | Application-owned root under which each artifact format writes its own subtree | ```yaml artifact: materialization_root: build/wippy ``` See: [Build-time artifacts](guides/artifacts.md#where-output-lands) ### Workspace Local module replacements, keyed by `org/module`. Values are directories; relative paths resolve against the first `--config` file's directory, and `null` disables a replacement inherited from an earlier config layer or profile. ```yaml workspace: replacements: acme/http: ../local-http acme/sql: null ``` Replacements are never written to `wippy.lock`. See [Local Development with Replacements](guides/dependency-management.md#local-development-with-replacements). ### Relay Message routing between processes across nodes. | Field | Type | Default | Description | |-------|------|---------|-------------| | `node_name` | string | derived per-instance ID | Identifier for this relay node (default: UUIDv5 of machine-id/hostname + working dir; overridable via `WIPPY_NODE_ID` / `WIPPY_RELAY_NODE_NAME`) | ```yaml relay: node_name: worker-1 ``` See: [Process Model](concepts/process-model.md) ### Supervisor Service lifecycle management. Controls the supervisor's internal control mailbox used to dispatch lifecycle events. | Field | Type | Default | Description | |-------|------|---------|-------------| | `host.buffer_size` | int | 1024 | Internal control mailbox capacity | | `host.worker_count` | int | 16 | Concurrent dispatcher workers | ```yaml supervisor: host: buffer_size: 2048 worker_count: 32 ``` See: [Supervision Guide](guides/supervision.md) Per-`process.host` workers and queues are configured on the entry itself (`workers`, `queue_size`, `local_queue_size`), not in this global section. See the [Process Host](system/process-host.md) entry kind. ### Lua Runtime Lua VM caching and expression evaluation. | Field | Type | Default | Description | |-------|------|---------|-------------| | `cache.enabled` | bool | `type_system.enabled` | Persist compiled bytecode/typecheck cache to disk; follows `type_system.enabled` unless set explicitly | | `cache.dir` | string | `.wippy/cache/lua` | Cache directory path (relative to the config/working directory) | | `cache.mode` | string | `readwrite` | Cache mode: `readwrite` (default), `readonly`, `off`; unknown values fall back to `readwrite` | | `cache.compile.enabled` | bool | true | Persist compiled bytecode (when `cache.enabled`) | | `cache.typecheck.enabled` | bool | true | Persist typecheck results (when `cache.enabled`) | | `cache.max_bytes` | int | 1073741824 | On-disk cache size ceiling in bytes | | `cache.max_entries` | int | 20000 | Maximum cached entries | | `cache.prune_interval` | int | 256 | Writes between cache prune passes | | `type_system.enabled` | bool | false | Enable static type checking | | `type_system.strict` | bool | false | Treat type warnings as errors | | `invalidation_wait_timeout` | duration | `registry.event_wait_timeout` (30s) | Wait for code invalidation to be acknowledged after an entry changes | | `eval.max_steps` | int | 10000 | Default scheduler-step budget for an `eval` run; negative values are rejected | | `eval.cache_size` | int | 256 | Compiled-program cache entries for evaluated source | | `eval.cache_ttl` | duration | 0 (no expiry) | Lifetime of a cached compiled program | ```yaml lua: cache: enabled: true dir: .cache/lua type_system: enabled: true ``` See: [Lua Overview](lua/overview.md) ### Scheduler Core partitioning for the WASM runtime. When enabled, `reserved_cores` CPUs are set aside for WASM execution and the rest serve the actor scheduler; an invalid split (for example more reserved cores than available) is logged and ignored. | Field | Type | Default | Description | |-------|------|---------|-------------| | `wasm_isolation.enabled` | bool | false | Partition cores between WASM and actor work | | `wasm_isolation.reserved_cores` | int | 1 | Cores reserved for WASM execution | ```yaml scheduler: wasm_isolation: enabled: true reserved_cores: 2 ``` ### Finder Registry search caching. Used internally for entry lookups. | Field | Type | Default | Description | |-------|------|---------|-------------| | `query_cache_size` | int | 1000 | Cached query results | | `regex_cache_size` | int | 100 | Compiled regex patterns | ```yaml finder: query_cache_size: 2000 ``` ### OpenTelemetry Distributed tracing and metrics export via OTLP. | Field | Type | Default | Description | |-------|------|---------|-------------| | `enabled` | bool | false | Enable OTEL | | `endpoint` | string | localhost:4318 | OTLP endpoint | | `protocol` | string | http/protobuf | Protocol: grpc, http/protobuf | | `service_name` | string | wippy-runtime | Service identifier | | `service_version` | string | | Service version tag | | `insecure` | bool | true | Allow plaintext OTLP connection | | `sample_rate` | float | 1.0 | Trace sampling (0.0-1.0) | | `propagators` | string[] | `[tracecontext, baggage]` | Context propagators | | `traces_enabled` | bool | true | Export traces | | `metrics_enabled` | bool | false | Export metrics | | `http.enabled` | bool | true | Trace HTTP requests | | `http.extract_headers` | bool | true | Extract trace context from inbound headers | | `http.inject_headers` | bool | true | Inject trace context into the HTTP response | | `process.enabled` | bool | true | Trace process lifecycle | | `process.trace_lifecycle` | bool | true | Emit spans for spawn/terminate | | `interceptor.enabled` | bool | true | Trace function calls | | `interceptor.order` | int | 100 | Decoded compatibility field; runtime v0.3.32a registers the interceptor at order 100 regardless of this value | | `queue.enabled` | bool | true | Trace queue publish/consume | | `temporal.enabled` | bool | false | Trace Temporal workflows | ```yaml otel: enabled: true endpoint: "http://jaeger:4318" traces_enabled: true process: trace_lifecycle: true ``` Standard OTEL environment variables (`OTEL_SDK_DISABLED`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_INSECURE`, `OTEL_SERVICE_NAME`, `OTEL_SERVICE_VERSION`, `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG`, `OTEL_PROPAGATORS`) override the matching fields. See: [Observability Guide](guides/observability.md) ### Shutdown Graceful shutdown behavior. | Field | Type | Default | Description | |-------|------|---------|-------------| | `timeout` | duration | 30s | Max wait for components to stop | ```yaml shutdown: timeout: 60s ``` ### Metrics Internal metrics collection buffer. | Field | Type | Default | Description | |-------|------|---------|-------------| | `buffer.size` | int | 10000 | Metrics buffer capacity | | `interceptor.enabled` | bool | true | Auto-track function calls | ```yaml metrics: buffer: size: 20000 interceptor: enabled: true ``` See: [Metrics Module](lua/system/metrics.md), [Observability Guide](guides/observability.md) ### Prometheus Prometheus metrics endpoint. | Field | Type | Default | Description | |-------|------|---------|-------------| | `enabled` | bool | false | Start metrics server | | `address` | string | | Listen address; must be set explicitly when `enabled: true`, otherwise the metrics server does not start | | `max_cardinality` | int | 1024 | Distinct label sets retained per metric (LRU); `0` or less uses the default | ```yaml prometheus: enabled: true address: "0.0.0.0:9090" ``` Exposes `/metrics` endpoint for Prometheus scraping, plus `/livez`. See: [Observability Guide](guides/observability.md) ### Cluster Multi-node clustering: gossip membership plus a bounded Raft consensus core. See the [Cluster Guide](guides/cluster.md) for the architecture and operational model; this section is the config-key reference. #### Top-level | Field | Type | Default | Description | |-------|------|---------|-------------| | `enabled` | bool | false | Enable clustering | | `name` | string | hostname | Node name; must be unique across the cluster | | `failure_domain` | string | | Zone/rack label; advertised in gossip so voters spread across domains | | `kv_crdt_tombstone_retention` | duration | 0 | Age after which `store.kv.crdt` delete tombstones are reclaimed; `0` disables age-based GC | | `kv_crdt_tombstone_gc_alive_peers` | bool | false | Use the current alive membership as the tombstone acknowledgement set | #### Membership (gossip) SWIM gossip via memberlist. Used for node discovery, failure detection, and metadata dissemination. | Field | Type | Default | Description | |-------|------|---------|-------------| | `membership.bind_addr` | string | 0.0.0.0 | Gossip bind address | | `membership.bind_port` | int | 7946 | Gossip bind port (TCP+UDP) | | `membership.advertise_addr` | string | | Address peers use to reach this node (NAT/k8s) | | `membership.join_addrs` | string | | Comma-separated seed `host:port` pairs | | `membership.secret_key` | string | | Base64-encoded gossip encryption key (inline) | | `membership.secret_file` | string | | Path to file holding the gossip encryption key | | `membership.gossip_interval` | duration | 500ms | Gossip dissemination period | | `membership.push_pull_interval` | duration | 5s | Full state sync period | | `membership.dead_node_reclaim_time` | duration | 30s | When a dead node's name/address can be reclaimed | | `membership.probe_interval` | duration | 1s | Failure-detection probe cycle | | `membership.probe_timeout` | duration | 200ms | Ack wait per probe | | `membership.tcp_timeout` | duration | 1s | TCP fallback probe timeout | | `membership.suspicion_mult` | int | 3 | Suspicion timeout multiplier | A gossip secret is required. Set `membership.secret_key` or `membership.secret_file` (the file wins if both are given); with neither, the cluster component fails to start. The value is base64-encoded. The four probe keys inherit memberlist's local-network defaults when unset; raise them for high-latency links (e.g. `probe_interval: 2s`, `probe_timeout: 500ms`, `suspicion_mult: 5`). #### Internode (transport) TCP mesh carrying the relay and Raft traffic between nodes. Raft rides this mesh over internode request/reply; there is no separate Raft port. | Field | Type | Default | Description | |-------|------|---------|-------------| | `internode.bind_addr` | string | 0.0.0.0 | Mesh bind address | | `internode.bind_port` | int | 0 | Mesh port (0 = auto: 7950-7959, then ephemeral) | | `internode.auto_port` | bool | true | Discover the actual port at boot, pin it, and advertise it in gossip | | `internode.advertise_addr` | string | | Additional relay endpoint (IP or DNS name) published for upgraded peers — for NAT or load-balanced reachability | | `internode.advertise_port` | int | 0 | Port for `advertise_addr` (0 = bind port; requires `advertise_addr`) | | `internode.identity_key` | string | | Base64-encoded ed25519 private key identifying this node (inline) | | `internode.identity_key_file` | string | | Path to a file holding that key | | `internode.trusted_peer_keys` | map | | Base64-encoded ed25519 public key per node name, including this node | | `internode.tls.enabled` | bool | false | Enable mutual TLS on the internode TCP mesh | | `internode.tls.cert_file` | string | | PEM node certificate; required with TLS | | `internode.tls.key_file` | string | | Matching private key; required with TLS | | `internode.tls.ca_file` | string | | PEM CA bundle for client and server verification; required with TLS | `advertise_addr`/`advertise_port` publish an additive endpoint in node metadata while the bind endpoint stays advertised unchanged, so mixed-version clusters keep connecting during a rolling upgrade. Internode identity is mandatory whenever clustering is enabled. `identity_key` and `identity_key_file` are mutually exclusive and one of them must be present; the value decodes (standard or raw base64) to either a 32-byte ed25519 seed or a 64-byte ed25519 private key. `trusted_peer_keys` maps each node name to that node's 32-byte ed25519 public key, and must contain an entry for the local `cluster.name` whose value matches the local identity — otherwise startup fails. See the [Cluster Guide](guides/cluster.md#internode-identity). TLS requires all three credential paths and TLS 1.2 or later. Unknown settings, invalid values, unreadable or invalid credentials, and credential paths supplied while TLS is disabled fail startup. The ed25519 identity configuration remains required. See [Internode TLS](guides/cluster.md#internode-tls) for an example. #### Raft (consensus) The bounded Raft core stores durable state under `raft.data_dir` by default (`~/.wippy/store`). A restarted node rejoins quorum from its peers. [`store.kv.raft`](system/store.md#cluster-kv-stores) entries replicate through this core, and gossip coordinates bootstrap using a `bootstrap_expect` model. | Field | Type | Default | Description | |-------|------|---------|-------------| | `raft.data_dir` | string | `~/.wippy/store` | Directory for fs-durable Raft state and durable CRDT snapshots (under `/_sys/`). Diskless only when no path resolves (no home dir and none set) | | `raft.enabled` | bool | true | Run a Raft node; `false` makes this a gossip-only client | | `raft.role` | string | server | `server` runs a Raft node; `client` is gossip-only | | `raft.eligible` | bool | true | Whether this node may be selected as a voter or standby; false keeps it outside Raft as a client | | `raft.priority` | int | 100 | Voter selection priority (lower is preferred) | | `raft.bootstrap_expect` | int | 1 | Initial quorum size: `0`=join existing, `1`=single-node, `N`=wait for N eligible nodes including the local node, then form quorum | | `raft.max_voters` | int | 5 | Voter ceiling (must be odd); up to `max_standbys` additional eligible nodes become standbys, and the rest remain clients | | `raft.max_standbys` | int | 4 | Non-voting members kept warm for promotion; nodes beyond voters+standbys are not Raft members | | `raft.reconcile_debounce` | duration | 2s | Coalesce window after a gossip event before the voter reconciler runs | | `raft.reconcile_timeout` | duration | 2s | Bound per reconcile pass | | `raft.heartbeat_timeout` | duration | 3s | Follower idle wait before starting an election | | `raft.election_timeout` | duration | 3s | Candidate election timeout (clamped to >= heartbeat) | | `raft.commit_timeout` | duration | 500ms | Idle leader heartbeat cadence | | `raft.snapshot_threshold` | uint64 | 8192 | Log entries since last snapshot before a new one | | `raft.snapshot_interval` | duration | 2m | Snapshot check interval | | `raft.snapshot_retain` | int | 3 | Snapshots retained | | `raft.trailing_logs` | uint64 | 10240 | Log entries retained after a snapshot | | `raft.max_append_entries` | int | 16 | Max entries per AppendEntries RPC | | `raft.leader_probe_interval` | duration | 3s | Global-registry leader-reachability probe cadence | | `raft.leader_probe_grace` | int | 3 | Consecutive probe failures before leader is declared unreachable | | `raft.registry_backend` | string | kv | Cluster name-registry implementation: `kv` (shared kv keyspace) or `fsm` (dedicated Raft FSM) | | `raft.global_dissem_tombstone_retention` | duration | 0 | How long the global-name dissemination cache keeps delete tombstones | Single-node (development) — clustering on, bootstraps itself immediately: ```yaml cluster: enabled: true name: dev membership: secret_key: "d2lwcHktZG9jcy1nb3NzaXAtc2VjcmV0LTMyYnl0ZXM=" internode: identity_key: "d2lwcHktZG9jcy1kZXYtbm9kZS1leGFtcGxlc2VlZCE=" trusted_peer_keys: dev: "rNqImcjOzef28dzvma80mSrCW1px5LBAc5TbaYqAgm0=" raft: bootstrap_expect: 1 ``` Three-node voting cluster — each node lists the others as seeds and waits for all three before forming quorum. Every node carries the same `trusted_peer_keys` map and its own private key: ```yaml cluster: enabled: true name: node-1 failure_domain: us-east-1a membership: bind_port: 7946 join_addrs: "node-2:7946,node-3:7946" secret_file: /etc/wippy/cluster.key internode: identity_key_file: /etc/wippy/node-1.key trusted_peer_keys: node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0=" node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8=" node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE=" raft: bootstrap_expect: 3 max_voters: 5 ``` Gossip-only client — joins the cluster for naming/messaging but never runs Raft. It still needs its own identity and must appear in every node's trusted map: ```yaml cluster: enabled: true name: edge-7 membership: join_addrs: "node-1:7946,node-2:7946" secret_file: /etc/wippy/cluster.key internode: identity_key_file: /etc/wippy/edge-7.key trusted_peer_keys: node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0=" node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8=" node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE=" edge-7: "7lzP4jBAkC3P+0jq4vtMsC45571BlVXk3mSlOD/Z0SA=" raft: role: client ``` ### LSP Language Server Protocol server for editor integrations. | Field | Type | Default | Description | |-------|------|---------|-------------| | `enabled` | bool | false | Enable the LSP service and TCP server; the HTTP transport also requires this | | `address` | string | :7777 | TCP listen address | | `http_enabled` | bool | false | Enable the HTTP transport | | `http_address` | string | :7778 | HTTP listen address | | `http_path` | string | /lsp | HTTP endpoint path | | `http_allow_origin` | string | * | CORS allowed origin | | `max_message_bytes` | int | 8388608 | Max incoming message size | ```yaml lsp: enabled: true address: ":7777" http_enabled: true ``` See: [LSP Guide](guides/lsp.md) ### Network Service Overlay network manager (SOCKS5, I2P, Tailscale drivers). | Field | Type | Default | Description | |-------|------|---------|-------------| | `state_dir` | string | .wippy/net | Driver state storage directory | | `default_network` | string | | Default network ID applied when entries omit `network` | ```yaml network_service: state_dir: /var/lib/wippy/net default_network: app:tailscale ``` See: [Network Overlays](system/network.md) ### HTTP Dispatcher Tuning for the shared HTTP client pool used by HTTP-dispatched functions and outbound requests. | Field | Type | Default | Description | |-------|------|---------|-------------| | `dispatcher.http.timeout` | duration | 0 (none) | Per-request timeout | | `dispatcher.http.max_idle_conns` | int | 0 (stdlib) | Max idle connections across all hosts | | `dispatcher.http.max_idle_per_host` | int | 0 (stdlib) | Max idle connections per host | | `dispatcher.http.idle_conn_timeout` | duration | 0 (stdlib) | Idle connection timeout | | `dispatcher.http.max_clients` | int | 0 (unbounded) | Max distinct pooled clients | ```yaml dispatcher: http: timeout: 30s max_idle_per_host: 32 ``` ### Modules Module registry client used by `wippy install`/`update`. | Field | Type | Default | Description | |-------|------|---------|-------------| | `registry_url` | string | https://hub.wippy.ai | Registry endpoint | ```yaml modules: registry_url: https://internal-registry.example.com ``` ### Extensions Native Go plugin extensions loaded at boot (Unix only). | Field | Type | Default | Description | |-------|------|---------|-------------| | `enabled` | bool | true | Load extensions | | `paths` | string[] | | Plugin file paths (relative to config dir) | ```yaml extensions: enabled: true paths: - ./extensions/myplugin.so ``` ### Environment Variables | Variable | Description | |----------|-------------| | `GOMEMLIMIT` | Memory limit fallback when `--memory-limit` flag is not set (precedence: `--memory-limit` flag > `GOMEMLIMIT` > 1G default) | ### See Also - [CLI Reference](guides/cli.md) — Command-line options - [Cluster Guide](guides/cluster.md) — Clustering architecture and operations - [Entry Kinds](guides/entry-kinds.md) — Entry types and fields - [Observability Guide](guides/observability.md) — Logging, metrics, and tracing --- # "Cluster" ## Cluster Wippy runs as a single node by default. Enabling clustering connects nodes through gossip membership and a bounded Raft consensus core, supporting cluster-wide process names, distributed locks, and process-group messaging. Clustering is disabled until `cluster.enabled` is set to `true`. ### Cluster Capabilities - **Membership** — every node knows the live set of peers through gossip, with fast failure detection. - **Cluster-wide process names** — register a process under a name that resolves from any node, with a choice of consistency guarantees (see [Naming](#naming-and-name-scopes)). - **Distributed locks** — `system.lock` provides cluster-wide mutual exclusion with automatic release when the holder dies (see [Distributed locks](#distributed-locks)). - **Process groups** — publish to every member of a named group across all nodes (see [Process groups](#process-groups)). - **Replicated key-value stores** — `store.kv.raft` (strong) and `store.kv.crdt` (eventual) replicate KV data across nodes (see [Store](system/store.md#cluster-kv-stores)). - **A consensus core** — a small, bounded Raft cluster provides the linearizable backbone the naming and lock primitives build on. ### Architecture: Bounded Raft Wippy limits Raft membership to a fixed-size core so the leader does not replicate every log entry to every node. Other nodes participate through gossip. Each node has one of three roles in the Raft configuration: | Role | Count (default) | In Raft config | Receives log replication | Votes | |------|-----------------|----------------|--------------------------|-------| | **Voter** | up to 5 (`max_voters`, odd) | yes | yes | yes | | **Standby** | up to 4 (`max_standbys`) | yes | yes | no | | **Client** | unbounded | no | no | no | - **Voters** form the quorum. Writes commit once a majority of voters acknowledge them. `max_voters` is normalized to an odd cap (default 5). With at least three eligible nodes, the reconciler also chooses an odd voter count. With two eligible nodes and a cap greater than one, both are voters; `max_voters: 1` keeps a single voter. - **Standbys** are non-voting members kept fully replicated and warm. When a voter departs, the leader promotes the highest-ranked standby into the open voter slot, so quorum recovers without waiting for a fresh node to catch up. - **Clients** are nodes beyond `voters + standbys`. They are not in the Raft configuration, so the leader does not send them log entries. They participate in gossip and route writes to a Raft member, keeping Raft replication bounded by the configured core size. The `max_voters` and `max_standbys` settings cap the consensus core independently of the total cluster size. #### Voter Selection The leader runs a reconciler that, on every membership change (debounced by `raft.reconcile_debounce`, default 2s), recomputes which nodes should be voters and applies the minimal set of promote/demote operations. Selection is deterministic — every node derives the same ordering from the same gossip view — and is driven by three gossip-advertised hints: - `raft.eligible` — a node with `eligible: false` is excluded from both voter and standby selection and remains outside Raft as a client. Keep a node eligible but below the voter cutoff when it should serve as a standby. - `raft.priority` — lower value is preferred when filling voter slots; ties break by node ID. - `failure_domain` — voters are spread across distinct domains (zones/racks) first, reducing the risk that one domain failure removes a voter majority. Operations are applied in a quorum-preserving order: adds and promotions first, then demotions, then removals. ### Membership and Gossip Membership uses SWIM gossip (HashiCorp memberlist). Each node binds a gossip port (default **7946**) and continuously exchanges small messages with peers to detect failures and disseminate metadata. A node joins by pointing at one or more existing nodes: ```yaml cluster: enabled: true name: node-2 membership: join_addrs: "node-1:7946" secret_file: /etc/wippy/cluster.key internode: identity_key_file: /etc/wippy/node-2.key trusted_peer_keys: node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0=" node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8=" ``` The first node needs no `join_addrs`; it starts as a seed. Joins retry with backoff, and an isolated node periodically attempts to rejoin. This supports nodes that restart with a new IP, as commonly occurs in Kubernetes. Gossip is always encrypted with a shared key. Supply it inline as `membership.secret_key` or from a file as `membership.secret_file`; a node started with neither fails to bring the cluster component up. The value is base64-encoded and identical on every node. Membership changes (`NodeJoined`, `NodeLeft`, `NodeUpdated`) are the events that drive Raft bootstrap, voter reconciliation, process-group sync, and automatic cleanup of names owned by a departed node. ### Internode identity Every node holds an ed25519 key pair, and every node carries the map of public keys it trusts. Both are mandatory when `cluster.enabled: true`. ```yaml cluster: internode: identity_key_file: /etc/wippy/node-1.key trusted_peer_keys: node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0=" node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8=" node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE=" ``` | Key | Content | |-----|---------| | `internode.identity_key` | The node's private key, inline | | `internode.identity_key_file` | Path to a file holding that key | | `internode.trusted_peer_keys` | Node name to public key, for every node in the mesh including this one | Key format: base64, standard or raw (unpadded) encoding. A private key decodes to either a 32-byte ed25519 seed or a full 64-byte ed25519 private key; a trusted peer key decodes to a 32-byte ed25519 public key. There is no key-generation subcommand — mint keys with any ed25519 tool and base64 the raw bytes: ```bash ## 32-byte seed and its public key, base64-encoded openssl genpkey -algorithm ed25519 -out node-1.pem openssl pkey -in node-1.pem -outform DER \ | tail -c 32 | base64 > node-1.key openssl pkey -in node-1.pem -pubout -outform DER \ | tail -c 32 | base64 ``` `identity_key` and `identity_key_file` are mutually exclusive, and one of them is required. `trusted_peer_keys` must contain an entry for the local `cluster.name` whose value is this node's own public key; a missing or mismatched self entry aborts startup. This makes the trusted map a single artifact you can distribute unchanged to every node. The mesh handshake is mutual. Each side proves knowledge of the shared gossip secret with an HMAC over a transcript binding both node IDs and both nonces, and signs that transcript with its identity key; the peer verifies the signature against the public key it has for that node ID and against the key the peer advertises in gossip. Either check failing closes the connection. Consequences to plan for: - The mesh does not interoperate with a node that has no identity. Every node in the cluster must be configured with one. - A peer whose node ID is absent from `trusted_peer_keys` is rejected, as is one whose gossip-advertised public key disagrees with the trusted entry. Adding a node means distributing its public key to the existing nodes. - A node ID must be present in the live gossip membership before its key resolves, so a peer that has not joined gossip cannot open a mesh connection. ### Internode TLS Enable mutual TLS for the internode TCP mesh under `cluster.internode.tls`. This also protects the relay and Raft traffic carried by that mesh. Add the following settings to your existing cluster configuration: ```yaml cluster: internode: tls: enabled: true cert_file: /etc/wippy/node.crt key_file: /etc/wippy/node.key ca_file: /etc/wippy/cluster-ca.pem ``` TLS is disabled by default. Enabling it requires all three nonempty file paths: a PEM certificate and matching private key, and a PEM CA bundle used to verify both server and client certificates. The minimum protocol version is TLS 1.2. Configure compatible certificates and trust roots on every connecting peer. The ed25519 internode identity and trusted peer map described above are still required; TLS adds transport protection to the existing mesh authentication. Unknown TLS settings, invalid types, missing credentials, malformed CA bundles, and credential paths supplied without `enabled: true` fail startup. An invalid explicit configuration never silently falls back to plaintext. ### Bootstrap The initial cluster forms through gossip rather than a static peer list. With the Consul/Nomad-style `bootstrap_expect` setting, each starting node waits until the configured number of eligible nodes, including itself, is visible before forming quorum. | `bootstrap_expect` | Behavior | |--------------------|----------| | `0` | Never self-bootstrap; only join a cluster that already exists | | `1` | Single-node; bootstrap immediately with self as the only voter | | `N` | Wait until `N` eligible nodes, including the local node, are stably visible in gossip, then all derive the same voter list and form quorum | For an `N`-node bootstrap, set the same `bootstrap_expect: N` on every initial node. Each advertises a "pre-bootstrap" status in gossip; once exactly `N` such nodes, including itself, are visible for a short stability window, every node independently computes the identical sorted voter set and forms the cluster. The stability window prevents a brief, partial view from triggering bootstrap early. Nodes that start later see an already-formed cluster and skip bootstrap entirely — the leader's reconciler adds them as voters or standbys. ### Raft Consensus Core Raft state is **fs-durable by default**: logs and snapshots are persisted under `cluster.raft.data_dir` (default `~/.wippy/store`, in `_sys/raft`), and [`store.kv.raft`](system/store.md#cluster-kv-stores) replicates through the same core. A restarting node still rejoins gossip and catches up from its peers, so the cluster also tolerates losing a node's disk; durability comes from both the live quorum and on-disk state. A node runs diskless only when no data directory resolves (no configured path and no home directory) — see [Recovery](#recovery-and-failure-modes). Raft does not open its own listening port. It rides the **internode mesh** — the same TCP connections used for relay traffic between nodes — carrying its RPCs as internode request/reply frames over the mesh's reliable per-class channels. The internode port is auto-selected at boot (range 7950-7959, then ephemeral), pinned, and advertised in gossip so peers can reach it. Nodes must be mutually reachable on both the gossip port and their advertised internode TCP ports. The Raft FSM holds the global name registry: active `name -> PID` bindings plus in-flight strong reservations. That is what the naming primitives below read and write. ### Naming and Name Scopes A process can be registered and addressed by name instead of its raw PID. Its **scope** selects the consistency and coordination behavior. Four scopes are available, ordered from local to strongest: | Scope | Backed by | Visibility | Guarantee | |-------|-----------|------------|-----------| | **Local** | per-node map | this node only | Instant, node-local; no coordination | | **Eventual** | gossip CRDT | cluster-wide | Eventually consistent; converges after gossip rounds | | **Consistent** | Raft | cluster-wide | Linearizable writes; unique singleton across the cluster | | **Strong** | Raft + all-node ack | cluster-wide | Consistent, plus every live node acknowledges before the name is active | How to choose: - **Local** — names meaningful only on one node, such as a per-node helper. Released when the process exits and requires no cluster coordination. - **Eventual** — cluster-wide service, group, and presence names where a brief stale window is acceptable. The binding set is fully replicated to every node, so it fits a bounded namespace — not one name per high-cardinality entity such as a per-session process (address those directly by PID). When two origins register the same name, conflict resolution picks a winner and the losing process receives a cancel event (`process.event.CANCEL`) carrying the reason `name revoked: `; it keeps running and can re-register. Names release when the owning node leaves. - **Consistent** — the standard choice for cluster-wide named singletons. First-write-wins: a second registration of the same name to a different PID fails with "already exists" and returns the current owner. Writes need a quorum, so they stall in a minority partition. Reads come from the local Raft replica and may lag a write by a few milliseconds. - **Strong** — the small set of control-plane singletons where even a momentary stale read is dangerous. On top of the Consistent guarantee, the registration opens a reservation that every live node must acknowledge before the name becomes authoritative; any node already holding a conflicting binding rejects it immediately. If the deadline passes before all nodes ack, the registration expires and reports which nodes were missing. Names are released automatically: Local on process exit; Consistent and Strong on process exit (via topology monitoring) and on node departure; Eventual on node departure. Resolution for messaging (`process.send`, `process.terminate`, and similar) consults the planes most-authoritative first — Consistent and Strong (Raft), then Eventual (gossip), then Local — so a cluster-wide name shadows a local one with the same string. The Lua surface for naming lives on `process.registry` (register/lookup/unregister with a scope) — see the [Process](lua/core/process.md) reference. ### Process Groups Process groups are a cluster-aware publish/subscribe and membership facility modeled on Erlang's `pg`. A process joins a named group; a broadcast fans out over the internode mesh to the group's members across all nodes, delivered best-effort. Groups are eventually consistent and independent of Raft — they use the gossip membership view to choose recipients — so they keep working even while the consensus core is converging. Typical operations: join/leave a group, broadcast to all members (or local members only), list members, and monitor a group for join/leave events. On a new node joining, groups reconcile their membership through a direct sync handshake, and a background anti-entropy loop repairs any divergence over time. See [Process Groups](lua/core/pg.md) for the Lua API and the [`pg.scope` entry kind](system/process-groups.md) for configuration. ### Distributed Locks `system.lock` is cluster-wide mutual exclusion built on a raft-linearizable conditional write in the shared key-value store. Acquiring a lock performs a set-if-absent of the holder PID at `_sys:lock:`; releasing deletes that entry if it is still held by the caller. Because the conditional write goes through Raft (with off-leader writes forwarded to the leader), it is linearizable, so at most one holder can exist cluster-wide. ```lua local ok, err = system.lock.acquire("orders.migration") if not ok then -- err has kind errors.ALREADY_EXISTS when another process holds the lock. -- Apply the caller's retry and backoff policy for that case if needed. return nil, err end -- critical section: only one holder cluster-wide local released, release_err = system.lock.release("orders.migration") if release_err then return nil, release_err end return released ``` Acquire is fail-fast (non-blocking): if the lock is held, it returns immediately, so callers provide their own retry and backoff. The lock is released if the holder process exits or its node leaves. See the [System](lua/system/system.md) reference for exact signatures. ### Configuration See [Configuration](guides/configuration.md#cluster) for related cluster settings. These minimal shapes include the mandatory internode identity settings. These are configuration fragments, not a complete deployment manifest. Replace node names, addresses, failure domains, paths, and every `${env:...}` identity placeholder with values generated and distributed for your own cluster. Single node (development): ```yaml cluster: enabled: true name: dev membership: secret_key: "d2lwcHktZG9jcy1nb3NzaXAtc2VjcmV0LTMyYnl0ZXM=" internode: identity_key: "d2lwcHktZG9jcy1kZXYtbm9kZS1leGFtcGxlc2VlZCE=" trusted_peer_keys: dev: "rNqImcjOzef28dzvma80mSrCW1px5LBAc5TbaYqAgm0=" raft: bootstrap_expect: 1 ``` Three-node voting cluster (`node-2` and `node-3` differ only in `name`, `identity_key_file`, and `join_addrs`): ```yaml cluster: enabled: true name: node-1 failure_domain: us-east-1a membership: join_addrs: "node-2:7946,node-3:7946" secret_file: /etc/wippy/cluster.key internode: identity_key_file: /etc/wippy/node-1.key trusted_peer_keys: node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0=" node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8=" node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE=" raft: bootstrap_expect: 3 ``` Gossip-only client (joins for naming/messaging, never runs Raft). It still needs an identity, and the voters need its public key in their own maps: ```yaml cluster: enabled: true name: edge-7 membership: join_addrs: "node-1:7946,node-2:7946" secret_file: /etc/wippy/cluster.key internode: identity_key_file: /etc/wippy/edge-7.key trusted_peer_keys: node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0=" node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8=" node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE=" edge-7: "7lzP4jBAkC3P+0jq4vtMsC45571BlVXk3mSlOD/Z0SA=" raft: role: client ``` ### Ports | Purpose | Port | Protocol | Config key | |---------|------|----------|------------| | Gossip (membership) | 7946 | TCP + UDP | `cluster.membership.bind_port` | | Internode mesh (relay + Raft) | auto | TCP | `cluster.internode.bind_port` | Raft is multiplexed over the internode mesh rather than using a separate port. The internode port is assigned automatically and advertised through gossip. The gossip port is predictable by default, but peers must also be able to reach each node's advertised internode TCP port. ### Observability Cluster health is exposed through the standard [Prometheus endpoint](guides/observability.md) and through liveness health checks. Key metrics to watch: | Metric | Meaning | |--------|---------| | `raft_state` | 0 = follower, 1 = candidate, 2 = leader | | `raft_term` | Current Raft term; rapid increases signal election churn | | `raft_voters` / `raft_non_voters` | Live voters and standbys in the configuration | | `raft_leader_changes_total` | Leader transitions; should be near-flat in a healthy cluster | | `raft_voter_churn_burst_total` | Bursts of voter add/remove operations; sustained churn indicates instability | | `gossip_members{state}` | Counts by state (alive/suspect/dead/left) | | `gossip_convergence_seconds` | Time between gossip events | Built-in liveness checks (wired to the liveness endpoint): - **gossip** — healthy while the node's gossip health score stays low, with a boot grace window so a rejoining node is not killed prematurely. - **raft last-contact** — a voting follower fails if it has not heard from a leader recently; a standby tolerates a much longer gap; leaders always pass. - **process-group broadcast** — fails when the process-group service has neither sent nor received any broadcast for its activity ceiling, catching a wedged service or a persistent partition. ### Recovery and Failure Modes Raft state is fs-durable, but the cluster's primary durability still comes from a live quorum. The practical rules: - Keep a voter majority alive. With 5 voters you tolerate 2 simultaneous voter failures; standbys are promoted to refill open slots. Drop below a majority and writes (new Consistent/Strong registrations and lock acquisitions) stall until quorum returns. Locally replicated state can still answer some reads, but do not treat those reads as proof that a partitioned node has the latest value. - The leader proactively evicts a voter that is both heartbeat-silent and gossip-dead, so a dead voter does not permanently block quorum while a standby is promoted in. - To recover a cluster that has lost quorum, restart the failed nodes. They rejoin gossip and the surviving members fold them back in. Spreading voters across `failure_domain`s reduces the chance that a single zone failure causes quorum loss. ### See Also - [Configuration](guides/configuration.md#cluster) — related cluster settings - [Process](lua/core/process.md) — registering and resolving processes by name - [System](lua/system/system.md) — `system.cluster`, `system.raft`, `system.node`, `system.lock` - [Observability](guides/observability.md) — metrics and health endpoints - [Process Model](concepts/process-model.md) — actors, PIDs, and messaging --- # "Linter" ## Linter Run `wippy lint` to type-check and statically analyze Lua entries. ### Usage ```bash wippy lint # Check all Lua entries wippy lint --level hint # Show all diagnostics including hints wippy lint --json # Output in JSON format wippy lint --ns app # Check only the app namespace wippy lint --summary # Group results by error code ``` ### What Gets Checked The linter validates all Lua entry kinds: - `function.lua` — Functions - `library.lua` — Libraries - `process.lua` — Processes - `workflow.lua` — Workflows Bytecode entries hold compiled bytecode (fs/path/hash), not source, so they cannot be parsed or type-checked; the linter only checks source-bearing Lua entries (their `.bc` variants are skipped, though they may still appear in the total entry count). Each entry is parsed, type-checked, and analyzed for correctness issues. ### Severity Levels Diagnostics have three severity levels: | Level | Description | |-------|-------------| | `error` | Type errors and correctness issues that must be fixed | | `warning` | Likely bugs or problematic patterns | | `hint` | Style suggestions and informational notes | Control which levels appear with `--level`: ```bash wippy lint --level error # Errors only wippy lint --level warning # Warnings and errors (default) wippy lint --level hint # Everything ``` #### Parse Errors | Code | Description | |------|-------------| | `P0001` | Lua syntax error - source cannot be parsed | #### Type Check Errors (E-series) Type checker errors (`E0001`+) report issues found by the type system: type mismatches, undefined variables, invalid operations, and similar correctness problems. These are always reported as errors. ```lua local x: number = "hello" -- E: string not assignable to number local function add(a: number, b: number): number return a + b end add("one", "two") -- E: string not assignable to number ``` #### Undeclared Requires A string-literal `require("name")` whose module is neither in the entry's `imports`/`modules` declarations nor an ambient builtin fails with: ``` require("name") is not declared in _index.yaml imports or modules ``` This check always runs (it is not gated behind `--rules`) and reports as an error. Declare the module to satisfy it: ```yaml imports: json: wippy.stdlib:json # alias -> registry id modules: - funcs # bare module name ``` Dynamic requires (`require(variable)`) are not inspected. The linter and runtime share the ambient module set, which includes modules available without declaration, such as `process` in executable kinds. #### Lint Rule Warnings (W-series) Lint rules provide style and quality checks. Enable them with `--rules`: ```bash wippy lint --rules ``` | Code | Rule | Description | |------|------|-------------| | `W0001` | no-empty-blocks | Empty block statements | | `W0002` | no-global-assign | Assignment to global variables | | `W0003` | no-self-compare | Comparison of a value with itself | | `W0004` | no-unused-vars | Unused local variables | | `W0005` | no-unused-params | Unused function parameters | | `W0006` | no-unused-imports | Unused import statements | | `W0007` | no-shadowed-vars | Variable shadowing outer scope | Without `--rules`, only type checking (P and E codes) is performed. #### By Namespace Check specific namespaces using `--ns`: ```bash wippy lint --ns app # Exact namespace match wippy lint --ns "app.*" # All under app wippy lint --ns app --ns lib # Multiple namespaces ``` Dependencies of selected entries are loaded for type checking but their diagnostics are not reported. #### By Error Code Filter diagnostics by code: ```bash wippy lint --code E0001 wippy lint --code E0001 --code E0004 ``` #### By Count Limit the number of diagnostics shown: ```bash wippy lint --limit 10 # Show first 10 issues ``` #### Table Format (Default) Each diagnostic is displayed with source context, file location, and the error message. Results are sorted by entry, severity, and line number. A summary line shows totals: ``` Checked 42 entries: 5 errors, 12 warnings ``` #### Summary Format Group diagnostics by namespace and error code: ```bash wippy lint --summary ``` ``` By namespace: app 15 issues (5 errors, 10 warnings) lib 2 issues (2 warnings) By error code: E0001 [error ] 5 occurrences E0004 [error ] 3 occurrences Checked 42 entries: 5 errors, 12 warnings ``` #### JSON Format Machine-readable output for CI/CD processing: ```bash wippy lint --json ``` ```json { "diagnostics": [ { "entry_id": "app:handler", "code": "E0001", "severity": "error", "message": "string not assignable to number", "line": 10, "column": 5 } ], "total_entries": 42, "error_count": 5, "warning_count": 12, "hint_count": 0 } ``` ### Caching The linter caches results between runs. Cache keys include the source hash, method name, dependencies, and type-system configuration. Clear the cache if results seem stale: ```bash wippy lint --cache-reset ``` ### CI Integration In table and summary modes, the command exits non-zero when the filtered result contains errors. Warnings and hints do not affect the exit code, even when `--level warning` or `--level hint` displays them. JSON mode is different: after successfully encoding the result, `wippy lint --json` exits with code 0 even when `error_count` is non-zero. A CI job using JSON output must parse `error_count` itself. To use the command's exit status as the gate, run a non-JSON invocation: ```bash wippy lint --level error ``` You can produce a report separately without treating its exit status as the lint result: ```bash wippy lint --json --level error > lint-results.json ``` Example GitHub Actions step: ```yaml - name: Lint run: wippy lint --level warning ``` ### Flags Reference | Flag | Short | Default | Description | |------|-------|---------|-------------| | `--level` | | warning | Minimum severity level (error, warning, hint) | | `--json` | | false | Output in JSON format | | `--ns` | | | Filter by namespace patterns | | `--code` | | | Filter by error codes | | `--limit` | | 0 | Max diagnostics to show (0 = unlimited) | | `--summary` | | false | Group by error code | | `--no-color` | | false | Disable colored output | | `--rules` | | false | Enable lint rules (W-series style/quality checks) | | `--cache-reset` | | false | Clear cache before linting | | `--profile` | | | Apply a workspace profile from merged runtime configuration; repeat to apply profiles in order | | `--set` | | | Override a merged configuration value as `section.path=value`; repeat for multiple overrides | | `--lock-file` | `-l` | wippy.lock | Path to lock file | | `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable, applied in order) | | `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) | ### See Also - [CLI](guides/cli.md) — Full CLI reference - [Types](lua/types.md) — Type-system documentation - [LSP](guides/lsp.md) — Editor integration with live diagnostics --- # "Language Server" ## Language Server Wippy includes a Language Server Protocol (LSP) server for Lua editor features. It runs as part of the Wippy runtime and accepts editor connections over TCP or HTTP. ### Features - Code completion with type-aware suggestions - Hover information showing types and signatures - Go to definition - Find references - Document and workspace symbols - Call hierarchy (incoming and outgoing calls) - Pull diagnostics for type errors in the current editor overlay after successful parsing - Signature help for function parameters ### Configuration Enable the LSP server in `.wippy.yaml`: ```yaml lsp: enabled: true address: ":7777" ``` #### Configuration Fields | Field | Default | Description | |-------|---------|-------------| | `enabled` | false | Enable the LSP service and TCP server | | `address` | :7777 | TCP listen address | | `http_enabled` | false | Enable the HTTP transport | | `http_address` | :7778 | HTTP listen address | | `http_path` | /lsp | HTTP endpoint path | | `http_allow_origin` | * | CORS allowed origin | | `max_message_bytes` | 8388608 | Max incoming message size (bytes) | #### TCP Transport The TCP server speaks JSON-RPC 2.0 with standard LSP message framing (Content-Length headers). This is the primary transport for editor integrations. #### HTTP Transport The HTTP transport accepts POST requests with JSON-RPC payloads. It supports browser-based editors and web tools, answers CORS preflight `OPTIONS` requests, and includes CORS headers for cross-origin access. ```yaml lsp: enabled: true http_enabled: true http_address: ":7778" http_path: "/lsp" http_allow_origin: "*" ``` ### Document URI Scheme The LSP server uses the `wippy://` URI scheme to identify registry entries: ``` wippy://namespace:entry_name ``` Editors map these URIs to entry IDs in the registry. Both `wippy://` scheme and raw `namespace:entry_name` formats are accepted. ### Indexing The LSP server maintains an index of code entries. Multiple workers update the index in the background. Key behaviors: - Entries are indexed in dependency order (dependencies first) - Changes trigger re-indexing of affected entries - Unsaved editor changes are stored in an overlay - Indexing is incremental; only changed entries are reprocessed ### Supported LSP Methods | Method | Description | |--------|-------------| | `initialize` | Capability negotiation | | `initialized` | Initialization-complete notification | | `shutdown` | Shut down the protocol session | | `exit` | Exit notification | | `textDocument/didOpen` | Track opened documents | | `textDocument/didChange` | Full document sync | | `textDocument/didClose` | Release documents | | `textDocument/hover` | Type info at cursor | | `textDocument/definition` | Jump to definition | | `textDocument/references` | Find all references | | `textDocument/completion` | Code completion | | `textDocument/signatureHelp` | Function signatures | | `textDocument/diagnostic` | File diagnostics | | `textDocument/documentSymbol` | File symbols | | `workspace/symbol` | Global symbol search | | `textDocument/prepareCallHierarchy` | Call hierarchy | | `callHierarchy/incomingCalls` | Find callers | | `callHierarchy/outgoingCalls` | Find callees | ### Completion The completion engine resolves types through the code graph. It provides: - Member completion after `.` and `:` (fields, methods) - Local variable completion - Module-level symbol completion - Trigger characters: `.`, `:` ### Diagnostics After a document parses successfully, indexing stores type-checking diagnostics such as mismatches and undefined symbols. Diagnostics use the standard error, warning, information, and hint severities. Full-document change notifications update the overlay used for diagnostics. Clients retrieve the current stored result with `textDocument/diagnostic`; this server does not push `textDocument/publishDiagnostics` notifications. A parse failure aborts re-indexing before new diagnostics are stored, so the pull result does not report that syntax error and can retain the previous successful result. ### See Also - [Linter](guides/linter.md) — CLI-based code checking - [Types](lua/types.md) — Type-system documentation - [Configuration](guides/configuration.md) — Runtime configuration --- # "Dependency Management" ## Dependency Management Wippy resolves module dependencies from source declarations and records exact versions in `wippy.lock`. Published modules are downloaded from the Hub into the project's module directory. The `acme/*` module names, versions, hashes, and local paths below are illustrative. Substitute modules and verified digests from your own project or the Hub. #### wippy.lock The lock file tracks your project's directory layout and pinned dependencies: ```yaml directories: modules: .wippy src: ./src modules: - name: acme/http version: v1.2.0 hash: 4ea816fe84ca58a1f0869e5ca6afa93d6ddd72fa09e1162d9e600a7fbf39f0a2 - name: acme/sql version: v2.0.1 hash: b3f9c8e12a456d7890abcdef1234567890abcdef1234567890abcdef12345678 ``` | Field | Description | |-------|-------------| | `directories.modules` | Where downloaded modules are stored (default: `.wippy`) | | `directories.src` | Where your source code lives (default: `./src`) | | `modules[].name` | Module identifier in `org/module` format | | `modules[].version` | Pinned semantic version | | `modules[].hash` | Artifact digest the downloaded pack must match; a bare hex value is read as `sha256` | | `modules[].root` | Marks the selected deployment root; at most one module may carry it | | `options.unpack_modules` | Extract packs into directories instead of loading them as `.wapp` files (default: `false`) | #### wippy.yaml Module metadata for publishing. Required only when you publish your own module: ```yaml organization: acme module: http version: 1.2.0 description: HTTP utilities for Wippy license: MIT repository: https://github.com/acme/wippy-http keywords: - http - web ``` | Field | Required | Description | |-------|----------|-------------| | `organization` | Yes | Lowercase, alphanumeric with hyphens | | `module` | Yes | Lowercase, alphanumeric with hyphens | | `version` | No | Semantic version (set at publish time) | | `description` | No | Module description | | `license` | No | SPDX license identifier | | `repository` | No | Source repository URL | | `homepage` | No | Project homepage | | `keywords` | No | Discovery keywords | | `authors` | No | Author list | ### Declaring Dependencies Add `ns.dependency` entries in your `_index.yaml`: ```yaml version: "1.0" namespace: app entries: - name: dependency.http kind: ns.dependency component: acme/http version: "^1.0.0" - name: dependency.sql kind: ns.dependency component: acme/sql version: ">=2.0.0" ``` #### Version Constraints | Constraint | Example | Matches | |------------|---------|---------| | Exact | `1.2.3` | Only 1.2.3 | | Caret | `^1.2.0` | >=1.2.0, <2.0.0 | | Tilde | `~1.2.0` | >=1.2.0, <1.3.0 | | Range | `>=1.0.0` | 1.0.0 and above | | Wildcard | `*` | Any version (picks highest) | | Combined | `>=1.0.0 <2.0.0` | Between 1.0.0 and 2.0.0 | #### Resolution Rules - Each module resolves against the **intersection of all declared ranges** across the dependency graph. Incompatible ranges (diamond conflicts) fail resolution with an explicit error rather than silently picking one side. - A full `wippy update` solves every module from its declared ranges; a targeted update and boot-time repair keep a pinned version that still satisfies every live range. - **Root parameters win over transitive ones**: when your app and a dependency both bind the same requirement, the parameters on your `ns.dependency` take precedence. Version ranges are never overridden; every declaration joins the intersection. - A component declared by several root `ns.dependency` entries is controlled by one of them — established declarations before new ones, parameter carriers before plain ones, ties on the lowest entry ID — and the others fold into references to it. A duplicate whose parameters disagree with the controlling declaration is rejected with a conflict error; update the existing dependency instead. Two resolution failures are reported distinctly. A constraint expression that cannot be satisfied by any release ever — the intersection of live ranges is empty — is a conflict, and the error names the module and every requester that contributed a range. A valid range set for which the hub currently publishes no matching version is an availability failure instead: a later release can make it resolvable without any change to the declarations. The runtime persists each resolved graph in its registry history and replays it at boot instead of re-solving, so a deployed application boots with exactly the versions that were resolved when the dependency change was applied. `wippy.lock` remains the portable snapshot for source projects. #### Entry provenance Provenance is registry-owned, not entry metadata. When entries are loaded, the registry stamps each one with the deployment source that supplied it: | Field | Description | |-------|-------------| | `registry.owner` | Module name (`org/module`) that supplied the entry; empty for application source | | `registry.root` | Set on `ns.dependency` entries supplied by the deployment root, marking them as root declarations | Entry authors never write these fields; they are assigned during loading and cannot be forged from an `_index.yaml`. Inspect them with `wippy registry list --registry-meta --json`. #### Starting a New Project ```bash wippy init ``` Creates a `wippy.lock` with default directories. #### Adding Dependencies ```bash wippy add acme/http # Latest version wippy add acme/http@1.2.3 # Exact version wippy add acme/http@latest # Latest label ``` This updates the lock file. Then install: ```bash wippy install ``` #### Resolving from Source If your source already declares `ns.dependency` entries: ```bash wippy update ``` This scans your source directory, resolves all dependency constraints, updates the lock file, and installs modules. #### Updating Dependencies ```bash wippy update # Re-resolve all dependencies wippy update acme/http # Update only acme/http wippy update acme/http acme/sql # Update specific modules ``` When updating specific modules, other modules stay pinned to their current versions. If the update would require changing non-target modules, you are prompted for confirmation. #### Installing from Lock File ```bash wippy install # Install all from lock wippy install --refresh # Re-fetch every module (--force and --repair are aliases) ``` ### Module Storage Downloaded modules are stored under the `.wippy/vendor/` directory: ``` project/ wippy.lock src/ _index.yaml .wippy/ vendor/ acme/ http-v1.2.0.wapp sql-v2.0.1.wapp ``` By default, modules are kept as `.wapp` files. To extract them into directories: ```yaml ## wippy.lock options: unpack_modules: true ``` With unpacking enabled: ``` .wippy/ vendor/ acme/ http-v1.2.0.wapp http/ wippy.yaml src/ _index.yaml ... ``` Unpacking never discards the pack. The canonical verified `.wapp` stays beside the extracted directory because it is the only content-addressed evidence for the module, and artifact materialization and repair read resources back out of it. The `.wapp` is what installation checks for: a directory whose pack is missing counts as not installed, and the module is downloaded again. Each install extracts the directory afresh from the verified archive, so hand-edits to a vendored directory do not survive. Modules resolved from a [workspace replacement](#local-development-with-replacements) are never downloaded or vendored; they load from the local path. ### Local Development with Replacements For local development, map Hub modules to local directories in the `workspace` section of a runtime configuration file. This is typically a private, ignored file composed over `.wippy.yaml`: ```yaml ## .wippy.workspace.yaml version: "1.0" workspace: replacements: acme/http: ../local-http acme/sql: ../local-sql ``` ```bash wippy run --config .wippy.yaml --config .wippy.workspace.yaml ``` Keys are `org/module`, values are directories (relative paths resolve against the first `--config` file's directory). Setting a replacement to `null` disables one inherited from an earlier config layer or profile. Replacements can also live inside a [profile](guides/configuration.md#profiles) so they activate only with `--profile workspace`. The path is required to exist, and to be a directory, only for a module the lock graph actually selects. A replacement declared for a module that nothing depends on is a resolution input, not a boot input: it can point at a directory that is not checked out on this machine without failing validation. A replacement changes where a module's source comes from, not which release was chosen. It keeps the selected version, while reconciliation snapshots the current local tree and records its digest and size as the replacement identity. Entries loaded from it shadow the vendored ones with the same ID. When a replacement is declared for a module the lock does not pin a version for, resolution asks the hub for a release version, and until stronger evidence selects one it holds a local-only zero version. Workspace replacements affect the load graph at boot and are never written to `wippy.lock`. Changes to the local source are reconciled directly, without contacting the hub. The module's source `exclude:` globs from `wippy.yaml` apply to replacement directories too, both when loading entries and when hashing content. The `replacements:` section in `wippy.lock` is deprecated. It still loads with a warning; move those entries to `workspace.replacements` in a configuration file. ### Load Order At boot, Wippy loads entries from directories in this order: 1. Source directory (`src`) 2. Replacement directories 3. Vendored module directories Modules with active replacements skip their vendor path. ### Integrity Verification Every module in the lock file carries an artifact digest. Boot refuses to load a module whose lock entry has none; `wippy install` accepts such an entry and records the digest the hub serves with the download. At boot, downloads are staged: the pack is written to a temporary file next to its final location, verified against both the digest pinned in `wippy.lock` and the digest the hub served with the download URL (plus the served size), and only then renamed into place. A staged file that fails verification is deleted. `wippy install` renames the download into its vendor path before verifying it, checks it against the served digest and size only, deletes it on failure, and replaces a lock digest that differs from the served one rather than enforcing it. A digest mismatch is a hard, non-retryable failure. At boot it is `PermissionDenied`, "module integrity verification failed", raised for a fresh download and for an already-vendored pack, which is re-verified against the lock digest before entries are loaded. `wippy install` reports it as `Internal`: "failed to store module" wrapping "verify cached WAPP: digest mismatch" for a pack already in the vendor directory, and "failed to download module" wrapping "verify downloaded WAPP: digest mismatch" for a fresh download. Nothing retries, re-downloads over the mismatch, or falls back to the served content. The same check guards resolution. When the hub serves a manifest whose digest differs from the one the lock pins, the manifest cache is refreshed once and re-compared; if it still disagrees, resolution fails naming both digests. Extracted directories carry their own recorded digest, size, and tree digest, and are re-verified against the recorded values, so a modified vendored tree is detected rather than loaded. Replacement sources are content-addressed per reconciliation attempt. The runtime snapshots the current local tree, then verifies that same digest and size before loading it. A concurrent change fails validation instead of mixing two source generations. A later reconciliation can accept a new local tree; its old recorded digest is a checkpoint, not an immutable Hub artifact identity. On restart, immutable historical artifacts are prefetched separately. Historical local replacements are reconciled against the final dependency declarations before loading, so a removed replacement does not require its old directory to remain on disk. A replacement still selected by the final graph must be present and valid. ### Build-time Artifacts A module can ship a filesystem resource marked with `meta.artifact.format` that consumers materialize onto disk instead of reading at runtime. Full and targeted `wippy install` and `wippy update`, cold boot, and runtime dependency operations reconcile those outputs as part of the same transaction that changes the module graph; `artifact.materialization_root` sets the output root. See [Build-time artifacts](guides/artifacts.md). ### See Also - [Build-time artifacts](guides/artifacts.md) - Declaring, materializing and reconciling artifact resources - [Building Components](guides/components.md) - The author side: `ns.requirement` and supplying values via `parameters` - [CLI](guides/cli.md) - Command reference - [Publishing](guides/publishing.md) - Publishing modules to the hub - [Project Structure](start/structure.md) - Project layout --- # "Build-time artifacts" ## Build-time artifacts A module can ship a directory that consumers use **at build time** rather than at runtime — most usefully, a package that other modules compile against. Wippy calls these **artifacts**: ordinary WAPP filesystem resources marked with `meta.artifact.format`. This is how a shared package reaches a module in a different repository. A path alias only resolves inside one repo; an artifact travels with the module. [The Design Layer](../frontend/design-layer.md) explains *what* belongs in such a package and what does not; this page is the mechanism that ships it. ### Declaring an artifact The producer declares a normal `fs.directory` and marks it with a format: ```yaml ## src/_index.yaml entries: - name: package_fs kind: fs.directory meta: comment: The npm package consumers materialize at build time. artifact: format: node-package directory: ./package ``` Nothing else changes: the resource is embedded into the WAPP like any other `fs.directory` — list it under `embed:` in `wippy.yaml` or pass `--embed` to `wippy publish` and `wippy pack`; a directory that is not embedded is neither packed nor validated. Declared artifacts are **validated during module publish and application pack**, so a malformed one fails at publish rather than in a consumer. ### Formats A format adapter decides how a directory is validated, what identity it has, and where it lands. Wippy ships one built-in: | Format | Owns subtree | Validates | |---|---|---| | `node-package` | `npm/` | `package.json` | `node-package` requires a `name` and a semantic `version`, and **rejects `preinstall`, `install`, `postinstall` and `prepare` lifecycle scripts** — a materialized package may not execute anything on install. It writes to `npm/` under the materialization root. The format must be registered in the binary doing the work. Hosts may register additional formats; duplicate names and overlapping roots are rejected. ### Materializing Most of the time you do not run anything. Materialized outputs are reconciled automatically during: - full and targeted `wippy install` and `wippy update` - cold boot - Hub-backed dynamic install, update and uninstall Full install, update, cold boot and runtime dependency reconciliation are *exact*: stale outputs are pruned. A **targeted** install overlays only the selected modules and preserves outputs belonging to modules it did not select. Local module replacements go through the same validation and materialization lifecycle as packed resources, so a replaced module's artifact behaves like a published one. #### Materializing explicitly For a build step that needs the artifact before the runtime is involved, the CLI exposes it directly: ```bash wippy artifacts materialize [--root ] ``` `--root` defaults to `.wippy`. The resource must declare `meta.artifact.format` and that format must be registered in this CLI. Be clear about what this command deliberately does **not** do: it does not resolve module dependencies, does not mutate `wippy.lock`, does not invoke package managers, and does not participate in runtime composition. It validates one artifact out of one WAPP and writes it to disk. #### Where output lands `artifact.materialization_root` configures the application-owned output root. Its default is the parent of the dependency vendor directory. Each format owns a non-overlapping subtree beneath it, so `node-package` output is always under `/npm/`. Materialization is transactional. Content is validated and staged, managed roots are swapped atomically under a process lock, a failure rolls back with the surrounding registry transaction, and an interrupted swap is recovered on the next run. ### Worked example: a shared frontend package A producer module whose only job is to publish a package — it serves nothing at runtime: ```yaml ## platform/ui-kit/src/_index.yaml version: "1.0" namespace: kickside.ui_kit entries: - name: package_fs kind: fs.directory meta: artifact: format: node-package directory: ./package ``` A consumer materializes it into its own tree before installing dependencies: ```bash wippy artifacts materialize kickside-ui-kit-1.5.0.wapp \ kickside.ui_kit:package_fs --root ./.wippy ``` That writes `./.wippy/npm/@kickside/ui-kit`. The consumer picks it up with an ordinary workspaces glob, so resolution is plain node resolution from there on: ```json { "workspaces": ["./.wippy/npm/@*/*"] } ``` ```bash npm install ``` Two things worth copying from this shape: - **The package is its own module, not a directory inside a bigger one.** The artifact carries its own `package.json` version, and tying it to a module that changes for unrelated reasons forces a release of one every time the other moves. - **The consumer resolves it as a normal dependency.** Once materialized there is no Wippy-specific import path, which is what lets the same source build inside the monorepo and outside it. #### Authoring the producer For a package artifact there is usually **nothing to build** — the directory is the deliverable. A CSS vocabulary package is just files plus a manifest: ```text platform/ui-kit/ ├── src/_index.yaml # declares package_fs as the artifact └── package/ # the directory that becomes the npm package ├── package.json ├── kx-card.css └── kx-state.css ``` ```json { "name": "@kickside/ui-kit", "version": "1.5.0", "type": "module", "sideEffects": ["*.css"], "exports": { "./kx-card.css": "./kx-card.css", "./kx-state.css": "./kx-state.css" }, "files": ["kx-card.css", "kx-state.css", "package.json"] } ``` `sideEffects` matters for a CSS-only package: without it a bundler is free to treat an imported stylesheet as dead code and drop it. **The package version must equal the module version.** `wippy publish` validates this and refuses a mismatch, so bump both together. This is also the reason to give a shared package its *own* module rather than nesting it inside a larger one — otherwise every unrelated change to the host module forces a release of the package, and vice versa. #### Publishing ```bash ## validate without publishing wippy publish --dry-run --version 1.5.0 --embed package_fs ## publish wippy publish --create --module-type library --module-visibility public --version 1.5.0 --embed package_fs ``` Declared artifacts are validated as part of publish, so a package.json that fails the format's rules is rejected here rather than in a consumer's build. #### The dev loop Publishing on every edit is not a dev loop. Pack the producer locally and point the consumer's materialize step at that file instead: ```bash ## from the producer module wippy pack /tmp/ui-kit-dev.wapp --embed package_fs ## consumers materialize from the local pack rather than the published one UI_KIT_WAPP=/tmp/ui-kit-dev.wapp make ui-kit MOD=workflows ``` Keep that override as the *only* difference between the dev path and CI — an environment variable that selects the pack file, with everything downstream identical. A dev loop that materializes differently from CI stops predicting CI. #### Wiring it into make and CI Make the materialize step a **prerequisite of the consumer's build**, not a thing a person remembers to run: ```make UI_KIT_WAPP ?= build: @case " $(UI_KIT_CONSUMERS) " in *" $(MOD) "*) $(MAKE) ui-kit MOD=$(MOD);; esac cd $(call fe_dir,$(MOD)) && npm run build ``` CI then needs no artifact-specific step at all: it runs the same `make build`, `UI_KIT_WAPP` is unset, so the fetch-and-materialize path runs against the published version pinned in `build-inputs`. A fresh checkout cannot compile against a stale or missing package, and a contributor who has never heard of artifacts still gets a correct build. ### What you still have to hand-roll `wippy artifacts materialize` is deliberately narrow, so a build that consumes an artifact currently glues four steps together itself. Knowing which four saves rediscovering them: **1. Getting the `.wapp`.** The command takes a *pack file path*, not a module reference, and does not resolve dependencies — so something has to fetch the producer first. The workable pattern is a tiny Wippy project whose only job is to pin and download it: ```yaml ## build-inputs/wippy.lock — a project that exists only to fetch directories: modules: .wippy src: ./src modules: - name: kickside/ui-kit version: 1.5.0 hash: be1eafd5… ``` ```bash ( cd build-inputs && wippy install ) wapp=$(ls build-inputs/.wippy/vendor/kickside/ui-kit-*.wapp | grep -v sha256 | sort | tail -1) ``` Pinning it here rather than in the application lock keeps a build-time input out of the runtime dependency graph. **2. Materializing once per consumer**, into a root the consumer's package manager can see: ```bash wippy artifacts materialize "$wapp" kickside.ui_kit:package_fs --root ./ui/.wippy ``` **3. Wiring the consumer's `package.json`.** Materializing writes files; it does not edit manifests. npm links the package only if the consumer declares *both* the workspace glob and the dependency: ```json { "workspaces": ["./.wippy/npm/@*/*"], "dependencies": { "@kickside/ui-kit": "*" } } ``` The version is `*` because the materialized package carries its own. Script this and make it idempotent — if the wiring is missing, the build fails much later with a bare `ENOENT` on a stylesheet, which reads as a missing file rather than as missing wiring. **4. Running the package manager.** `materialize` does not invoke one, so `npm install` is yours to call, after step 3. Together, in a target that takes the consuming module as a parameter: ```make ui-kit: @set -e; \ ( cd build-inputs && $(WIPPY) install ); \ wapp=$$(ls build-inputs/.wippy/vendor/kickside/ui-kit-*.wapp | grep -v sha256 | sort | tail -1); \ test -n "$$wapp" || { echo "no ui-kit .wapp; is the module published?"; exit 1; }; \ $(WIPPY) artifacts materialize "$$wapp" kickside.ui_kit:package_fs --root $(DIR)/.wippy; \ cd $(DIR) && node ../../scripts/wire-ui-kit.mjs && npm install --no-audit --no-fund ``` Make the whole target a prerequisite of the consumer's build, so a fresh checkout cannot compile against a stale or absent package. ### Out of scope Artifacts intentionally do not introduce a second resolver, package registry, archive format, lock schema, Hub API, or module manifest. Build-only dependency semantics, redistribution policy and host ABI validation are separate concerns and are not solved here. ### Related - [Dependency Management](./dependency-management.md) — resolving modules and local replacements - [Publishing](./publishing.md) — what a published module contains - [The Design Layer](../frontend/design-layer.md) — why a shared frontend vocabulary ships as a package in the first place --- # "Building Components" ## Building Components A **component** is a reusable Wippy module published to the Hub and mounted into a host application. A component can depend on a database, process host, or router without knowing the host's entry IDs. It declares these dependencies through a **requirement interface**, and the host supplies their values. This guide covers the author side: declaring that interface and understanding how values flow into your entries. For the consumer side (lock files, version constraints, `wippy add`/`update`) see [Dependency Management](guides/dependency-management.md). For how a component is structured internally see [Application Architecture](concepts/architecture.md). ### The Three Entry Kinds | Kind | Side | Role | |------|------|------| | `ns.definition` | component | Module metadata; required to publish. | | `ns.requirement` | component | A hole the host must fill, and where to inject the value. | | `ns.dependency` | host | Mounts a component and supplies values for its requirements. | ### ns.definition Each published module must have exactly one definition. The definition can carry module metadata, a README reference, and wiki page references. ```yaml - name: definition kind: ns.definition module: jobs # optional module metadata readme: file://README.md # path to the module's documentation meta: title: Durable Jobs description: Leased job queue with retry and dead-lettering. ``` `module`, `readme`, and `wiki` are definition data; all are optional. `meta` is ordinary entry metadata for management UIs. Release notes are supplied at publish time, not here. ### ns.requirement A requirement is a **named value with a list of injection targets**. The host supplies the value, and the runtime writes it into each target entry at the specified path. ```yaml - name: target_db kind: ns.requirement meta: description: SQL database backing every table in this module. default: app:db targets: - entry: app.jobs.migrations:schema path: .meta.target_db - entry: app.jobs.persist:lifecycle path: .db ``` #### `default`: Mandatory or Optional The `default` field decides whether the host *must* supply a value: - **`default` present with a non-null value** (including an empty string) → the requirement is **optional**. If the host supplies nothing, the default is used. - **`default` absent** → the requirement is **mandatory**. With nothing supplied, linking fails under strict mode (and warns otherwise). An explicitly empty default (default: "") is distinct from an absent or null default. Empty-string means "optional, falls back to nothing"; absent and default: null both mean "the host must provide this." Use a non-null default for infrastructure that has a sane in-app convention (app:db, app:processes); omit it for values only the host can know. #### `targets`: Injection Locations Each target is an `{entry, path}` pair: - **`entry`** — the entry the value is injected into. A bare name (`schema`) resolves within the requirement's own namespace; a fully-qualified id (`app.jobs.migrations:schema`) targets that entry exactly, across namespaces. - **`path`** — a dot path into the target entry, e.g. `.meta.target_db`, `.host`, `.database.url`. The leading dot is conventional. A requirement must declare at least one target. Append instead of set with the `+=` suffix on the path — useful when several requirements contribute to one list (e.g. middleware): ```yaml targets: - entry: app.api:router path: .middleware+= # appends the value to the list at .middleware ``` #### One Requirement, Multiple Targets Group targets that need the same value under one requirement. For example, `target_db` can supply every migration's `.meta.target_db` and persistence library's `.db`; `process_host` can supply each supervised service's `.host`; and `api_router` can supply each endpoint's `.meta.router`: ```yaml - name: process_host kind: ns.requirement default: app:processes targets: - { entry: app.jobs.service:worker.service, path: .host } - { entry: app.jobs.service:sweeper.service, path: .host } ``` The host supplies one value, and the runtime writes it to every declared target. The requirement entry contains this wiring directly. ### Consuming a Component The host mounts a component with `ns.dependency` and fills its requirements through `parameters`: ```yaml version: "1.0" namespace: app entries: - name: dep.jobs kind: ns.dependency component: acme/jobs version: "^1.0.0" parameters: - name: target_db value: app:db - name: process_host value: app:processes - name: api_router value: app:api ``` Each `parameter.name` matches a requirement; its `value` is what gets injected into that requirement's targets. Requirements with a default may be omitted; mandatory ones must be supplied. #### Parameter Name Matching How a parameter name binds to a requirement: - **Bare name** (`target_db`) matches a requirement of that name belonging to the component being mounted. It does not cross into a different module's requirements. - **Qualified name** (`acme.jobs:target_db`) matches that requirement id exactly. Use this to disambiguate when wiring transitive dependencies. If two dependencies supply **different** values for the same requirement, that is a conflict and is reported (identical values are fine). ### When Values Resolve Injection happens at the **Link stage** of the build pipeline — at publish, during dependency expansion, and at boot — not at runtime. The stage: 1. Collects every `ns.requirement` and every `ns.dependency` with its parameters. 2. For each requirement, resolves a value: a matching parameter wins; otherwise the default; otherwise (no default) it is unresolved. 3. Writes the resolved value into each target entry at its path (set, or append for `+=`). Under **strict requirements** an unresolved mandatory requirement fails the build; otherwise it logs a warning and proceeds. By the time entries reach the runtime, every filled requirement has already been baked into its targets. ### Verify Integration with a Mount Test Unit tests do not verify the assembled module's registry relationships. Add a packaging or mount test against the requirement-injected registry to verify that: - every supervised `service` points at a process entry that exists, - every spawned or scheduled id resolves to a real entry, - every `env.variable`'s storage is registered. This catches unresolved relationships such as a supervisor referencing an unregistered worker or a test fixture using a harness-only storage ID. See [Supervision](guides/supervision.md) and the [Testing](framework/testing.md) framework. ### See Also - [Application Architecture](concepts/architecture.md) — how a component is structured internally - [Dependency Management](guides/dependency-management.md) — lock files, versions, the consumer workflow - [Publishing Modules](guides/publishing.md) — putting a component on the hub - [Entry Kinds Guide](guides/entry-kinds.md) — `ns.definition`, `ns.requirement`, `ns.dependency` reference --- # "Entry Kinds Reference" ## Entry Kinds Reference This page summarizes the available entry kinds and links to their detailed module and system references. The YAML and Lua blocks are reference fragments, not one application. Registry IDs, credentials, data objects, and helpers such as `get_users` or `delete_user` are illustrative; use the linked module pages for complete return and error contracts. > Entries reference one another using `namespace:name`. The registry uses these references to resolve dependencies and initialization order. ### See Also - [Registry](concepts/registry.md) — How entries are stored and resolved - [Configuration](guides/configuration.md) — YAML configuration format ### Lua Runtime | Kind | Description | |------|-------------| | `function.lua` | Lua function entry point | | `process.lua` | Long-running Lua process | | `workflow.lua` | Temporal workflow (deterministic) | | `library.lua` | Shared Lua library | | `module.lua` | Lua module surface | | `function.lua.bc` | Precompiled function bytecode | | `library.lua.bc` | Precompiled library bytecode | | `process.lua.bc` | Precompiled process bytecode | | `workflow.lua.bc` | Precompiled workflow bytecode | ```yaml - name: handler kind: function.lua source: file://handler.lua method: main modules: - http - json imports: utils: app.lib:helpers # Import another entry as module ``` Use imports to reference other Lua entries. They become available via require("alias_name") in your code. ### HTTP Services | Kind | Description | |------|-------------| | `http.service` | HTTP server (binds port) | | `http.router` | Route prefix and middleware | | `http.endpoint` | HTTP endpoint (method + path) | | `http.static` | Static file serving | ```yaml ## HTTP server - name: gateway kind: http.service addr: ":8080" lifecycle: auto_start: true ## Router with middleware - name: api kind: http.router meta: server: gateway prefix: /api middleware: - cors - ratelimit ## Endpoint - name: users_list kind: http.endpoint meta: router: app:api method: GET path: /users func: list_handler ``` **Lua API:** See [HTTP Module](lua/http/http.md) ```lua local http = require("http") local req = http.request() local resp = http.response() resp:set_status(200) resp:write_json({users = get_users()}) ``` ### Databases | Kind | Description | |------|-------------| | `db.sql.sqlite` | SQLite database | | `db.sql.postgres` | PostgreSQL database | | `db.sql.mysql` | MySQL database | | `db.cdc.postgres` | Postgres Change Data Capture source (see [CDC](system/cdc.md)) | | `db.cdc.sqlite` | SQLite Change Data Capture source (see [CDC](system/cdc.md)) | #### SQLite ```yaml - name: database kind: db.sql.sqlite file: "./data/app.db" lifecycle: auto_start: true ## In-memory for testing - name: testdb kind: db.sql.sqlite file: ":memory:" ``` #### PostgreSQL ```yaml - name: database kind: db.sql.postgres host: localhost port: 5432 database: dbname username: user password: pass options: sslmode: disable pool: max_open: 25 max_idle: 5 max_lifetime: "30m" lifecycle: auto_start: true ``` #### MySQL ```yaml - name: database kind: db.sql.mysql host: localhost port: 3306 database: dbname username: user password: pass options: parseTime: "true" lifecycle: auto_start: true ``` See [Database](system/database.md) for `${env:NAME}` secret references, TLS options, and connection pool tuning. When an env-backed value behind a database entry changes, the pool swaps live — active borrows finish against the old connection settings. **Lua API:** See [SQL Module](lua/storage/sql.md) ```lua local sql = require("sql") local db, err = sql.get("app:database") local rows, err = db:query("SELECT * FROM users WHERE id = ?", {user_id}) db:execute("INSERT INTO logs (msg) VALUES (?)", {message}) ``` ### Key-Value Stores | Kind | Description | |------|-------------| | `store.memory` | In-memory key-value store | | `store.sql` | SQL-backed key-value store | | `store.kv.raft` | Cluster-replicated, strongly-consistent KV (shared Raft) | | `store.kv.crdt` | Cluster-replicated, eventually-consistent KV (gossip/CRDT) | ```yaml ## Memory store - name: cache kind: store.memory lifecycle: auto_start: true ## SQL-backed store - name: persistent_store kind: store.sql database: app:database table_name: kv_store lifecycle: auto_start: true ## Cluster-replicated store (requires clustering) - name: deployments kind: store.kv.raft namespace: deploy ``` The `store.kv.*` kinds need [clustering](guides/cluster.md) enabled. See [Store](system/store.md#cluster-kv-stores) for the consistency tradeoffs. **Lua API:** See [Store Module](lua/storage/store.md) ```lua local store = require("store") local s, err = store.get("app:cache") s:set("user:123", user_data, 3600) -- TTL in seconds local data = s:get("user:123") ``` ### Queues | Kind | Description | |------|-------------| | `queue.driver.memory` | In-memory queue driver | | `queue.driver.amqp` | AMQP (RabbitMQ) driver | | `queue.driver.sqs` | AWS SQS driver | | `queue.queue` | Queue declaration | | `queue.consumer` | Queue consumer | ```yaml ## Driver - name: queue_driver kind: queue.driver.memory lifecycle: auto_start: true ## Queue - name: jobs kind: queue.queue driver: queue_driver ## Consumer - name: job_consumer kind: queue.consumer queue: app:jobs func: job_handler concurrency: 4 prefetch: 10 lifecycle: auto_start: true ``` **Lua API:** See [Queue Module](lua/storage/queue.md) ```lua local queue = require("queue") -- Publish a message queue.publish("app:jobs", {task = "process", id = 123}) -- In a consumer handler: the message body is the handler's argument local function main(data) -- access delivery metadata via the current message local msg = queue.message() local id = msg:id() local priority = msg:header("priority") msg:ack() end ``` The consumer's func is invoked once per message with the message body as its argument. Use queue.message() inside the handler for the delivery's id(), header()/headers(), and ack()/nack(). ### Process Management | Kind | Description | |------|-------------| | `process.host` | Process execution host | | `process.service` | Supervised process (wraps process.lua) | | `terminal.host` | Terminal/CLI host | | `pg.scope` | Process-group scope (see [Process Groups](system/process-groups.md)) | ```yaml ## Process host (where processes run) - name: processes kind: process.host host: workers: 32 # Worker goroutines (default: NumCPU) queue_size: 1024 # Global queue capacity local_queue_size: 256 # Per-worker queue lifecycle: auto_start: true ## Process definition - name: worker_process kind: process.lua source: file://worker.lua method: main ## Supervised process service - name: worker kind: process.service process: app:worker_process host: app:processes input: ["arg1", "arg2"] lifecycle: auto_start: true restart: max_attempts: 10 - name: terminal kind: terminal.host lifecycle: auto_start: true ``` Use process.service when you need a process to run as a supervised service with automatic restart. The process field references a process.lua entry. Updating a live `process.host` entry rescales `host.workers` in place — running processes, PIDs, and queues are preserved. `host.queue_size`, `host.local_queue_size`, and `lifecycle` are fixed at construction: a live update changing them is rejected, as is resizing workers on a host whose workers are affinity-managed. #### Process security `process.lua` and `process.lua.bc` entries accept a top-level `security:` block. It is part of the entry, so it applies to every spawn of that process, on both `process.host` and `terminal.host`: ```yaml - name: worker_process kind: process.lua source: file://worker.lua method: main security: actor: id: system.worker meta: tenant: acme policies: - app.security:worker_policy groups: - app.security:background_jobs ``` | Field | Description | |-------|-------------| | `actor.id` | Actor identity the process runs as; replaces the inherited actor | | `actor.meta` | Actor attributes policies evaluate | | `policies` | Registry IDs (`namespace:name`) of policies merged into the scope | | `groups` | Registry IDs of policy groups whose policies are merged into the scope | Resolution happens as the process starts and is atomic: if any listed policy or group cannot be resolved, the spawn fails and no partial context is installed. Omitting `actor` inherits the spawner's actor; omitting both `policies` and `groups` inherits the spawner's scope. `function.lua`, `function.lua.bc`, `process.lua`, and `process.lua.bc` all accept the block. A command entry can additionally declare `meta.command.security`, which applies only when the entry is launched as a CLI command — see [Command security](guides/cli.md#command-security). It does not affect ordinary spawns. See [Security](system/security.md). ### Temporal (Workflows) | Kind | Description | |------|-------------| | `temporal.client` | Temporal client connection | | `temporal.worker` | Temporal worker | ```yaml - name: temporal_client kind: temporal.client address: "localhost:7233" namespace: "default" auth: type: none # none, api_key, mtls lifecycle: auto_start: true - name: temporal_worker kind: temporal.worker client: temporal_client task_queue: "main-queue" lifecycle: auto_start: true ``` ### Cloud Storage | Kind | Description | |------|-------------| | `config.aws` | AWS configuration | | `cloudstorage.s3` | S3 bucket access | ```yaml - name: aws kind: config.aws region: "us-east-1" access_key_id: ${env:AWS_ACCESS_KEY_ID} secret_access_key: ${env:AWS_SECRET_ACCESS_KEY} - name: uploads kind: cloudstorage.s3 config: app:aws bucket: "my-uploads" endpoint: "" # Optional, for S3-compatible services ``` **Lua API:** See [Cloud Storage Module](lua/storage/cloud.md) ```lua local cloudstorage = require("cloudstorage") local storage, err = cloudstorage.get("app:uploads") storage:upload_object("files/doc.pdf", file_content) local url = storage:presigned_get_url("files/doc.pdf", {expiration = 3600}) -- seconds, default 3600 ``` Use endpoint to connect to S3-compatible services like MinIO or DigitalOcean Spaces. ### File Systems | Kind | Description | |------|-------------| | `fs.directory` | Directory access | | `fs.embed` | Read-only embedded filesystem | ```yaml - name: data_dir kind: fs.directory directory: "./data" auto_init: true # Create if not exists mode: "0755" # Permissions ``` **Lua API:** See [Filesystem Module](lua/storage/filesystem.md) ```lua local fs = require("fs") local filesystem, err = fs.get("app:data_dir") local file = filesystem:open("output.txt", "w") file:write("Hello, World!") file:close() ``` ### Environment | Kind | Description | |------|-------------| | `env.storage.memory` | In-memory env storage | | `env.storage.file` | File-based env storage | | `env.storage.os` | OS environment | | `env.storage.static` | Read-only static key-value storage | | `env.storage.router` | Env router (multiple storages) | | `env.variable` | Environment variable | ```yaml - name: os_env kind: env.storage.os - name: file_env kind: env.storage.file file_path: ".env" auto_create: true - name: defaults kind: env.storage.static values: PUBLIC_API_HOST: "https://api.example.com" APP_ENV: "production" - name: app_env kind: env.storage.router storages: - app:os_env - app:file_env - app:defaults ``` **Lua API:** See [Env Module](lua/system/env.md) ```lua local env = require("env") local api_key = env.get("API_KEY") env.set("CACHE_TTL", "3600") ``` The router tries storages in order. First match wins for reads; writes go to the first storage in the list. ### Templates | Kind | Description | |------|-------------| | `template.jet` | Individual Jet template | | `template.set` | Template set configuration | ```yaml ## Template set with engine configuration - name: templates kind: template.set engine: development_mode: false extensions: - ".jet" - ".html.jet" ## Individual template - name: email_template kind: template.jet source: file://templates/email.jet set: app:templates ``` **Lua API:** See [Template Module](lua/text/template.md) ```lua local templates = require("templates") local set, err = templates.get("app:templates") local html = set:render("email", { user = "Alice", message = "Welcome!" }) ``` ### Security | Kind | Description | |------|-------------| | `security.policy` | Security policy with conditions | | `security.policy.expr` | Expression-based policy | | `security.token_store` | Token storage | ```yaml ## Condition-based policy - name: admin_policy kind: security.policy policy: actions: "*" resources: "*" effect: allow conditions: - field: "actor.meta.role" operator: eq value: "admin" ## Expression-based policy - name: owner_policy kind: security.policy.expr policy: actions: "*" resources: "*" effect: allow expression: 'actor.id == meta.owner_id || actor.meta.role == "admin"' groups: - operators ``` Policy groups are formed by the policies themselves: a policy lists the group IDs it belongs to under `groups:`, and a group is the set of policies naming it. There is no separate group entry kind. Group IDs are registry IDs — a bare name resolves in the declaring policy's namespace, so `operators` above becomes `app.security:operators` when declared in namespace `app.security`. Entries reference groups by their full `namespace:name`. **Lua API:** See [Security Module](lua/security/security.md) ```lua local security = require("security") -- Check permission before action if security.can("delete", "users", {user_id = id}) then delete_user(id) end -- Get current actor local actor = security.actor() ``` Every policy in scope is evaluated. A deny from any matching policy wins over every allow; with no deny, a matching allow grants access. Order does not matter. ### Contracts (Dependency Injection) | Kind | Description | |------|-------------| | `contract.definition` | Interface with method specifications | | `contract.binding` | Maps contract methods to function implementations | ```yaml ## Define the contract interface - name: greeter kind: contract.definition methods: - name: greet description: Returns a greeting message - name: greet_with_name description: Returns a personalized greeting input_schemas: - format: "application/schema+json" definition: {"type": "string"} output_schemas: - format: "application/schema+json" definition: {"type": "string"} ## Implementation functions - name: greeter_greet kind: function.lua source: file://greeter_greet.lua method: main - name: greeter_greet_name kind: function.lua source: file://greeter_greet_name.lua method: main ## Bind contract methods to implementations - name: greeter_impl kind: contract.binding contracts: - contract: app:greeter default: true methods: greet: app:greeter_greet greet_with_name: app:greeter_greet_name ``` Usage from Lua: ```lua local contract = require("contract") -- Open binding by ID local greeter, err = contract.open("app:greeter_impl") -- Call methods local result = greeter:greet() local personalized = greeter:greet_with_name("Alice") -- Check if instance implements contract local is_greeter = contract.is(greeter, "app:greeter") ``` **Lua API:** See [Contract Module](lua/core/contract.md) Mark one binding as default: true to use it when opening a contract without specifying a binding ID. A contract may have only one default binding. ### Execution | Kind | Description | |------|-------------| | `exec.native` | Native command execution | | `exec.docker` | Docker container execution | ```yaml - name: native_exec kind: exec.native default_work_dir: "/app" command_whitelist: - "ls" - "cat" - name: docker_exec kind: exec.docker image: "python:3.11-slim" default_work_dir: "/workspace" auto_remove: true memory_limit: 536870912 # 512MB command_whitelist: - "python" ``` ### WASM Runtime | Kind | Description | |------|-------------| | `function.wat` | WebAssembly function (WAT text format) | | `function.wasm` | WebAssembly function (binary) | | `process.wasm` | WebAssembly process | ```yaml ## WAT text is inline source - name: sum_wat kind: function.wat source: file://sum.wat method: sum transport: payload # or wasi-http ## Binary WASM is loaded from a filesystem entry and verified by hash - name: sum kind: function.wasm fs: app:modules path: sum.wasm hash: sha256:2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae method: sum transport: payload ``` `function.wasm` and `process.wasm` take `fs`, `path`, and `hash` — there is no `source` field on a binary entry; `source` belongs to `function.wat` only. `hash` is required and must be `sha256:`; the module is rejected if the bytes do not match. See [WASM Overview](wasm/overview.md). ### Networks | Kind | Description | |------|-------------| | `network` | Base network overlay | | `network.socks5` | SOCKS5 proxy overlay | | `network.i2p` | I2P network overlay | | `network.tailscale` | Tailscale overlay | Referenced by `http.service` via `network:`, by `funcs`/`process` via the `network` option, and by `http_client` via the `overlay_network` option. See [Network](system/network.md). ### Registry Primitives | Kind | Description | |------|-------------| | `registry.entry` | Plain data entry with no service behind it (app-specific config) | | `ns.definition` | Namespace definition | | `ns.requirement` | Namespace requirement declaration | | `ns.dependency` | Namespace dependency | The `ns.*` kinds are authored like any other entry: a component declares `ns.definition` and `ns.requirement`, and a host declares `ns.dependency`. See [Building Components](guides/components.md). ### Lifecycle Configuration Supervisor-managed service entries expose lifecycle configuration. The block below belongs inside a service entry that supports it: ```yaml lifecycle: auto_start: true # Start automatically start_timeout: 10s # Max startup time stop_timeout: 10s # Max shutdown time stable_threshold: 5s # Uninterrupted run time before retry accounting resets requires: - app:database restart: # Retry policy initial_delay: 1s max_delay: 90s backoff_factor: 2.0 max_attempts: 0 # 0 = infinite ``` Use depends_on to ensure entries start in the correct order. The supervisor starts a dependent entry only after each of its dependencies has completed its own start. ### Entry Reference Format Entries are referenced using `namespace:name` format: ```yaml ## Definition namespace: app.users entries: - name: handler kind: function.lua ## Reference from another entry func: app.users:handler ``` ### Overriding Entries Any entry's fields — including its `kind` — can be overridden at launch without editing the source YAML, using the `override:` config section or the `-o` CLI flag. Keys use `namespace:entry:path` format: ```yaml override: app:gateway:addr: ":9090" # data field (a bare path targets data.*) app:worker:meta.priority: high # meta field app:db:kind: db.sql.postgres # the entry's typed kind app:db:data.kind: custom # a payload field literally named "kind" ``` | Path | Targets | |------|---------| | `kind` | The entry's typed kind (must be a non-empty string) | | `data.` or bare `` | A field in the entry's data payload | | `meta.` | A field in the entry's metadata | The same overrides apply from the CLI: ```bash wippy run -o app:db:kind=db.sql.postgres -o app:gateway:addr=:9090 ``` CLI (`-o`) values coerce by shape (`true`/`false` to bool, numbers to numbers, otherwise string); `override:` section values keep their YAML type. To override global [configuration](guides/configuration.md) sections instead of entries, use `--set`. --- # "Observability" ## Observability Wippy exposes application and runtime behavior through logging, metrics, distributed tracing, and runtime statistics. ### Overview Three observability areas are configured at boot: | Pillar | Backend | Configuration | |--------|---------|---------------| | Logging | Zap (JSON structured) | `logger` and `logmanager` | | Metrics | Prometheus | `prometheus` | | Tracing | OpenTelemetry | `otel` | #### Logger Encoding ```yaml logger: encoding: json # json or console ``` Level and output are controlled by CLI flags (`-v`, `-c`, `-s`); only `encoding` is read from YAML. #### Log Manager The log manager controls log propagation and event streaming: ```yaml logmanager: propagate_downstream: true # Propagate to child components stream_to_events: false # Forward logs to event bus min_level: 0 # -1=debug, 0=info, 1=warn, 2=error (wippy run sets 0, or -1 with -v) ``` When `stream_to_events` is enabled, log entries become events that processes can subscribe to via the event bus. The embedded log-manager default is `-1`, but `wippy run` applies its CLI logging choice at startup: info (`0`) by default and debug (`-1`) with `-v` or `--very-verbose`. #### Automatic Context Logs emitted from Lua via the [logger module](lua/system/logger.md) automatically include: - `pid` - Current process PID - `location` - Entry ID and caller line (e.g., `app.api:handler:45`) ### Prometheus Metrics ```yaml prometheus: enabled: true address: "localhost:9090" ``` Metrics are exposed at `/metrics` on the configured address; the same listener serves `/livez`. `max_cardinality` (default 1024) caps the number of live label sets per exporter; the least recently updated series are evicted beyond it. #### Scrape Configuration ```yaml ## prometheus.yml scrape_configs: - job_name: 'wippy' static_configs: - targets: ['localhost:9090'] scrape_interval: 15s ``` For the Lua metrics API, see [Metrics Module](lua/system/metrics.md). ### OpenTelemetry OpenTelemetry (OTEL) provides distributed tracing and optional metrics export. #### Basic Configuration ```yaml otel: enabled: true endpoint: "localhost:4318" protocol: http/protobuf # grpc or http/protobuf service_name: my-app service_version: "1.0.0" insecure: true # Use plaintext for a local collector sample_rate: 1.0 # 0.0 to 1.0 traces_enabled: true metrics_enabled: false propagators: - tracecontext - baggage ``` #### Trace Sources All trace sources are on by default once `otel.enabled` is true; each can be disabled individually: ```yaml otel: enabled: true endpoint: "localhost:4318" service_name: my-app # HTTP request tracing http: enabled: true extract_headers: true # Read incoming trace context inject_headers: true # Write trace context to the HTTP response # Process lifecycle tracing process: enabled: true trace_lifecycle: true # Trace spawn/exit events # Queue message tracing queue: enabled: true # Function call tracing interceptor: enabled: true ``` When OTEL is enabled, HTTP tracing and propagation, process tracing and lifecycle spans, function interception, queue tracing, and trace export are enabled by default. Temporal tracing and metric export default to disabled. The pinned runtime registers the function interceptor at order 100; although an `interceptor.order` value can be decoded from configuration, it does not change that registration order. #### Temporal Workflows Enable tracing for Temporal workflows: ```yaml otel: enabled: true endpoint: "localhost:4318" service_name: my-app temporal: enabled: true ``` When enabled, the Temporal SDK's tracing interceptor is registered for both client and worker operations. Traced operations include: - Workflow starts and completions - Activity executions - Child workflow calls - Signal and query handling #### What Gets Traced | Component | Span Name | Attributes | |-----------|-----------|------------| | HTTP requests | `{METHOD} {route}` | http.method, http.url, http.host, http.route | | Function calls | Function ID | process.pid, frame.id | | Process lifecycle | `{source}.started/terminated` | process.pid | | Queue messages | `{queue}.publish` | messaging.operation, messaging.destination.name | | Temporal workflows | Workflow/Activity name | workflow.id, run.id | #### Context Propagation The configured integrations propagate trace context through: - **HTTP → Function**: W3C Trace Context headers - **Function → Function**: Frame context inheritance - **Process → Process**: Spawn context - **Queue publish → consume**: Message headers #### Environment Variables OTEL can be configured via environment: | Variable | Description | |----------|-------------| | `OTEL_SDK_DISABLED` | Set to `true` to disable OTEL | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Collector endpoint; an `http://` or `https://` scheme is removed before exporter setup | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` or `http/protobuf` | | `OTEL_EXPORTER_OTLP_INSECURE` | Set to `true` to use a plaintext collector connection | | `OTEL_SERVICE_NAME` | Service name | | `OTEL_SERVICE_VERSION` | Service version | | `OTEL_TRACES_SAMPLER` | `always_on`, `always_off`, `traceidratio`, or `parentbased_traceidratio` | | `OTEL_TRACES_SAMPLER_ARG` | Sample rate (0.0-1.0) | | `OTEL_TRACES_SAMPLER` | `always_on`, `always_off`, `traceidratio`, or `parentbased_traceidratio` (ratio from `OTEL_TRACES_SAMPLER_ARG`) | | `OTEL_EXPORTER_OTLP_INSECURE` | Set to `true` to allow non-TLS connections | | `OTEL_PROPAGATORS` | Propagator list | ### Runtime Statistics The `system` module provides internal runtime statistics: ```lua local system = require("system") -- Memory statistics local mem, mem_err = system.memory.stats() -- mem.alloc, mem.heap_alloc, mem.heap_objects, etc. -- Goroutine count local count, count_err = system.runtime.goroutines() -- Supervisor states local states, states_err = system.supervisor.states() ``` These functions return `value, error`. They require the `system.read` permission in the current security scope. ### See Also - [Logger Module](lua/system/logger.md) — Lua logging API - [Metrics Module](lua/system/metrics.md) — Lua metrics API - [System Module](lua/system/system.md) — Runtime statistics --- # "Queue Consumers" ## Queue Consumers Queue consumers deliver messages from a queue to function handlers through a configurable worker pool. ### Overview ```mermaid flowchart LR subgraph Consumer QD[Queue Driver] --> DC[Delivery Channel
prefetch=10] DC --> WP[Worker Pool
concurrency] WP --> FH[Function Handler] FH --> AN[Ack/Nack] end ``` ### Configuration | Option | Default | Max | Description | |--------|---------|-----|-------------| | `queue` | Required | - | Queue registry ID | | `func` | Required | - | Handler function registry ID | | `concurrency` | 1 | 1000 | Worker count | | `prefetch` | 10 | 10000 | Message buffer size | | `auto_ack` | false | - | Driver-level auto-ack (AMQP `Consume` autoAck; ignored by the memory driver) | | `driver_options` | `{}` | - | Driver-specific consumer options | ### Entry Definition ```yaml - name: order_consumer kind: queue.consumer queue: app:orders func: app:process_order concurrency: 5 prefetch: 20 lifecycle: auto_start: true requires: - app:orders ``` ### Handler Function The handler function receives the body after the queue's codec decodes it. Use `queue.message()` to access the current delivery and its metadata: ```lua -- process_order.lua local queue = require("queue") local logger = require("logger") local function main(order) local msg, msg_err = queue.message() if msg_err then return nil, msg_err end logger:info("processing order", { message_id = msg:id(), order_id = order.id }) return {processed = true, order_id = order.id} end return {main = main} ``` ```yaml - name: process_order kind: function.lua source: file://process_order.lua method: main modules: - queue - logger ``` ### Acknowledgment Unless the handler explicitly settles the delivery, the consumer uses the function invocation result: | Handler outcome | Action | Effect | |-----------------|--------|--------| | Completes without an invocation error | Ack | Message removed from queue | | Returns or raises an invocation error | Nack | Redelivery is driver-dependent | Ordinary return values, including `false`, do not select acknowledgment behavior. Call `msg:ack()` or `msg:nack()` to settle explicitly. Settlement is single-shot: the first settlement wins. With AMQP `auto_ack: true`, the broker acknowledges on delivery, so a later handler failure cannot cause broker redelivery. The handler can settle the message itself with `queue.message()` and `msg:ack()` / `msg:nack()`; the consumer then skips its own ack/nack. ### Worker Pool - Workers run as concurrent goroutines. - Each worker processes one message at a time. - Workers pull from a shared delivery channel. The next idle worker receives the next message, without guaranteed ordering or rotation across workers. - The prefetch buffer allows the driver to deliver messages ahead of processing. #### Example ``` concurrency: 3 prefetch: 10 Flow: 1. Driver delivers up to 10 messages to buffer 2. 3 workers pull from buffer concurrently 3. As workers finish, buffer refills 4. Backpressure when all workers busy and buffer full ``` ### Graceful Shutdown During shutdown, the consumer: 1. Stops accepting new deliveries 2. Cancels worker contexts 3. Waits for in-flight handlers, up to the stop timeout 4. Returns a timeout error if workers do not finish ### Queue Declaration ```yaml ## Queue driver (memory for dev/test) - name: queue_driver kind: queue.driver.memory lifecycle: auto_start: true ## Queue definition - name: orders kind: queue.queue driver: app:queue_driver queue_name: orders # Override name (default: entry name) codec: json/plain # Payload codec (optional; json/plain is the default) dead_letter: # Accepted configuration; not enforced by built-in drivers queue: app:dlq max_attempts: 5 driver_options: memory: max_length: 10000 # Memory driver: bounded queue size ``` | Field | Description | |-------|-------------| | `queue_name` | Override queue name (default: entry ID name) | | `codec` | Payload codec name | | `dead_letter.queue` | Registry ID accepted for a dead-letter queue; not enforced by built-in drivers | | `dead_letter.max_attempts` | Attempt count accepted in configuration; not enforced by built-in drivers | | `driver_options` | Driver-specific settings keyed by driver name | No built-in driver currently counts attempts or routes messages from the `dead_letter` block. The runtime does not translate that block into AMQP queue arguments, and ordinary AMQP consumer failures request requeue. Broker-side dead-lettering must therefore be configured and triggered outside this block. The memory driver does not route to a DLQ. ### Memory Driver The built-in in-memory driver is intended for development and testing: - Its kind is `queue.driver.memory`. - Messages are stored in memory. - Nack attempts to re-enqueue a cloned message at the end of the queue; that attempt can fail when the bounded queue is full. - Messages do not persist across restarts. ### See Also - [Message Queue](lua/storage/queue.md) — Queue module reference - [Queue Configuration](system/queue.md) — Queue drivers and entry definitions - [Supervision](guides/supervision.md) — Consumer lifecycle - [Process Management](lua/core/process.md) — Process spawning and communication --- # "Supervision" ## Supervision The supervisor manages service startup, dependency order, restarts, and graceful shutdown. Services with `auto_start: true` start when the application boots. ### Lifecycle Configuration Services register with the supervisor using a `lifecycle` block. For processes, use `process.service` to wrap a process definition: ```yaml ## Process definition (the code) - name: worker_process kind: process.lua source: file://worker.lua method: main ## Supervised service (wraps the process with lifecycle management) - name: worker kind: process.service process: app:worker_process host: app:processes lifecycle: auto_start: true startup: required start_timeout: 30s stop_timeout: 10s stable_threshold: 5s requires: - app:database restart: initial_delay: 2s max_delay: 60s max_attempts: 10 ``` `host` must reference a configured process host. The `requires` entry must resolve either to another supervised service or, through registry dependency extraction, to a supervised service that owns the referenced resource. | Field | Default | Description | |-------|---------|-------------| | `auto_start` | `false` | Start automatically when supervisor starts | | `startup` | `required` | Startup policy for an auto-start root: `required` blocks boot on failure; `optional` may fail and keep retrying without blocking independent branches | | `start_timeout` | `10s` | Maximum time allowed for startup | | `stop_timeout` | `10s` | Maximum time for graceful shutdown | | `stable_threshold` | `5s` | Runtime after which a later failure resets the retry counter | | `requires` | `[]` | Services that must be running first (legacy alias: `depends_on`) | | `startup` | `required` | `required` reports a failed or blocked auto-start as a transaction error; `optional` lets the service keep retrying in the background without failing the batch | ### Dependency Resolution The supervisor resolves dependencies from two sources: 1. **Explicit dependencies** declared in `requires` (or the legacy `depends_on`) 2. **Registry-extracted dependencies** from entry references (e.g., `database: app:db` in your config) ```mermaid graph LR A[HTTP Server] --> B[Router] B --> C[Handler Function] C --> D[Database] C --> E[Cache] ``` Dependencies start before their dependents. If Service C depends on A and B, both dependencies must reach the `Running` state before C starts. You do not need to repeat an infrastructure reference in requires when registry dependency extraction can trace that reference to a supervised service. Use requires for lifecycle dependencies that are not already expressed by entry references. ### Restart Policy When a service fails, the supervisor retries according to its `restart` block: ```yaml lifecycle: restart: initial_delay: 1s # First retry wait max_delay: 90s # Accepted backoff cap; see current behavior below backoff_factor: 2.0 # Accepted multiplier; see current behavior below jitter: 0.1 # ±10% randomization max_attempts: 0 # 0 = infinite retries ``` In runtime v0.3.32a, the supervisor constructs a new backoff calculator for each retry and takes only its first interval. Each retry therefore waits `initial_delay` with the configured jitter (0.9s–1.1s for the values above). `backoff_factor` and `max_delay` are accepted configuration fields but do not change this schedule in the pinned runtime. `max_attempts` counts the initial failed start. A value of `1` permits no retry, and `10` permits at most nine follow-up starts. A value of `0` allows unlimited attempts. When a service runs longer than `stable_threshold`, its retry counter resets, so later failures start from the initial retry delay. #### Terminal Errors These errors stop retry attempts: - Context cancellation - Explicit termination request - Errors marked as non-retryable ### Security Context Services can run with a specific security identity: ```yaml ## Process definition - name: admin_worker_process kind: process.lua source: file://admin_worker.lua method: main ## Supervised service with security context - name: admin_worker kind: process.service process: app:admin_worker_process host: app:processes lifecycle: auto_start: true security: actor: id: "service:admin-worker" meta: role: admin groups: - app:admin_policies policies: - app:data_access ``` The security context defines: | Field | Description | |-------|-------------| | `actor.id` | Identity string for this service | | `actor.meta` | Key-value metadata (role, permissions, etc.) | | `groups` | Policy groups to apply | | `policies` | Individual policies to apply | Code running in the service inherits this security context. The `security` module can use it for permission checks: ```lua local security = require("security") if security.can("delete", "users") then -- allowed end ``` When no security block is configured, the supervisor adds no service-specific actor or policy scope; any security values already present in the parent context remain inherited. In strict mode (default), a check with an incomplete resulting security context is denied. Configure a complete service security context for services that need authorization. ### Re-registration and Replacement A registry change can re-register an ID that already has a running controller. If the registration carries the same service instance, nothing is disturbed. If it carries a **different** instance — the manager rebuilt the service because its configuration changed — the supervisor retires the existing controller and adopts the replacement. Retirement covers more than the one service. A running dependent captured the superseded instance, so it cannot keep running against a service that is being replaced underneath it; the retirement closure is the replaced service plus every running service that depends on it, stopped in dependency order (dependents first). Services already stopped are not stopped a second time — a manager that stops its own instance before re-registering does not see a redundant `Stop`. The handover is transactional: 1. The plan is computed without touching anything, so a planning failure leaves the running set intact. 2. The stop batch runs. **If any stop fails, the handover is rejected**: the services the batch already stopped are brought back up and the error is reported. A service that could not be brought back is named in that error. The supervisor ends up owning the same running set it had before the commit, never a half-retired one. 3. Only after the batch succeeds are the retired controllers dropped and canceled, freeing the superseded service instances. 4. The replacement is created and started through the same dependency-aware sequencer as any other start, and the dependents that were stopped for the handover come back up against the adopted instance. A service that was running before the replacement is restarted afterwards even when the new registration sets `auto_start: false` — replacing an active service is an update, not an implicit stop. Restarting a stopped dependent is governed by its own restart policy and does not gate the commit. ### Service States ```mermaid stateDiagram-v2 [*] --> Unknown Unknown --> Starting Starting --> Running Running --> Stopping Stopping --> Stopped Stopping --> Failed : timeout/cancel Stopped --> [*] Running --> Failed Starting --> Failed Failed --> Starting : retry Running --> Exited Starting --> Exited Exited --> [*] ``` The supervisor transitions services through these states: | State | Description | |-------|-------------| | `Unknown` | Registered but not started | | `Starting` | Startup in progress | | `Running` | Operating normally | | `Stopping` | Graceful shutdown in progress | | `Stopped` | Stop operation completed; service-reported stop details may still contain an error | | `Exited` | Terminated by explicit request or a non-retryable/terminal error | | `Failed` | Error occurred, may retry | ### Startup and Shutdown Order **Startup:** Dependencies start before dependents. Services at the same dependency level can start in parallel. **Shutdown:** Dependents stop before dependencies, allowing dependent services to finish first. ``` Startup: database → cache → handler → http_server Shutdown: http_server → handler → cache → database ``` On SIGINT or SIGTERM the runtime begins a graceful shutdown and the whole sequence runs under a single budget, `shutdown.timeout` in the runtime config (default 30s). That budget is a fresh deadline that does not inherit the interrupted context, so a Ctrl-C does not cut component shutdown short; per-service `stop_timeout` still bounds each individual stop within it. A second signal skips the sequence and exits immediately. ```yaml ## .wippy.yaml shutdown: timeout: 60s ``` ### See Also - [Process Model](concepts/process-model.md) — Process lifecycle - [Configuration](guides/configuration.md) — YAML configuration format - [Security Module](lua/security/security.md) — Permission checks in Lua --- # "Publishing Modules" ## Publishing Modules Publishing packages a module and makes a version or mutable label available through the Wippy Hub. This is a publishing workflow and reference. The `acme/*` modules, URLs, tokens, credentials, and example source are illustrative; replace them with resources owned by your organization. ### Prerequisites 1. Create an account on [hub.wippy.ai](https://hub.wippy.ai) 2. Create an organization or join one 3. Have permission to create modules in that organization — the first `wippy publish` registers the module automatically ### Module Structure ``` mymodule/ ├── wippy.yaml # Module manifest ├── src/ │ ├── _index.yaml # Entry definitions │ └── *.lua # Source files └── README.md # Documentation (optional) ``` ### wippy.yaml Define module metadata in `wippy.yaml`: ```yaml organization: acme module: http-utils type: library description: HTTP utilities and helpers license: MIT repository: https://github.com/acme/http-utils homepage: https://acme.dev keywords: - http - utilities authors: - Acme Engineering embed: - acme.http:assets exclude: - test/** - "*.test.lua" - acme.http:debug_handler exclude_meta: stage: - experimental metadata: support_url: https://acme.dev/support ``` | Field | Required | Description | |-------|----------|-------------| | `organization` | Yes | Organization name on the Hub | | `module` | Yes | Module name | | `type` | No | Module type: `library`, `application`, `agent`, or `plugin` | | `description` | No | Short description | | `license` | No | SPDX identifier (MIT, Apache-2.0) | | `repository` | No | Source repository URL | | `homepage` | No | Project homepage | | `keywords` | No | Search keywords | | `authors` | No | Author list | | `version` | No | Semantic version; `--version` overrides it | | `exclude` | No | Patterns to drop: values containing `:` are entry IDs, everything else is a source-file glob | | `embed` | No | Default `fs.directory` embed patterns when `--embed` is not passed | | `exclude_meta` | No | Metadata field to values map; entries whose metadata matches are dropped | | `metadata` | No | Arbitrary key/value metadata carried with the published module | | `publish.profiles` | No | Which config profiles to ship in the pack (see [Publishing Profiles](#publishing-profiles)) | | `publish.runtime` | No | Which runtime config sections to ship as pack defaults; `type: application` only | `exclude` splits by shape rather than by a separate field. `_old/**`, `test/**` and `*.test.lua` filter source files as they are collected; `acme.http:debug_handler` disables a registry entry after entries are decoded. A `**` segment spans any number of directory segments. `type` controls how the Hub classifies the module and can be changed in a later publish. The `--module-type` flag overrides it for one publish. When omitted, a newly created module defaults to `application` with a deprecation warning. ### Entry Definitions Define the module's entries in `_index.yaml`: ```yaml version: "1.0" namespace: acme.http entries: - name: definition kind: ns.definition meta: title: HTTP Utilities description: Helpers for HTTP operations readme: file://README.md wiki: GUIDE.md: file://docs/GUIDE.md examples/auth.md: file://docs/auth.md - name: client kind: library.lua source: file://client.lua modules: - http_client - json ``` The `wiki:` map on `ns.definition` publishes documentation pages alongside the README. Keys are page paths, and values are `file://` references. Contents are inlined during packing and served by the Hub as a module wiki. ### Dependencies Declare dependencies on other modules: ```yaml entries: - name: __dependency.wippy.test kind: ns.dependency meta: description: Testing framework component: wippy/test version: ">=0.3.0" ``` Version constraints: | Constraint | Meaning | |------------|---------| | `*` | Any version | | `1.0.0` | Exact version | | `>=1.0.0` | Minimum version | | `^1.0.0` | Compatible (same major) | ### Requirements Define configuration that consumers must provide: ```yaml entries: - name: api_endpoint kind: ns.requirement meta: description: API endpoint URL targets: - entry: acme.http:client path: ".meta.endpoint" default: "https://api.example.com" ``` Targets specify where the value is injected: - `entry` — Full entry ID to configure - `path` — Dot path into the target entry for value injection `default` accepts any scalar type — `default: 20` flows into a numeric target as a number, not a string. The same applies to `parameters[].value` on `ns.dependency` entries, and both accept `${env:NAME}` references, carried verbatim and resolved when the target entry is decoded. Consumers can configure the target through an override. The `-o` flag accepts a `namespace:entry:field=value` value: ```bash wippy run -o acme.http:client:meta.endpoint=https://custom.api.com ``` ### Imports Reference other entries: ```yaml - name: handler kind: function.lua source: file://handler.lua modules: - json imports: client: acme.http:client # Same namespace utils: acme.utils:helpers # Different namespace base_registry: :registry # Built-in ``` In Lua: ```lua local client = require("client") local utils = require("utils") ``` ### Contracts Define public interfaces: ```yaml - name: http_contract kind: contract.definition meta: name: HTTP Client Contract methods: - name: get description: Perform GET request - name: post description: Perform POST request - name: http_contract_binding kind: contract.binding contracts: - contract: acme.http:http_contract methods: get: acme.http:get_handler post: acme.http:post_handler ``` #### 1. Authenticate ```bash wippy auth login ``` #### 2. Prepare ```bash wippy init wippy update wippy lint ``` #### 3. Validate ```bash wippy publish --dry-run ``` Publish builds the pack the same way with or without `--dry-run`, so validation covers everything the real publish would produce: - `organization` and `module` must be lowercase alphanumeric with interior hyphens, `version` must be semver, and `type` must be one of the four module types. - `publish.runtime` is application-owned: declaring `source`, `sections`, or `vars` under it without `type: application` fails. - Every resource declaring `meta.artifact.format` is inspected by that format. A malformed artifact fails here rather than in a consumer, and two artifacts whose outputs would land in overlapping directories are rejected. - The `node-package` format additionally requires `package.json` to carry a semantic `version` that **equals the module version being published**, a valid package `name`, and no `preinstall`, `install`, `postinstall`, or `prepare` lifecycle script. The last rule is the one that bites during a release: bump `version` in `wippy.yaml` and in the artifact's `package.json` together, or the publish stops. #### 4. Publish ```bash wippy publish --version 1.0.0 ``` With release notes: ```bash wippy publish --version 1.0.0 --release-notes "Initial release" ``` #### Publish Flags | Flag | Description | |------|-------------| | `--label ` | Publish as a mutable label (e.g. `latest`, `beta`) instead of an immutable version | | `--protected` | Mark the published version as protected (cannot be deleted or overwritten) | | `--registry ` | Override the registry URL for this publish | | `--config ` | Directory containing `wippy.yaml` (default: current dir) | | `--create` | Register the module on the hub if it does not exist yet, then publish | | `--module-visibility ` | Visibility for `--create`: `private` (default) or `public` | | `--module-type ` | Module type: `library`, `application`, `agent`, or `plugin` (overrides `type:` in wippy.yaml) | | `--module-display-name ` | Display name for `--create` | #### Embed Static Files Modules with `fs.directory` entries (static assets, templates, public files) must use `--embed` to include them in the published package. Without it, an `fs.directory` entry is packed without its directory contents. ```bash wippy publish --version 1.0.0 --embed app:public_files wippy publish --version 1.0.0 --embed app:assets,app:templates ``` The manifest list and `--embed` flag accept entry IDs or names matching `fs.directory` entries. The same CLI flag is available on `wippy pack`; a CLI selection overrides the manifest list for that invocation. #### First Publish On its first publish, a module is registered on the Hub as private by default, and the publish retries once. Use `--create` to register it before publishing and set its properties: ```bash wippy publish --create --version 0.1.0 \ --module-visibility public \ --module-type library \ --module-display-name "HTTP Utils" ``` `--create` is idempotent — for an already-registered module the create step is a no-op. If your account cannot create modules in the organization, the hub returns a permission error instead of publishing. #### Publishing to a Local Hub Point `--registry` at a locally running Hub to publish and install without using the public registry. Plain HTTP is allowed only for local hosts: `localhost`, `127.0.0.1`, and the container aliases `host.docker.internal` (Docker Desktop or OrbStack) and `host.containers.internal` (Podman). Other hosts must use HTTPS. ```bash wippy auth login --registry http://localhost:8080 --token wpy_xxx wippy publish --registry http://localhost:8080 --create --version 0.1.0 ``` The registry and token can also come from the `WIPPY_REGISTRY` and `WIPPY_TOKEN` environment variables. When unset, the registry defaults to `https://hub.wippy.ai`. #### Quotas If the organization's private-module quota is exhausted, publishing fails with a message such as `cannot publish: Private-module quota exhausted (5 of 5)...`. Make the module public or ask an organization administrator to raise the quota. Uploads and downloads retry automatically after transient network errors. ### Publishing Runtime Defaults Applications with `type: application` can include runtime configuration defaults in their packs through `publish.runtime` in `wippy.yaml`: ```yaml type: application publish: runtime: source: .wippy.yaml # default: .wippy.yaml sections: [security, registry, override] vars: [public_url] ``` | Field | Description | |-------|-------------| | `source` | Config file the sections are read from (default: `.wippy.yaml`) | | `sections` | Runtime config sections copied into pack metadata as defaults | | `vars` | Explicit allowlist of variables to pack even when unreferenced | Rules: - Only variables referenced by the selected sections or published profiles are packed (followed transitively); everything else needs a `vars` entry. - `${env:...}` references in exported config are rejected — publisher environment never leaks into a pack. - The machine-local sections `boot`, `extensions`, and `workspace` cannot be exported. - Only the main application pack provides host runtime defaults; runtime metadata in dependency packs is ignored. At the destination, configuration precedence runs from application-pack defaults through runtime defaults, local configuration files, selected profiles, and finally CLI overrides. ### Publishing Profiles Root application profiles are exported into the pack's `runtime.profiles` metadata. Publishing does not select or bake a profile — consumers pick one at run time with `wippy run --profile `: ```yaml publish: profiles: enabled: true source: config/profiles.yaml # default: .wippy.yaml include: [production] # omit to publish all non-workspace profiles ``` `include: []` publishes none; an unknown name fails the publish. `workspace` sub-sections are never exported, even inside a published profile. See [Configuration](guides/configuration.md#profiles) for declaring profiles. #### Add Dependency ```bash wippy add acme/http-utils wippy add acme/http-utils@1.0.0 wippy install ``` #### Configure Requirements Override values at runtime: ```bash wippy run -o acme.http:client:meta.endpoint=https://my.api.com ``` Or in `.wippy.yaml`: ```yaml override: acme.http:client:meta.endpoint: "https://my.api.com" ``` #### Import in Your Code ```yaml ## your src/_index.yaml entries: - name: __dependency.acme.http kind: ns.dependency component: acme/http-utils version: ">=1.0.0" - name: my_handler kind: function.lua source: file://handler.lua imports: http: acme.http:client ``` ### Example Module **wippy.yaml:** ```yaml organization: acme module: cache type: library description: In-memory caching with TTL license: MIT keywords: - cache - memory ``` **src/_index.yaml:** ```yaml version: "1.0" namespace: acme.cache entries: - name: definition kind: ns.definition meta: title: Cache Module - name: cache kind: library.lua source: file://cache.lua modules: - time ``` **src/cache.lua:** ```lua local time = require("time") local cache = {} local store = {} function cache.set(key, value, ttl) store[key] = { value = value, expires = ttl and (time.now():unix() + ttl) or nil } end function cache.get(key) local entry = store[key] if not entry then return nil end if entry.expires and time.now():unix() > entry.expires then store[key] = nil return nil end return entry.value end return cache ``` Publish: ```bash wippy init wippy update wippy lint wippy publish --version 1.0.0 ``` ### See Also - [CLI Reference](guides/cli.md) — Publishing commands and flags - [Entry Kinds](guides/entry-kinds.md) — Module and dependency entries - [Configuration](guides/configuration.md) — Runtime configuration and profiles --- # "Compute Units" ## Compute Units Wippy provides three ways to run code: functions, processes, and workflows. They share the same underlying machinery but differ in how long they live, where their state goes, and what happens when things fail. ### Functions Functions run when called and return a result. Treat each call as stateless: durable or shared state belongs in a database or store. Function pools can reuse Lua states, so module globals and closure upvalues are worker-local and are not a reliable cross-call store. ```lua local funcs = require("funcs") local result, err = funcs.call("app.math:add", 2, 3) if err then return nil, err end ``` Functions execute in the caller's context. If the caller is canceled or exits, its running function calls are canceled as well. Use functions for HTTP handlers, data transformations, and anything that should complete quickly and return a result. ### Processes Processes are actors. They maintain state across multiple messages, run independently of whoever started them, and communicate through message passing. ```lua local pid, err = process.spawn("app.workers:handler", "app:processes") if err then return nil, err end local ok, send_err = process.send(pid, "job", {task = "process_data"}) if send_err then return nil, send_err end return ok ``` After being spawned, a process runs independently of the code that created it. Processes can monitor or link to one another and can participate in supervision trees that restart failed children. The scheduler multiplexes thousands of processes across a worker pool. Each process yields when waiting for I/O, letting others run. Use processes for background jobs, service daemons, and anything that needs to outlive its creator or maintain state across messages. ### Workflows Workflows are for durable operations that must recover from interruptions. A workflow provider such as Temporal records execution history and replays it to rebuild state after crashes, restarts, or infrastructure changes. ```lua -- The provider records this workflow so a worker restart can replay it. local pid, err = process.spawn("app.orders:process", "app:temporal_worker", order_id) if err then return nil, err end return pid ``` Durability adds latency because workflow operations are recorded. Use workflows when recovery is more important than the lower latency of functions or processes, such as for multi-step business processes and long-running orchestration. Wippy records supported workflow operations so they produce the same results during replay. Workflow code uses the same Lua syntax as other compute units. ### How They Compare | | Functions | Processes | Workflows | |---|---|---|---| | **State** | Call-local; do not depend on worker reuse | In memory | Rebuilt from persisted history | | **Lifetime** | Single call | Until exit or crash | Persists across restarts | | **Communication** | Return value + messages | Message passing | Activity calls + messages | | **Failure handling** | Caller handles | Supervision trees | Provider recovery; retries follow policy | | **Latency** | Lowest | Low | Higher | ### Same Code, Different Behavior Many modules adapt to their context automatically. For example, `time.sleep()` yields in both functions and processes so other work can run; in a workflow, the provider also records the timer so replay does not start a second timer. --- # "Application Architecture" ## Application Architecture A Wippy application is a **graph of registry entries** represented by source files. Code lives in entries such as `function.lua` and `process.lua`; `_index.yaml` files declare how functions, routes, services, and libraries connect. Application structure determines how that graph is divided into namespaces so it remains composable, testable, and bootable as it grows. This page explains one way to organize that graph. For file format, naming, and `_index.yaml` placement, see [YAML & Project Structure](start/structure.md). For entry definitions, see the [Entry Kinds Guide](guides/entry-kinds.md). ### Feature Slices A useful default is to organize by **feature** rather than file type. A slice owns one capability end to end—its database access, long-running processes, HTTP surface, and shared vocabulary—and lives under one namespace prefix: ``` src/app/jobs/ namespace: app.jobs src/app/auth/ namespace: app.auth src/app/billing/ namespace: app.billing ``` Feature slices keep related behavior within one folder, making a capability easier to read, test, change, or remove without tracing it across top-level `handlers/`, `models/`, and `services/` directories. ### Layers within a slice For larger slices, separate code by **what touches the outside world**. This applies ports-and-adapters (hexagonal) architecture through **sub-namespaces**: ``` src/app/jobs/ namespace: app.jobs ← shared vocabulary consts.lua config.lua types.lua persist/ namespace: app.jobs.persist ← database adapters (sql) service/ namespace: app.jobs.service ← processes, workers api/ namespace: app.jobs.api ← http.endpoints ``` Keep imports flowing from outer layers toward inner layers: ``` api → service → persist → { consts, config, types } ``` The slice root contains shared vocabulary and does not import its own children. Children may import the root. Avoid direct imports between slices; place shared definitions in a common parent namespace such as `app.core:types`. Namespaces organize entry IDs but do not create dependencies or injection seams by themselves. Explicit imports, kind-specific references, and ns.requirement targets create those relationships. A consistent direction keeps the resulting graph explicit. See Why this shape. A small slice can use one `_index.yaml` for its libraries and endpoint. The important property is the **import direction**, not the number of folders. ### Shared Vocabulary Three files commonly appear at the root of a slice. They contain definitions shared by the slice's layers: | File | Holds | Capabilities | |------|-------|--------------| | `consts.lua` | State machines, enums, queue tiers, registry IDs of processes. The values that mirror your database `CHECK` constraints. | none | | `config.lua` | Env-tunable knobs with a helper that applies a code default only when `env.get(KEY)` returns `errors.NOT_FOUND` and propagates permission or backend errors. No `env.variable` entry is required for a value to be optional. | `env` | | `types.lua` | Entity shapes (`type Job = { ... }`) — the rows the persistence layer returns. | none | `consts` and `types` declare **no host capabilities**; they are pure `library.lua` entries that return a table. Keeping domain vocabulary free of I/O also makes it testable without a database or process host. Keep this vocabulary **slice-private**. Place constants and types shared across slices in a common parent namespace and import them rather than copying them. ### Capabilities by Layer Lua entries declare non-ambient modules in `modules:` and registry-backed dependencies in `imports:`. A layered slice can keep those dependencies aligned with responsibility: - `persist/*` declares `sql`, keeping database access in the persistence layer. - `service/*` keeps process orchestration and service dependencies in the service layer. The `process` and `channel` globals are ambient and do not need `modules:` declarations. - `api/*` declares modules such as `http` and imports the functions or libraries it calls. - The root vocabulary needs no non-ambient modules or infrastructure imports. This limits module visibility to a known layer. It is not an authorization grant: ABAC policies independently decide whether guarded operations such as `db.get` are allowed at runtime. To review code that can request a database handle, inspect `persist/`, its declared modules, and the policies attached to its execution context. ### Applications and Components The same shape can support a single application or a published library; the difference is **who supplies its dependencies**. An **application** is the top-level, deployable graph. It owns the concrete infrastructure — the `http.service`, the `process.host`, the database connection — under a root namespace (conventionally `app`), and wires everything together itself. A **component** is a publishable module mounted into a host. Because it does not know the host's database or router IDs, it declares an interface of `ns.requirement` entries that the host supplies. Internally, a component can use the same layers, vocabulary, and import direction as an application slice. These are two points on a spectrum: - **Single app, internal slices** — slices live under `src/app/`, share the app's infrastructure directly by referencing `app:db`, `app:processes`. No requirement interface is needed because nothing external mounts them. - **Multi-component composition** — each component is its own publishable module with an `ns.definition` and an `ns.requirement` interface, composed by a host through `ns.dependency`. The host fills each requirement (database, process host, router) once. Choose based on whether the slice will be **consumed by a host you do not control**. Reusable components need a requirement interface; internal slices can reference application infrastructure directly. The packaging changes with reuse, while the internal layering can remain the same. See [Building Components](guides/components.md) for the requirement/dependency mechanism, and [Dependency Management](guides/dependency-management.md) for the lock-file side. ### Why Use This Shape :id=why-this-shape This structure supports composition, capability review, and boot-order analysis: **Requirement targets are the injection seam.** Distinct namespaces make target IDs legible, but `ns.requirement.targets` performs the injection. A host can supply a database ID to persistence entries and a process-host ID to service entries. Directly referencing `app:db` instead couples the component to that host convention. **One-way references keep registry transitions resolvable.** The registry extracts declared dependency paths and topologically orders changes so dependencies are created before their dependents and deleted after them. The direction `api → service → persist → root` helps keep that graph acyclic. A parent namespace is only an organizational convention; the shared entries still need explicit references. **Modules scoped by layer have a clear boundary.** Each Lua chunk can resolve its declared imports and non-ambient modules; undeclared registry modules fail closed at module resolution. Runtime policy checks remain a separate boundary. When persistence entries alone declare `sql`, the code that can request a database handle is easier to identify and audit. **The layering supports different test scopes.** Vocabulary can be tested without infrastructure. Persistence tests can use a database without starting workers. A whole-module **mount test** then checks the integration seams: every supervised service points to a process, every spawned ID resolves, and every requirement is filled. ### See Also - [YAML & Project Structure](start/structure.md) — file format, naming, namespaces - [Building Components](guides/components.md) — `ns.definition`, `ns.requirement`, mounting - [Dependency Management](guides/dependency-management.md) — lock files, consuming modules - [Registry](concepts/registry.md) — how entries are stored and resolved - [Entry Kinds Guide](guides/entry-kinds.md) — every entry kind - [Process Model](concepts/process-model.md) — services, supervision, hosts --- # "Registry" ## Registry The registry is Wippy's versioned store for entry points, services, resources, and other runtime definitions. Most runtime entry kinds are reconciled through event-bus transactions; internal kinds such as `registry.entry` and namespace metadata bypass event dispatch by default. ### Entries The registry holds **entries**—typed definitions with unique IDs: ``` app.api:get_user → HTTP handler app.workers:email_sender → Background process app:database → Database connection app:templates → Template set ``` Each entry has an `ID` (namespace:name format), a `kind` that determines its handler, arbitrary `meta` fields, and kind-specific `data`. Alongside that authored content the registry keeps its own provenance for each entry: the `owner`, meaning the deployment source the entry came from, and `root`, marking a dependency declaration the deployment selected. This state is assigned by the registry, not written by the entry author, and it is kept separate from `meta` so the two can never be confused. It is read through the snapshot state API rather than the ordinary entry APIs—see [Registry module](lua/core/registry.md#snapshot-state). For how the registry functions as an authorization layer, see the [Security Model](concepts/security-model.md). ### Kind Handlers When a dispatched entry is submitted, its `kind` selects the registered handler. The handler validates and reconciles the corresponding runtime resource: an `http.service` entry manages an HTTP server, a `function.lua` entry manages a function pool, and a `db.sql.postgres` entry manages a connection pool. See the [Entry Kinds Guide](guides/entry-kinds.md) for available kinds and [Custom Entry Kinds](internals/kinds.md) for handler implementation. ### Live Updates Entries can be added, updated, or removed while the system runs. For dispatched kinds, a registry transaction asks participating handlers to accept or reject each operation before commit. A rejection discards the transaction and applies the inverse transition. Related topology changes produce one new registry version. Version history supports backward and forward transitions when history is enabled. Memory history is the default and lasts for the process lifetime; SQLite and PostgreSQL backends persist history across restarts. YAML and JSON definition files are source manifests that the boot loader converts into entries. They are not serialized registry snapshots. See [Registry module](lua/core/registry.md) for programmatic access. ### See Also - [YAML & Project Structure](start/structure.md) — Definition files - [Custom Entry Kinds](internals/kinds.md) — Implement kind handlers - [Process Model](concepts/process-model.md) — Understand process execution --- # Security Model - Process Isolation, Capability Control, and Data Boundaries ## Security Model Wippy's security model defines what your code can access, what it cannot, and who enforces those boundaries. It is worth reading before you build, because it works at two layers that most frameworks collapse into one: the runtime isolates each process so dangerous capabilities are simply absent, and an attribute-based policy layer governs which registry capabilities a process is allowed to use. Understanding both changes how you structure an application. ### Trust Model Wippy's isolation layer gives a process no ambient authority. A fresh Lua or WASM process cannot touch the file system, the network, the host OS, or other processes' memory, because those capabilities are not present in its environment. Capabilities arrive only through the registry: functions, tools, connections, and configuration the process is explicitly granted. On top of that, access to registry capabilities is governed by attribute-based access control (ABAC). Every guarded operation is checked against the current actor's security scope, a set of policies that allow or deny an action on a resource, optionally conditioned on actor and resource metadata. This is declarative: you define policies in configuration, not in application code. When a process runs with both an actor and a scope, access is deny-by-default: a request is allowed only if a policy explicitly permits it and none denies it. **Strict mode** governs the incomplete case, when no actor or scope is established. It is **on by default**, so an incomplete context is denied; setting `security.strict_mode: false` in the runtime config opts into the permissive behavior instead. The consequence to plan for is that a process with no declared security context fails every check under the default — give such a process a `security:` block on its entry, or start it through a path that supplies one. Combined with least-privilege policies, this gives you fail-closed authorization on top of deny-by-absence isolation. See the [Security reference](system/security.md) for policy syntax, evaluation rules, and the shape of the `security:` block. ### Process Isolation Every unit of execution in Wippy runs in an isolated process with its own embedded interpreter (Lua or WASM). **What a process has:** its own memory space (a baseline overhead of ~13 KB for Lua). A scoped view of the registry. An actor identity and a security scope. A supervised lifecycle with crash recovery and restart limits. **What a process does not have:** access to the file system (except through registry-controlled filesystem entries). Access to the network (except through granted HTTP client or tool modules). Access to other processes' memory. Access to the Go runtime hosting it. Access to environment variables (except through granted environment entries). **How isolation is enforced:** each Lua process starts from a minimal standard library. File I/O, OS process access, dynamic code loading, and networking are never loaded, so they are not present in the environment, and the process cannot restore what does not exist. Module loading is restricted: `require` resolves only the modules and registry entries the process is explicitly granted, with no file system search path. WASM processes achieve equivalent isolation through WASI: only the host functions and mounted filesystem entries configured for that entry are reachable. This is not sandboxing via runtime permissions (like seccomp or AppArmor). It is sandboxing via absence. Dangerous capabilities are never loaded, so they cannot be exploited, bypassed, or escalated. ### Capability Control The registry is Wippy's capability store, and security policies are its authorization layer. **Every capability is a registry entry.** Functions, tools, agent definitions, database connections, environment references, configuration values, and scheduled tasks are all registry entries with a declared kind, schema, and metadata. Entries are validated by their kind handler when registered. **Entry IDs are namespaced.** An ID has the form `namespace:name` with a single colon, and namespaces are hierarchical via dot-separated segments, for example `tenant_acme.tools:read` (namespace `tenant_acme.tools`, name `read`). Policies match actions and resources, and resource patterns can target a namespace prefix, so a single rule can cover an entire namespace. **Policies decide access.** Each capability access (a registry lookup, a function call, a database handle, a file open) is checked against the actor's scope. A policy declares the actions and resources it covers, an allow or deny effect, and optional conditions on actor and resource metadata. Evaluation happens per access, not once at startup: if any policy denies, access is denied; if at least one allows and none denies, it is allowed; if no policy matches, access is denied. (When the context has no actor or scope at all, that incomplete case is resolved by strict mode rather than by policy evaluation.) **A context is declared, not inherited from thin air.** Functions inherit the caller's actor and scope. A spawned process inherits them too: its frame is forked from the spawner's, and the `security:` block on its own entry then modifies that inherited context — an `actor` it names replaces the inherited actor, and the policies and policy groups it lists by registry ID are merged into the inherited scope. Resolution is atomic — if any named policy or group is missing, the spawn fails rather than proceeding with a partial scope. A CLI command can additionally declare `meta.command.security`, applied only on the trusted launch path where the operator started the command themselves. **Tool arguments are schema-shaped.** A tool declares a JSON Schema for its inputs. That schema is given to the model so it generates conforming arguments, and access to the tool is policy-checked before the call runs. ### Data Boundaries **Database connections are registry entries.** A process does not assemble its own connection string. It requests a connection by registry ID, and that request is policy-checked before a handle is returned. A process whose policies do not grant Tenant B's database entry cannot obtain a handle to it. **LLM API keys live in the environment system.** Keys for Claude, GPT, and other providers are read from the environment system (for example OS environment variables exposed through an `env.storage.os` entry, referenced by `env.variable` entries whose reads are policy-checked via the `env.get` action). The provider reads them internally; they are not passed in process arguments or returned to calling code. **File and blob storage follow the same model.** A process reads or writes through filesystem or cloud-storage registry entries, each access policy-checked. WASM processes access files only through filesystem entries explicitly mounted for that entry. ### Agent Security Agents are LLM-powered processes with tool use. They make decisions at runtime that your code does not directly control, so their boundaries matter. Wippy handles this through the same registry and policy mechanisms as any other process. **Tool access.** An agent can only invoke tools listed in its definition, and each tool execution runs through `funcs.call`, which is policy-checked. A denied call fails before the tool function runs. An agent designed to read customer data but not delete it either has no delete tool in its definition or is denied that action by policy. **External and MCP tools.** Wippy can consume external tools and expose its own over the Model Context Protocol. Consumed tools run through the same function-call path and policy checks as native tools. Tools Wippy exposes to external MCP clients are gated by scoped, revocable access tokens that limit which actions a client may perform. **Structured output.** The LLM module can request schema-constrained (structured) output using the provider's native structured-output support, so an agent's output can be held to a declared shape. **Observability.** With OpenTelemetry enabled, LLM provider calls and tool invocations are traced, and token usage is recorded through the usage-tracker contract. This gives you an audit trail of what an agent called and what it spent. See [Observability](guides/observability.md). **Self-modification boundaries.** An agent permitted to create tools in one namespace can be denied write access to its own definition in another. Registry writes are policy-checked actions, so a deny policy on the agent's own namespace prevents it from editing itself or granting itself new access. ### Multi-Tenant Enforcement For deployments where multiple customers share a single Wippy instance, isolation is enforced by policy evaluation before any operation runs, not by application code checking tenant IDs. **Tenant isolation is policy-enforced.** Give each tenant an actor and a scope whose policies cover only that tenant's namespaces. With strict mode on, a tenant's process is denied access to resources outside its scope before its code runs. Effective isolation depends on writing those per-tenant policies; the runtime enforces them, but it does not infer tenancy for you. **Cross-tenant access is explicit.** A capability shared across tenants lives in a shared namespace that each tenant's policies allow. Sharing is opt-in per namespace. **Concurrency is bounded at the host.** Process hosts bound concurrency through worker pools. Process groups (`pg.scope`) provide isolated, cluster-wide membership and broadcast namespaces and can cap group and member counts. Per-tenant CPU or memory ceilings are not a built-in runtime feature; enforce those at the infrastructure layer. A dedicated Multi-Tenant Architecture guide is planned. ### Scope and Limitations Wippy's security model covers process isolation, capability control, and data boundaries. The following are outside the runtime's scope and remain your infrastructure's responsibility. **Data encryption at rest.** Database, disk, and blob-storage encryption are handled by the underlying infrastructure (PostgreSQL TDE, disk encryption, and similar). Wippy assumes the storage layer handles encryption. **Network-level isolation.** Process isolation happens at the application layer. Network segmentation between Wippy and its dependencies (database, LLM APIs, external services) is handled by infrastructure: VPCs, security groups, firewalls. **Identity management.** Authentication (verifying who a user is) is handled by your auth layer. Wippy's security model starts after authentication: it controls what an authenticated user's processes can do, not who the user is. Tokens that carry an actor and scope can be issued and validated through a token store. **Infrastructure audit logs.** Wippy's tracing covers process-level operations: function calls, tool calls, process activity. Infrastructure-level access (SSH to the server, database admin operations) should be audited by infrastructure tools. ### Common Questions **Can one tenant's agent access another tenant's data?** Not when each tenant's resources are scoped by policy. With per-tenant policies and strict mode, the runtime denies access to resources outside the tenant's scope before the agent's code runs. **Can an agent escalate its own permissions?** Only if its policies allow writing to its own definition. Registry writes are policy-checked, so a deny policy on the agent's own namespace prevents self-modification. An agent that can create tools in one namespace cannot grant itself access to namespaces its scope does not already cover. **How do I see what an agent did?** With OpenTelemetry enabled, LLM and tool calls are traced, and token usage is recorded through the usage-tracker contract. See [Observability](guides/observability.md). **What happens if an agent behaves unexpectedly?** It is contained by the sandbox: no file system, no network, no OS, no access to other processes beyond what it was granted. It can only call tools in its definition that policy permits, and those calls are logged. **Is tenant isolation enforced by my code or by the runtime?** By the runtime. The policy engine evaluates each access before the operation runs. Your job is to write the per-tenant policies; the runtime enforces them. **How are external MCP tools secured?** Tools consumed over MCP run through the same function-call path and policy checks as native tools. Tools Wippy exposes to external MCP clients are gated by scoped, revocable access tokens. Connecting an MCP service does not bypass the security model. ### Security Reference | Concern | Wippy's approach | |---------|------------------| | Process isolation | Separate interpreter per process (Lua or WASM), no shared memory | | Default access | Unmatched policies deny when both an actor and a scope are set; strict mode, on by default, denies when no actor or scope is established | | Context declaration | `security:` block on the entry (actor, policies, groups); resolution is atomic and fail-closed | | Supply chain | Module packs verified by digest at install and at boot; a mismatch refuses the module | | Node-to-node trust | Mutually authenticated internode mesh; ed25519 identity per node, explicit trusted-peer map | | Workflow propagation | Actor and scope carried to Temporal as a signed, audience-bound header; verification failure fails the execution | | Capability control | Registry entries governed by attribute-based security policies (actor, scope, action, resource) | | Data boundaries | Connections and storage are registry entries; each access is policy-checked by entry ID | | API key management | Stored in the environment system, read internally by providers, not exposed to process code | | Agent tool control | Tools limited to the agent's definition; each call checked via `funcs.call` policy | | External tools (MCP) | Same function-call path and policy checks; exposed tools gated by scoped tokens | | Agent audit trail | OpenTelemetry tracing (when enabled) plus usage-tracker records | | Multi-tenant isolation | Per-tenant policies and scopes evaluated by the runtime before each operation | | Concurrency limits | Bounded by host worker pools; no per-tenant CPU/memory ceilings built in | | Self-modification | Deny policies on registry-write actions prevent agents from editing their own definitions | ### See Also - [Security reference](system/security.md) - Policies, scopes, actors, token stores, and the `security:` block - [Dependency Management](guides/dependency-management.md#integrity-verification) - Module digest verification - [Cluster](guides/cluster.md#internode-identity) - Internode identity and peer trust - [Temporal Workflows](temporal/workflows.md#security-context) - Signed context propagation - [Registry](concepts/registry.md) - The capability store - [Process Model](concepts/process-model.md) - Process isolation and lifecycle - [Agents](framework/agents.md) - Agent definitions and tool use --- # "Process Model" ## Process Model Wippy executes code in isolated processes: lightweight state machines that communicate through messages rather than shared memory. This actor model gives each process its own state and lifecycle. This page explains the lifecycle and isolation model. Use the [Process Management reference](../lua/core/process.md) for spawn, messaging, monitoring, registry, and upgrade APIs. See [Process Host and Services](../system/process-host.md) for runtime-managed service fields. ### State Machine Execution Each process initializes, advances through execution, yields on blocking operations, and closes when complete. The scheduler multiplexes processes across a worker pool and runs other work while a process waits for I/O. Processes support multiple concurrent yields, allowing code to start several asynchronous operations and wait for any or all of them without spawning additional processes. ```mermaid flowchart LR Ready --> Running Running --> Blocked Running --> Idle Blocked --> Running Idle --> Running Running --> Complete ``` Processes are not limited to Lua. The runtime also supports WebAssembly modules through the `process.wasm` kind, and its process architecture can support other state-machine implementations. Processes are lightweight but not free. Each process carries a small baseline cost for its state, inbox, and scheduler bookkeeping, and dynamic allocations grow that footprint during execution. ### Process Hosts Wippy can run multiple process hosts within one runtime, each with its own capabilities and security boundaries. Privileged system processes can run in a host separate from hosts that execute user sessions. Some hosts are specialized. The Terminal host, for example, uses one scheduler worker and supplies terminal I/O context to accepted processes; it does not enforce a one-process lifetime limit. Separate hosts allow one deployment to run processes with different trust levels. ### Security Model Each process executes under an actor identity and security policy. This is typically the user who initiated the call, while system processes use a system actor with different privileges. Access control applies at multiple levels. Security policy can restrict individual process operations and message delivery between hosts. The policy attached to the current actor determines which operations are permitted. For the security implications of process isolation, see the [Security Model](concepts/security-model.md). ### Spawning Processes Create background processes with `process.spawn()`: ```lua local pid, err = process.spawn("app.workers:handler", "app:processes", arg1, arg2) if err then return nil, err end return pid ``` The first argument is the registry entry, the second is the process host, and remaining arguments pass to the process. Spawn variants control lifecycle relationships: | Function | Behavior | |----------|----------| | `spawn` | Start an independent process | | `spawn_monitored` | Receive EXIT events when child exits | | `spawn_linked` | Abnormal exit propagates in either direction; with `trap_links: true`, the peer receives `LINK_DOWN` instead of failing | ### Message Passing Processes communicate through messages rather than shared memory: ```lua local ok, err = process.send(target_pid, "topic", payload) if err then return nil, err end return ok ``` Messages from the same sender arrive in order. Messages from different senders may interleave. Delivery is fire-and-forget—use request-response patterns when you need confirmation. Processes can register in a local name registry and be addressed by name instead of PID (e.g., `session_manager`). Names can also be registered cluster-wide for cross-node addressing via `process.registry` using EVENTUAL (gossip-based), CONSISTENT, or STRONG (both Raft-backed) scopes. ### Supervision Any process can supervise other processes by monitoring them. A supervisor starts monitored children, watches for EXIT events, and decides whether to restart them after failure. ```lua local worker, spawn_err = process.spawn_monitored("app.workers:handler", "app:processes") if spawn_err then return nil, spawn_err end local event, open = process.events():receive() if not open then return nil, errors.new("process event channel closed") end if event.kind == process.event.EXIT and event.result.error then local replacement, restart_err = process.spawn_monitored("app.workers:handler", "app:processes") if restart_err then return nil, restart_err end worker = replacement end ``` At the runtime level, services can start and supervise long-running processes. Define a `process.service` entry to have the runtime manage a process: ```yaml - name: worker.service kind: process.service process: app.workers:handler host: app:processes lifecycle: auto_start: true restart: max_attempts: 5 initial_delay: 1s ``` The service starts automatically and integrates with the runtime's lifecycle management. At the pinned runtime, the initial failed start counts toward `max_attempts`, so `5` permits at most four follow-up starts. Each retry waits for `initial_delay` with jitter; the delay does not increase between attempts. ### Process Upgrading Running processes can upgrade their code without losing identity. Call `process.upgrade()` to swap to a new definition while preserving PID, mailbox, and supervision relationships: ```lua process.upgrade("app.workers:v2", current_state) ``` The first argument is the new registry entry (or nil to reload the current definition). Additional arguments pass to the new version, letting you carry state across the upgrade. The process resumes execution with the new code immediately. The runtime caches compiled prototypes to avoid repeated compilation. If an upgrade fails, the process crashes and normal supervision behavior applies; a monitoring parent can restart it or escalate the failure. ### Scheduling The actor scheduler uses work-stealing across CPU cores. Each worker has a local queue for cache locality, plus a global queue for distributing work. Processes yield on blocking operations so other processes can run on the worker pool. --- # "Cluster" ## Cluster A single Wippy node is a complete runtime. A **cluster** connects several nodes so processes can use cluster-wide names, route messages across nodes, and coordinate through locks, groups, and a shared consensus core. Clustering is opt-in (`cluster.enabled`). This page explains the model your code sees; for topology, configuration, and operations see the [Cluster Guide](guides/cluster.md). ### Cluster Model Nodes discover one another through **gossip** (SWIM). A node joins through a seed, after which membership and failure information converge without a central coordinator. A bounded **Raft** core provides linearizable consensus through a dynamically reconciled voter set, while other nodes participate through gossip. The application-facing model has three parts: **names**, **routing**, and **coordination primitives**. ### Naming A process is normally addressed by its PID. In a cluster, it can also be registered under a **name** and reached by that name from other nodes. The selected **scope** determines the consistency guarantee and coordination cost: | Scope | Visibility | Guarantee | Use it for | |-------|------------|-----------|------------| | **Local** | this node | instant, no coordination | node-local helpers | | **Eventual** | cluster-wide | converges after gossip; conflicts resolve and notify the loser | service, group, and bounded presence names | | **Consistent** | cluster-wide | linearizable singleton via Raft | the standard cluster-wide named service | | **Strong** | cluster-wide | Consistent, plus every live node acknowledges before the name is active | control-plane singletons and locks | The scopes are ordered as `Local < Eventual < Consistent < Strong` by consistency and coordination cost. Select the least costly scope that meets the required guarantee. Names are registered through [`process.registry`](lua/core/process.md). Local names are removed when the process exits; Consistent and Strong names are also reaped on process exit or node departure. Eventual names are removed explicitly or when their origin node leaves, not automatically when only the owning process exits. ### Routing Routing connects a registered name to the process that owns it: - **Reads are local.** Every node resolves a name from its own replica or gossip-disseminated cache — no network round-trip to look up a name. This keeps resolution fast and keeps working during partitions. - **Resolution has a fixed order.** A name is resolved across the planes most-authoritative first — Consistent and Strong (Raft), then Eventual (gossip), then Local — so a cluster-wide name shadows a local one of the same string. - **Writes route to the authority.** A Consistent or Strong registration goes through the Raft leader; a node that isn't the leader forwards the write and waits for the result. Once committed, the active binding is disseminated over gossip so every node — including those not in the Raft core — can resolve the name locally afterward. - **Messaging routes by PID.** When you `process.send` to a name, it resolves to a PID and the relay delivers the message to the owning node. Your code addresses a process the same way whether it lives on this node or another — location is transparent. Applications register and resolve names without addressing the authority node directly. After resolution, messages route to the node that owns the target PID. ### Primitives Clustering exposes a small set of coordination building blocks: - **Membership and identity** — the live set of nodes and this node's identity and role. Use it to discover peers or shard work. See [`system.cluster`](lua/system/system.md) and [`system.node`](lua/system/system.md). - **Consensus state** — the Raft leader, term, and this node's role, for diagnostics and leader-aware logic. See [`system.raft`](lua/system/system.md). - **Cluster-wide names** — register and resolve processes by name and scope, the foundation everything else builds on. See [`process.registry`](lua/core/process.md). - **Distributed locks** — cluster-wide mutual exclusion with at most one holder, released automatically if the holder dies. See [`system.lock`](lua/system/system.md). - **Process groups** — join named groups and broadcast to every member across all nodes, Erlang-style. See [Process Groups](lua/core/pg.md). These primitives share membership and routing infrastructure. Consistent and Strong names and distributed locks use the Raft core. Process groups use gossip membership to discover peers, send changes over the relay, and periodically exchange full state for convergence. ### See Also - [Cluster Guide](guides/cluster.md) — Topology, configuration, and operations - [Process Management](lua/core/process.md) — Spawning, messaging, and the name registry - [Process Groups](lua/core/pg.md) — Named groups and broadcast - [System](lua/system/system.md) — `system.cluster`, `system.node`, `system.raft`, `system.lock` - [Process Model](concepts/process-model.md) — Processes, PIDs, and messaging --- # "Functions" ## Functions Functions are call-and-return entry points. A function inherits its caller's context and is canceled when the caller is canceled. Pools can reuse Lua states, so module globals and closure upvalues may survive on one worker but are not shared consistently across calls. Store durable or shared state outside the function. Use functions for HTTP handlers, API endpoints, and other operations that complete within a request lifecycle. ### Calling Functions Call functions synchronously with `funcs.call()`: ```lua local funcs = require("funcs") local result, err = funcs.call("app.api:get_user", user_id) if err then return nil, err end return result ``` For non-blocking execution, use `funcs.async()`: ```lua local future, err = funcs.async("app.process:analyze", data) if err then return nil, err end local ch = future:response() local payload, open = ch:receive() if not open then return nil, "future response channel closed" end local result, err = payload:data() if err then return nil, err end ``` See the [funcs module](lua/core/funcs.md) for function invocation and executor options. ### Context Propagation Each call creates a frame with its own context scope. Child functions inherit parent context without explicit passing: ```lua local ctx = require("ctx") local trace_id = ctx.get("trace_id") local user_id = ctx.get("user_id") ``` Add context when calling: ```lua local funcs = require("funcs") local exec, err = funcs.new():with_context({trace_id = "abc-123"}) if err then return nil, err end local result, err = exec:call("app.api:process", data) if err then return nil, err end return result ``` Security context propagates the same way. Called functions see the caller's actor and can check permissions. See the [security module](lua/security/security.md) for access control APIs. ### Registry Definition At the registry level, a function entry looks like this: ```yaml - name: get_user kind: function.lua source: file://handlers/user.lua method: get pool: type: lazy max_size: 16 ``` Functions can be invoked by other runtime components—HTTP handlers, queue consumers, scheduled jobs—and are subject to permission checks based on the caller's security context. ### Pools Functions run on pools that manage execution. The pool type determines scaling behavior. **Inline** runs in the caller's goroutine without a worker pool. It is used for embedded contexts. **Static** maintains a fixed number of workers. Requests queue when all workers are busy, which keeps worker concurrency fixed. ```yaml pool: type: static size: 8 buffer: 512 ``` **Lazy** starts without workers and creates them on demand. Idle workers are removed after a timeout. ```yaml pool: type: lazy max_size: 32 ``` **Adaptive** adjusts the worker count based on measured throughput and current load. ```yaml pool: type: adaptive max_size: 256 ``` If you don't specify a pool type, the runtime selects one based on your configuration. Set `workers` for static, `max_size` for lazy, or explicitly set `type` for full control. With neither set, the pool is lazy with a maximum of 16 workers. ### Interceptors Function calls pass through an interceptor chain. Interceptors can handle cross-cutting concerns separately from the function implementation. ```yaml - name: my_function kind: function.lua source: file://handler.lua method: main meta: options: retry: max_attempts: 3 initial_delay: 100 backoff_factor: 2.0 ``` Built-in interceptors include retry with exponential backoff. Runtime integrations written in Go can register additional interceptors for logging, metrics, tracing, authorization, circuit breaking, or request transformation; Lua application entries can configure only interceptors installed by the runtime. The chain runs before and after each call. Each interceptor can modify the request, short-circuit execution, or wrap the response. ### Contracts Functions can expose their input/output schemas as contracts. Contracts define method signatures that enable runtime validation and documentation generation. ```lua local contract = require("contract") local sender, err = contract.get("app.email:sender") if err then return nil, err end local email, err = sender:open("app.email:sender_impl") if err then return nil, err end local result, err = email:send({to = "user@example.com", subject = "Hello"}) if err then return nil, err end return result ``` Contracts allow callers to use an interface while selecting an implementation separately. This supports testing, multi-tenant deployments, and gradual migrations. ### Functions vs Processes Functions inherit the caller's context and lifecycle. When the caller is canceled, its function calls are canceled as well. This suits execution within HTTP handlers and queue consumers. Processes run independently with host context. They outlive their creator and communicate through messages. Use processes for background work; use functions for request-scoped operations. --- # "Workflows" ## Workflows Workflows persist the state of long-running operations so execution can recover after crashes and restarts. They suit processes such as payments, order fulfillment, and multi-step approvals. ### Why Use Workflows Functions keep in-flight state in memory, while workflows persist execution state: | Aspect | Functions | Workflows | |--------|-----------|-----------| | State | Call-local | Rebuilt from persisted history | | Worker crash | In-flight call fails | Replays from recorded history | | Duration | Seconds to minutes | Hours to months | | Application failure | Returned to caller | Ends or retries according to provider policy | ### How Workflows Work Workflow code looks like regular Lua code: ```lua local funcs = require("funcs") local time = require("time") local result, err = funcs.call("app.api:charge_card", payment) if err then return nil, err end time.sleep("24h") local status, err = funcs.call("app.api:check_status", result.id) if err then return nil, err end if status == "failed" then local _, refund_err = funcs.call("app.api:refund", result.id) if refund_err then return nil, refund_err end end return status ``` The workflow engine intercepts calls and records their results. After a crash, it replays execution from the recorded history. Inside a workflow, each `funcs.call()` target runs as a Temporal activity. A target `function.*` entry must register with a worker through `meta.temporal.activity.worker`; unregistered entries are not available to the workflow. A `process.*` activity target additionally needs `meta.options.default_host` (or the legacy `meta.default_host`) so it is registered in the function registry used by the Temporal worker. See [Activities](../temporal/activities.md) for the function activity example and activity options. Workflow authors must still write deterministic code. Wippy limits workflow modules to those classified as Deterministic or Workflow and supplies replay-safe implementations for supported operations. funcs.call() runs a recorded activity, time.sleep() uses a workflow timer, uuid.v4() records a side effect, and time.now() reads the workflow's deterministic time reference. #### Saga Pattern Compensate on failure: ```lua local funcs = require("funcs") local inventory, err = funcs.call("app.inventory:reserve", items) if err then return nil, err end local payment, err = funcs.call("app.payments:charge", amount) if err then local _, compensation_err = funcs.call("app.inventory:release", inventory.id) return nil, compensation_err or err end local shipping, err = funcs.call("app.shipping:create", order) if err then local _, refund_err = funcs.call("app.payments:refund", payment.id) local _, release_err = funcs.call("app.inventory:release", inventory.id) return nil, refund_err or release_err or err end return {inventory = inventory, payment = payment, shipping = shipping} ``` #### Waiting for Signals Wait for external events (approval decisions, webhooks, user actions): ```lua local funcs = require("funcs") local _, err = funcs.call("app.approvals:submit", request) if err then return nil, err end local inbox = process.inbox() local msg, open = inbox:receive() -- blocks until signal arrives if not open then return nil, errors.new("workflow inbox closed") end local decision, payload_err = msg:payload():data() if payload_err then return nil, payload_err end if decision.approved then return funcs.call("app.orders:fulfill", request.order_id) else return funcs.call("app.notifications:send_rejection", request) end ``` ### Choosing a Compute Model | Use Case | Choose | |----------|--------| | HTTP request handling | Functions | | Data transformation | Functions | | Background jobs | Processes | | User session state | Processes | | Real-time messaging | Processes | | Payment processing | Workflows | | Order fulfillment | Workflows | | Multi-day approvals | Workflows | ### Starting Workflows Workflows use `process.spawn()` with a workflow host: ```lua -- Spawn workflow on temporal worker local pid, err = process.spawn("app.workflows:order_processor", "app:temporal_worker", order_data) if err then return nil, err end -- Send signals to workflow local ok, err = process.send(pid, "update", {status = "approved"}) if err then return nil, err end return ok ``` The caller uses the same spawn API. The host determines whether the entry runs on a `temporal.worker` or a `process.host`. Persisted history and replay apply only to the Temporal-hosted path. A workflow entry run through an ordinary process host has in-memory process semantics and does not gain Temporal durability. When a workflow spawns children via process.spawn(), they become child workflows on the same provider, maintaining durability guarantees. ### Failure and Supervision Processes can run as supervised services using `process.service`: ```yaml ## Process definition - name: session_handler kind: process.lua source: file://session_handler.lua method: main ## Supervised service wrapping the process - name: session_manager kind: process.service process: app:session_handler host: app:processes lifecycle: auto_start: true restart: max_attempts: 10 ``` Workflows do not use process supervision trees. The workflow provider manages persistence and recovery; application-level retries follow the configured workflow and activity policies. ### Configuration Workflow definition (spawned dynamically): ```yaml - name: order_processor kind: workflow.lua source: file://order_processor.lua method: main meta: temporal: workflow: worker: app:temporal_worker modules: - funcs - time ``` Every function or process invoked through `funcs.call()` also declares the activity worker. For example: ```yaml - name: charge_card kind: function.lua source: file://charge_card.lua method: main meta: temporal: activity: worker: app:temporal_worker ``` Workflow provider: ```yaml - name: temporal_worker kind: temporal.worker client: app:temporal_client task_queue: "orders" lifecycle: auto_start: true ``` See [Temporal](https://temporal.io) for production workflow infrastructure. ### See Also - [Functions](concepts/functions.md) — Request-scoped calls - [Process Model](concepts/process-model.md) — Stateful background work - [Supervision](guides/supervision.md) — Process restart policies --- # "Lua Runtime" ## Lua Runtime Lua is Wippy's primary runtime for I/O-bound work and business logic. Code runs in isolated processes that communicate through message passing rather than shared memory. This page is a conceptual overview. Its code blocks are isolated reference snippets; names such as `inbox`, `events`, and `handle_message` stand for values or callbacks supplied by the surrounding application. For the design tradeoffs behind Lua and its relationship to WebAssembly, see [Why Wippy Uses Lua](why-lua.md). ### Processes Lua code runs inside **processes**: isolated execution contexts managed by the scheduler. Each process: - has its own memory space; - yields during blocking operations such as I/O and channel access; - can be monitored and supervised; and - can run alongside thousands of other processes on one machine. ```lua local pid, err = process.spawn("app.workers:handler", "app:processes") if err then return nil, err end local sent, send_err = process.send(pid, "task", {data = "work"}) if send_err then return nil, send_err end ``` Executable Lua entries receive `process` as an ambient global. It can also be loaded with `require("process")` without adding it to the entry's `modules` list. See [Process Management](lua/core/process.md) for spawning, linking, and supervision. ### Channels Channels provide communication between concurrent tasks: ```lua local sync_ch = channel.new() -- unbuffered local buffered = channel.new(10) buffered:send("work") -- completes while buffer space is available local val, ok = buffered:receive() -- val is "work" and ok is true ``` See [Channels](lua/core/channel.md) for select and patterns. ### Coroutines Within a process, use lightweight coroutines for concurrent work: ```lua coroutine.spawn(function() local data = fetch_data() ch:send(data) end) do_other_work() -- continues immediately ``` The scheduler manages spawned coroutines, so callers do not manually yield or resume them. ### Select Use `channel.select` to wait for multiple event sources: ```lua local r = channel.select { inbox:case_receive(), events:case_receive(), timeout:case_receive() } if r.channel == timeout then -- timed out elseif r.channel == events then handle_event(r.value) else handle_message(r.value) end ``` ### Globals The following globals are available without `require` and do not need to be listed in `modules:`: - `channel` - Go-style channels - `payload` - the entry's input payload - `process` - process spawning, messaging, monitoring, and lifecycle operations - `print`, `subscribe`, `unsubscribe` - logging and pub/sub - `os`, `table`, `math`, `string`, `coroutine`, `errors` - standard libraries ### Modules Built-in runtime modules that are not ambient are loaded with `require()` and must appear in the entry's `modules:` allowlist. Executable entries receive `process` as an ambient global; `require("process")` is also allowed and does not require a `modules:` declaration. ```lua local process = require("process") local json = require("json") local sql = require("sql") local http = require("http_client") ``` Available modules depend on entry configuration. See [Entry Definitions](lua/entries.md). Registry libraries use the same `require("alias")` syntax but are declared separately in the entry's `imports:` map. ### Language and Library Support Wippy uses Lua 5.3 syntax with a [gradual type system](lua/types.md) inspired by Luau. Types are first-class runtime values that can be used for validation, passed as arguments, and inspected at runtime. External Lua libraries (LuaRocks, etc.) are not supported. The runtime provides its own module system with built-in extensions for I/O, networking, and system integration. For custom extensions, see [Modules](internals/modules.md) in the internals documentation. ### Error Handling Functions commonly return `result, error` pairs: ```lua local data, err = json.decode(input) if err then return nil, errors.wrap(err, "decode failed") end ``` This snippet assumes `json` is enabled in the entry's `modules` list and `input` contains the string to decode. See [Error Handling](lua/core/errors.md) for patterns. ### What's Next - [Entry Definitions](lua/entries.md) - Configure entry points - [Channels](lua/core/channel.md) - Channel patterns - [Process Management](lua/core/process.md) - Spawning and supervision - [Functions](lua/core/funcs.md) - Cross-process calls --- # "Why Wippy Uses Lua" ## Why Wippy Uses Lua Wippy uses Lua as its primary runtime language because it fits the platform's process-isolation and embedding requirements. This page explains that design choice and its tradeoffs; it is not a general ranking of programming languages. This is a conceptual design note rather than a runnable tutorial. It describes runtime properties and points to the reference pages that define the concrete APIs. ### Runtime Requirements Wippy runs user-defined logic in isolated processes. Each process has its own memory and receives only the capabilities exposed by the runtime. Because many processes can run concurrently, the embedded language must support: - **Low per-process overhead.** Memory use must remain practical as process counts grow. - **Capability isolation.** The runtime must control the modules, functions, and system operations available to each process. - **In-process embedding.** Wippy's Go core must be able to create, configure, and stop a language environment for each process. - **Controlled module loading.** Modules must come from the runtime's allowlist or declared registry imports rather than arbitrary file-system paths. - **A small language surface.** Application code should remain readable and straightforward to generate, review, and lint. #### Python Python offers a large application and data ecosystem, but its interpreter, import model, and package assumptions do not match Wippy's per-process embedding and capability model. Python services can still integrate with Wippy over explicit service boundaries. #### JavaScript JavaScript runtimes offer several embedding options. Their module and package ecosystems, however, require a separate integration layer to provide the registry-scoped loading model Wippy uses. Wippy chose Lua's smaller host-controlled runtime surface for application code. #### Go Go is used for Wippy's core runtime. Compiled Go code and plugins do not provide the same isolated, per-process embedded environment required for user-defined application logic. #### WebAssembly WebAssembly fills a complementary role rather than replacing Lua as the primary authoring language. Its division of responsibilities is described in [Lua and WebAssembly](#lua-and-webassembly). #### Host-Controlled Embedding Lua is designed to run inside a host application. Wippy creates an environment for each process, connects it to the scheduler and registry, and controls its globals and module loader. `require` reads only modules already installed in that environment: the always-available base modules and standard libraries, the executable entry's ambient `process` module, built-in runtime modules allowed by `modules:`, and registry libraries declared through `imports:`. It does not search file-system paths or install packages from the network. Different entries can therefore receive different module sets without application-level loading rules. #### Language Surface Lua has a compact syntax and a small standard environment. Wippy adds type annotations and linting so code can be checked incrementally without changing the underlying execution model. #### Cooperative Scheduling Lua coroutines map to Wippy's cooperative scheduling model. A process can yield during channel or I/O operations while the scheduler runs other work. ### Tradeoffs Lua does not provide an in-process package ecosystem comparable to pip or npm. Wippy supplies built-in runtime modules through an allowlist and application libraries through registry imports rather than installing packages from the network. Workloads that depend on large external libraries can run as services or as WebAssembly components. Lua may also be unfamiliar to developers coming from other languages. The syntax is compact, but teams still need conventions, review, and linting for production code. ### Lua and WebAssembly Wippy provides two complementary runtimes: - **Lua** is the primary runtime for application logic, tools, and agents. - **WebAssembly** runs compiled workloads and existing code that can target WASM. Lua and WASM process entries use Wippy's process model; Lua and WASM functions are exposed through registered function entries. Both integrations are configured through the registry and runtime security policies. Lua code can call registered WASM functions, and WASM processes can call registered Lua functions. ### See Also - [Lua Runtime Overview](lua/overview.md) - The Lua runtime and its modules - [Types](lua/types.md) - Type annotations, generics, and unions - [Linter](guides/linter.md) - Static analysis for Lua - [WASM Runtime](wasm/overview.md) - Running compiled code in the sandbox --- # "Type System" ## Type System > **Experimental.** The type system is still evolving, and some limitations are expected. Wippy's gradual type system supports incremental annotations and flow-sensitive checking. Types are non-nullable by default. This page is a language reference, not a complete program. Each code block is an isolated type-checking example, and alternatives within a block are not necessarily meant to be combined. Names such as `get_data`, `get_user`, `call`, and `User` represent application code; lines marked `ERROR` intentionally demonstrate diagnostics. These examples use language syntax and built-in type values, so they do not require runtime modules. ### Primitives ```lua local n: number = 3.14 local i: integer = 42 -- integer is subtype of number local s: string = "hello" local b: boolean = true local a: any = "anything" -- dynamic member and method access local u: unknown = { source = "example" } -- must narrow before use ``` #### `any` and `unknown` ```lua -- any: dynamic member and method access local a: any = get_data() a.foo.bar.baz() -- no error, may crash at runtime local s: string = a -- ERROR: any is not assignable to string -- unknown: safe unknown, must narrow before use as a concrete type local u: unknown = get_data() u.foo -- no error: member access on unknown behaves like any local n: number = u -- ERROR: unknown not assignable to number, narrow first if type(u) == "table" then -- u narrowed to table here end ``` ### Nil Safety Types are non-nullable by default. Use `?` for optional values: ```lua local x: number = nil -- ERROR: nil not assignable to number local y: number? = nil -- OK: number? means "number or nil" local z: number? = 42 -- OK ``` #### Control Flow Narrowing The type checker tracks control flow: ```lua local function process(x: number?): number if x ~= nil then return x -- x is number here end return 0 end -- Early return pattern local user, err = get_user(123) if err then return nil, err end -- user narrowed to non-nil here -- Or default local val = get_value() or 0 -- val: number ``` ### Union Types ```lua local val: number | string = get_value() if type(val) == "number" then print(val + 1) -- val: number else print(val:upper()) -- val: string end ``` #### Literal Types ```lua type Status = "pending" | "active" | "done" local s: Status = "pending" -- OK local s: Status = "invalid" -- ERROR ``` ### Function Types ```lua local function add(a: number, b: number): number return a + b end -- Multiple returns local function div_mod(a: number, b: number): (number, number) return math.floor(a / b), a % b end -- Error returns (Lua idiom) local function fetch(url: string): (string?, error?) -- returns (data, nil) or (nil, error) end -- First-class function types local double: (number) -> number = function(x: number): number return x * 2 end ``` #### Variadic Functions ```lua local function sum(...: number): number local total: number = 0 for _, v in ipairs({...}) do total = total + v end return total end ``` ### Record Types ```lua type User = {name: string, age: number} local u: User = {name = "alice", age = 25} ``` #### Optional Fields ```lua type Config = { host: string, port: number, timeout?: number, debug?: boolean } local cfg: Config = {host = "localhost", port = 8080} -- OK ``` ### Generics ```lua local function identity(x: T): T return x end local n: number = identity(42) local s: string = identity("hello") ``` #### Constrained Generics ```lua type HasName = {name: string} local function greet(obj: T): string return "Hello, " .. obj.name end greet({name = "Alice"}) -- OK greet({age = 30}) -- ERROR: missing 'name' ``` ### Intersection Types Combine multiple types: ```lua type Named = {name: string} type Aged = {age: number} type Person = Named & Aged local p: Person = {name = "Alice", age = 30} ``` ### Tagged Unions ```lua type Result = {ok: true, value: T} | {ok: false, error: E} type LoadState = {status: "loading"} | {status: "loaded", data: User} | {status: "error", message: string} local function render(state: LoadState): string if state.status == "loading" then return "Loading..." elseif state.status == "loaded" then return "Hello, " .. state.data.name elseif state.status == "error" then return "Error: " .. state.message end end ``` ### The `never` Type `never` is the bottom type: it has no possible values. ```lua function fail(msg: string): never error(msg) end ``` ### Error Handling Pattern The checker understands the common Lua `value, error` return pattern: ```lua local value, err = call() if err then -- value is nil here return nil, err end -- value is non-nil here, err is nil print(value) ``` ### Non-Nil Assertion Use `!` to assert an expression is non-nil: ```lua local user: User? = get_user() local name = (user!).name -- assert user is non-nil ``` `!` is a type-checker assertion only - it narrows the type to non-nil but emits no runtime check. If the value is actually nil, the following operation fails with the usual error (e.g. indexing nil). Use when you know a value cannot be nil but the type checker cannot prove it. #### Runtime Validation Call a type as a function to validate a value. Validation returns the original value with the requested static type; it does not convert or coerce the value: ```lua local data: any = get_json() local user = User(data) -- validates and returns User local name = user.name -- safe field access ``` This works with primitives and custom types: ```lua local x: any = get_value() local s = string(x) -- requires an existing string local n = integer(x) -- requires an existing integer local b = boolean(x) -- requires an existing boolean type Point = {x: number, y: number} local p = Point(data) -- validates record structure ``` For example, `string(42)` raises a validation error; use `tostring(42)` when conversion is intended. #### Type:is() Method `Type:is` validates without throwing and returns either `(value, nil)` or `(nil, error)`: ```lua type Point = {x: number, y: number} local data: any = get_input() local p, err = Point:is(data) if p then local sum = p.x + p.y -- p is valid Point else return nil, err -- validation failed end ``` The result narrows in conditionals: ```lua if Point:is(data) then local p: Point = data -- data narrowed to Point end ``` #### Unsafe Cast Use `::` or `as` for unchecked casts: ```lua local data: any = get_data() local user = data :: User -- no runtime check local user = data as User -- same as :: ``` Use sparingly. Unsafe casts bypass validation and can cause runtime errors if the value doesn't match the type. ### Type Reflection Types are first-class values that provide introspection methods. #### Kind and Name ```lua type Num = number print(Num:kind()) -- "number" print(Point:kind()) -- "record" print(Point:name()) -- "Point" ``` #### Record Fields Iterate over record fields: ```lua type User = {name: string, age: number} for name, typ in User:fields() do print(name, typ:kind()) end -- name string -- age number ``` Access individual field types: ```lua local nameType = User.name -- type of 'name' field print(nameType:kind()) -- "string" ``` #### Collection Types ```lua type NumberList = {number} print(NumberList:elem():kind()) -- "number" type ScoreMap = {[string]: number} print(ScoreMap:key():kind()) -- "string" print(ScoreMap:val():kind()) -- "number" ``` #### Optional Types ```lua type MaybeNumber = number? print(MaybeNumber:kind()) -- "optional" print(MaybeNumber:inner():kind()) -- "number" ``` #### Union Types ```lua type Status = "pending" | "active" | "done" for variant in Status:variants() do print(variant) end ``` #### Function Types ```lua type Predicate = (number, string) -> boolean for param in Predicate:params() do print(param:kind()) end print(Predicate:ret():kind()) -- "boolean" ``` `typeof(expression)` is type syntax, not a runtime reflection function. Use it in an alias such as `type Config = typeof(default_config)`; the resulting alias is the runtime type value. #### Type Comparison ```lua type Num = number type Int = integer print(Num == Num) -- true print(Int <= Num) -- true (subtype) print(Int < Num) -- true (strict subtype) ``` #### Types as Table Keys ```lua type Point = {x: number, y: number} type Line = {from: Point, to: Point} local handlers = {} handlers[Point] = function() return "point handler" end handlers[Line] = function() return "line handler" end local h = handlers[Point] if h then h() end ``` ### Type Annotations Add types to function signatures: ```lua -- Parameter and return types local function process(input: string): number return #input end -- Local variable types local count: number = 0 -- Type aliases type StringArray = {string} type StringMap = {[string]: number} ``` ### Type Validators Attach validation constraints to type aliases with annotations, then call the type or use `Type:is()` to enforce them at runtime: ```lua type NonNegative = number @min(0) type Percentage = number @min(0) @max(100) type Email = string @pattern("^.+@.+$") local x = NonNegative(1) local percent, err = Percentage:is(50) local email = Email("test@example.com") ``` An annotation on a local variable is checked statically by the linter. It does not insert an automatic runtime check at assignment; runtime enforcement occurs when a type value validates a value. #### Built-in Validators | Validator | Applies to | Example | |-----------|------------|---------| | `@min(n)` | number | `type Positive = number @min(1)` | | `@max(n)` | number | `type Percentage = number @max(100)` | | `@min_len(n)` | string, array | `type NonEmpty = string @min_len(1)` | | `@max_len(n)` | string, array | `type ShortName = string @max_len(10)` | | `@pattern(regex)` | string | `type Email = string @pattern("^.+@.+$")` | #### Record Field Validators ```lua type User = { age: number @min(0) @max(150), name: string @min_len(1) @max_len(100) } ``` #### Array Element Validators ```lua local scores: {number @min(0) @max(100)} = {85, 90} ``` #### Union Member Validators ```lua local id: number @min(1) | string @min_len(1) = 1 ``` ### Variance Rules | Position | Variance | Description | |----------|----------|-------------| | Readonly field | Covariant | Can use subtype | | Mutable field | Quasi-invariant | Normally invariant; fresh literals and refinements may widen to their base type | | Function parameter | Contravariant | Can use supertype | | Function return | Covariant | Can use subtype | ### Subtyping - `integer` is a subtype of `number` - `never` is a subtype of all types - All types are subtypes of `any` - Union subtyping: `A` is subtype of `A | B` ### Gradual Adoption Types can be added incrementally; untyped code continues to work: ```lua -- Existing code works unchanged function old_function(x) return x + 1 end -- New code gets types function new_function(x: number): number return x + 1 end ``` Useful starting points include: 1. Function signatures at API boundaries 2. HTTP handlers and queue consumers 3. Critical business logic ### Type Checking Run the type checker with: ```bash wippy lint ``` The command reports type errors without executing the code. --- # "Lua Entry Kinds" ## Lua Entry Kinds Lua entry kinds define how source code is loaded and executed as a function, process, workflow, or library. This page is a configuration reference. YAML blocks are partial entry definitions intended to be placed under an `entries:` mapping in a Wippy index; they are not complete applications by themselves. Referenced source files, imports, dependencies, process hosts, and security policies must exist in the surrounding project. ### Entry Kinds | Kind | Description | |------|-------------| | `function.lua` | Stateless function, runs on demand | | `process.lua` | Long-running actor with state | | `workflow.lua` | Durable workflow (Temporal) | | `library.lua` | Shared code imported by other entries | Each kind has a precompiled bytecode counterpart (`function.lua.bc`, `library.lua.bc`, `process.lua.bc`, `workflow.lua.bc`) produced by `wippy pack --bytecode '**'` (or a pattern like `--bytecode 'app:**'`). Authors write `.lua` entries; the bytecode kinds are emitted when packing with that flag. `module.lua` is reserved for built-in module definitions created by the runtime. It is not an authorable source entry and has no bytecode counterpart. ### Common Fields All Lua entries share these fields: | Field | Required | Description | |-------|----------|-------------| | `name` | yes | Unique name within namespace | | `kind` | yes | One of the Lua kinds above | | `source` | yes | Inline Lua source or a `file://path.lua` reference resolved when the registry is loaded | | `method` | function/process/workflow | Function to export (libraries don't use it) | | `modules` | no | Allowed modules for `require()` | | `imports` | no | Other entries as local modules | | `meta` | no | Searchable metadata | `pool` applies only to `function.lua`. `security` applies to `function.lua` and `process.lua`. ### `function.lua` A `function.lua` entry runs on demand, with each invocation handled independently. ```yaml - name: handler kind: function.lua source: file://handler.lua method: main modules: - http - json ``` Use functions for HTTP handlers, data transformations, and utilities. ### `process.lua` A `process.lua` entry is a long-running actor that maintains state and communicates through messages. ```yaml - name: worker kind: process.lua source: file://worker.lua method: main modules: - sql ``` Choose a process for background workers, service daemons, and stateful actors. To run as a supervised service: ```yaml - name: worker_service kind: process.service process: app:worker host: app:processes lifecycle: auto_start: true restart: max_attempts: 10 ``` ### `workflow.lua` A `workflow.lua` entry defines a durable workflow whose state is persisted to Temporal. ```yaml - name: order_processor kind: workflow.lua source: file://order_workflow.lua method: main modules: - workflow - time ``` Use workflows for multi-step business processes and long-running orchestration. ### `library.lua` A `library.lua` entry provides shared code that other entries can import. ```yaml - name: helpers kind: library.lua source: file://helpers.lua modules: - json - base64 ``` Other entries reference it via `imports`: ```yaml - name: handler kind: function.lua source: file://handler.lua method: main imports: helpers: app.lib:helpers ``` In Lua code: ```lua local helpers = require("helpers") helpers.format_date(timestamp) ``` ### Modules The `modules` field controls which modules can be loaded with `require()`: ```yaml modules: - http - json - sql ``` `channel`, `payload`, `print`, `process`, `subscribe`, and `unsubscribe` are loaded as Lua globals — they don't need to appear in `modules:`. `require("process")` is also allowed without a `modules:` declaration. Only listed built-in modules and aliases declared under `imports` are available. The module allowlist limits access to runtime capabilities, makes dependencies explicit, and restricts workflows to workflow-compatible module classes. See [Lua Runtime](lua/overview.md) for available modules. ### Imports Import other entries as local modules: ```yaml imports: utils: app.lib:utils # require("utils") auth: app.auth:helpers # require("auth") ``` The key becomes the module name in Lua code. The value is the entry ID (`namespace:name`). ### Function Pools Use `pool` to configure how a function entry executes: ```yaml - name: handler kind: function.lua source: file://handler.lua method: main pool: type: adaptive # explicit; omit to use auto-select (lazy) max_size: 16 # cap for elastic growth ``` | Field | Pools | Description | |-------|-------|-------------| | `type` | all | Scheduler implementation (see table below) | | `workers` | static | Worker thread count (falls back to `size`, then 8) | | `size` | static | Worker count when `workers` is unset; with `type` omitted, `size` without `max_size` selects an inline pool | | `buffer` | static | Task queue capacity (default: `workers * 64`) | | `max_size` | lazy, adaptive | Upper bound for elastic growth (default: 16; 100 when `type` is omitted) | | Type | Behavior | |------|----------| | `inline` | Synchronous execution in the caller's goroutine. No isolation between calls. | | `lazy` | Zero idle workers, spawn on demand, tear down when idle. | | `static` | Fixed-size channel-based pool. Predictable under steady load. | | `adaptive` | Auto-scaling pool — grows under load, shrinks when idle. | When `type` is omitted, the pool is auto-selected from the other fields: a lazy pool by default, a static pool if `workers` is set, an inline pool if only `size` is set. ### Metadata Use `meta` to attach searchable routing and discovery fields: ```yaml - name: api_handler kind: function.lua meta: type: handler version: "2.0" tags: [api, users] source: file://api.lua method: handle modules: - http - json - registry ``` Metadata is searchable via the registry: ```lua local registry = require("registry") local handlers, err = registry.find({["meta.type"] = "handler"}) if err then return nil, err end ``` The query returns all matching registry entries. The Lua code belongs to an executable entry whose `modules` list includes `registry`, such as the `api_handler` entry above. ### See Also - [Entry Kinds](guides/entry-kinds.md) - Reference for all entry kinds - [Compute Units](concepts/compute-units.md) - Functions vs processes vs workflows - [Lua Runtime](lua/overview.md) - Available modules --- # "Standard Lua Libraries" ## Standard Lua Libraries These core Lua libraries are available in every executable Lua entry without `require()`. This is an API reference. Signature blocks list available functions, while the longer blocks are isolated examples or partial patterns rather than complete entries. Names such as `check_health` and `process_request` represent application callbacks. #### Type and Conversion ```lua type(value) -- Returns: "nil", "number", "string", "boolean", "table", "function", "thread", "userdata" tonumber(s [,base]) -- Convert to number, optional base (2-36) tostring(value) -- Convert to string, calls __tostring metamethod ``` #### Assertions and Errors ```lua assert(v [,msg]) -- Raises error if v is false/nil, returns v otherwise error(msg [,level]) -- Raises error at specified stack level (default 1) pcall(fn, ...) -- Protected call, returns ok, result_or_error xpcall(fn, errh) -- Protected call with error handler function ``` #### Table Iteration ```lua pairs(t) -- Iterate all key-value pairs ipairs(t) -- Iterate array portion (1, 2, 3, ...) next(t [,index]) -- Get next key-value pair after index ``` #### Metatables ```lua getmetatable(obj) -- Get metatable (or __metatable field if protected) setmetatable(t, mt) -- Set metatable, returns t ``` #### Raw Table Access Bypass metamethods for direct table access: ```lua rawget(t, k) -- Get t[k] without __index rawset(t, k, v) -- Set t[k]=v without __newindex rawequal(a, b) -- Compare without __eq ``` #### Utilities ```lua select(index, ...) -- Return args from index onwards select("#", ...) -- Return number of args unpack(t [,i [,j]]) -- Return t[i] through t[j] as multiple values print(...) -- Print values (uses structured logging in Wippy) ``` #### Global Variables ```lua _G -- The global environment table _VERSION -- Lua version string ``` ### Table Manipulation The `table` library provides in-place array operations, sorting, concatenation, and unpacking: ```lua table.insert(t, [pos,] value) -- Insert value at pos (default: end) table.remove(t [,pos]) -- Remove and return element at pos (default: last) table.concat(t [,sep [,i [,j]]]) -- Concatenate array elements with separator table.sort(t [,comp]) -- Sort in place, comp(a,b) returns true if a < b table.unpack(t [,i [,j]]) -- Unpack table elements as multiple values table.create(narr, nhash) -- Preallocate table with array and hash capacity table.freeze(t) -- Make table immutable, returns t table.isfrozen(t) -- true if table is immutable ``` ```lua local items = {"a", "b", "c"} table.insert(items, "d") -- {"a", "b", "c", "d"} table.insert(items, 2, "x") -- {"a", "x", "b", "c", "d"} table.remove(items, 2) -- {"a", "b", "c", "d"}, returns "x" local csv = table.concat(items, ",") -- "a,b,c,d" table.sort(items, function(a, b) return a > b -- Descending order end) ``` ### String Operations String functions are also available as methods on string values. #### Pattern Matching ```lua string.find(s, pattern [,init [,plain]]) -- Find pattern, returns start, end, captures string.match(s, pattern [,init]) -- Extract matching substring string.gmatch(s, pattern) -- Iterator over all matches string.gsub(s, pattern, repl [,n]) -- Replace matches, returns string, count ``` #### Case Conversion ```lua string.upper(s) -- Convert to uppercase string.lower(s) -- Convert to lowercase ``` #### Substrings and Characters ```lua string.sub(s, i [,j]) -- Substring from i to j (negative indexes from end) string.len(s) -- String length (or use #s) string.byte(s [,i [,j]]) -- Numeric codes of characters string.char(...) -- Create string from character codes string.rep(s, n) -- Repeat string n times string.reverse(s) -- Reverse string ``` #### Formatting ```lua string.format(fmt, ...) -- Printf-style formatting string.pack(fmt, ...) -- Pack values into a binary string string.unpack(fmt, s [,pos]) -- Unpack binary string, returns values and next position string.packsize(fmt) -- Size in bytes of a packed format ``` Format specifiers: `%d` (integer), `%f` (float), `%s` (string), `%q` (quoted), `%x` (hex), `%o` (octal), `%e` (scientific), `%%` (literal %) ```lua local s = "Hello, World!" -- Pattern matching local start, stop = string.find(s, "World") -- 8, 12 local word = string.match(s, "%w+") -- "Hello" -- Substitution local new = string.gsub(s, "World", "Wippy") -- "Hello, Wippy!" -- Method syntax local upper = s:upper() -- "HELLO, WORLD!" local part = s:sub(1, 5) -- "Hello" ``` #### Patterns | Pattern | Matches | |---------|---------| | `.` | Any character | | `%a` | Letters | | `%d` | Digits | | `%w` | Alphanumeric | | `%s` | Whitespace | | `%p` | Punctuation | | `%c` | Control characters | | `%x` | Hexadecimal digits | | `%z` | Zero (null) | | `[set]` | Character class | | `[^set]` | Negated class | | `*` | 0 or more (greedy) | | `+` | 1 or more (greedy) | | `-` | 0 or more (lazy) | | `?` | 0 or 1 | | `^` | Start of string | | `$` | End of string | | `%b()` | Balanced pair | | `(...)` | Capture group | Uppercase versions (`%A`, `%D`, etc.) match the complement. ### Math Functions The `math` library provides numeric constants and common mathematical operations. #### Constants {id="math-constants"} ```lua math.pi -- 3.14159... math.huge -- Largest representable float math.mininteger -- Minimum integer math.maxinteger -- Maximum integer ``` #### Basic Operations ```lua math.abs(x) -- Absolute value math.min(...) -- Minimum of arguments math.max(...) -- Maximum of arguments math.floor(x) -- Round down math.ceil(x) -- Round up math.modf(x) -- Integer and fractional parts math.fmod(x, y) -- Floating-point remainder ``` #### Powers and Roots ```lua math.sqrt(x) -- Square root math.pow(x, y) -- x^y (or use x^y operator) math.exp(x) -- e^x math.log(x) -- Natural log math.log10(x) -- Base-10 log math.frexp(x) -- Mantissa and exponent math.ldexp(m, e) -- m * 2^e ``` #### Trigonometry ```lua math.sin(x) math.cos(x) math.tan(x) -- Radians math.asin(x) math.acos(x) math.atan(x) math.atan2(y, x) -- Arc tangent of y/x math.sinh(x) math.cosh(x) math.tanh(x) -- Hyperbolic math.deg(r) -- Radians to degrees math.rad(d) -- Degrees to radians ``` #### Random Numbers ```lua math.random() -- Random float [0,1) math.random(n) -- Random integer [1,n] math.random(m, n) -- Random integer [m,n] math.randomseed(x) -- No effect; the generator is auto-seeded ``` `math.random` is nondeterministic. Do not use it for decisions that must replay identically in a workflow; `math.randomseed` cannot make it deterministic. #### Type Conversion ```lua math.tointeger(x) -- Convert to integer or nil math.type(x) -- "integer", "float", or nil math.ult(m, n) -- Unsigned less-than comparison ``` ### Coroutines The `coroutine` library provides coroutine creation and control. See [Channels and Coroutines](lua/core/channel.md) for channel-based concurrency patterns. ```lua coroutine.create(fn) -- Create coroutine from function coroutine.resume(co, ...) -- Start/continue coroutine coroutine.yield(...) -- Suspend coroutine, return values to resume coroutine.status(co) -- "running", "suspended", "normal", "dead" coroutine.running() -- Current coroutine (nil if main thread) coroutine.wrap(fn) -- Create coroutine as callable function ``` #### Spawning Concurrent Coroutines Wippy adds `coroutine.spawn` for scheduler-managed concurrent work: ```lua coroutine.spawn(fn) -- Spawn function as concurrent coroutine ``` ```lua local time = require("time") -- Spawn background task coroutine.spawn(function() while true do check_health() time.sleep("30s") end end) -- Continue main execution immediately process_request() ``` This partial pattern assumes the entry lists `time` in `modules:` and provides the `check_health` and `process_request` functions. The spawned coroutine runs concurrently in the same Lua process; `process_request()` is reached immediately, and each health check is followed by a 30-second sleep. ### Error Handling The global `errors` table creates and classifies structured errors. See [Error Handling](lua/core/errors.md) for the complete API. #### Constants {id="error-constants"} ```lua errors.UNKNOWN -- Unclassified error errors.INVALID -- Invalid argument or input errors.NOT_FOUND -- Resource not found errors.ALREADY_EXISTS -- Resource already exists errors.PERMISSION_DENIED -- Permission denied errors.TIMEOUT -- Operation timed out errors.CANCELED -- Operation cancelled errors.UNAVAILABLE -- Service unavailable errors.INTERNAL -- Internal error errors.CONFLICT -- Conflict (e.g., concurrent modification) errors.RATE_LIMITED -- Rate limit exceeded ``` #### Functions {id="error-functions"} ```lua -- Create error from string local err = errors.new("something went wrong") -- Create error with metadata local err = errors.new({ message = "User not found", kind = errors.NOT_FOUND, retryable = false, details = {user_id = 123} }) -- Wrap existing error with context local wrapped = errors.wrap(err, "failed to load profile") -- Check error kind if errors.is(err, errors.NOT_FOUND) then -- handle not found end -- Get call stack from error local stack = errors.call_stack(err) ``` #### Error Methods ```lua err:message() -- Get error message string err:kind() -- Get error kind (e.g., "NOT_FOUND") err:retryable() -- true, false, or nil (unknown) err:details() -- Get details table or nil err:stack() -- Get stack trace as string ``` ### Restricted Features The following standard Lua features are unavailable in Wippy processes: | Feature | Alternative | |---------|-------------| | `load`, `loadstring`, `loadfile`, `dofile` | Use [Dynamic Evaluation](lua/dynamic/eval.md) module | | `collectgarbage` | Automatic GC | | `rawlen` | Use `#` operator | | Standard `io.*` file library | Use [File System](lua/storage/filesystem.md) module; the `io` module in Wippy is [Terminal I/O](lua/system/io.md) | | `os.execute`, `os.exit`, `os.getenv`, `os.remove`, `os.rename`, `os.tmpname` | Use [Command Execution](lua/dynamic/exec.md), [Environment](lua/system/env.md) modules | | `string.dump` | Not available | | `debug.*` | Not available | | `utf8.*` | Not available | | `package.loadlib` | Native libraries not supported | ### See Also - [Channels and Coroutines](lua/core/channel.md) - Go-style channels for concurrency - [Error Handling](lua/core/errors.md) - Creating and handling structured errors - [OS Time](lua/system/ostime.md) - System time functions --- # "Errors" ## Errors The global `errors` table creates and inspects structured errors with categories, details, and retry metadata. It is available without `require`. This is an API reference. Each code block is an isolated snippet, not a complete entry. Variables such as `err` refer to an error returned or created by surrounding application code; the wrapping example assumes `db` is an application-provided database client. ### Creating Errors ```lua -- Simple message (kind defaults to UNKNOWN) local err = errors.new("something went wrong") -- With kind, retryable, and details local err = errors.new({ message = "user not found", kind = errors.NOT_FOUND, retryable = false, details = {user_id = 123} }) ``` `errors.new` accepts either a string message or a table with at least a `message` field. The `(kind, message)` form is not supported. ### Wrapping Errors Wrap an error to add context while preserving its kind, retry metadata, and details: ```lua local data, err = db:query("SELECT * FROM users") if err then return nil, errors.wrap(err, "failed to load users") end ``` ### Error Methods | Method | Returns | Description | |--------|---------|-------------| | `err:kind()` | string | Error category | | `err:message()` | string | Error message | | `err:retryable()` | boolean/nil | Whether operation can be retried | | `err:details()` | table/nil | Structured metadata | | `err:stack()` | string | Lua stack trace | | `tostring(err)` | string | Full representation | ### Checking Kind ```lua if errors.is(err, errors.INVALID) then -- handle invalid input end -- Or compare directly if err:kind() == errors.NOT_FOUND then -- handle missing resource end ``` ### Error Kinds | Constant | Use Case | |----------|----------| | `errors.NOT_FOUND` | Resource doesn't exist | | `errors.ALREADY_EXISTS` | Resource already exists | | `errors.INVALID` | Bad input or arguments | | `errors.PERMISSION_DENIED` | Access denied | | `errors.UNAVAILABLE` | Service temporarily down | | `errors.INTERNAL` | Internal error | | `errors.CANCELED` | Operation was canceled | | `errors.CONFLICT` | Resource state conflict | | `errors.TIMEOUT` | Operation timed out | | `errors.RATE_LIMITED` | Too many requests | | `errors.UNKNOWN` | Unspecified error | ### Call Stack Use `errors.call_stack` to inspect a structured call stack: ```lua local stack = errors.call_stack(err) if stack then print("Thread:", stack.thread) for _, frame in ipairs(stack.frames) do print(frame.source .. ":" .. frame.line, frame.name) end end ``` ### Retryable Errors Retryability is error metadata, not a property guaranteed by an error kind. Check the value returned by `err:retryable()` rather than inferring it from `err:kind()`. A result of `nil` means the error does not specify whether retrying is appropriate. ```lua if err:retryable() then -- safe to retry end ``` ### Error Details ```lua local err = errors.new({ message = "validation failed", kind = errors.INVALID, details = { errors = { {field = "email", message = "invalid format"}, {field = "age", message = "must be positive"} } } }) local details = err:details() for _, e in ipairs(details.errors) do print(e.field, e.message) end ``` --- # "Time & Duration" ## Time & Duration The `time` module provides time values, durations, time-zone handling, parsing, formatting, sleeps, and timers. Supported workflow time calls are recorded so they can replay deterministically. This is an API reference. Code blocks are isolated examples or partial scheduling patterns, not a complete entry. Names such as `do_work`, `try_operation`, `make_request`, `send_reminder`, `user_activity`, `check_health`, and `process` represent application callbacks, channels, or data. Where a snippet assigns an error return to `_`, it assumes the shown literal is valid; handle errors when values can come from input or configuration. ### Loading ```lua local time = require("time") ``` Add `time` to the executable entry's `modules:` list before requiring it. The ambient `channel` and `errors` globals used by scheduling examples need no module declaration. #### `now` Returns the current time. In workflows, it returns the recorded workflow time reference so execution can replay deterministically. ```lua local t = time.now() print(t:format_rfc3339()) -- "2024-12-29T15:04:05Z" -- Measure elapsed time local start = time.now() do_work() local elapsed = time.now():sub(start) print("Took " .. elapsed:milliseconds() .. "ms") ``` The timestamp and elapsed-time output are illustrative; `time.now()` supplies the current or recorded workflow time. **Returns:** `Time` #### Create from Components ```lua -- Create specific date/time in UTC local t = time.date(2024, time.DECEMBER, 25, 10, 30, 0, 0, time.utc) print(t:format_rfc3339()) -- "2024-12-25T10:30:00Z" -- Create in specific timezone local ny, err = time.load_location("America/New_York") if err then return nil, err end local meeting = time.date(2024, time.JANUARY, 15, 14, 0, 0, 0, ny) -- Defaults to local timezone if not specified local t = time.date(2024, 1, 15, 12, 0, 0, 0) ``` | Parameter | Type | Description | |-----------|------|-------------| | `year` | number | Year | | `month` | number | Month (1-12 or `time.JANUARY` etc) | | `day` | number | Day of month | | `hour` | number | Hour (0-23) | | `minute` | number | Minute (0-59) | | `second` | number | Second (0-59) | | `nanosecond` | number | Nanosecond (0-999999999) | | `location` | Location | Timezone (optional, defaults to local) | **Returns:** `Time` #### Create from a Unix Timestamp ```lua -- From seconds since epoch local t = time.unix(1703862245, 0) print(t:utc():format_rfc3339()) -- "2023-12-29T15:04:05Z" -- With nanoseconds local t = time.unix(1703862245, 500000000) -- +500ms -- Convert JavaScript timestamp (milliseconds) local js_timestamp = 1703862245000 local t = time.unix(js_timestamp // 1000, (js_timestamp % 1000) * 1000000) ``` | Parameter | Type | Description | |-----------|------|-------------| | `sec` | number | Unix seconds | | `nsec` | number | Nanoseconds offset | **Returns:** `Time` #### Parse from a String Parse time strings using Go's reference time format: `Mon Jan 2 15:04:05 MST 2006`. ```lua -- Parse RFC3339 local t, err = time.parse(time.RFC3339, "2024-12-29T15:04:05Z") if err then return nil, err end -- Parse custom format local t, err = time.parse("2006-01-02", "2024-12-29") local t, err = time.parse("15:04:05", "14:30:00") local t, err = time.parse("2006-01-02 15:04:05 MST", "2024-12-29 14:30:00 EST") -- Parse in specific timezone local ny, _ = time.load_location("America/New_York") local t, err = time.parse("2006-01-02 15:04", "2024-12-29 14:30", ny) ``` | Parameter | Type | Description | |-----------|------|-------------| | `layout` | string | Go time format layout | | `value` | string | String to parse | | `location` | Location | Default time zone (optional) | **Returns:** `Time, error` #### Arithmetic ```lua local t = time.now() -- Add duration (accepts number, string, or Duration) local tomorrow = t:add("24h") local later = t:add(5 * time.MINUTE) local d, _ = time.parse_duration("1h30m") local future = t:add(d) -- Subtract time to get duration local diff = tomorrow:sub(t) -- returns Duration print(diff:hours()) -- 24 -- Add calendar units (handles month boundaries correctly) local next_month = t:add_date(0, 1, 0) -- add 1 month local next_year = t:add_date(1, 0, 0) -- add 1 year local last_week = t:add_date(0, 0, -7) -- subtract 7 days ``` | Method | Parameters | Returns | Description | |--------|------------|---------|-------------| | `add(duration)` | number/string/Duration | Time | Add duration | | `sub(time)` | Time | Duration | Difference between times | | `add_date(years, months, days)` | numbers | Time | Add calendar units | #### Comparison ```lua local t1 = time.date(2024, 1, 1, 0, 0, 0, 0, time.utc) local t2 = time.date(2024, 1, 2, 0, 0, 0, 0, time.utc) t1:before(t2) -- true t2:after(t1) -- true t1:equal(t1) -- true ``` | Method | Parameters | Returns | Description | |--------|------------|---------|-------------| | `before(time)` | Time | boolean | Whether this time is before the other value | | `after(time)` | Time | boolean | Whether this time is after the other value | | `equal(time)` | Time | boolean | Whether the two values represent the same time | #### Formatting ```lua local t = time.now() t:format_rfc3339() -- "2024-12-29T15:04:05Z" t:format(time.DATE_ONLY) -- "2024-12-29" t:format(time.TIME_ONLY) -- "15:04:05" t:format("Mon Jan 2, 2006") -- "Sun Dec 29, 2024" ``` | Method | Parameters | Returns | Description | |--------|------------|---------|-------------| | `format(layout)` | string | string | Format using Go layout | | `format_rfc3339()` | - | string | Format as RFC3339 | #### Unix Timestamps ```lua local t = time.now() t:unix() -- seconds since epoch t:unix_nano() -- nanoseconds since epoch ``` #### Components ```lua local t = time.now() -- Get date parts local year, month, day = t:date() -- Get time parts local hour, min, sec = t:clock() -- Individual accessors t:year() -- e.g., 2024 t:month() -- 1-12 t:day() -- 1-31 t:hour() -- 0-23 t:minute() -- 0-59 t:second() -- 0-59 t:nanosecond() -- 0-999999999 t:weekday() -- 0=Sunday .. 6=Saturday t:year_day() -- 1-366 t:is_zero() -- true if zero value ``` #### Time-Zone Conversion ```lua local t = time.now() t:utc() -- convert to UTC t:in_local() -- convert to local timezone t:in_location(ny) -- convert to specific timezone t:location() -- get current Location t:location():string() -- get timezone name ``` | Method | Parameters | Returns | Description | |--------|------------|---------|-------------| | `utc()` | - | Time | Convert to UTC | | `in_local()` | - | Time | Convert to the local time zone | | `in_location(loc)` | Location | Time | Convert to a specified time zone | | `location()` | - | Location | Return the current time zone | #### Rounding Round or truncate to duration boundaries. **Requires Duration userdata** (not number or string). ```lua local t = time.now() local hour_duration, _ = time.parse_duration("1h") local minute_duration, _ = time.parse_duration("15m") t:round(hour_duration) -- round to nearest hour t:truncate(minute_duration) -- truncate to 15-minute boundary ``` | Method | Parameters | Returns | Description | |--------|------------|---------|-------------| | `round(duration)` | Duration | Time | Round to nearest multiple | | `truncate(duration)` | Duration | Time | Truncate to multiple | #### Create a Duration ```lua -- Parse from string local d, err = time.parse_duration("1h30m45s") local d, err = time.parse_duration("500ms") local d, err = time.parse_duration("2h30m45s500ms") -- From number (nanoseconds) local d, err = time.parse_duration(time.SECOND) local d, err = time.parse_duration(5 * time.MINUTE) -- Valid units: ns, us, ms, s, m, h ``` | Parameter | Type | Description | |-----------|------|-------------| | `value` | number/string/Duration | Duration to parse | **Returns:** `Duration, error` #### Duration Methods ```lua local d, _ = time.parse_duration("1h30m45s500ms") d:hours() -- 1.5126... d:minutes() -- 90.75... d:seconds() -- 5445.5 d:milliseconds() -- 5445500 d:microseconds() -- 5445500000 d:nanoseconds() -- 5445500000000 ``` #### Named Locations Load a time zone by its IANA name, such as `America/New_York`, `Europe/London`, or `Asia/Tokyo`. ```lua local ny, err = time.load_location("America/New_York") if err then return nil, err end local tokyo, _ = time.load_location("Asia/Tokyo") local london, _ = time.load_location("Europe/London") -- Convert between timezones local t = time.now():utc() print("UTC:", t:format(time.TIME_ONLY)) print("New York:", t:in_location(ny):format(time.TIME_ONLY)) print("Tokyo:", t:in_location(tokyo):format(time.TIME_ONLY)) ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | IANA time-zone name | **Returns:** `Location, error` #### Fixed-Offset Locations Create a time zone with a fixed UTC offset. ```lua -- UTC+5:30 (India Standard Time) local ist = time.fixed_zone("IST", 5*3600 + 30*60) -- UTC-8 (Pacific Standard Time) local pst = time.fixed_zone("PST", -8*3600) local t = time.date(2024, 1, 15, 12, 0, 0, 0, ist) ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Zone name | | `offset` | number | UTC offset in seconds | **Returns:** `Location` #### Built-In Locations ```lua time.utc -- UTC timezone time.localtz -- Local system timezone ``` #### `sleep` Suspend execution for the specified duration. Workflow execution records the sleep for deterministic replay. ```lua time.sleep("5s") time.sleep(500 * time.MILLISECOND) -- Backoff pattern for attempt = 1, 3 do local ok = try_operation() if ok then break end time.sleep(tostring(attempt) .. "s") end ``` | Parameter | Type | Description | |-----------|------|-------------| | `duration` | number/string/Duration | Sleep time | #### `after` Returns a channel that receives one value after the duration. The channel can be used with `channel.select`. ```lua -- Simple timeout local timeout, err = time.after("5s") if err then return nil, err end timeout:receive() -- blocks for 5 seconds -- Timeout with select local response_ch = make_request() local timeout_ch, err = time.after("30s") if err then return nil, err end local result = channel.select{ response_ch:case_receive(), timeout_ch:case_receive() } if result.channel == timeout_ch then return nil, errors.new({message = "Request timed out", kind = errors.TIMEOUT}) end ``` | Parameter | Type | Description | |-----------|------|-------------| | `duration` | number/string/Duration | Time to wait | **Returns:** `Channel, error` #### `timer` Creates a one-shot timer that fires after the specified duration and can be stopped or reset. ```lua local timer, err = time.timer("5s") if err then return nil, err end -- Wait for timer timer:response():receive() send_reminder() -- Reset on activity local idle_timer, err = time.timer("5m") if err then return nil, err end local idle_ch = idle_timer:response() while true do local r = channel.select{ user_activity:case_receive(), idle_ch:case_receive() } if r.channel == idle_ch then logout_user() break end idle_timer:reset("5m") end -- Stop timer timer:stop() ``` | Parameter | Type | Description | |-----------|------|-------------| | `duration` | number/string/Duration | Time until fire | **Returns:** `Timer, error` | Timer Method | Parameters | Returns | Description | |--------------|------------|---------|-------------| | `response()` | - | Channel | Get timer channel | | `channel()` | - | Channel | Alias for response() | | `stop()` | - | boolean | Cancel timer | | `reset(duration)` | number/string/Duration | boolean | Reset with new duration | #### `ticker` Creates a repeating timer that fires at regular intervals. ```lua -- Periodic task local ticker, err = time.ticker("30s") if err then return nil, err end local ch = ticker:response() while true do local tick_time = ch:receive() check_health() end ``` The loop above is intended for a long-running process. A separate finite rate-limiting pattern is: ```lua -- Rate limiting local ticker, err = time.ticker("100ms") if err then return nil, err end for _, item in ipairs(items) do ticker:response():receive() process(item) end ticker:stop() ``` | Parameter | Type | Description | |-----------|------|-------------| | `duration` | number/string/Duration | Interval between ticks | **Returns:** `Ticker, error` | Ticker Method | Parameters | Returns | Description | |---------------|------------|---------|-------------| | `response()` | - | Channel | Get ticker channel | | `channel()` | - | Channel | Alias for response() | | `stop()` | - | boolean | Stop ticker | #### Duration Units Duration constants are expressed in nanoseconds and can be combined with arithmetic. ```lua time.NANOSECOND -- 1 time.MICROSECOND -- 1,000 time.MILLISECOND -- 1,000,000 time.SECOND -- 1,000,000,000 time.MINUTE -- 60 * SECOND time.HOUR -- 60 * MINUTE -- Example usage time.sleep(5 * time.SECOND) local timeout, err = time.after(30 * time.SECOND) if err then return nil, err end ``` #### Format Layouts ```lua time.RFC3339 -- "2006-01-02T15:04:05Z07:00" time.RFC3339NANO -- "2006-01-02T15:04:05.999999999Z07:00" time.RFC822 -- "02 Jan 06 15:04 MST" time.RFC822Z -- "02 Jan 06 15:04 -0700" time.RFC850 -- "Monday, 02-Jan-06 15:04:05 MST" time.RFC1123 -- "Mon, 02 Jan 2006 15:04:05 MST" time.RFC1123Z -- "Mon, 02 Jan 2006 15:04:05 -0700" time.DATE_TIME -- "2006-01-02 15:04:05" time.DATE_ONLY -- "2006-01-02" time.TIME_ONLY -- "15:04:05" time.KITCHEN -- "3:04PM" time.STAMP -- "Jan _2 15:04:05" time.STAMP_MILLI -- "Jan _2 15:04:05.000" time.STAMP_MICRO -- "Jan _2 15:04:05.000000" time.STAMP_NANO -- "Jan _2 15:04:05.000000000" ``` #### Months ```lua time.JANUARY -- 1 time.FEBRUARY -- 2 time.MARCH -- 3 time.APRIL -- 4 time.MAY -- 5 time.JUNE -- 6 time.JULY -- 7 time.AUGUST -- 8 time.SEPTEMBER -- 9 time.OCTOBER -- 10 time.NOVEMBER -- 11 time.DECEMBER -- 12 ``` #### Weekdays ```lua time.SUNDAY -- 0 time.MONDAY -- 1 time.TUESDAY -- 2 time.WEDNESDAY -- 3 time.THURSDAY -- 4 time.FRIDAY -- 5 time.SATURDAY -- 6 ``` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Invalid duration format | `errors.INVALID` | no | | Parse failed | `errors.INVALID` | no | | Empty location name | `errors.INVALID` | no | | Location not found | `errors.NOT_FOUND` | no | | Duration <= 0 (timer/ticker) | `errors.INVALID` | no | ```lua local t, err = time.parse(time.RFC3339, "invalid") if err then if errors.is(err, errors.INVALID) then print("Invalid format:", err:message()) end return nil, err end local loc, err = time.load_location("Unknown/Zone") if err then if errors.is(err, errors.NOT_FOUND) then print("Location not found:", err:message()) end return nil, err end ``` See [Error Handling](lua/core/errors.md) for working with errors. --- # "Channels and Coroutines" ## Channels and Coroutines Channels exchange values between concurrent tasks. They can be buffered or unbuffered and can be combined with `channel.select` to coordinate multiple operations. This is an API reference. The basic blocks are isolated snippets; the timeout, fan-in, and non-blocking sections are partial patterns whose named channels and callbacks come from the surrounding application. The worker-pool block is a complete in-process example. The `channel` and `coroutine` globals are always available. Channels coordinate coroutines within one Lua process; use process messaging, functions, or queues across process boundaries. ### Creating Channels An unbuffered channel (size 0) requires a sender and receiver to be ready before a transfer completes. A buffered channel allows sends to complete while buffer space is available. ```lua -- Unbuffered: synchronizes sender and receiver local sync_ch = channel.new() -- Buffered: queue up to 10 messages local work_queue = channel.new(10) ``` | Parameter | Type | Description | |-----------|------|-------------| | `size` | integer | Buffer capacity (default: 0 for unbuffered) | **Returns:** `channel` ### Sending Values Sending blocks until a receiver is ready on an unbuffered channel or until buffer space is available on a buffered channel. ```lua -- Send work to a worker pool local tasks = {"task-a", "task-b"} local jobs = channel.new(100) for i, task in ipairs(tasks) do jobs:send(task) -- Blocks if buffer full end jobs:close() -- Signal no more work ``` | Parameter | Type | Description | |-----------|------|-------------| | `value` | any | Value to send | **Returns:** `boolean` Sending to a closed channel raises an error. ### Receiving Values Receiving blocks until a value is available or the channel is closed. ```lua -- Worker consuming from job queue while true do local job, ok = jobs:receive() if not ok then break -- Channel closed, no more work end process(job) end ``` Here, `jobs` is the application-provided queue and `process` is its task-processing callback. **Returns:** `any, boolean` - `value, true` — a value was received - `nil, false` — the channel is closed and empty ### Closing Channels Closing a channel causes pending senders to receive an error and pending receivers to receive `nil, false`. Closing an already closed channel is a no-op. ```lua local results = channel.new(10) -- Producer fills results for _, item in ipairs(data) do results:send(process(item)) end results:close() -- Signal completion ``` This isolated producer snippet assumes `data` and the `process` callback are provided by the application. ### Selecting from Multiple Channels `channel.select` waits on multiple channel operations at the same time. It can coordinate event sources, timeouts, and non-blocking checks. ```lua local result = channel.select(cases) ``` | Parameter | Type | Description | |-----------|------|-------------| | `cases` | table | Array of select cases | | `default` | boolean | If true, returns immediately when no case ready | **Returns:** `table` - For a channel case: `{channel, value, ok}` — `channel` is the case's channel, `value` is the received/sent value, `ok` is false for a closed-channel receive. - For the default branch (when no case is ready and `default = true`): `{default = true, ok = true}`. #### Timeout Pattern Use `time.after()` to add a timeout to a channel wait. ```lua local time = require("time") local result_ch = application_response_channel local timeout, err = time.after("5s") if err then return nil, err end local r = channel.select { result_ch:case_receive(), timeout:case_receive() } if r.channel == timeout then return nil, errors.new({ kind = errors.TIMEOUT, message = "Operation timed out" }) end return r.value ``` This partial pattern assumes the entry lists `time` in `modules:` and the application supplies `application_response_channel`. `time.after` returns one channel on success; invalid or non-positive durations return `nil, error`. #### Fan-in Pattern Handle values from multiple sources in one loop. This process-entry pattern uses ambient `process`, while the application supplies the shutdown signal and the two handler functions. ```lua local events = process.events() local inbox = process.inbox() local shutdown = channel.new() while true do local r = channel.select { events:case_receive(), inbox:case_receive(), shutdown:case_receive() } if r.channel == shutdown then break elseif r.channel == events then handle_event(r.value) else handle_message(r.value) end end ``` #### Non-Blocking Check Use a default case to check for available data without blocking. In this isolated pattern, `ch` and the `process` callback come from the application. ```lua local r = channel.select { ch:case_receive(), default = true } if r.default then -- Nothing available, do something else elseif not r.ok then -- The channel is closed else process(r.value) end ``` ### Creating Select Cases Create send and receive cases for `channel.select`: ```lua -- Send case - completes when channel can accept value ch:case_send(value) -- Receive case - completes when value available ch:case_receive() ``` Values in the cases table that are not send or receive cases are ignored. Make sure the table contains at least one valid case unless it also has a default branch. ### Worker Pool Pattern ```lua local items = {1, 2, 3, 4} local num_workers = 2 local function process_item(item) return item * 2 end local work = channel.new(#items) local results = channel.new(#items) -- Spawn workers for _ = 1, num_workers do coroutine.spawn(function() while true do local item, ok = work:receive() if not ok then return end results:send(process_item(item)) end end) end -- Feed work for _, item in ipairs(items) do work:send(item) end work:close() -- Collect results local processed = {} while #processed < #items do local result, ok = results:receive() if not ok then break end table.insert(processed, result) end ``` After the loop, `processed` contains `2`, `4`, `6`, and `8`; result order depends on coroutine scheduling. The workers share channels because they are coroutines in the same Lua process. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Send on closed channel | runtime error | no | | `cases` argument to select is not a table | runtime error | no | ### See Also - [Process Management](lua/core/process.md) - Process spawning and communication - [Message Queue](lua/storage/queue.md) - Queue-based messaging - [Functions](lua/core/funcs.md) - Function invocation --- # "Process Management" ## Process Management The `process` global provides process spawning, messaging, monitoring, linking, naming, and lifecycle control. It is available without `require()` and does not need to be listed in `modules:`. This is an API reference. Its call-form blocks use placeholders such as `id`, `host`, `destination`, `topic`, and `name` for values supplied by application code; they are not standalone programs. Calls shown with an `err` result return their documented value on success or a failure sentinel plus `error`; the sentinel is normally `nil`, while `process.set_options` returns `false`. Application control flow should handle the error. ### Process Information Read the current frame ID or process ID: ```lua local frame_id, err = process.id() -- Registry ID of the current function, process, or workflow definition if err then return nil, err end local pid, err = process.pid() -- Process ID if err then return nil, err end ``` ### Sending Messages Send one or more payload values to a process by PID or registered name: ```lua local ok, err = process.send(destination, topic, ...) ``` | Parameter | Type | Description | |-----------|------|-------------| | `destination` | string | PID or registered name | | `topic` | string | Topic name (cannot start with `@`) | | `...` | any | Payload values | **Permission:** `process.send` on target PID ### Spawning Processes ```lua -- Basic spawn local pid, err = process.spawn(id, host, ...) -- With monitoring (receive EXIT events) local pid, err = process.spawn_monitored(id, host, ...) -- With linking (receive LINK_DOWN on abnormal exit) local pid, err = process.spawn_linked(id, host, ...) -- Both linked and monitored local pid, err = process.spawn_linked_monitored(id, host, ...) ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Process source ID (e.g., `"app.workers:handler"`) | | `host` | string | Host ID (e.g., `"app:processes"`) | | `...` | any | Arguments passed to spawned process | All variants require `process.spawn` on the process ID. The monitored variants also require `process.spawn.monitored`, and the linked variants require `process.spawn.linked`. At runtime v0.3.32a, only the module-level `spawn()` checks `process.host` on the host ID; the specialized module-level variants do not perform that host permission check. ### Process Control ```lua -- Forcefully terminate a process local ok, err = process.terminate(destination) -- Request graceful cancellation with an optional reason local ok, err = process.cancel(destination, "shutting down") ``` | Parameter | Type | Description | |-----------|------|-------------| | `destination` | string | PID or registered name | | `reason` | string | Optional reason delivered to the target | **Permissions:** `process.terminate`, `process.cancel` on target PID ### Monitoring and Linking Add or remove monitoring and links for an existing process: ```lua -- Monitoring: receive EXIT events when target exits local ok, err = process.monitor(destination) local ok, err = process.unmonitor(destination) -- Linking: bidirectional, receive LINK_DOWN on abnormal exit local ok, err = process.link(destination) local ok, err = process.unlink(destination) ``` **Permissions:** `process.monitor`, `process.unmonitor`, `process.link`, `process.unlink` on target PID ### Process Options ```lua local options = process.get_options() local ok, err = process.set_options({trap_links = true}) ``` | Field | Type | Description | |-------|------|-------------| | `trap_links` | boolean | Whether LINK_DOWN events are delivered to events channel | | `upgradable` | boolean | Opt in to OUTDATED events when the process's code is invalidated | ### Inbox and Events Use the inbox and event channels to receive messages and lifecycle events: ```lua local inbox = process.inbox() -- Message objects from @inbox topic local events = process.events() -- Lifecycle events from @events topic ``` #### Event Types | Constant | Description | |----------|-------------| | `process.event.CANCEL` | Cancellation requested | | `process.event.EXIT` | Monitored process exited | | `process.event.LINK_DOWN` | Linked process terminated abnormally | | `process.event.OUTDATED` | The process's code or an imported dependency changed in the registry | #### Event Fields | Field | Type | Description | |-------|------|-------------| | `kind` | string | Event type constant | | `from` | string | Source PID (absent for OUTDATED) | | `result` | table | For EXIT/LINK_DOWN: a {value, error} record; the process return value is at `result.value` and any error at `result.error` | | `reason` | string | For CANCEL: why the process is being cancelled | | `sources` | string[] | For OUTDATED: registry IDs that changed or were transitively affected | `OUTDATED` is delivered only to processes that opt in with `process.set_options({upgradable = true})`. Multiple invalidations are combined into one pending event containing the union of their `sources`. Handle the event by calling [`process.upgrade`](#process-upgrade). ### Topic Subscription Subscribe to a custom message topic: ```lua local ch, err = process.listen(topic, options) if err then return nil, err end local ok, err = process.unlisten(ch) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `topic` | string | Topic name (cannot start with `@`) | | `options.message` | boolean | If true, receive Message objects; if false, raw payloads | ### Message Objects The inbox and listeners configured with `{message = true}` return message objects: ```lua local msg = inbox:receive() msg:topic() -- string: topic name msg:from() -- string: sender PID (empty string when unknown) msg:payload() -- Payload: wrapper (call :data() to extract); nil when empty, table of wrappers for several values msg:payload():data() -- any: actual payload value ``` ### Synchronous Call `process.exec` spawns a process and waits for its result: ```lua local result, err = process.exec(id, host, ...) ``` **Permissions:** `process.exec` on process id, `process.host` on host id ### Process Upgrade Upgrade the current process while preserving its PID: The two snippets below are alternative call forms, not sequential operations. ```lua -- Upgrade to new version, passing state process.upgrade(id, ...) ``` ```lua -- Keep same definition, re-run with new state process.upgrade(nil, preserved_state) ``` `process.upgrade` is a terminal control transfer: it clears the current execution and starts the requested definition with the same PID. Code after the call does not run in the old execution. ### Context Spawner Create a spawner that supplies custom context to child processes: ```lua local spawner = process.with_context({request_id = "123"}) ``` **Permission:** `process.context` on "context" #### Spawner with Options `process.with_options(options)` creates a spawner with spawn-time options, such as a network selector, rather than context values: ```lua local spawner = process.with_options({network = "app:tor_proxy"}) ``` | Option | Type | Description | |--------|------|-------------| | `network` | string | Registry ID of a `network.*` entry to use for the child's outbound connections | | `terminal` | string | Viewport grant that attaches a virtual terminal to the child | **Permission:** `process.context` on "context"; selecting a network additionally requires `network.select` on that network ID. #### Terminal Attachment A `terminal` grant comes from `viewport:grant()` and gives the child a terminal port of its own, so it can use the [TTY](lua/system/tty.md) module exactly as it would on a terminal host: ```lua local view = assert(tty.viewport({width = 80, height = 24})) local child = assert(process.with_options({terminal = assert(view:grant())}) :spawn_monitored("app:child", "app:workers")) ``` The grant is one-shot and is consumed at admission: a rejected start leaves it unresolved and reusable, a child that resolves the port consumes it permanently, and a host that does not support terminal attachments rejects the spawn rather than dropping the option. The spawning process keeps reading the child's frames through the viewport it created. See [Terminal](system/terminal.md#composable-terminals). #### SpawnBuilder Methods `SpawnBuilder` is immutable; each configuration method returns a new instance: ```lua spawner:with_context(values) -- Add context values spawner:with_actor(actor) -- Set security actor spawner:with_scope(scope) -- Set security scope spawner:with_name(name) -- Register name at start; if taken, spawn returns the existing PID and queued messages go to it spawner:with_message(topic, ...) -- Queue message to send after spawn spawner:with_options(options) -- Merge spawn-time options (e.g. network) ``` **Permission:** `process.security` on "security" for `:with_actor()` and `:with_scope()` #### Spawner Spawn Methods ```lua spawner:spawn(id, host, ...) spawner:spawn_monitored(id, host, ...) spawner:spawn_linked(id, host, ...) spawner:spawn_linked_monitored(id, host, ...) ``` All `SpawnBuilder` spawn methods require `process.host` on the host ID in addition to the applicable `process.spawn`, `process.spawn.monitored`, and `process.spawn.linked` permissions. #### Spawner Exec ```lua local result, err = spawner:exec(id, host, ...) ``` This method runs the target process synchronously with the builder's context, actor, and scope, then returns its result. A deferred worker can use `with_actor` and `with_scope` to execute with an owner's identity. **Permissions:** `process.exec` on process id, `process.host` on host id ### Name Registry Register a process under a name so callers can use the name instead of its PID. Functions that accept a `destination`, including `send`, `terminate`, `cancel`, `monitor`, and `link`, also accept registered names. ```lua local ok, err = process.registry.register(name) -- self, local scope local pid, err = process.registry.lookup(name) local ok, err = process.registry.unregister(name) ``` #### Scope The optional `scope` argument selects the name's consistency guarantee and defaults to `LOCAL`. See the [Cluster Guide](guides/cluster.md#naming-and-name-scopes) for the complete model. | Constant | Visibility | Guarantee | |----------|------------|-----------| | `process.registry.LOCAL` | this node only | Instant, node-local | | `process.registry.EVENTUAL` | cluster-wide | Eventually consistent (gossip) | | `process.registry.CONSISTENT` | cluster-wide | Linearizable singleton (Raft) | | `process.registry.STRONG` | cluster-wide | Consistent + every live node acknowledges | On a standalone node, only `LOCAL` is available; cluster scopes require [clustering](guides/cluster.md). #### register ```lua local ok, err = process.registry.register(name, pid, scope) ``` | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `name` | string | yes | | Name to register | | `pid` | string | no | self | PID to register; defaults to the calling process | | `scope` | number | no | `LOCAL` | One of the scope constants above | Returns `true` on success, or `nil, error` on failure. Conflicts (name already registered to a different PID) return `errors.ALREADY_EXISTS`. Registering the same name to the same PID is idempotent. A `STRONG` registration blocks until every live node acknowledges or the reservation deadline expires; on timeout it returns an error. Registering on behalf of a different PID additionally requires the `process.registry.foreign` permission on the target PID. #### lookup ```lua local pid, err = process.registry.lookup(name) ``` Returns the registered PID string, or `nil, error` with kind `errors.NOT_FOUND` when the name is not registered. #### unregister ```lua local ok, err = process.registry.unregister(name, scope) ``` `scope` defaults to `LOCAL` and must match the scope the name was registered under. For `CONSISTENT` and `STRONG`, the owning process is the one allowed to unregister; unregistering a name owned by another PID returns `false`. Names also release automatically when the owning process exits (and, for cluster scopes, when its node leaves), so explicit unregister is for early release. ### Permissions Permission checks evaluate the caller's security actor against the target resource. #### Policy Evaluation Policies can allow or deny an operation based on: - **Actor**: The security principal making the request - **Action**: The operation being performed (e.g., `process.send`) - **Resource**: The target (PID, process id, host id, or name) - **Attributes**: Additional context including `pid` (caller's process ID) #### Permission Reference | Permission | Functions | Resource | |------------|-----------|----------| | `process.spawn` | `spawn*()` | process id | | `process.spawn.monitored` | `spawn_monitored()`, `spawn_linked_monitored()` | process id | | `process.spawn.linked` | `spawn_linked()`, `spawn_linked_monitored()` | process id | | `process.host` | module-level `spawn()`, all `SpawnBuilder` spawn methods, `exec()` | host id | | `process.send` | `send()` | target PID | | `process.exec` | `exec()` | process id | | `process.terminate` | `terminate()` | target PID | | `process.cancel` | `cancel()` | target PID | | `process.monitor` | `monitor()` | target PID | | `process.unmonitor` | `unmonitor()` | target PID | | `process.link` | `link()` | target PID | | `process.unlink` | `unlink()` | target PID | | `process.context` | `with_context()`, `with_options()` | "context" | | `process.security` | `:with_actor()`, `:with_scope()` | "security" | | `process.registry.register` | `registry.register()` | name | | `process.registry.unregister` | `registry.unregister()` | name | | `process.registry.foreign` | `registry.register()` | target PID | Cluster name scopes are authorized by scope-suffixed variants of these actions (`process.registry.register.eventual`, `.consistent`, `.strong`, and the matching `unregister` actions), so a policy can grant local naming separately from cluster-wide naming. #### Multiple Permissions Some operations require multiple permissions: | Operation | Required Permissions | |-----------|---------------------| | `spawn()` | `process.spawn` + `process.host` | | module-level `spawn_monitored()` | `process.spawn` + `process.spawn.monitored` | | module-level `spawn_linked()` | `process.spawn` + `process.spawn.linked` | | module-level `spawn_linked_monitored()` | `process.spawn` + `process.spawn.monitored` + `process.spawn.linked` | | `SpawnBuilder:spawn()` | `process.spawn` + `process.host` | | `SpawnBuilder:spawn_monitored()` | `process.spawn` + `process.spawn.monitored` + `process.host` | | `SpawnBuilder:spawn_linked()` | `process.spawn` + `process.spawn.linked` + `process.host` | | `SpawnBuilder:spawn_linked_monitored()` | `process.spawn` + `process.spawn.monitored` + `process.spawn.linked` + `process.host` | | `exec()` | `process.exec` + `process.host` | | spawn with custom actor/scope | spawn permissions + `process.security` | ### Errors | Condition | Kind | |-----------|------| | No context found | `errors.INTERNAL` | | Frame context not found | `errors.INTERNAL` | | Missing required arguments | `errors.INVALID` | | Reserved topic prefix (`@`) | `errors.INVALID` | | Destination is neither a PID nor a registered name | `errors.NOT_FOUND` | | Name not registered | `errors.NOT_FOUND` | | Permission denied | `errors.PERMISSION_DENIED` | | Name already registered | `errors.ALREADY_EXISTS` | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Channels](lua/core/channel.md) - In-process coroutine coordination - [Message Queue](lua/storage/queue.md) - Queue-based messaging - [Functions](lua/core/funcs.md) - Function invocation - [Supervision](guides/supervision.md) - Process lifecycle management - [Cluster](guides/cluster.md) - Name scopes and cluster-wide naming --- # "Process Groups" ## Process Groups Process groups organize processes under dynamic names and broadcast messages to group members across the cluster. A process can join multiple groups, and cluster-wide membership is eventually consistent. This is an API reference. Its snippets assume an existing `pg.scope`, an executable entry running with process context, and policies that authorize the documented operations. The blocks demonstrate individual calls or partial subscription flows rather than a standalone application. For the scope entry kind and its configuration, see [Process Groups](system/process-groups.md). For the broader clustering model, see the [Cluster Guide](guides/cluster.md). ### Loading ```lua local pg = require("pg") ``` Add `pg` to the executable entry's `modules:` list before requiring it. ### Opening a Scope A process group belongs to a **scope**, represented by a `pg.scope` registry entry. Open the scope to obtain an instance for group operations: ```lua local group, err = pg.open("app:pg") if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Scope entry ID (format: `"namespace:name"`) | **Returns:** `pg.Instance, error` **Permission:** `pg.open` on the scope `id` The instance is released automatically during execution-frame cleanup. Call `release()` to release it earlier. Other operations are methods on the instance and use `:` syntax. ### Joining and Leaving The calls below are independent forms; choose the single-group or batch join needed by the application and pair it with the corresponding leave operations. ```lua local ok, err = group:join("workers") -- single group if err then return nil, err end ``` ```lua local ok, err = group:join({"workers", "all"}) -- batch if err then return nil, err end ``` ```lua local ok, err = group:leave("workers") if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `group` | string \| string[] | Group name, or a list of names for a batch operation | **Returns:** `boolean, error` A process can join the same group more than once and must leave the same number of times to depart fully. For a batch, `leave` is best-effort and returns an error only when the process was not a member of any named group. **Permissions:** `pg.join` / `pg.leave` on each group name ### Listing Members ```lua local members, err = group:get_members("workers") -- all nodes if err then return nil, err end local local_members, err = group:get_local_members("workers") -- this node only if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `group` | string | Group name | **Returns:** `string[], error` — an array of PID strings (empty for an unknown group) **Permissions:** `pg.get_members` / `pg.get_local_members` on the group name ### Listing Groups ```lua local groups, err = group:which_groups() -- all groups in the cluster if err then return nil, err end local local_groups, err = group:which_local_groups() -- groups with a local member if err then return nil, err end ``` **Returns:** `string[], error` — group names that currently have at least one member **Permissions:** `pg.which_groups` / `pg.which_local_groups` ### Broadcasting Broadcast sends a message from the calling process to every group member under `topic`. Members receive it with `process.listen(topic)`. ```lua local ok, err = group:broadcast("workers", "task", {id = 42}) -- all nodes if err then return nil, err end ok, err = group:broadcast_local("workers", "task", {id = 42}) -- this node only if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `group` | string | Target group | | `topic` | string | Message topic | | `...` | any | Zero or more payload values | **Returns:** `boolean, error` **Permissions:** `pg.broadcast` / `pg.broadcast_local` on the group name ### Monitoring a Group `monitor` subscribes to join and leave events for one group and returns an atomic snapshot of its current members. No membership change can occur between the snapshot and subscription setup without being observed. ```lua local sub, members, err = group:monitor("workers") if err then return nil, err end for _, pid in ipairs(members) do -- current members at subscription time end local ch = sub:channel() local event, open = ch:receive() -- {kind = "member.joined" | "member.left", path = "workers", data = {...}} if not open then return nil, errors.new("Process-group subscription closed") end sub:close() -- unsubscribe; sub:close({flush = true}) drains queued events first ``` | Parameter | Type | Description | |-----------|------|-------------| | `group` | string | Group to watch | **Returns:** `pg.Subscription, string[], error` — the subscription and a snapshot of current members **Permission:** `pg.monitor` on the group name ### Watching All Groups `events` subscribes to membership changes for every group in the scope and returns a snapshot mapping groups to their members. ```lua local sub, snapshot, err = group:events() if err then return nil, err end -- snapshot: { ["workers"] = {pid, ...}, ["all"] = {pid, ...} } local event, open = sub:channel():receive() if not open then return nil, errors.new("Process-group subscription closed") end sub:close() ``` **Returns:** `pg.Subscription, table, error` **Permission:** `pg.events` #### Event Fields Events delivered on a subscription channel carry: | Field | Type | Description | |-------|------|-------------| | `system` | string | Always `"pg"` | | `kind` | string | `"member.joined"` or `"member.left"` | | `path` | string | The group name | | `data` | table | `{Group = string, PIDs = string[]}` — the affected members | Subscription channels are buffered (capacity 64). If a slow consumer fills the buffer, further events are retained in the process mailbox in order and delivered once the consumer drains the channel (the subscription stalls rather than dropping events). ### Releasing ```lua group:release() ``` `release` frees the instance immediately and is idempotent. After release, every other group operation returns an error. Cleanup also runs automatically at the end of the execution frame. **Returns:** `boolean` ### Permissions | Permission | Method | Resource | |------------|--------|----------| | `pg.open` | `pg.open()` | scope id | | `pg.join` | `join()` | group name | | `pg.leave` | `leave()` | group name | | `pg.get_members` | `get_members()` | group name | | `pg.get_local_members` | `get_local_members()` | group name | | `pg.which_groups` | `which_groups()` | (none) | | `pg.which_local_groups` | `which_local_groups()` | (none) | | `pg.broadcast` | `broadcast()` | group name | | `pg.broadcast_local` | `broadcast_local()` | group name | | `pg.monitor` | `monitor()` | group name | | `pg.events` | `events()` | (none) | ### Errors | Condition | Kind | |-----------|------| | Permission denied | `errors.PERMISSION_DENIED` | | Missing or empty argument | `errors.INVALID` | | Scope not found | `errors.INTERNAL` | | Leave a group with no membership | `errors.NOT_FOUND` | | Instance released | `errors.INVALID` | | Group/member or action-queue limit reached | `errors.RATE_LIMITED` (retryable) | | Service stopped, backpressure, or open circuit | `errors.UNAVAILABLE` | | Broadcast timed out | `errors.TIMEOUT` (retryable) | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Process Groups](system/process-groups.md) - Scope entry kind and configuration - [Cluster](guides/cluster.md) - Membership, naming, and the clustering model - [Process Management](lua/core/process.md) - Spawning and messaging individual processes --- # "Function Invocation" ## Function Invocation The `funcs` module calls registered functions synchronously or asynchronously. An executor can propagate request context, security identity, and implementation-specific call options. This page is an API reference; target IDs, arguments, and application data represent surrounding code. ### Loading ```lua local funcs = require("funcs") ``` ### `call` Calls a registered function synchronously and waits for its result. ```lua local result, err = funcs.call("app.api:get_user", user_id) if err then return nil, err end print(result.name) ``` | Parameter | Type | Description | |-----------|------|-------------| | `target` | string | Function ID in format "namespace:name" | | `...args` | any | Arguments passed to the function | **Returns:** `result, error` The target uses the `namespace:name` format. ### `async` Starts a function call and returns a `Future` immediately. Futures allow other work to continue while the call runs and support multiple concurrent calls. ```lua -- Start heavy computation without blocking local future, err = funcs.async("app.process:analyze_data", large_dataset) if err then return nil, err end -- Do other work while computation runs... -- Wait for result when ready local ch = future:response() local _, open = ch:receive() if not open then return nil, errors.new("future response channel closed") end local payload, result_err = future:result() if result_err then return nil, result_err end local result, data_err = payload:data() if data_err then return nil, data_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `target` | string | Function ID in format "namespace:name" | | `...args` | any | Arguments passed to the function | **Returns:** `Future, error` ### `new` Creates an `Executor` for calls that need custom context, security identity, or call options. ```lua local exec = funcs.new() ``` **Returns:** `Executor` ### Executor An executor stores call context and options. Its configuration methods return new executor instances, allowing a base configuration to be reused. #### `with_context` Adds request-scoped values that will be available to the called function, such as trace IDs, session data, or feature flags. ```lua local ctx = require("ctx") -- Propagate request context to downstream services local request_id, ctx_err = ctx.get("request_id") if ctx_err then return nil, ctx_err end local exec, err = funcs.new():with_context({ request_id = request_id, feature_flags = {dark_mode = true} }) if err then return nil, err end local user, err = exec:call("app.api:get_user", user_id) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `values` | table | Key-value pairs to add to context | **Returns:** `Executor, error` #### `with_actor` Sets the security actor used for authorization checks in the called function. ```lua local security = require("security") local actor = security.actor() -- Get current user's actor -- Call admin function with user's credentials local exec, err = funcs.new():with_actor(actor) if err then return nil, err end local result, err = exec:call("app.admin:delete_record", record_id) if err and err:kind() == errors.PERMISSION_DENIED then return nil, errors.new({kind = errors.PERMISSION_DENIED, message = "User cannot delete records"}) end ``` | Parameter | Type | Description | |-----------|------|-------------| | `actor` | Actor | Security actor (from security module) | **Returns:** `Executor, error` #### `with_scope` Sets the security scope for called functions. The scope defines the permissions available to the call. ```lua local security = require("security") local scope = security.new_scope() local exec, err = funcs.new():with_scope(scope) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `scope` | Scope | Security scope (from security module) | **Returns:** `Executor, error` #### `with_options` Sets call options such as the retry policy or the overlay network. Options are merged over any preset options of the target function entry. ```lua -- Retry transient failures up to 5 times with exponential backoff local exec = funcs.new():with_options({ retry = { max_attempts = 5, initial_delay = 100 } }) local result, err = exec:call("app.external:fetch_data", query) if err then -- All attempts failed, or the error was not retryable end ``` | Parameter | Type | Description | |-----------|------|-------------| | `options` | table | Call options | | Option | Type | Description | |--------|------|-------------| | `retry.max_attempts` | int | Maximum attempts including the first (1 disables retry) | | `retry.initial_delay` | int/duration | Delay before first retry (ms or duration string), default `100` | | `retry.max_delay` | int/duration | Upper bound for the backoff delay (ms or duration string), default `10s` | | `retry.backoff_factor` | number | Multiplier applied to the delay after each attempt, default `2.0` | | `retry.jitter` | number | Random jitter fraction applied to each delay, default `0.1` | | `retry.retry_kinds` | string[] | Only retry errors of these kinds; by default every kind except `Invalid`, `PermissionDenied` and `Internal` is retried | | `retry.skip_kinds` | string[] | Never retry errors of these kinds | | `network` | string | Registry ID of an overlay network to route the call's outbound traffic through; requires the `network.select` permission | Only retryable errors trigger retries; non-retryable errors surface immediately. Temporal activity options are described in [Activities](temporal/activities.md). The runtime-defined option is: | Recognized option | Type | Description | |-------------------|------|-------------| | `network` | string | Registry ID of the outbound `network.*` entry | **Returns:** `Executor, error` Selecting a network requires `network.select` permission on that network ID. #### `call` and `async` The executor versions of `call` and `async` use its configured context and options. ```lua -- Build reusable executor with context local exec = funcs.new() :with_context({trace_id = "abc-123"}) :with_options({retry = {max_attempts = 3}}) -- Make multiple calls with same context local users, users_err = exec:call("app.api:list_users") if users_err then return nil, users_err end local posts, posts_err = exec:call("app.api:list_posts") if posts_err then return nil, posts_err end ``` ### Future Invocation Summary `async()` returns a future representing an in-progress invocation. The methods below cover the caller-facing steps for receiving, inspecting, or canceling that invocation. See [Future](./future.md) for the Future object reference. #### `response` and `channel` Returns the channel used to receive the result. ```lua local time = require("time") local future, err = funcs.async("app.api:slow_operation", data) if err then return nil, err end local ch = future:response() -- or future:channel() local timeout, err = time.after("5s") if err then return nil, err end local result = channel.select { ch:case_receive(), timeout:case_receive() } ``` **Returns:** `Channel` The response channel signals completion. After it becomes ready, call `future:result()` to obtain the cached value or the called function's error. #### `is_complete` Checks whether the future has completed without blocking. ```lua while not future:is_complete() do -- do other work local _, sleep_err = time.sleep("100ms") if sleep_err then return nil, sleep_err end end local result, err = future:result() ``` **Returns:** `boolean` #### `is_canceled` Returns `true` if the future has been marked canceled by its provider. See the cancellation limitation below. ```lua if future:is_canceled() then print("Operation was canceled") end ``` **Returns:** `boolean` #### `result` Returns the cached result when complete or `nil` while the operation is pending. ```lua local value, err = future:result() if err then print("Failed:", err:message()) elseif value then local data, data_err = value:data() if data_err then return nil, data_err end print("Got:", data) end ``` **Returns:** `Payload|table|nil, error|nil` #### `error` Returns the operation error when the future has failed. ```lua local err, has_error = future:error() if has_error then print("Error kind:", err:kind()) end ``` **Returns:** `error|nil, boolean` This method returns a non-retryable `INTERNAL` wrapper for a failed operation. Use `result()` to preserve the called function's original error metadata. #### `cancel` Requests cancellation of the asynchronous operation. ```lua local canceled, err = future:cancel() if err then return nil, err end ``` **Returns:** `boolean, error` In runtime v0.3.32a, function and contract futures share one process-global cancellation callback. When both providers are loaded, cancel() and is_canceled() are not a stable cross-provider contract. Do not use cancellation for application correctness; time out locally and ignore a late result until the runtime separates provider cancellation. ### Parallel Operations Combine `async` with `channel.select` to run and collect multiple calls concurrently. ```lua -- Start multiple operations in parallel local f1, err = funcs.async("app.api:get_user", user_id) if err then return nil, err end local f2, err = funcs.async("app.api:get_orders", user_id) if err then return nil, err end local f3, err = funcs.async("app.api:get_preferences", user_id) if err then return nil, err end -- Wait for all to complete using channels local user_ch = f1:channel() local orders_ch = f2:channel() local prefs_ch = f3:channel() local pending = { [user_ch] = {name = "user", future = f1}, [orders_ch] = {name = "orders", future = f2}, [prefs_ch] = {name = "preferences", future = f3} } local results = {} while next(pending) do local cases = {} for ch in pairs(pending) do cases[#cases + 1] = ch:case_receive() end local r = channel.select(cases) local completed = pending[r.channel] pending[r.channel] = nil local payload, result_err = completed.future:result() if result_err then return nil, result_err end local data, data_err = payload:data() if data_err then return nil, data_err end results[completed.name] = data end ``` ### Permissions Function operations are subject to security policy evaluation. | Action | Resource | Description | |--------|----------|-------------| | `funcs.call` | Function ID | Call a specific function | | `funcs.context` | `context` | Use `with_context()` to set custom context | | `funcs.security` | `security` | Use `with_actor()` or `with_scope()` | | `network.select` | Network ID | Use `with_options({network = ...})` to select an overlay network | ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Target empty | `errors.INVALID` | no | | Namespace missing | `errors.INVALID` | no | | Name missing | `errors.INVALID` | no | | Permission denied | `errors.PERMISSION_DENIED` | no | | Async outside a process | `errors.INTERNAL` | no | | Subscribe failed | `errors.INTERNAL` | no | | Async start dispatch failed | `errors.INTERNAL` | no | | Function error | varies | varies | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Futures" ## Futures Futures represent asynchronous operation results. They are returned by `funcs.async()` and asynchronous contract calls. This page is an API reference; the target IDs and arguments in its patterns are application-defined. ### Loading Futures are not loaded as a module; asynchronous operations create them: ```lua local funcs = require("funcs") local future, err = funcs.async("app.compute:task", data) if err then return nil, err end ``` ### Response Channel Use the response channel to wait for completion, then read the cached result from the future: ```lua local ch = future:response() local _, open = ch:receive() if not open then return nil, errors.new("future response channel closed") end local payload, err = future:result() if err then return nil, err end local result, data_err = payload:data() if data_err then return nil, data_err end ``` `channel()` is an alias for `response()`. The channel value is the operation's payload, payload table, or error. Calling `result()` after the channel becomes ready provides one consistent success/error interface and returns the cached value even after the channel is drained. ### Completion Check Check whether the future has completed without blocking: ```lua if future:is_complete() then local result, err = future:result() end ``` ### Cancellation Check Check whether the future has been marked canceled by its provider: ```lua if future:is_canceled() then print("Operation was canceled") end ``` ### Getting Result Read the cached result without blocking: ```lua local val, err = future:result() ``` **Returns:** - Not complete: `nil, nil` - Canceled: `nil, error` (kind `CANCELED`) - Error: `nil, error` - Success: `Payload, nil` or `table, nil` (multiple payloads) ### Getting Error Read the error when the future has failed: ```lua local err, has_error = future:error() if has_error then print("Failed:", err:message()) end ``` **Returns:** `error, boolean` When an operation fails, `error()` returns a non-retryable `INTERNAL` wrapper. Use `result()` when the called function's original error kind and retryability must be preserved. ### Canceling Request cancellation of the asynchronous operation on a best-effort basis: ```lua local canceled, err = future:cancel() ``` **Returns:** `boolean, error` Operation may still complete if already in progress. ### Timeout Pattern ```lua local time = require("time") local future, err = funcs.async("app.compute:slow", data) if err then return nil, err end local timeout, err = time.after("5s") if err then return nil, err end local r = channel.select { future:channel():case_receive(), timeout:case_receive() } if r.channel == timeout then future:cancel() return nil, errors.new({ kind = errors.TIMEOUT, message = "Operation timed out" }) end local payload, result_err = future:result() if result_err then return nil, result_err end local value, data_err = payload:data() if data_err then return nil, data_err end return value ``` ### First-to-Complete ```lua local f1, err = funcs.async("app.cache:get", key) if err then return nil, err end local f2, err = funcs.async("app.db:get", key) if err then return nil, err end local ch1 = f1:channel() local ch2 = f2:channel() local r = channel.select { ch1:case_receive(), ch2:case_receive() } -- The slower operation may still complete; this caller ignores its result. local winner if r.channel == ch1 then winner = f1 else winner = f2 end local payload, result_err = winner:result() if result_err then return nil, result_err end local value, data_err = payload:data() if data_err then return nil, data_err end return value ``` ### Errors | Condition | Kind | |-----------|------| | Operation canceled | `CANCELED` | | Async operation failed | `result()` preserves the operation's kind; `error()` reports `INTERNAL` | --- # "Streams" ## Streams Streams provide incremental I/O for HTTP, filesystem, and other modules. The modules that own the underlying data create stream objects. This page is an API reference; the scanner loop uses an application-defined `process(token)` callback. ### Obtaining a Stream ```lua -- From HTTP request body local stream, err = req:stream() if err then return nil, err end -- From filesystem local fs = require("fs") local volume, err = fs.get("app:data") if err then return nil, err end local stream, err = volume:open("/file.txt", "r") if err then return nil, err end ``` ### Reading ```lua local chunk, err = stream:read(size) ``` | Parameter | Type | Description | |-----------|------|-------------| | `size` | integer | Bytes to read (0 = default 32KB chunk) | **Returns:** `string, error` — `nil, nil` on EOF ### Writing ```lua local bytes, err = stream:write(data) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to write | **Returns:** `integer, error` — bytes written ### Seeking ```lua local pos, err = stream:seek(whence, offset) ``` | Parameter | Type | Description | |-----------|------|-------------| | `whence` | string | `"set"`, `"cur"`, or `"end"` | | `offset` | integer | Offset in bytes | **Returns:** `integer, error` — new position ### Flushing ```lua local ok, err = stream:flush() ``` `flush` writes buffered data to the underlying destination. ### Stream Information ```lua local info, err = stream:stat() ``` | Field | Type | Description | |-------|------|-------------| | `size` | integer | Total size (-1 if unknown) | | `position` | integer | Current position | | `readable` | boolean | Can read | | `writable` | boolean | Can write | | `seekable` | boolean | Can seek | ### Closing ```lua local ok, err = stream:close() ``` `close` releases the stream's resources and can be called more than once. ### Scanner Create a scanner that tokenizes stream content: ```lua local scanner, err = stream:scanner(split) ``` | Parameter | Type | Description | |-----------|------|-------------| | `split` | string | `"lines"`, `"words"`, `"bytes"`, `"runes"` | #### Scanner Methods ```lua local has_more, err = scanner:scan() -- advance to next token local token = scanner:text() -- current token local err_msg = scanner:err() -- scanner error if any ``` ```lua while true do local has_token, err = scanner:scan() if err then return nil, err end if not has_token then local scan_err = scanner:err() if scan_err then return nil, scan_err end -- raw scanner error string break -- clean EOF end process(scanner:text()) end ``` When `scan()` returns `false`, check `scanner:err()` before treating the result as EOF. Tokenization and underlying read failures are stored on the scanner and do not appear in `scan()`'s second return value. ### Errors | Condition | Kind | |-----------|------| | Invalid whence/split type | raised as a Lua error (not returned) | | Stream closed | `INTERNAL` | | Not readable/writable | `INTERNAL` | | Read/write failure | `INTERNAL` | --- # "Request Context" ## Request Context The `ctx` module reads request-scoped values propagated through [function calls](lua/core/funcs.md) or [process operations](lua/core/process.md). This page is an API reference; the snippets show individual calls inside an executable Lua entry. ### Loading ```lua local ctx = require("ctx") ``` #### Get a Value ```lua local value, err = ctx.get("key") ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Context key | **Returns:** `any, error` #### Get All Values ```lua local values, err = ctx.all() ``` **Returns:** `table, error` `ctx.all()` returns an empty table when an execution context is present but has no request values. A missing execution context returns `nil, errors.INTERNAL`. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty key | `errors.INVALID` | no | | Key not found | `errors.NOT_FOUND` | no | | No execution context available | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Event Bus" ## Event Bus The event bus publishes runtime and application activity for monitoring, logging, metrics, and reactive side effects. This page is an API reference; the snippets assume an executable Lua entry with the listed module and permissions. The event bus is a best-effort publish/subscribe channel, not a reliable transport. Do not depend on it for business-critical delivery. Use process messaging (`process.send`), channels, or the [message queue](lua/storage/queue.md) when delivery is part of application correctness. ### Loading ```lua local events = require("events") ``` ### Subscribing to Events Subscribe to one system or a system pattern, with an optional event-kind filter: ```lua -- Subscribe to all order events local sub, err = events.subscribe("orders.*") if err then return nil, err end -- Process events local ch = sub:channel() while true do local evt, ok = ch:receive() if not ok then break end print(evt.system, evt.kind, evt.path) -- Process evt.data when the publisher supplied a payload. end ``` Pass a second argument to restrict delivery to one kind, for example `events.subscribe("users", "user.created")`. An omitted kind accepts every kind from the matching system. | Parameter | Type | Description | |-----------|------|-------------| | `system` | string | System pattern (supports wildcards like "test.*") | | `kind` | string | Event kind filter (optional) | **Returns:** `Subscription, error` ### Publishing Events Publish an event to the event bus: ```lua -- Send order created event local ok, err = events.send("orders", "order.created", "/orders/123", { order_id = "123", customer_id = "456", total = 99.99 }) if err then return nil, err end -- Send without data local heartbeat_sent, heartbeat_err = events.send("system", "heartbeat", "/health") if heartbeat_err then return nil, heartbeat_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `system` | string | System identifier | | `kind` | string | Event kind/type | | `path` | string | Event path for routing | | `data` | any | Event payload (optional) | **Returns:** `boolean, error` A successful return confirms that the runtime accepted the send. It does not confirm that any subscriber received or processed the event. #### Receive Channel Use the subscription channel to receive events: ```lua local json = require("json") local ch = sub:channel() local evt, ok = ch:receive() if ok then print("System:", evt.system) print("Kind:", evt.kind) print("Path:", evt.path) local encoded, encode_err = json.encode(evt.data) if encode_err then return nil, encode_err end print("Data:", encoded) end ``` Each event contains `system`, `kind`, and `path`. The `data` field is present only when the publisher supplied a non-nil payload. #### Close a Subscription Close the subscription to unsubscribe and close its channel: ```lua local closed = sub:close() -- true ``` Closing is idempotent. After the channel is closed, `receive()` returns `nil, false` once buffered events are drained. ### Permissions | Action | Resource | Description | |--------|----------|-------------| | `events.subscribe` | system | Subscribe to events from a system | | `events.send` | system | Send events to a system | ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty system | `errors.INVALID` | no | | Empty send kind | `errors.INVALID` | no | | Empty path | `errors.INVALID` | no | | Policy denied | `errors.INVALID` | no | | Missing execution or process context | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Entry Registry" ## Entry Registry Query and modify registered entries. Access metadata, snapshots, and version history. ### Loading ```lua local registry = require("registry") ``` ### Entry Structure ```lua { id = "app.lib:assert", -- string: "namespace:name" kind = "function.lua", -- string: entry type meta = {type = "test"}, -- table: searchable metadata data = {...} -- any: entry payload } ``` Entries read back from `registry.get`, `registry.find`, `snap:entries()`, `snap:get()`, `snap:namespace()` and `snap:find()` carry only these four author-facing fields. `dependency_root` is a write-side field accepted by `changes:create()` and `changes:update()`. It is a boolean that marks an `ns.dependency` entry as a deployment root. It is never returned by the entry APIs; registry-owned state is read through [`snap:state()`](lua/core/registry.md#snapshot-state). ### Get Entry ```lua local entry, err = registry.get("app.lib:assert") ``` **Permission:** `registry.get` on entry ID ### Find Entries ```lua local entries, err = registry.find({[".kind"] = "function.lua"}) local entries, err = registry.find({[".kind"] = "http.endpoint", [".ns"] = "app.api"}) ``` Keys prefixed with `.` match entry fields (`.kind`, `.ns`, `.name`, `.id`) and accept `*` globs. Keys prefixed with `meta.` match entry metadata; a leading `~`, `*`, `^` or `$` on a `meta.` key selects regex, contains, prefix or suffix matching. Keys with neither prefix are ignored. ### Parse ID ```lua local id = registry.parse_id("app.lib:assert") -- id.ns = "app.lib", id.name = "assert" ``` ### Snapshots Point-in-time view of the registry: ```lua local snap, err = registry.snapshot() -- current state local snap, err = registry.snapshot_at(5) -- at version 5 ``` #### Snapshot Methods | Method | Returns | Description | |--------|---------|-------------| | `snap:entries()` | `Entry[], error` | All accessible entries | | `snap:state()` | `State, error` | Entries with registry-owned metadata, plus the resolved module graph | | `snap:get(id)` | `Entry, error` | Single entry by ID | | `snap:find(filter)` | `Entry[]` | Filter entries | | `snap:namespace(ns)` | `Entry[]` | Entries in namespace | | `snap:version()` | `Version` | Snapshot version | | `snap:changes()` | `Changes` | Create changeset | #### Snapshot State `snap:state()` returns the entry state together with the module graph selected for the snapshot version. Registry-owned provenance is carried on each entry rather than merged into `meta`, so it cannot be confused with authored metadata. ```lua local snap, err = registry.snapshot() local state, err = snap:state() for _, entry in ipairs(state.entries) do print(entry.id, entry.registry.owner, entry.registry.root) end if state.resolution then print(state.resolution.digest, state.resolution.input_digest) for _, module in ipairs(state.resolution.modules) do print(module.name, module.version) end end ``` Each entry in `state.entries` has the four author-facing fields plus: - `registry.owner` - deployment source that supplied the entry - `registry.root` - `true` when the entry is a dependency declaration selected by the deployment `state.resolution` describes the module graph of a `registry.snapshot()` view. It is absent on snapshots that carry no graph of their own, including `registry.snapshot_at()` and overlay snapshots: | Field | Type | Description | |-------|------|-------------| | `digest` | string | Content digest of the complete immutable selection | | `input_digest` | string | Digest of the declared root set | | `baseline_digest` | string | Digest of the deployment baseline the graph was solved against; omitted when unbound | | `roots` | array | Authored dependency declarations used as solver inputs | | `references` | array | Root-shaped declarations folded into an existing root for the same component; omitted when empty | | `modules` | array | Selected modules | `roots` and `references` entries have `id`, `component` and `version`. `modules` entries have `name` and `version`, plus `version_id`, `source`, `digest`, `size_bytes` and `protected` when set. ### Versions ```lua local version, err = registry.current_version() local versions, err = registry.versions() print(version:id()) -- numeric ID print(version:string()) -- display string local prev = version:previous() -- previous version or nil local next = version:next() -- next version or nil ``` ### History ```lua local hist, err = registry.history() local versions, err = hist:versions() local version, err = hist:get_version(5) local snap, err = hist:snapshot_at(version) ``` ### Changesets Build and apply modifications: ```lua local snap, err = registry.snapshot() local changes = snap:changes() changes:create({ id = "test:new_entry", kind = "test.kind", meta = {type = "test"}, data = {config = "value"} }) changes:update({ id = "test:existing", kind = "test.kind", meta = {updated = true}, data = {new_value = true} }) changes:delete("test:old_entry") local new_version, err = changes:apply() ``` **Permission:** `registry.apply` for `changes:apply()` #### Deleting Entries `changes:delete()` accepts an ID string, a table with an `id` string, a table with `ns` and `name` strings, or an array of any of those. Arrays may nest, and duplicate IDs collapse into a single delete operation. ```lua changes:delete("test:old_entry") changes:delete({id = "test:old_entry"}) changes:delete({ns = "test", name = "old_entry"}) changes:delete({"test:a", {ns = "test", name = "b"}, {"test:c"}}) ``` An empty list, a table that references itself, and a value that is neither a string nor a table are rejected with `errors.INVALID`. #### Changes Methods | Method | Description | |--------|-------------| | `changes:create(entry)` | Add create operation | | `changes:update(entry)` | Add update operation | | `changes:delete(id)` | Add delete operation | | `changes:ops()` | Get pending operations | | `changes:apply()` | Apply changes, returns new Version | ### Apply Version Roll back or forward to a specific version: ```lua local prev = current_version:previous() local ok, err = registry.apply_version(prev) ``` **Permission:** `registry.apply_version` ### Build Delta Compute operations to transition between states: ```lua local from = {{id = "test:a", kind = "test", meta = {}, data = {}}} local to = {{id = "test:b", kind = "test", meta = {}, data = {}}} local ops, err = registry.build_delta(from, to) for _, op in ipairs(ops) do print(op.kind, op.entry.id) -- "entry.create", "entry.update", "entry.delete" end ``` ### Overlays An overlay is a process-local set of registry entries owned by a logical identity. Overlay entries take part in ordinary topology and handler transitions, so services start and stop for them exactly as for durable entries, but they never advance registry history and never appear in a version. They exist only in the running process and are empty after a cold boot, so the owning control service reconciles them on startup. ```lua local snap, err = registry.overlay("data-sources:crm") ``` **Returns:** `Snapshot, error` The snapshot exposes the owner's overlay entries through the usual methods and reports the current registry version from `snap:version()`. It also captures the overlay generation at the moment it is opened, which is what makes writes safe. ```lua local snap, err = registry.overlay("data-sources:crm") if err then return nil, err end local changes = snap:changes() changes:create({ id = "data.crm:connection", kind = "registry.entry", meta = {}, data = {endpoint = "https://crm.internal"} }) local version, err = changes:apply() ``` `changes:apply()` on an overlay snapshot writes the overlay and returns the current registry version. No history version is created, so the returned version is unchanged unless a durable change happened concurrently. #### Concurrency Each overlay carries a generation counter that increases on every successful apply. `changes:apply()` succeeds only if the generation still matches the one captured when the snapshot was opened. A concurrent apply to the same overlay fails with `errors.CONFLICT` marked retryable: reopen the overlay and rebuild the changeset. ```lua local last_err for _ = 1, 3 do local snap, err = registry.overlay("data-sources:crm") if err then return nil, err end local _, apply_err = snap:changes():delete("data.crm:connection"):apply() if not apply_err then return true end if not apply_err:retryable() then return nil, apply_err end last_err = apply_err end return nil, last_err ``` #### Restrictions - The owner string is required and must not be blank. - A changeset must be non-empty and must not name the same entry twice. - `create` fails when the ID already exists in durable state or in any overlay. - `update` and `delete` only work on entries this owner created; any other ID fails with `errors.NOT_FOUND`. - Overlay entries cannot set `dependency_root` or any other registry-owned metadata. - Overlay entries cannot use kinds owned by a registry directive, such as `ns.dependency`. - A delete that removes an entry a surviving entry depends on is rejected. - Dependencies cannot cross overlay owner boundaries, and durable entries cannot depend on overlay entries. The rest surface as `errors.CONFLICT` or `errors.INVALID`, and none are retryable: only the generation mismatch above is. **Permissions:** `registry.overlay.get` on the owner to open and read, `registry.overlay.apply` on the owner to write, and `registry.overlay..` on each entry ID in the changeset. ### Permissions | Permission | Resource | Description | |------------|----------|-------------| | `registry.get` | entry ID | Read entry (also filters find/entries results) | | `registry.apply` | - | Apply changeset | | `registry.apply_version` | - | Apply/rollback version | | `registry.overlay.get` | owner ID | Open and read an overlay snapshot | | `registry.overlay.apply` | owner ID | Apply an overlay changeset | | `registry.overlay.create.` | entry ID | Create an overlay entry of that kind | | `registry.overlay.update.` | entry ID | Update an overlay entry of that kind | | `registry.overlay.delete.` | entry ID | Delete an overlay entry of that kind | ### Errors | Condition | Kind | |-----------|------| | Entry not found | `errors.NOT_FOUND` | | Version not found | `errors.NOT_FOUND` | | Permission denied | `errors.PERMISSION_DENIED` | | Invalid parameter | `errors.INVALID` | | No changes to apply | `errors.INVALID` | | Overlay changed during apply | `errors.CONFLICT` (retryable) | | Overlay entry owned elsewhere or conflicts with durable state | `errors.CONFLICT` | | Registry not available | `errors.INTERNAL` | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Contracts" ## Contracts The `contract` module opens typed service bindings for remote APIs, workflows, and functions. Contracts support schema validation, asynchronous calls, and call-context propagation. This page is an API reference; IDs and values such as `current_user` represent application-owned entries and surrounding handler state. ### Loading ```lua local contract = require("contract") ``` ### Opening a Binding Open a binding by its registry ID: ```lua local greeter, err = contract.open("app.services:greeter") if err then return nil, err end local result, err = greeter:say_hello("Alice") if err then return nil, err end ``` Bindings can also receive scope values, query parameters, or call options: ```lua -- With scope table local svc, err = contract.open("app.services:user", { tenant_id = "acme", region = "us-east" }) -- With query parameters (auto-converted: "true"→bool, numbers→int/float) local api, err = contract.open("app.services:api?debug=true&timeout=5000") -- With call options (third argument) local inst, err = contract.open("app.services:flaky", nil, { retry = { max_attempts = 5, initial_delay = 100 } }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `binding_id` | string | Binding ID; query parameters are supported | | `scope` | table | Context values (optional, overrides query params) | | `options` | table | Call options (optional) — e.g. `retry.max_attempts`, `retry.initial_delay` | **Returns:** `Instance, error` ### Getting a Contract Retrieve a contract definition for introspection: ```lua local c, err = contract.get("app.services:greeter") if err then return nil, err end print(c:id()) -- "app.services:greeter" local methods = c:methods() for _, m in ipairs(methods) do print(m.name, m.description) end local method, err = c:method("say_hello") if err then return nil, err end ``` #### Method Definition | Field | Type | Description | |-------|------|-------------| | `name` | string | Method name | | `description` | string | Method description | | `input_schemas` | table[] | Input schema definitions (absent when the method declares none) | | `output_schemas` | table[] | Output schema definitions (absent when the method declares none) | ### Finding Implementations List the bindings that implement a contract: ```lua local bindings, err = contract.find_implementations("app.services:greeter") if err then return nil, err end for _, binding_id in ipairs(bindings) do print(binding_id) end ``` The same lookup is available on a contract object: ```lua local c, err = contract.get("app.services:greeter") if err then return nil, err end local bindings, err = c:implementations() if err then return nil, err end ``` ### Checking Implementation Check whether an already opened instance implements a contract: ```lua if contract.is(instance, "app.services:greeter") then instance:say_hello("World") end ``` ### Calling Methods A synchronous method call blocks until it completes: ```lua local calc, err = contract.open("app.services:calculator") if err then return nil, err end local sum, err = calc:add(10, 20) if err then return nil, err end local product, err = calc:multiply(5, 6) if err then return nil, err end ``` ### Async Calls Append `_async` to a method name to start it asynchronously: ```lua local processor, err = contract.open("app.services:processor") if err then return nil, err end local future, err = processor:process_async(large_dataset) if err then return nil, err end -- Do other work... -- Wait for result local ch = future:response() local _, open = ch:receive() if not open then return nil, errors.new("future response channel closed") end local payload, result_err = future:result() if result_err then return nil, result_err end local result, data_err = payload:data() if data_err then return nil, data_err end ``` See [Futures](lua/core/future.md) for future methods. ### Opening via Contract Open a binding through a contract object. The calls below are alternatives; check the error returned by `contract.get()` and by the selected `open()` call before using the instance. ```lua local c, err = contract.get("app.services:user") if err then return nil, err end -- Default binding local instance, err = c:open() -- Specific binding local instance, err = c:open("app.services:user_impl") -- With scope local instance, err = c:open(nil, {user_id = 123}) local instance, err = c:open("app.services:user_impl", {user_id = 123}) ``` ### Adding Context Create a wrapper with preconfigured context values: ```lua local ctx = require("ctx") local c, err = contract.get("app.services:user") if err then return nil, err end local request_id, ctx_err = ctx.get("request_id") if ctx_err then return nil, ctx_err end local wrapped, err = c:with_context({ request_id = request_id, user_id = current_user.id }) if err then return nil, err end local instance, err = wrapped:open() ``` ### Call Options Use `with_options` to configure retries and other call behavior: ```lua local c, err = contract.get("app.services:flaky") if err then return nil, err end local configured = c:with_options({ retry = { max_attempts = 5, initial_delay = 100 } }) local inst, err = configured:open("app.services:flaky_impl") if err then return nil, err end local result, err = inst:call() ``` Options apply to every method call on the returned instance. Only retryable errors trigger retries; non-retryable errors return immediately. `with_options` can be chained with `with_context`, `with_actor`, and `with_scope`. | Option | Type | Description | |--------|------|-------------| | `retry.max_attempts` | int | Maximum attempts including the first (1 disables retry) | | `retry.initial_delay` | int/duration | Delay before first retry (ms or duration string), default `100` | | `retry.max_delay` | int/duration | Upper bound for the backoff delay (ms or duration string), default `10s` | | `retry.backoff_factor` | number | Multiplier applied to the delay after each attempt, default `2.0` | | `retry.jitter` | number | Random jitter fraction applied to each delay, default `0.1` | | `retry.retry_kinds` | string[] | Only retry errors of these kinds; by default every kind except `Invalid`, `PermissionDenied` and `Internal` is retried | | `retry.skip_kinds` | string[] | Never retry errors of these kinds | ### Security Context Set the actor and scope used for authorization: ```lua local security = require("security") local c, err = contract.get("app.services:admin") if err then return nil, err end local secured, err = c:with_actor(security.actor()) if err then return nil, err end secured, err = secured:with_scope(security.scope()) if err then return nil, err end local admin, err = secured:open() if err then return nil, err end ``` Without explicit `with_actor`/`with_scope`, an opened contract inherits the caller's ambient actor and scope. When set, they propagate to the bound implementation functions — every method call on the instance executes under that identity. ### Permissions | Permission | Resource | Functions | |------------|----------|-----------| | `contract.get` | contract id | `get()` | | `contract.open` | binding id | `open()`, `Contract:open()` | | `contract.implementations` | contract id | `find_implementations()`, `Contract:implementations()` | | `contract.call` | method name | sync and async method calls | | `contract.context` | "context" | `Contract:with_context()` | | `contract.security` | "security" | `Contract:with_actor()`, `Contract:with_scope()` | ### Errors | Condition | Kind | |-----------|------| | Invalid binding ID format | `errors.INVALID` | | Contract not found | `errors.NOT_FOUND` | | Binding not found | `errors.NOT_FOUND` | | Method not found | `errors.NOT_FOUND` | | No default binding | `errors.NOT_FOUND` | | Permission denied | `errors.PERMISSION_DENIED` | | Call failed | kind of the implementation's error (preserved); `errors.INTERNAL` for dispatch failures | --- # "JSON Encoding" ## JSON Encoding The `json` module encodes Lua values as JSON, decodes JSON strings, and validates data with JSON Schema. This is an API reference. Short expression examples show successful return values; examples that consume the result capture the optional second `error` return. ### Loading ```lua local json = require("json") ``` Add `json` to the executable entry's `modules:` list before requiring it. #### `encode` Encode a Lua value as a JSON string: ```lua -- Simple values json.encode("hello") -- '"hello"' json.encode(42) -- '42' json.encode(true) -- 'true' json.encode(nil) -- 'null' -- Arrays (sequential numeric keys) json.encode({1, 2, 3}) -- '[1,2,3]' json.encode({"a", "b"}) -- '["a","b"]' -- Objects (string keys) local user = {name = "Alice", age = 30} json.encode(user) -- JSON object with name="Alice" and age=30; member order is unspecified -- Nested structures local order = { id = "ord-123", items = { {sku = "ABC", qty = 2}, {sku = "XYZ", qty = 1} }, total = 99.50 } json.encode(order) -- Structurally equivalent JSON; object-member order is unspecified ``` | Parameter | Type | Description | |-----------|------|-------------| | `value` | any | Lua value to encode | **Returns:** `string, error` Encoding follows these rules: - `nil` becomes `null` - Empty tables become `[]` (or `{}` if created with string keys) - Tables with sequential 1-based keys become arrays - Tables with string keys become objects - Mixed numeric and string keys cause an error - Sparse arrays (gaps in indices) cause an error - Inf/NaN numbers become `null` - Recursive table references cause an error - Maximum nesting depth is 128 levels #### `decode` Decode a JSON string into a Lua value: ```lua -- Parse object local user, err = json.decode('{"name":"Bob","active":true}') if err then return nil, err end print(user.name) -- "Bob" print(user.active) -- true -- Parse array local items, items_err = json.decode('[10, 20, 30]') if items_err then return nil, items_err end print(items[1]) -- 10 print(#items) -- 3 -- Parse nested data local response, response_err = json.decode([[ { "status": "ok", "data": { "users": [ {"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"} ] } } ]]) if response_err then return nil, response_err end print(response.data.users[1].name) -- "Alice" -- Handle errors local data, err = json.decode("not valid json") if err then print(err:kind()) -- "Internal" (errors.INTERNAL) print(err:message()) -- parse error details end ``` | Parameter | Type | Description | |-----------|------|-------------| | `str` | string | JSON string to decode | **Returns:** `any, error` #### `validate` Validate a Lua value against a JSON Schema: ```lua -- Define a schema local user_schema = { type = "object", properties = { name = {type = "string", minLength = 1}, email = {type = "string", format = "email"}, age = {type = "integer", minimum = 0, maximum = 150} }, required = {"name", "email"} } -- Valid data passes local valid, err = json.validate(user_schema, { name = "Alice", email = "alice@example.com", age = 30 }) if err then return nil, err end print(valid) -- true -- Invalid data fails with details local valid, err = json.validate(user_schema, { name = "", email = "not-an-email" }) if not valid then print(err:message()) -- validation error details end -- Schema can also be a JSON string local schema_json = '{"type":"number","minimum":0}' local valid, schema_err = json.validate(schema_json, 42) if schema_err then return nil, schema_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `schema` | table or string | JSON Schema definition | | `data` | any | Value to validate | **Returns:** `boolean, error` Schemas are cached by content hash for performance. #### `validate_string` Validate a JSON string against a schema without first returning a decoded value: ```lua local schema = { type = "object", properties = { action = {type = "string", enum = {"create", "update", "delete"}} }, required = {"action"} } -- Validate raw JSON from request body local body = '{"action":"create","data":{}}' local valid, err = json.validate_string(schema, body) if not valid then return nil, errors.new("Invalid request: " .. err:message()):kind(errors.INVALID) end -- Now safe to decode local request, decode_err = json.decode(body) if decode_err then return nil, decode_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `schema` | table or string | JSON Schema definition | | `json_str` | string | JSON string to validate | **Returns:** `boolean, error` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Recursive table reference | `errors.INTERNAL` | no | | Sparse array (gaps in indices) | `errors.INTERNAL` | no | | Mixed key types in table | `errors.INTERNAL` | no | | Nesting exceeds 128 levels | `errors.INTERNAL` | no | | Invalid JSON syntax | `errors.INTERNAL` | no | | Input not a string or empty string (decode) | `errors.INVALID` | no | | Schema compilation failed | `errors.INVALID` | no | | Validation failed | `errors.INVALID` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "YAML Encoding" ## YAML Encoding The `yaml` module serializes Lua tables as YAML and parses YAML documents into Lua values. This is an API reference. Output-only expressions illustrate successful encoding; examples that consume a value capture the optional second `error` return. ### Loading ```lua local yaml = require("yaml") ``` Add `yaml` to the executable entry's `modules:` list before requiring it. #### `encode` Encode a Lua table as YAML: ```lua -- Simple key-value local config = { name = "myapp", port = 8080, debug = true } local out, err = yaml.encode(config) if err then return nil, err end -- YAML mapping containing name, port, and debug. -- Arrays become YAML lists local items = {"apple", "banana", "cherry"} yaml.encode(items) -- - apple -- - banana -- - cherry -- Nested structures local server = { http = { address = ":8080", timeout = "30s" }, database = { host = "localhost", port = 5432 } } yaml.encode(server) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | table | Lua table to encode | | `options` | table? | Optional encoding options | ##### Options | Field | Type | Description | |-------|------|-------------| | `field_order` | string[] | Custom field order; listed fields appear in this order | | `sort_unordered` | boolean | Sort fields not in `field_order` alphabetically | ```lua -- Control field order in output local entry = { zebra = 1, alpha = 2, name = "test", kind = "demo" } -- Fields appear in specified order, remaining sorted alphabetically local result, encode_err = yaml.encode(entry, { field_order = {"name", "kind"}, sort_unordered = true }) if encode_err then return nil, encode_err end -- name: test -- kind: demo -- alpha: 2 -- zebra: 1 -- Just sort all fields alphabetically yaml.encode(entry, {sort_unordered = true}) -- alpha: 2 -- kind: demo -- name: test -- zebra: 1 ``` **Returns:** `string, error` #### `decode` Parse a YAML string into a Lua value: ```lua -- Parse configuration local config, err = yaml.decode([[ server: host: localhost port: 8080 features: - auth - logging - metrics ]]) if err then return nil, err end print(config.server.host) -- "localhost" print(config.server.port) -- 8080 print(config.features[1]) -- "auth" -- Parse from file content local content = fs.get("app:config"):readfile("config.yaml") local settings, err = yaml.decode(content) if err then return nil, errors.wrap(err, "invalid config file") end -- Handle mixed types local data, data_err = yaml.decode([[ name: test count: 42 ratio: 3.14 enabled: true tags: - lua - wippy ]]) if data_err then return nil, data_err end print(type(data.count)) -- "number" print(type(data.enabled)) -- "boolean" print(type(data.tags)) -- "table" ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | YAML string to parse | **Returns:** `any, error` — the value type depends on the YAML content ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Input not a table (encode) | `errors.INVALID` | no | | Input not a string (decode) | `errors.INVALID` | no | | Empty string (decode) | `errors.INVALID` | no | | Invalid YAML syntax | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Base64 Encoding" ## Base64 Encoding The `base64` module encodes strings and binary data using standard RFC 4648 Base64 and decodes them back to bytes. This is an API reference. Output-only expressions show successful values; filesystem and transport examples check the optional second `error` return before consuming data. Names such as `username`, `password`, `encoded_image`, and `user_input` are application-supplied strings. Base64 is an encoding, not encryption or authentication. Do not use it to conceal secrets or to verify that data has not been modified. Send Basic authentication credentials only over TLS and obtain them from application-owned secret storage rather than literals. ### Loading ```lua local base64 = require("base64") ``` Add `base64` to the executable entry's `modules:` list before requiring it. Filesystem and JSON examples also require `fs` and `json` respectively. #### `encode` Encodes a string, including binary data, as Base64. ```lua -- Encode text local encoded, err = base64.encode("Hello, World!") if err then return nil, err end print(encoded) -- "SGVsbG8sIFdvcmxkIQ==" -- Encode binary data (e.g., from file) local image_data = fs.get("app:data"):readfile("photo.jpg") local image_b64 = base64.encode(image_data) -- Encode JSON for transport local json = require("json") local payload, json_err = json.encode({user = "alice", action = "login"}) if json_err then return nil, json_err end local token_part, token_err = base64.encode(payload) if token_err then return nil, token_err end -- Encode credentials local credentials, credentials_err = base64.encode(username .. ":" .. password) if credentials_err then return nil, credentials_err end local auth_header = "Basic " .. credentials ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to encode (text or binary) | **Returns:** `string, error` — an empty input returns an empty string #### `decode` Decodes a Base64 string to its original bytes. ```lua -- Decode text local decoded, decode_err = base64.decode("SGVsbG8sIFdvcmxkIQ==") if decode_err then return nil, decode_err end print(decoded) -- "Hello, World!" -- Decode with error handling local data, err = base64.decode(user_input) if err then return nil, errors.new("Invalid base64 data"):kind(errors.INVALID) end -- Decode binary data local image_data, err = base64.decode(encoded_image) if err then return nil, err end fs.get("app:data"):writefile("output.jpg", image_data) -- Decode a base64-wrapped JSON document local json = require("json") local doc = json.decode(base64.decode(encoded_json)) ``` The final block demonstrates delimiter handling only. It does not parse or verify a signed token format. | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Base64-encoded string | **Returns:** `string, error` — an empty input returns an empty string ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Input not a string | `errors.INVALID` | no | | Invalid base64 characters | `errors.INVALID` | no | | Corrupted padding | `errors.INVALID` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Compression" ## Compression The `compress` module encodes and decodes strings with gzip, Brotli, Zstandard, raw DEFLATE, and zlib. This is an API reference with partial HTTP and storage recipes. Every operation materializes its complete input and output as Lua strings; use the archive or stream APIs when data must remain streaming. The examples assume the entry enables `compress` and any separately required modules such as `json` or `http`. ### Loading ```lua local compress = require("compress") ``` Add `compress` to the executable entry's `modules:` list before requiring it. ### GZIP Gzip is defined by RFC 1952. #### Compress {id="gzip-compress"} ```lua -- Compress for HTTP response local body, json_err = json.encode(large_response) if json_err then return nil, json_err end local compressed, err = compress.gzip.encode(body) if err then return nil, err end -- Set Content-Encoding header local header_err = res:set_header("Content-Encoding", "gzip") if header_err then return nil, header_err end local write_err = res:write(compressed) if write_err then return nil, write_err end -- Maximum compression for storage local archived, archive_err = compress.gzip.encode(data, {level = 9}) if archive_err then return nil, archive_err end -- Fast compression for real-time local fast, fast_err = compress.gzip.encode(data, {level = 1}) if fast_err then return nil, fast_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to compress | | `options` | table? | Optional encoding options | ##### Options {id="gzip-compress-options"} | Field | Type | Description | |-------|------|-------------| | `level` | integer | Compression level 1-9 (default: 6) | **Returns:** `string, error` #### Decompress {id="gzip-decompress"} ```lua -- Decompress HTTP request local content_encoding, header_err = req:header("Content-Encoding") if header_err then return nil, header_err end if content_encoding == "gzip" then local body, body_err = req:body() if body_err then return nil, body_err end local decompressed, err = compress.gzip.decode(body) if err then return nil, errors.new("Invalid gzip data"):kind(errors.INVALID) end body = decompressed end -- Decompress with size limit (prevent zip bombs) local decompressed, err = compress.gzip.decode(data, {max_size = 10 * 1024 * 1024}) if err then return nil, errors.new("Decompressed size exceeds 10MB limit"):kind(errors.INVALID) end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | GZIP compressed data | | `options` | table? | Optional decoding options | ##### Options {id="gzip-decompress-options"} | Field | Type | Description | |-------|------|-------------| | `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) | **Returns:** `string, error` ### Brotli Brotli is defined by RFC 7932 and is commonly used for compressed text content. #### Compress {id="brotli-compress"} ```lua -- Best for static assets and text content local compressed, err = compress.brotli.encode(html_content, {level = 11}) if err then return nil, err end -- Store `compressed` through the application's cache contract if needed. -- Moderate compression for API responses local compressed, err = compress.brotli.encode(json_data, {level = 4}) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to compress | | `options` | table? | Optional encoding options | ##### Options {id="brotli-compress-options"} | Field | Type | Description | |-------|------|-------------| | `level` | integer | Compression level 0-11 (default: 6) | **Returns:** `string, error` #### Decompress {id="brotli-decompress"} ```lua local decompressed, err = compress.brotli.decode(compressed_data) if err then return nil, err end -- With size limit local decompressed, err = compress.brotli.decode(data, {max_size = 50 * 1024 * 1024}) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Brotli compressed data | | `options` | table? | Optional decoding options | ##### Options {id="brotli-decompress-options"} | Field | Type | Description | |-------|------|-------------| | `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) | **Returns:** `string, error` ### Zstandard Zstandard is a general-purpose compression format defined by RFC 8878. #### Compress {id="zstd-compress"} ```lua -- Good balance of speed and ratio local compressed, err = compress.zstd.encode(binary_data) if err then return nil, err end -- Higher compression for archival local archived, archive_err = compress.zstd.encode(data, {level = 19}) if archive_err then return nil, archive_err end -- Fast mode for latency-sensitive payloads local fast, fast_err = compress.zstd.encode(data, {level = 1}) if fast_err then return nil, fast_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to compress | | `options` | table? | Optional encoding options | ##### Options {id="zstd-compress-options"} | Field | Type | Description | |-------|------|-------------| | `level` | integer | Compression level 1-22 (default: 3) | | `dict` | string? | Zstd dictionary bytes from `train_dict` (default: none) | **Returns:** `string, error` #### Decompress {id="zstd-decompress"} ```lua local decompressed, err = compress.zstd.decode(compressed_data) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Zstandard compressed data | | `options` | table? | Optional decoding options | ##### Options {id="zstd-decompress-options"} | Field | Type | Description | |-------|------|-------------| | `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) | | `dict` | string? | Zstd dictionary bytes (must match the dict used to encode) | **Returns:** `string, error` #### Dictionaries {id="zstd-dictionaries"} Train a dictionary from similar sample payloads, then pass it through the `dict` option to `encode` and `decode`. Decoding requires the same dictionary used for encoding. ```lua local dict, err = compress.zstd.train_dict(samples, { size = 112640 }) if err then return nil, err end local packed, pack_err = compress.zstd.encode(data, { dict = dict }) if pack_err then return nil, pack_err end local original, decode_err = compress.zstd.decode(packed, { dict = dict }) if decode_err then return nil, decode_err end ``` ##### train_dict(samples, options?) | Parameter | Type | Description | |-----------|------|-------------| | `samples` | string[] | Training samples (at least one >= 8 bytes) | | `options` | table? | `size` (integer, target dict bytes, 256-1048576, default 114688), `id` (integer, default 0), `level` (integer, 1-22) | **Returns:** `string, error` (the dictionary bytes) ##### inspect_dict(dict) | Parameter | Type | Description | |-----------|------|-------------| | `dict` | string | Dictionary bytes | **Returns:** `table, error` — `{id: integer, content_size: integer}` ### Deflate Raw DEFLATE is defined by RFC 1951 and is also used inside other formats. #### Compress {id="deflate-compress"} ```lua local compressed, err = compress.deflate.encode(data, {level = 6}) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to compress | | `options` | table? | Optional encoding options | ##### Options {id="deflate-compress-options"} | Field | Type | Description | |-------|------|-------------| | `level` | integer | Compression level 1-9 (default: 6) | **Returns:** `string, error` #### Decompress {id="deflate-decompress"} ```lua local decompressed, err = compress.deflate.decode(compressed) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | DEFLATE compressed data | | `options` | table? | Optional decoding options | ##### Options {id="deflate-decompress-options"} | Field | Type | Description | |-------|------|-------------| | `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) | **Returns:** `string, error` ### Zlib Zlib wraps DEFLATE data with a header and checksum as defined by RFC 1950. #### Compress {id="zlib-compress"} ```lua local compressed, err = compress.zlib.encode(data, {level = 6}) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to compress | | `options` | table? | Optional encoding options | ##### Options {id="zlib-compress-options"} | Field | Type | Description | |-------|------|-------------| | `level` | integer | Compression level 1-9 (default: 6) | **Returns:** `string, error` #### Decompress {id="zlib-decompress"} ```lua local decompressed, err = compress.zlib.decode(compressed) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Zlib compressed data | | `options` | table? | Optional decoding options | ##### Options {id="zlib-decompress-options"} | Field | Type | Description | |-------|------|-------------| | `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) | **Returns:** `string, error` ### Choosing an Algorithm | Algorithm | Best For | Speed | Ratio | Level Range | |-----------|----------|-------|-------|-------------| | gzip | HTTP, wide compatibility | Medium | Good | 1-9 | | brotli | Static assets, text | Slow | Best | 0-11 | | zstd | Binary payloads, fast compression | Fast | Good | 1-22 | | deflate/zlib | Low-level, specific protocols | Medium | Good | 1-9 | ```lua -- HTTP response based on Accept-Encoding local accept, header_err = req:header("Accept-Encoding") if header_err then return nil, header_err end accept = accept or "" local body, json_err = json.encode(response_data) if json_err then return nil, json_err end local qualities = {} for item in accept:gmatch("[^,]+") do local coding = item:match("^%s*([^;%s]+)") local has_q = item:match(";%s*[qQ]%s*=") ~= nil local q_text = item:match(";%s*[qQ]%s*=%s*([^;%s,]+)") local q if not has_q then q = 1 elseif q_text == "0" or q_text == "1" or (q_text and q_text:match("^0%.%d?%d?%d?$")) or (q_text and q_text:match("^1%.0?0?0?$")) then q = tonumber(q_text) end if coding and q and q >= 0 and q <= 1 then coding = coding:lower() qualities[coding] = math.max(qualities[coding] or 0, q) end end local function quality(coding) if qualities[coding] ~= nil then return qualities[coding] end if coding == "identity" then return qualities["*"] == 0 and 0 or 1 end return qualities["*"] or 0 end local selected, selected_q = nil, -1 for _, coding in ipairs({"br", "gzip", "identity"}) do local q = quality(coding) if q > selected_q then selected, selected_q = coding, q end end -- Include every field used by this handler or its surrounding middleware. local vary_fields = {"Accept-Encoding"} local vary_err = res:set_header("Vary", table.concat(vary_fields, ", ")) if vary_err then return nil, vary_err end if selected_q <= 0 then local status_err = res:set_status(http.STATUS.NOT_ACCEPTABLE) if status_err then return nil, status_err end local write_err = res:write("No acceptable content encoding") if write_err then return nil, write_err end elseif selected == "br" then local compressed, compress_err = compress.brotli.encode(body) if compress_err then return nil, compress_err end local set_err = res:set_header("Content-Encoding", "br") if set_err then return nil, set_err end local write_err = res:write(compressed) if write_err then return nil, write_err end elseif selected == "gzip" then local compressed, compress_err = compress.gzip.encode(body) if compress_err then return nil, compress_err end local set_err = res:set_header("Content-Encoding", "gzip") if set_err then return nil, set_err end local write_err = res:write(compressed) if write_err then return nil, write_err end else local write_err = res:write(body) if write_err then return nil, write_err end end ``` This partial handler parses exact coding tokens and RFC q-values, honors explicit rejections such as `br;q=0`, and emits `Vary: Accept-Encoding`. `set_header` replaces an existing `Vary` value, so add every other field used by surrounding middleware to `vary_fields` before setting it. A full HTTP stack may provide a shared negotiation helper instead. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty input | `errors.INVALID` | no | | Level out of range | `errors.INVALID` | no | | Invalid compressed data | `errors.INVALID` | no | | Decompressed size exceeds limit | `errors.INTERNAL` (gzip, zlib, zstd) / `errors.INVALID` (deflate, brotli) | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Archive" ## Archive Read and write zip/tar archives with bounded memory. Archives are never loaded into RAM nor extracted to disk — peak memory is independent of archive and entry size, so multi-GB archives run on a low-RAM server. ### Loading ```lua local archive = require("archive") ``` ### Formats Built-in formats are detected by magic bytes, or forced with `opts.format`: | Format | Random read | Sequential scan | Write | |--------|:-----------:|:---------------:|:-----:| | `zip` | yes | yes (local headers) | yes | | `tar` | yes | yes | yes | | `tar.gz` | no | yes | yes | | `tar.zst` | no | yes | yes | `archive.formats()` returns the list of registered format names. ```lua local names = archive.formats() -- {"zip", "tar", "tar.gz", "tar.zst", ...} ``` ### Options All entrypoints accept an optional `opts` table: | Key | Default | Meaning | |-----|---------|---------| | `format` | auto | `"zip"`, `"tar"`, `"tar.gz"`, `"tar.zst"`; auto = sniff magic, else extension | | `max_entries` | 100000 | Reject archives with more entries (decompression-bomb defense) | | `max_total_bytes` | 2 GiB | Cap on cumulative uncompressed output during read/extract | | `max_file_bytes` | 1 GiB | Cap on a single entry's uncompressed size | | `max_inline_bytes` | 16 MiB | Hard cap for the RAM-materializing `read()` call; above it, use `stream()`/`extract()` | | `buffer_bytes` | 64 KiB | Streaming copy buffer for read/extract/add | `max_total_bytes`/`max_file_bytes` are work caps, not RAM caps — streaming an entry never holds more than `buffer_bytes` plus the codec's decompression window. The only RAM-sizing knob is `max_inline_bytes`. ### Reading — Random Access `archive.open(source, ...)` opens a **seekable** source for full random access (zip central directory is read up front; entries decompress on demand). The source may be an `fs.FS` handle plus a path, an open `fs.File`, raw bytes (bytes hold the whole archive in RAM — small archives only), or any random-access reader handed over by another module. A reader from another module qualifies when it implements `io.ReaderAt` and reports its `Size`; an optional `Name` is used for extension sniffing when `opts.format` is omitted. [`cloudstorage`](lua/storage/cloud.md) `open_reader` is one, which reads a multi-GB archive directly out of object storage. The archive opens nothing in that case and never closes the reader — its owner does. ```lua local fs = require("fs") local archive = require("archive") -- Open by fs handle + path (the module opens the file and owns its lifecycle) local r, err = archive.open(fs.get("app:uploads"), "incoming.zip") -- Or from an already-open seekable fs.File -- local r = archive.open(fs:get("app:uploads"):open("x.zip")) -- Or from raw bytes (small archives only) -- local r = archive.open(zip_bytes, { format = "zip" }) -- Or from a random-access reader owned by another module -- local reader = cloudstorage.get("app:files"):open_reader("incoming.zip") -- local r = archive.open(reader) ``` **Returns:** `Reader, error` **Permission:** `archive.read` #### entries Iterate the directory (metadata only — no decompression): ```lua for e in r:entries() do -- e: name, size, compressed_size, is_dir, mode, modified, method, crc32, type print(e.name, e.size, e.is_dir) end ``` #### stat Get entry metadata by name (no decompression): ```lua local info, err = r:stat("docs/readme.md") ``` #### read Materialize a single entry as a Lua string. Errors (`kind = Invalid`) above `max_inline_bytes` — for anything large, use `stream()` or `extract()`: ```lua local data, err = r:read("docs/readme.md") -- small entries only ``` #### stream Return the entry as a `stream.Stream` that decompresses on demand. Composes everywhere a stream does — `:scanner()`, `fs:writefile()`, or handed to another module: ```lua local es, err = r:stream("big.csv") while true do local chunk = es:read(65536) if not chunk then break end process(chunk) end es:close() ``` #### extract Stream one entry into a destination filesystem: ```lua local ok, err = r:extract("docs/readme.md", fs.get("app:out")) -- optional destination path: -- r:extract("docs/readme.md", fs.get("app:out"), "readme.md") ``` #### extract_all Stream every entry into a destination filesystem: ```lua local count, err = r:extract_all(fs.get("app:out"), { prefix = "job123/", -- prepend to each destination path strip = 1, -- drop N leading path components filter = function(e) return not e.is_dir end, }) ``` Entry names are sanitized on extract — `..` segments, absolute paths, and Windows drive/UNC prefixes are rejected (zip-slip defense). #### close Close the reader. Idempotent; also auto-closed at task scope. ```lua r:close() ``` ### Reading — Sequential Scan `archive.scan(source, opts?)` opens a **forward-only** stream (an HTTP upload body, a multipart file stream). Entries are visited in archive order; each entry's reader is valid only until you advance. No random `read(name)`. ```lua local up = form.files.upload[1]:stream() -- stream.Stream local s, err = archive.scan(up, { format = "zip" }) for e, entry in s:walk() do -- entry is a stream.Stream if not e.is_dir then fs.get("app:uploads"):writefile("job123/" .. e.name, entry) end end s:close() ``` **Returns:** `Walker, error` **Permission:** `archive.read` A walker also supports `extract_all` with the same options as the random-access reader, streaming every entry into a destination filesystem in one call: ```lua local count, err = s:extract_all(fs.get("app:uploads"), { prefix = "job123/" }) ``` `tar`, `tar.gz`, and `tar.zst` stream natively. `zip` is parsed via per-entry local headers; entries written with a streaming data descriptor (size/CRC trailing the data) are read by decompressing to the entry boundary. For robust zip handling of large uploads, land the upload as a file first (a bounded sequential copy) then use `archive.open`: ```lua local dst = fs.get("app:tmp") dst:writefile("u.zip", req:stream()) -- streaming copy upload → fs file local r = archive.open(dst, "u.zip") -- robust random access -- ... entries / extract_all ... r:close() dst:remove("u.zip") ``` ### Writing `archive.create(dest, ...)` builds an archive by streaming entries into a destination — a file in an fs (with a path) or a writable `stream.Stream` (e.g. an HTTP response), so a download `.zip` is generated straight to the wire with bounded memory. ```lua local w, err = archive.create(fs.get("app:tmp"), "out.zip", { format = "zip" }) -- or stream to a response: -- local w = archive.create(res:stream(), { format = "zip" }) ``` **Returns:** `Writer, error` **Permission:** `archive.write` #### add Add an entry from a string, bytes, reader, or `stream.Stream`: ```lua w:add("notes.txt", "hello") w:add("from_upload", some_stream, { method = "deflate", mode = tonumber("644", 8) }) ``` #### add_file Stream an entry from a file in a filesystem: ```lua w:add_file("data/big.bin", fs.get("app:data"), "big.bin") ``` #### add_dir Add a directory entry: ```lua w:add_dir("empty/") ``` #### close Finalize the archive (writes the central directory for zip). Idempotent; also auto-closed at task scope. ```lua w:close() ``` `add*` options: `{ method = "store"|"deflate", mode, size }`. Tar formats need the entry size up front, so `add()` from a stream or reader into a `tar*` archive requires `size` (strings and `add_file` supply it). The zip writer streams to non-seekable writers using data descriptors, so writing to a response stream works. ### Errors | Condition | Kind | |-----------|------| | Source is not an fs handle, an fs file, bytes or a random-access reader | `errors.INVALID` | | Unknown / mismatched format | `errors.INVALID` | | Corrupt or truncated archive | `errors.INVALID` | | Limit exceeded (entries / total / file / inline) | `errors.INVALID` | | Random access on a stream-only format (use `scan`) | `errors.UNAVAILABLE` | | Entry name not found | `errors.NOT_FOUND` | | Source not readable / destination not writable | `errors.PERMISSION_DENIED` | | Read a stale streamed entry after the walk advanced | `errors.INTERNAL` | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Filesystem](lua/storage/filesystem.md) - Source and destination filesystems - [Stream](lua/core/stream.md) - Stream objects handed to and from archives - [Compression](lua/data/compress.md) - In-memory gzip/deflate/zstd - [Cloud Storage](lua/storage/cloud.md) - `open_reader` as a random-access archive source --- # "Payload Encoding" ## Payload Encoding Payloads carry typed values between functions, processes, services, and workflows. They can be inspected, extracted, or transcoded between supported formats. This is an API reference with partial transport recipes. Values such as `p`, `input_data`, and the asynchronous target entry come from the surrounding application. ### Loading `payload` is a global namespace and does not require `require()`. ```lua payload.new(...) -- direct access ``` ### Format Constants The following constants identify payload formats: ```lua payload.format.JSON -- "json/plain" payload.format.YAML -- "yaml/plain" payload.format.STRING -- "text/plain" payload.format.BYTES -- "application/octet-stream" payload.format.MSGPACK -- "application/msgpack" payload.format.LUA -- "lua/any" payload.format.GOLANG -- "golang/any" payload.format.ERROR -- "golang/error" ``` ### Creating Payloads Create a payload from a Lua value: ```lua -- From table local p = payload.new({ user_id = 123, name = "Alice", roles = {"admin", "user"} }) -- From string local str_p = payload.new("Hello, World!") -- From number local num_p = payload.new(42.5) -- From boolean local bool_p = payload.new(true) -- From nil local nil_p = payload.new(nil) -- From error local err_p = payload.new(errors.new("something failed")) ``` | Parameter | Type | Description | |-----------|------|-------------| | `value` | any | Lua value (string, number, boolean, table, nil, or error) | **Returns:** `Payload` ### Getting Format Read the payload's format identifier: ```lua local p = payload.new({name = "test"}) local format = p:get_format() -- "lua/any" local str_p = payload.new("hello") local format2 = str_p:get_format() -- "lua/any" local err_p = payload.new(errors.new("failed")) local format3 = err_p:get_format() -- "golang/error" ``` **Returns:** `string` - one of `payload.format.*` constants ### Extracting Data Extract the payload's Lua value, transcoding when needed: ```lua local p = payload.new({ items = {1, 2, 3}, total = 100 }) local data, err = p:data() if err then return nil, err end print(data.total) -- 100 print(data.items[1]) -- 1 ``` **Returns:** `any, error` ### Transcoding Payloads Transcode a payload to another supported format: ```lua local p = payload.new({ name = "test", value = 123 }) -- Convert to JSON local json_p, err = p:transcode(payload.format.JSON) if err then return nil, err end print(json_p:get_format()) -- "json/plain" -- Convert to MessagePack (compact binary) local msgpack_p, err = p:transcode(payload.format.MSGPACK) if err then return nil, err end -- Convert to YAML local yaml_p, yaml_err = p:transcode(payload.format.YAML) if yaml_err then return nil, yaml_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `format` | string | Target format from `payload.format.*` | **Returns:** `Payload, error` ### Unmarshalling Decode a payload to a Lua value regardless of its source format: ```lua local data, err = p:unmarshal() if err then return nil, err end ``` `unmarshal()` behaves like `data()`: both transcode non-Lua payloads to the Lua format and return the resulting Lua value. The only difference is that `unmarshal()` returns an error when the transcoded data is not a valid Lua value, whereas `data()` returns `nil`. **Returns:** `any, error` ### Async Results Asynchronous function calls return their values in payloads: This example assumes `app.process:compute` returns exactly one value. With no result, `future:result()` returns `nil`; with multiple results, it returns a Lua table rather than one `Payload`, so callers must handle those shapes separately. ```lua local funcs = require("funcs") local future, err = funcs.async("app.process:compute", input_data) if err then return nil, err end -- Wait for result local ch = future:response() local _, ok = ch:receive() if not ok then return nil, errors.new("channel closed") end local result_payload, result_err = future:result() if result_err then return nil, result_err end if result_payload == nil then return nil, errors.new("compute returned no result") end -- Extract data from payload local result, err = result_payload:data() if err then return nil, err end print(result.computed_value) ``` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Transcoding failure | `errors.INTERNAL` | no | | Result not valid Lua value | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Excel Spreadsheets" ## Excel Spreadsheets Read and write Microsoft Excel files (.xlsx). Create workbooks, manage sheets, read cell values, and generate reports with formatting support. ### Loading ```lua local excel = require("excel") ``` #### New Workbook Creates a new empty Excel workbook. ```lua local wb, err = excel.new() if err then return nil, err end -- Create sheets and add data wb:new_sheet("Report") wb:set_cell_value("Report", "A1", "Title") wb:close() ``` **Returns:** `Workbook, error` #### Open Workbook Opens an Excel workbook from a reader object. ```lua local fs = require("fs") local vol, err = fs.get("app:data") if err then return nil, err end local file, err = vol:open("/reports/sales.xlsx", "r") if err then return nil, err end local wb, err = excel.open(file) if err then file:close() return nil, err end -- Read data from workbook local rows = wb:get_rows("Sheet1") for i, row in ipairs(rows) do print("Row " .. i .. ": " .. table.concat(row, ", ")) end wb:close() file:close() ``` | Parameter | Type | Description | |-----------|------|-------------| | `reader` | File | Must implement io.Reader (e.g., fs.File) | **Returns:** `Workbook, error` #### Create Sheet Creates a new sheet or returns existing sheet index. ```lua local wb = excel.new() -- Create sheets local idx1 = wb:new_sheet("Summary") local idx2 = wb:new_sheet("Details") local idx3 = wb:new_sheet("Charts") -- If sheet exists, returns its index local existing = wb:new_sheet("Summary") -- returns same as idx1 ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Sheet name | **Returns:** `integer, error` #### List Sheets Returns list of all sheet names in workbook. ```lua local wb = excel.new() wb:new_sheet("Sales") wb:new_sheet("Expenses") wb:new_sheet("Summary") local sheets = wb:get_sheet_list() -- sheets = {"Sheet1", "Sales", "Expenses", "Summary"} for _, name in ipairs(sheets) do print("Sheet:", name) end ``` **Returns:** `string[], error` #### Set Cell Value Sets value of a single cell. ```lua local wb = excel.new() wb:new_sheet("Data") -- Set different value types wb:set_cell_value("Data", "A1", "Product Name") -- string wb:set_cell_value("Data", "B1", "Price") -- string wb:set_cell_value("Data", "C1", "In Stock") -- string wb:set_cell_value("Data", "A2", "Widget") wb:set_cell_value("Data", "B2", 29.99) -- number wb:set_cell_value("Data", "C2", true) -- boolean wb:set_cell_value("Data", "A3", "Gadget") wb:set_cell_value("Data", "B3", 49.99) wb:set_cell_value("Data", "C3", false) -- Cell references support columns beyond Z wb:set_cell_value("Data", "AA1", "Extended Column") wb:set_cell_value("Data", "AB100", "Far cell") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sheet` | string | Sheet name | | `cell` | string | Cell reference ("A1", "B2", "AA100") | | `value` | any | string, integer, number, or boolean | **Returns:** `error` #### Get All Rows Gets all rows from a sheet as 2D array. ```lua local wb = excel.new() wb:new_sheet("Report") wb:set_cell_value("Report", "A1", "Name") wb:set_cell_value("Report", "B1", "Score") wb:set_cell_value("Report", "A2", "Alice") wb:set_cell_value("Report", "B2", 95) wb:set_cell_value("Report", "A3", "Bob") wb:set_cell_value("Report", "B3", 87) local rows, err = wb:get_rows("Report") if err then return nil, err end -- rows[1] = {"Name", "Score"} -- rows[2] = {"Alice", "95"} -- rows[3] = {"Bob", "87"} for i, row in ipairs(rows) do if i == 1 then print("Headers:", row[1], row[2]) else print("Data:", row[1], "scored", row[2]) end end ``` | Parameter | Type | Description | |-----------|------|-------------| | `sheet` | string | Sheet name | **Returns:** `string[][], error` All cell values returned as strings. Booleans as "TRUE" or "FALSE", numbers as string representation. #### Stream Rows `wb:rows(sheet)` opens a streaming cursor over one sheet. The sheet is decoded incrementally in constant memory, unlike `get_rows` which materializes the entire sheet: ```lua local cursor, err = wb:rows("Report") if err then return nil, err end while true do local batch, err = cursor:read(500) if err then cursor:close() return nil, err end if not batch then break -- end of sheet end for _, row in ipairs(batch) do process(row) end end cursor:close() ``` | Method | Description | |--------|-------------| | `cursor:read(n?)` | Read the next batch of up to `n` rows (default 1, max 10000). Returns `string[][], error`; `nil, nil` at end of sheet | | `cursor:close()` | Release the cursor (idempotent; cursors also close with the workbook) | Cell values format identically to `get_rows`. Empty rows come back as empty tables, and trailing empty rows are preserved rather than trimmed. After end-of-sheet or an error, subsequent reads keep returning that same state. #### Write to File Writes workbook to a writer object. ```lua local fs = require("fs") local wb = excel.new() -- Build report wb:new_sheet("Monthly Report") wb:set_cell_value("Monthly Report", "A1", "Month") wb:set_cell_value("Monthly Report", "B1", "Revenue") wb:set_cell_value("Monthly Report", "A2", "January") wb:set_cell_value("Monthly Report", "B2", 45000) wb:set_cell_value("Monthly Report", "A3", "February") wb:set_cell_value("Monthly Report", "B3", 52000) -- Write to file local vol, err = fs.get("app:output") if err then wb:close() return nil, err end local file, err = vol:open("/reports/monthly.xlsx", "w") if err then wb:close() return nil, err end local err = wb:write_to(file) file:close() wb:close() if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `writer` | File | Must implement io.Writer (e.g., fs.File) | **Returns:** `error` #### Serialize to a String Renders the workbook into an `xlsx` byte string, without a filesystem or a writer. Use it to hand a workbook to an HTTP response, an object store or a queue message. ```lua local cloudstorage = require("cloudstorage") local wb = excel.new() wb:new_sheet("Report") wb:set_cell_value("Report", "A1", "Total") wb:set_cell_value("Report", "B1", 45000) local data, err = wb:bytes() wb:close() if err then return nil, err end local storage = cloudstorage.get("app.infra:files") storage:upload_object("reports/monthly.xlsx", data, { content_type = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", }) storage:release() ``` **Returns:** `string, error` The whole workbook is materialized in memory. `write_to` builds the same in-memory buffer and then copies it to the writer, so it saves the Lua string but does not stream a large workbook. Calling `bytes()` on a closed workbook returns an `errors.INTERNAL` error. #### Close Workbook Closes workbook and releases resources. ```lua local wb = excel.new() -- ... work with workbook ... wb:close() -- Safe to call multiple times wb:close() ``` **Returns:** `error` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | No context | `errors.INTERNAL` | no | | Invalid workbook | `errors.INVALID` | no | | Workbook closed | `errors.INTERNAL` (`errors.INVALID` from `rows`) | no | | Not a writer (`write_to`) | `errors.INTERNAL` | no | | Not a reader (`open`) | raised as an argument error | no | | Invalid Excel file | `errors.INTERNAL` | no | | Non-existent sheet | `errors.INTERNAL` (`errors.INVALID` from `rows`) | no | | Invalid cell reference | `errors.INTERNAL` | no | | Write failed | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Filesystem](lua/storage/filesystem.md) - File operations for reading/writing Excel files --- # "HTTP" ## HTTP The `http` module reads the current server-side request and builds its response, including headers, route data, body content, streaming output, and Server-Sent Events. This is an API reference with partial handler recipes. Names such as `id`, `data`, `token`, and application callbacks come from the surrounding handler. Request accessors generally return `value, error`, and response mutations return `error`; examples that consume a result check those errors. For server configuration, see [HTTP Server](http/server.md). ### Loading ```lua local http = require("http") ``` Add `http` to the executable entry's `modules:` list before requiring it. Examples using `uuid`, `fs`, or `time` require those modules separately. ### Accessing the Request Read the current HTTP request context: ```lua local req, err = http.request({ timeout = 5000, -- 5 second body read timeout max_body = 10485760 -- 10MB max body }) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `options.timeout` | integer | Body read timeout in ms (default: 300000 / 5 min) | | `options.max_body` | integer | Max body size in bytes (default: 120MB) | **Returns:** `Request, error` ### Accessing the Response Read the current HTTP response context: ```lua local res, err = http.response() if err then return nil, err end ``` **Returns:** `Response, error` #### `method` Return the request's HTTP method. ```lua local method, method_err = req:method() if method_err then return nil, method_err end if method == http.METHOD.GET then return get_resource(id) elseif method == http.METHOD.POST then local data, body_err = req:body_json() if body_err then return nil, body_err end return create_resource(data) elseif method == http.METHOD.PUT then local data, body_err = req:body_json() if body_err then return nil, body_err end return update_resource(id, data) elseif method == http.METHOD.DELETE then return delete_resource(id) end ``` #### `path` Return the request path. ```lua local path, err = req:path() if err then return nil, err end print(path) -- "/api/users/123" -- Route based on path if path:match("^/api/") then return handle_api(req) end ``` #### `query` Return one query parameter: ```lua -- GET /search?q=hello&page=2&limit=10 local query, query_err = req:query("q") if query_err then return nil, query_err end -- With defaults local page_text, page_err = req:query("page") if page_err then return nil, page_err end local page = tonumber(page_text) or 1 ``` #### `query_params` Return all query parameters. Multiple values for one key are joined with commas. ```lua -- GET /search?tags=lua&tags=go&active=true local params, err = req:query_params() if err then return nil, err end -- {tags = "lua,go", active = "true"} for key, value in pairs(params) do print(key .. ": " .. value) end ``` #### `header` Return one request header by name. ```lua local uuid = require("uuid") local auth, auth_err = req:header("Authorization") if auth_err then return nil, auth_err end if not auth then local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.UNAUTHORIZED) if status_err then return nil, status_err end return res:write_json({error = "Missing authorization"}) end local correlation_id, correlation_err = req:header("X-Correlation-ID") if correlation_err then return nil, correlation_err end if not correlation_id then correlation_id, correlation_err = uuid.v4() if correlation_err then return nil, correlation_err end end ``` Lookup is case-insensitive: `req:header("content-type")` and `req:header("Content-Type")` return the same value. A header sent more than once returns its values joined with `", "`. A header that is not present returns `nil`. #### headers Gets every request header. ```lua local headers, err = req:headers() for name, value in pairs(headers) do print(name .. ": " .. value) end ``` **Returns:** `table, error` Keys are canonical header names (`Content-Type`, `X-Correlation-ID`), regardless of the casing the client sent. Repeated headers are joined with `", "`, as in `req:header()`. #### content_type Return the `Content-Type` header: ```lua local ct, type_err = req:content_type() -- "application/json; charset=utf-8" or nil if type_err then return nil, type_err end ``` #### `content_length` Return the `Content-Length` header value: ```lua local length, length_err = req:content_length() -- number of bytes if length_err then return nil, length_err end ``` #### `host` Return the `Host` header: ```lua local host, host_err = req:host() -- "example.com:8080" if host_err then return nil, host_err end ``` #### `param` Return one route parameter from a path pattern such as `/users/:id`: ```lua -- Route: /users/:id/posts/:post_id local id, param_err = req:param("id") if param_err then return nil, param_err end local valid = false if id then local validate_err valid, validate_err = uuid.validate(id) if validate_err then return nil, validate_err end end if not valid then local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.BAD_REQUEST) if status_err then return nil, status_err end return res:write_json({error = "Invalid ID format"}) end ``` #### `params` Return all route parameters: ```lua -- Route: /orgs/:org/repos/:repo/issues/:issue local p, err = req:params() if err then return nil, err end -- {org = "acme", repo = "widget", issue = "123"} local issue = get_issue(p.org, p.repo, p.issue) ``` #### `body` Read the full request body as a string: ```lua local body, err = req:body() if err then return nil, err end -- Parse XML manually local is_xml, type_err = req:is_content_type("application/xml") if type_err then return nil, type_err end if is_xml then local data = parse_xml(body) end -- Avoid logging raw request bodies; record only non-sensitive metadata. logger.debug("Request body read", {length = #body}) ``` `body()`, `body_json()`, `stream()`, and `parse_multipart()` consume the same request body. Choose one body-reading path per handler. `body()` and `body_json()` enforce the request object's timeout and size limit; `stream()` is incremental and does not apply those two options. #### `body_json` Read and parse the request body as JSON: ```lua local data, err = req:body_json() if err then local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.BAD_REQUEST) if status_err then return nil, status_err end return res:write_json({error = "Invalid JSON: " .. err:message()}) end -- Validate required fields if not data.name or not data.email then local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.BAD_REQUEST) if status_err then return nil, status_err end return res:write_json({error = "Missing required fields"}) end local user = create_user(data) ``` #### `has_body` Check whether the request has a body. ```lua local has_body, body_state_err = req:has_body() if body_state_err then return nil, body_state_err end if has_body then local data, body_err = req:body_json() if body_err then return nil, body_err end process(data) else local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.BAD_REQUEST) if status_err then return nil, status_err end return res:write_json({error = "Request body required"}) end ``` `has_body()` returns `true` only when the request has a body object and a positive `Content-Length`. A chunked request or another request with unknown length can return `false`; handlers that permit such bodies should attempt their chosen body reader and handle its error instead. #### `is_content_type` Check whether the request has the specified content type. ```lua local is_json, type_check_err = req:is_content_type("application/json") if type_check_err then return nil, type_check_err end if not is_json then local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(415) if status_err then return nil, status_err end return res:write_json({error = "Content-Type must be application/json"}) end ``` #### `accepts` Check whether the request accepts the specified content type. ```lua local accepts_json, json_accept_err = req:accepts("application/json") if json_accept_err then return nil, json_accept_err end local accepts_html, html_accept_err = req:accepts("text/html") if html_accept_err then return nil, html_accept_err end if accepts_json then return res:write_json(data) elseif accepts_html then local type_err = res:set_content_type("text/html; charset=utf-8") if type_err then return nil, type_err end return res:write(render_html(data)) else local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.NOT_ACCEPTABLE) if status_err then return nil, status_err end return res:write_json({error = "Cannot produce acceptable response"}) end ``` The pinned `accepts()` helper performs exact comma-separated matches plus `*/*`; it does not process media-type parameters, subtype wildcards, or quality weights, and a missing `Accept` header returns `false`. Use application-owned negotiation when those HTTP semantics matter. #### `remote_addr` Return the client's remote network address. ```lua local addr, addr_err = req:remote_addr() -- "192.168.1.100:54321" if addr_err then return nil, addr_err end -- Extract the host from IPv4 and bracketed IPv6 addresses local ip = addr:match("^%[([^%]]+)%]:%d+$") or addr:match("^([^:]+):%d+$") or addr -- Rate limiting by IP if rate_limiter:is_limited(ip) then local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.TOO_MANY_REQUESTS) if status_err then return nil, status_err end return res:write_json({error = "Too many requests"}) end ``` #### `parse_multipart` Parse multipart form data, including file uploads. The optional `max_memory` argument sets the number of bytes held in memory before data spills to temporary files; the default is 32 MB. ```lua local uuid = require("uuid") local form, err = req:parse_multipart() -- or req:parse_multipart(8 * 1024 * 1024) if err then local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.BAD_REQUEST) if status_err then return nil, status_err end return res:write_json({error = "Invalid form data"}) end -- Access form values local title = form.values.title local description = form.values.description -- Access uploaded files if form.files.avatar then local file = form.files.avatar[1] local filename, name_err = file:name() -- untrusted client metadata if name_err then return nil, name_err end local size, size_err = file:size() if size_err then return nil, size_err end local content_type, header_err = file:header("Content-Type") -- "image/jpeg" if header_err then return nil, header_err end -- Read file content local stream = file:stream() local parts = {} while true do local chunk, err = stream:read(65536) if err or not chunk then break end parts[#parts + 1] = chunk end stream:close() local content = table.concat(parts) local stream, stream_err = file:stream() if stream_err then return nil, stream_err end local stored_name, id_err = uuid.v7() if id_err then stream:close() return nil, id_err end local _, write_err = uploads:writefile(stored_name, stream, "wx") local _, close_err = stream:close() if write_err then return nil, write_err end if close_err then return nil, close_err end end -- Handle multiple files if form.files.documents then for _, file in ipairs(form.files.documents) do process_document(file) end end ``` Multipart field values are strings when a field occurs once and arrays when it occurs repeatedly. Treat uploaded filenames and `Content-Type` values as untrusted metadata; generate the storage name and inspect the content independently when its type matters. The exclusive `wx` write prevents overwriting an existing object. A failed write does not prove that the target belongs to this request, so the failure path must not remove it blindly. Applications that need cleanup after partial writes should stage uploads under an ownership-tracked temporary name and promote them only after the write succeeds. #### `stream` Read the request body as a stream: ```lua local stream, stream_err = req:stream() if stream_err then return nil, stream_err end -- Process in chunks local read_err while true do local chunk chunk, read_err = stream:read(65536) -- 64KB chunks if read_err or not chunk then break end process_chunk(chunk) end local _, close_err = stream:close() if read_err then return nil, read_err end if close_err then return nil, close_err end ``` #### `set_status` Set the response status code. `set_status()` writes the status and commits the response headers immediately. Call `set_header()`, `set_content_type()`, or `set_transfer()` first; later header changes return `errors.INVALID`. ```lua local status_err = res:set_status(http.STATUS.CREATED) if status_err then return nil, status_err end -- Other common choices: 204 No Content, 400 Bad Request, -- 401 Unauthorized, 403 Forbidden, 404 Not Found, and 500 Internal Error. ``` #### `set_header` Set one response header. ```lua local request_id_err = res:set_header("X-Request-ID", correlation_id) if request_id_err then return nil, request_id_err end local cache_err = res:set_header("Cache-Control", "max-age=3600") if cache_err then return nil, cache_err end local rate_err = res:set_header("X-RateLimit-Remaining", tostring(remaining)) if rate_err then return nil, rate_err end -- CORS headers local origin_err = res:set_header("Access-Control-Allow-Origin", "*") if origin_err then return nil, origin_err end local methods_err = res:set_header("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE") if methods_err then return nil, methods_err end local headers_err = res:set_header("Access-Control-Allow-Headers", "Content-Type, Authorization") if headers_err then return nil, headers_err end ``` #### `set_content_type` Set the response content type. ```lua local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end -- Other examples: "text/html; charset=utf-8" or "application/pdf". ``` #### `write` Write to the response body: ```lua local write_err = res:write("Hello, World!") if write_err then return nil, write_err end -- Build response incrementally for _, fragment in ipairs({ "", "

Title

", "

Content

", "" }) do local fragment_err = res:write(fragment) if fragment_err then return nil, fragment_err end end ``` #### `write_json` Encode a value as JSON and write it to the response: ```lua -- Success response local write_err = res:write_json({ data = users, total = count, page = page }) if write_err then return nil, write_err end -- Error response local type_err = res:set_content_type(http.CONTENT.JSON) if type_err then return nil, type_err end local status_err = res:set_status(http.STATUS.BAD_REQUEST) if status_err then return nil, status_err end local error_write_err = res:write_json({ error = "Validation failed", details = { {field = "email", message = "Invalid format"}, {field = "age", message = "Must be positive"} } }) if error_write_err then return nil, error_write_err end ``` `write()`, `write_json()`, `flush()`, and `write_event()` also commit headers. `write_json()` sets `Content-Type: application/json` only when headers have not already been committed. #### `flush` Flush buffered response data to the client: -- Stream progress updates for i = 1, 100 do local write_err = res:write(string.format("Progress: %d%%\n", i)) if write_err then return nil, write_err end local flush_err = res:flush() if flush_err then return nil, flush_err end local _, sleep_err = time.sleep("100ms") if sleep_err then return nil, sleep_err end end #### `set_transfer` Set the transfer mode for a streaming response: ```lua -- Chunked transfer local transfer_err = res:set_transfer(http.TRANSFER.CHUNKED) if transfer_err then return nil, transfer_err end for chunk in get_chunks() do local write_err = res:write(chunk) if write_err then return nil, write_err end local flush_err = res:flush() if flush_err then return nil, flush_err end end -- Server-Sent Events local sse_err = res:set_transfer(http.TRANSFER.SSE) if sse_err then return nil, sse_err end ``` #### `write_event` Write a Server-Sent Event: ```lua -- Real-time updates local transfer_err = res:set_transfer(http.TRANSFER.SSE) if transfer_err then return nil, transfer_err end local connected_err = res:write_event({name = "connected", data = {client_id = client_id}}) if connected_err then return nil, connected_err end for progress in task:progress() do local event_err = res:write_event({name = "progress", data = {percent = progress}}) if event_err then return nil, event_err end end local complete_err = res:write_event({name = "complete", data = {result = result}}) if complete_err then return nil, complete_err end -- Chat messages local message_err = res:write_event({name = "message", data = { from = "alice", text = "Hello!", timestamp = time.now():unix() }}) if message_err then return nil, message_err end ``` #### HTTP Methods ```lua http.METHOD.GET http.METHOD.POST http.METHOD.PUT http.METHOD.DELETE http.METHOD.PATCH http.METHOD.HEAD http.METHOD.OPTIONS ``` #### Status Codes ```lua -- Success (2xx) http.STATUS.OK -- 200 http.STATUS.CREATED -- 201 http.STATUS.ACCEPTED -- 202 http.STATUS.NO_CONTENT -- 204 http.STATUS.PARTIAL_CONTENT -- 206 -- Redirect (3xx) http.STATUS.MOVED_PERMANENTLY -- 301 http.STATUS.FOUND -- 302 http.STATUS.SEE_OTHER -- 303 http.STATUS.NOT_MODIFIED -- 304 http.STATUS.TEMPORARY_REDIRECT -- 307 http.STATUS.PERMANENT_REDIRECT -- 308 -- Client Error (4xx) http.STATUS.BAD_REQUEST -- 400 http.STATUS.UNAUTHORIZED -- 401 http.STATUS.PAYMENT_REQUIRED -- 402 http.STATUS.FORBIDDEN -- 403 http.STATUS.NOT_FOUND -- 404 http.STATUS.METHOD_NOT_ALLOWED -- 405 http.STATUS.NOT_ACCEPTABLE -- 406 http.STATUS.CONFLICT -- 409 http.STATUS.GONE -- 410 http.STATUS.UNPROCESSABLE -- 422 http.STATUS.TOO_MANY_REQUESTS -- 429 -- Server Error (5xx) http.STATUS.INTERNAL_ERROR -- 500 (alias: INTERNAL_SERVER_ERROR) http.STATUS.NOT_IMPLEMENTED -- 501 http.STATUS.BAD_GATEWAY -- 502 http.STATUS.SERVICE_UNAVAILABLE -- 503 http.STATUS.GATEWAY_TIMEOUT -- 504 http.STATUS.VERSION_NOT_SUPPORTED -- 505 ``` #### Content Types ```lua http.CONTENT.JSON -- "application/json" http.CONTENT.FORM -- "application/x-www-form-urlencoded" http.CONTENT.MULTIPART -- "multipart/form-data" http.CONTENT.TEXT -- "text/plain" http.CONTENT.STREAM -- "application/octet-stream" ``` #### Transfer Modes ```lua http.TRANSFER.CHUNKED -- "chunked" http.TRANSFER.SSE -- "sse" ``` #### Legacy Error-Type Constants The module exports these compatibility strings, but current request and response methods do not return them. Runtime failures use the structured `errors.*` kinds described below. ```lua http.ERROR.PARSE_FAILED -- Form/multipart parse error http.ERROR.INVALID_STATE -- Invalid response state http.ERROR.WRITE_FAILED -- Response write error http.ERROR.STREAM_ERROR -- Body stream error ``` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | No HTTP context | `errors.INTERNAL` | no | | Body too large | `errors.INVALID` | no | | Read timeout | `errors.INTERNAL` | no | | Invalid JSON | `errors.INVALID` | no | | Not multipart | `errors.INVALID` | no | | Headers already sent | `errors.INVALID` | no | | Write failed | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "HTTP Client" ## HTTP Client The `http_client` module sends HTTP requests with headers, query parameters, forms, file uploads, authentication, TLS options, streaming responses, and concurrent batches. This is an API reference with partial request recipes. URLs, tokens, credentials, request data, and certificate material come from the surrounding application. Examples check `Response, error` before consuming a response and close streamed bodies explicitly. ### Loading ```lua local http_client = require("http_client") ``` Add `http_client` to the executable entry's `modules:` list before requiring it. JSON and filesystem recipes also require `json` and `fs`. ### HTTP Methods Convenience methods use the `method(url, options?)` signature and return `Response, error`. #### GET Send a `GET` request. ```lua local resp, err = http_client.get("https://api.example.com/users") if err then return nil, err end print(resp.status_code) -- 200 print(resp.body) -- response body ``` #### POST Send a `POST` request. ```lua local json = require("json") local body, body_err = json.encode({name = "Alice", email = "alice@example.com"}) if body_err then return nil, body_err end local resp, err = http_client.post("https://api.example.com/users", { headers = {["Content-Type"] = "application/json"}, body = body }) if err then return nil, err end ``` #### PUT Send a `PUT` request. ```lua local body, body_err = json.encode({name = "Alice Smith"}) if body_err then return nil, body_err end local resp, err = http_client.put("https://api.example.com/users/123", { headers = {["Content-Type"] = "application/json"}, body = body }) if err then return nil, err end ``` #### PATCH Send a `PATCH` request. ```lua local body, body_err = json.encode({status = "active"}) if body_err then return nil, body_err end local resp, err = http_client.patch("https://api.example.com/users/123", { headers = {["Content-Type"] = "application/json"}, body = body }) if err then return nil, err end ``` #### DELETE Send a `DELETE` request. ```lua local resp, err = http_client.delete("https://api.example.com/users/123", { headers = {["Authorization"] = "Bearer " .. token} }) if err then return nil, err end ``` #### HEAD A `HEAD` request returns headers without a response body. ```lua local resp, err = http_client.head("https://cdn.example.com/file.zip") if err then return nil, err end local size = resp.headers["Content-Length"] ``` #### Custom Methods Send a request using an explicit HTTP method string. ```lua local resp, err = http_client.request("PROPFIND", "https://dav.example.com/folder", { headers = {["Depth"] = "1"} }) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `method` | string | HTTP method | | `url` | string | Request URL | | `options` | table | Request options (optional) | ### Request Options | Field | Type | Description | |-------|------|-------------| | `headers` | table | Request headers `{["Name"] = "value"}` | | `body` | string | Request body | | `query` | table | Query parameters `{key = "value"}` | | `form` | table | Form data (sets Content-Type automatically) | | `files` | table | File uploads (array of file definitions) | | `cookies` | table | Request cookies `{name = "value"}` | | `auth` | table | Basic auth `{user = "name", pass = "secret"}` | | `timeout` | number/string | Timeout: number in seconds, or string like `"30s"`, `"1m"` | | `stream` | boolean | Stream response body instead of buffering | | `max_response_body` | number | Max response size in bytes (0 = default) | | `unix_socket` | string | Connect via Unix socket path | | `tls` | table | Per-request TLS configuration (see [TLS Options](#tls-options)) | | `overlay_network` | string | Route through a [network overlay](system/network.md) — registry ID of a `network.socks5` / `network.tailscale` / `network.i2p` entry | Selecting `overlay_network` requires `network.select` permission on that network ID. #### Query Parameters ```lua local resp, err = http_client.get("https://api.example.com/search", { query = { q = "lua programming", page = "1", limit = "20" } }) if err then return nil, err end ``` #### Headers and Authentication ```lua local resp, err = http_client.get("https://api.example.com/data", { headers = { ["Authorization"] = "Bearer " .. token, ["Accept"] = "application/json" } }) if err then return nil, err end -- Or use basic auth local resp, err = http_client.get("https://api.example.com/data", { auth = {user = service_user, pass = service_password} }) if err then return nil, err end ``` Load authentication values from application-owned secret storage and send them only over TLS. #### Form Data ```lua local resp, err = http_client.post("https://api.example.com/login", { form = { username = username, password = password } }) if err then return nil, err end ``` #### File Upload ```lua local resp, err = http_client.post("https://api.example.com/upload", { form = {title = "My Document"}, files = { { name = "attachment", -- form field name filename = "report.pdf", -- original filename content = pdf_data, -- file content content_type = "application/pdf" } } }) if err then return nil, err end ``` | File Field | Type | Required | Description | |------------|------|----------|-------------| | `name` | string | yes | Form field name | | `filename` | string | no | Original filename | | `content` | string | yes* | File content | | `reader` | userdata | yes* | Alternative: io.Reader for content | | `content_type` | string | no | Currently ignored: each uploaded part is always sent with `Content-Type: application/octet-stream` regardless of this field | \* Either `content` or `reader` is required. The pinned runtime fully reads a `reader` into memory before dispatch, does not close it, and does not surface a non-EOF read failure separately; it can send the bytes accumulated before that failure. Prefer `content` for already-bounded data, and close caller-owned readers after the request. The `content_type` field is parsed but not forwarded by runtime `v0.3.32a`, so uploaded parts use the transport default. Reader-backed files are supported only by single-request calls in this release. `request_batch` forwards the `content` field but drops a parsed `reader`, so batch file uploads must provide `content`. #### Timeout ```lua -- Number: seconds local resp, err = http_client.get(url, {timeout = 30}) if err then return nil, err end -- String alternatives use Go duration format: "30s", "1m30s", or "1h". ``` #### TLS Options Configure mutual TLS and custom CA certificates for one request. | Field | Type | Description | |-------|------|-------------| | `cert` | string | Client certificate in PEM format | | `key` | string | Client private key in PEM format | | `ca` | string | Custom CA certificate in PEM format | | `server_name` | string | Server name for SNI verification | | `insecure_skip_verify` | boolean | Skip TLS certificate verification | For mutual TLS, provide `cert` and `key` together. The `ca` field replaces the system certificate pool with a custom CA. ##### mTLS Authentication ```lua local fs = require("fs") local certs, volume_err = fs.get("app:certs") if volume_err then return nil, volume_err end local cert_pem, cert_err = certs:readfile("client.crt") if cert_err then return nil, cert_err end local key_pem, key_err = certs:readfile("client.key") if key_err then return nil, key_err end local resp, err = http_client.get("https://secure.example.com/api", { tls = { cert = cert_pem, key = key_pem, } }) if err then return nil, err end ``` ##### Custom CA ```lua local fs = require("fs") local certs, volume_err = fs.get("app:certs") if volume_err then return nil, volume_err end local ca_pem, ca_err = certs:readfile("internal-ca.crt") if ca_err then return nil, ca_err end local resp, err = http_client.get("https://internal.example.com/api", { tls = { ca = ca_pem, server_name = "internal.example.com", } }) if err then return nil, err end ``` ##### Insecure Skip Verify `insecure_skip_verify` disables TLS verification and requires the `http_client.insecure_tls` security permission. ```lua local resp, err = http_client.get("https://localhost:8443/api", { tls = { insecure_skip_verify = true, } }) if err then return nil, err end ``` Use `insecure_skip_verify` only for a controlled diagnostic endpoint. It disables both certificate-chain and hostname verification. ### Response Object | Field | Type | Description | |-------|------|-------------| | `status_code` | number | HTTP status code | | `body` | string | Response body (if not streaming) | | `body_size` | number | Body size in bytes (-1 if streaming) | | `headers` | table | Response headers | | `cookies` | table | Response cookies | | `url` | string | Final URL (after redirects) | | `stream` | Stream | Stream object (if `stream = true`) | ```lua local resp, err = http_client.get("https://api.example.com/data") if err then return nil, err end if resp.status_code == 200 then local data, decode_err = json.decode(resp.body) if decode_err then return nil, decode_err end print("Content-Type:", resp.headers["Content-Type"]) end ``` ### Streaming Responses Set `stream = true` to process a response incrementally rather than buffering its full body. ```lua local resp, err = http_client.get("https://cdn.example.com/large-file.zip", { stream = true }) if err then return nil, err end -- Process in chunks local read_err while true do local chunk chunk, read_err = resp.stream:read(65536) if read_err or not chunk then break end -- process chunk end local _, close_err = resp.stream:close() if read_err then return nil, read_err end if close_err then return nil, close_err end ``` | Stream Method | Returns | Description | |---------------|---------|-------------| | `read(n?)` | string, error | Read up to `n` bytes (default: implementation buffer) | | `close()` | boolean, error | Close the stream | `resp.stream` is a full [stream](lua/core/stream.md) object — `seek`, `stat`, and `scanner` are also available. The caller owns a streamed response body and should close it on every exit; task cleanup is a fallback, not a substitute for prompt release. ### Batch Requests `request_batch` executes multiple requests concurrently. ```lua local requests = { {"GET", "https://api.example.com/users"}, {"GET", "https://api.example.com/products"}, {"POST", "https://api.example.com/log", {body = "event"}} } local responses, batch_errors = http_client.request_batch(requests) if not responses then return nil, batch_errors -- whole-batch dispatch or validation failure end if batch_errors then for i = 1, #requests do local err = batch_errors[i] if err then print("Request " .. i .. " failed:", err) end end else -- All succeeded for i, resp in ipairs(responses) do print("Response " .. i .. ":", resp.status_code) end end ``` | Parameter | Type | Description | |-----------|------|-------------| | `requests` | table | Array of `{method, url, options?}` | **Returns:** `responses, errors` — arrays indexed by request position **Notes:** - Requests execute concurrently - Streaming (`stream = true`) is not supported in batch - Reader-backed file uploads are not supported in batch; use `files[].content` - Result arrays match request order (1-indexed) #### Encode Encode a string for inclusion in a URL. ```lua local encoded = http_client.encode_uri("hello world") -- "hello+world" local url = "https://api.example.com/search?q=" .. http_client.encode_uri(query) ``` #### Decode Decode a string previously encoded with `http_client.encode_uri`. ```lua local decoded, err = http_client.decode_uri("hello+world") if err then return nil, err end -- "hello world" ``` ### Permissions HTTP requests are evaluated against the active security policy. #### Security Actions | Action | Resource | Description | |--------|----------|-------------| | `http_client.request` | URL | Allow/deny requests to specific URLs | | `http_client.unix_socket` | Socket path | Allow/deny Unix socket connections | | `http_client.private_ip` | IP address | Allow/deny access to private IP ranges | | `http_client.insecure_tls` | URL | Allow/deny insecure TLS (skip verification) | | `network.select` | Network entry ID | Allow/deny routing through the `overlay_network` given in the request | #### Checking Access ```lua local security = require("security") if security.can("http_client.request", "https://api.example.com/users") then local resp, request_err = http_client.get("https://api.example.com/users") if request_err then return nil, request_err end end ``` #### SSRF Protection Non-public IP ranges are blocked by default. Access requires the `http_client.private_ip` permission on the address: - loopback, private (10.x, 172.16-31.x, 192.168.x), link-local unicast and multicast, and the unspecified address - carrier-grade NAT `100.64.0.0/10`, `192.0.0.0/24`, multicast `224.0.0.0/4`, reserved `240.0.0.0/4` - documentation and benchmarking ranges `192.0.2.0/24`, `198.18.0.0/15`, `198.51.100.0/24`, `203.0.113.0/24`, `2001:db8::/32` - IPv6 multicast `ff00::/8` ```lua local resp, err = http_client.get("http://192.168.1.1/admin") -- Error: not allowed: private IP 192.168.1.1 ``` The check runs at dial time, not on the URL string, and it covers every address the host resolves to. A hostname that resolves to several addresses is checked address by address: a denied address is skipped and the next one is tried, and the request fails only when every candidate is denied or unreachable. A public hostname that resolves to a private address is therefore blocked exactly like a private IP literal. #### Redirects Up to nine redirects are followed; the tenth fails with `stopped after 10 redirects`, a count that includes the original request. Every hop is authorized on its own. Before following a redirect the client evaluates `http_client.request` against the target URL and applies the private-IP check to it, so a permitted URL cannot be used to reach a denied one by redirection. A hop that fails either check aborts the request. See [Security Model](system/security.md) for policy configuration. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Security policy denied | `errors.PERMISSION_DENIED` | no | | Private IP blocked | `errors.PERMISSION_DENIED` | no | | Unix socket denied | `errors.PERMISSION_DENIED` | no | | Insecure TLS denied | `errors.PERMISSION_DENIED` | no | | Invalid batch item, batch streaming, or invalid URI escape | `errors.INVALID` | no | | No context | `errors.INTERNAL` | no | | Malformed transport URL or network failure | `errors.INTERNAL` | yes | | Timeout | `errors.INTERNAL` | yes | Many unsupported option values are ignored rather than returned as structured errors. Invalid Lua argument types and an empty batch raise Lua argument errors. Validate application-supplied option tables before calling the client. ```lua local resp, err = http_client.get(url) if err then if errors.is(err, errors.PERMISSION_DENIED) then print("Access denied:", err:message()) elseif err:retryable() then print("Temporary error:", err:message()) end return nil, err end ``` See [Error Handling](lua/core/errors.md) for working with errors. --- # "WebSocket Client" ## WebSocket Client The `websocket` module creates bidirectional client connections to WebSocket servers. This is an API reference with partial connection and subscription recipes. Endpoint URLs, tokens, message handlers, and application data come from the surrounding application. The lifecycle examples close the client on every terminal or checked error path; smaller method snippets assume an enclosing owner performs that cleanup. ### Loading ```lua local websocket = require("websocket") ``` Add `websocket` to the executable entry's `modules:` list before requiring it. The `channel` global is always available; JSON and timeout recipes also require `json` and `time`. #### `connect` Open a WebSocket connection with the default options: ```lua local client, err = websocket.connect("wss://api.example.com/ws") if err then return nil, err end ``` Pass an options table to configure the connection: ```lua local client, err = websocket.connect("wss://api.example.com/ws", { headers = { ["Authorization"] = "Bearer " .. token }, protocols = {"graphql-ws"}, dial_timeout = "10s", read_timeout = "30s", compression = websocket.COMPRESSION.CONTEXT_TAKEOVER }) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `url` | string | WebSocket URL (ws:// or wss://) | | `options` | table | Connection options (optional) | **Returns:** `Client, error` ##### Connection Options | Option | Type | Description | |--------|------|-------------| | `headers` | table | HTTP headers for handshake | | `protocols` | table | WebSocket subprotocols | | `dial_timeout` | number/string | Connection timeout (ms or "5s") | | `read_timeout` | number/string | Read timeout | | `write_timeout` | number/string | Write timeout | | `compression` | number/string | Compression mode (see Constants), or `"disabled"`, `"context_takeover"`, `"no_context_takeover"` | | `compression_threshold` | number | Min size to compress (0-100MB) | | `read_limit` | number | Max message size (0-128MB) | | `channel_capacity` | number | Receive channel buffer (1-10000) | **Timeout format:** Numbers are milliseconds. Strings use Go duration syntax such as `"5s"` or `"1m"`. Invalid timeout strings and out-of-range or unsupported option values are ignored, leaving the corresponding default in effect. #### Text Messages Send a text message. ```lua local json = require("json") client:send("Hello, Server!") -- Send JSON local payload, encode_err = json.encode({ type = "subscribe", channel = "orders" }) if encode_err then return nil, encode_err end client:send(payload) ``` #### Binary Messages Send a binary message by specifying `websocket.BINARY`. ```lua client:send(binary_data, websocket.BINARY) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Message content | | `type` | number | `websocket.TEXT` (1) or `websocket.BINARY` (2) | Yields until the message is sent. Returns no values. #### Ping Send a ping frame. ```lua client:ping() ``` Yields until the ping is sent. Returns no values. ### Receiving Messages `channel()` returns the receive channel, and `receive()` is an alias. The first call yields while the runtime creates the subscription; later calls return the same channel immediately. A subscription failure returns `nil, error`. The channel can be used with `channel.select`. #### Basic Receive ```lua local ch, err = client:channel() if err then client:close() return nil, err end local msg, ok = ch:receive() if ok then print("Type:", msg.type) -- "text" or "binary" print("Data:", msg.data) end local _, close_err = client:close() if close_err then return nil, close_err end ``` #### Message Loop ```lua local json = require("json") local ch, err = client:channel() if err then client:close() return nil, err end while true do local msg, ok = ch:receive() if not ok then break -- Connection closed end if msg.type == "text" then local data, decode_err = json.decode(msg.data) if decode_err then client:close() return nil, decode_err end handle_message(data) end end local _, close_err = client:close() if close_err then return nil, close_err end ``` #### With Select ```lua local json = require("json") local time = require("time") local ch, ch_err = client:channel() if ch_err then client:close() return nil, ch_err end local timeout, timeout_err = time.after("30s") if timeout_err then client:close() return nil, timeout_err end while true do local r = channel.select { ch:case_receive(), timeout:case_receive() } if r.channel == timeout then client:ping() -- Keep-alive timeout, timeout_err = time.after("30s") if timeout_err then client:close() return nil, timeout_err end elseif not r.ok then break else local data, decode_err = json.decode(r.value.data) if decode_err then client:close() return nil, decode_err end process(data) end end local _, close_err = client:close() if close_err then return nil, close_err end ``` #### Message Object | Field | Type | Description | |-------|------|-------------| | `type` | string | `"text"` or `"binary"` | | `data` | string? | Message content (nil for unknown payload types) | ### Closing Connection Close the connection with an optional status code and reason: ```lua local _, close_err = client:close(websocket.CLOSE_CODES.NORMAL, "Session ended") if close_err then return nil, close_err end -- Omitting both arguments also uses normal close code 1000. -- Use INTERNAL_ERROR with an application-owned reason for a failed session. ``` | Parameter | Type | Description | |-----------|------|-------------| | `code` | number | Close code (1000-4999), default 1000 | | `reason` | string | Close reason (optional) | The call yields until the close command completes. Success returns no values; a close failure returns `nil, error`. Capture two results when checking it, because the error is the second result. Values outside the accepted numeric range are ignored and the default code `1000` is used. The receive channel is owned by the client; do not close it directly. A remote terminal event closes the channel. Calling `client:close()` unsubscribes the receive channel and stops the client-side producer, so use it promptly rather than relying on process shutdown cleanup. #### Message Types ```lua -- Numeric (for send) websocket.TEXT -- 1 websocket.BINARY -- 2 -- Compatibility string constants websocket.TYPE_TEXT -- "text" websocket.TYPE_BINARY -- "binary" websocket.TYPE_PING -- "ping" websocket.TYPE_PONG -- "pong" websocket.TYPE_CLOSE -- "close" ``` Receive-channel message objects use only `"text"` and `"binary"`. Ping and pong frames are handled by the transport, and a terminal event closes the channel instead of producing a `"close"` message object. #### Compression Modes ```lua websocket.COMPRESSION.DISABLED -- 0 (no compression) websocket.COMPRESSION.CONTEXT_TAKEOVER -- 1 (sliding window) websocket.COMPRESSION.NO_CONTEXT -- 2 (per-message) ``` #### Close Codes | Constant | Code | Description | |----------|------|-------------| | `NORMAL` | 1000 | Normal closure | | `GOING_AWAY` | 1001 | Server shutting down | | `PROTOCOL_ERROR` | 1002 | Protocol error | | `UNSUPPORTED_DATA` | 1003 | Unsupported data type | | `RESERVED` | 1004 | Reserved | | `NO_STATUS` | 1005 | No status received | | `ABNORMAL_CLOSURE` | 1006 | Connection lost | | `INVALID_PAYLOAD` | 1007 | Invalid frame payload | | `POLICY_VIOLATION` | 1008 | Policy violation | | `MESSAGE_TOO_BIG` | 1009 | Message too large | | `MANDATORY_EXTENSION` | 1010 | Required extension not negotiated | | `INTERNAL_ERROR` | 1011 | Server error | | `SERVICE_RESTART` | 1012 | Server restarting | | `TRY_AGAIN_LATER` | 1013 | Server overloaded | | `BAD_GATEWAY` | 1014 | Gateway error | | `TLS_HANDSHAKE` | 1015 | TLS handshake failure | ```lua local _, close_err = client:close(websocket.CLOSE_CODES.NORMAL, "Done") if close_err then return nil, close_err end ``` #### Real-Time Chat ```lua local json = require("json") local function connect_chat(room_id, token, on_message) local client, err = websocket.connect("wss://chat.example.com/ws", { headers = {["Authorization"] = "Bearer " .. token} }) if err then return nil, err end -- Join room. Runtime v0.3.32a does not expose transport send failures. local join_payload, encode_err = json.encode({ type = "join", room = room_id }) if encode_err then client:close() return nil, encode_err end client:send(join_payload) -- Message loop local ch, channel_err = client:channel() if channel_err then client:close() return nil, channel_err end while true do local msg, ok = ch:receive() if not ok then break end local data, decode_err = json.decode(msg.data) if decode_err then client:close() return nil, decode_err end on_message(data) end local _, close_err = client:close() if close_err then return nil, close_err end return true end ``` #### Price Stream with Keep-Alive ```lua local json = require("json") local time = require("time") local client, err = websocket.connect("wss://stream.example.com/prices") if err then return nil, err end local subscribe_payload, encode_err = json.encode({ action = "subscribe", symbols = {"BTC-USD", "ETH-USD"} }) if encode_err then client:close() return nil, encode_err end client:send(subscribe_payload) local ch, channel_err = client:channel() if channel_err then client:close() return nil, channel_err end local heartbeat, heartbeat_err = time.after("30s") if heartbeat_err then client:close() return nil, heartbeat_err end while true do local r = channel.select { ch:case_receive(), heartbeat:case_receive() } if r.channel == heartbeat then client:ping() heartbeat, heartbeat_err = time.after("30s") if heartbeat_err then client:close() return nil, heartbeat_err end elseif not r.ok then break -- Connection closed else local price, decode_err = json.decode(r.value.data) if decode_err then client:close() return nil, decode_err end update_price(price.symbol, price.value) end end local _, close_err = client:close() if close_err then return nil, close_err end ``` ### Permissions WebSocket connections are evaluated against the active security policy. #### Security Actions | Action | Resource | Description | |--------|----------|-------------| | `websocket.connect` | - | Allow/deny WebSocket connections | | `websocket.connect.url` | URL | Allow/deny connections to specific URLs | See [Security Model](system/security.md) for policy configuration. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Connections disabled | `errors.PERMISSION_DENIED` | no | | URL not allowed | `errors.PERMISSION_DENIED` | no | | No context | `errors.INTERNAL` | no | | Connection failed | `errors.INTERNAL` | yes | | Invalid connection ID returned by the dispatcher | `errors.INTERNAL` | no | | Subscription failed | `errors.INTERNAL` | yes | | Missing process context during subscription | `errors.INTERNAL` | no | | Close failed | `errors.INTERNAL` | no | An empty URL, a non-table options value, invalid argument types, and a missing execution context or process PID when requesting the receive channel raise Lua errors. They are not returned as structured errors. Runtime `v0.3.32a` does not expose send or ping transport failures to Lua callers. ```lua local client, err = websocket.connect(url) if err then if errors.is(err, errors.PERMISSION_DENIED) then print("Access denied:", err:message()) elseif err:retryable() then print("Temporary error:", err:message()) end return nil, err end ``` See [Error Handling](lua/core/errors.md) for working with errors. --- # "HTML Sanitization" ## HTML Sanitization The `html` module sanitizes untrusted HTML with policies based on [bluemonday](https://github.com/microcosm-cc/bluemonday). Sanitization parses an HTML fragment and filters it through an allowlist policy. Elements and attributes that the policy does not allow are removed, and the remaining fragment is normalized during serialization. This is an API reference. Constructor blocks are self-contained policy snippets; later method blocks are partial configuration snippets that assume `policy` is an already-created policy. Sanitized output is suitable only for an HTML element-content context. It is not safe for JavaScript, CSS, URL, or HTML attribute interpolation; use an encoder for the actual output context. ### Loading ```lua local html = require("html") ``` Add `html` to the executable entry's `modules:` list before requiring it. ### Preset Policies The module provides three preset policy constructors: | Policy | Use Case | Allows | |--------|----------|--------| | `new_policy` | Custom sanitization | Nothing (build from scratch) | | `ugc_policy` | User comments, forums | Common formatting (`p`, `b`, `i`, `a`, lists, etc.) | | `strict_policy` | Plain text extraction | Nothing (strips all HTML) | All three constructors return `Policy, nil`; policy construction does not currently fail. #### Empty Policy Create an empty policy, then add the elements and attributes it should allow: ```lua local policy, err = html.sanitize.new_policy() if err then return nil, err end policy:allow_elements("p", "strong", "em") policy:allow_attrs("class"):globally() local clean = policy:sanitize(user_input) ``` **Returns:** `Policy, error` #### User Content Policy Create a policy configured for common user-generated formatting: ```lua local policy, err = html.sanitize.ugc_policy() if err then return nil, err end local safe = policy:sanitize('

Hello world

') -- '

Hello world

' local xss = policy:sanitize('

Hello

') -- '

Hello

' ``` **Returns:** `Policy, error` #### Strict Policy Create a strict policy that removes HTML and returns plain text: ```lua local policy, err = html.sanitize.strict_policy() if err then return nil, err end local text = policy:sanitize('

Hello world!

') -- 'Hello world!' ``` **Returns:** `Policy, error` #### Allow Elements Allow specific HTML elements: ```lua local policy, err = html.sanitize.new_policy() if err then return nil, err end policy:allow_elements("p", "strong", "em", "br") policy:allow_elements("h1", "h2", "h3") policy:allow_elements("a", "img") local result = policy:sanitize('

Hello world

') -- '

Hello world

' ``` | Parameter | Type | Description | |-----------|------|-------------| | `...` | string | Element tag names | **Returns:** `Policy` #### Allow Attributes Start an attribute rule, then apply it with `on_elements()` or `globally()`: ```lua policy:allow_attrs("href"):on_elements("a") policy:allow_attrs("src", "alt"):on_elements("img") policy:allow_attrs("class", "id"):globally() ``` | Parameter | Type | Description | |-----------|------|-------------| | `...` | string | Attribute names | **Returns:** `AttrBuilder` #### On Specific Elements Allow attributes only on specified elements: ```lua policy:allow_elements("a", "img") policy:allow_attrs("href", "target"):on_elements("a") policy:allow_attrs("src", "alt", "width", "height"):on_elements("img") ``` | Parameter | Type | Description | |-----------|------|-------------| | `...` | string | Element tag names | **Returns:** `Policy` #### On All Elements Allow attributes on every permitted element: ```lua policy:allow_attrs("class"):globally() policy:allow_attrs("id"):globally() ``` **Returns:** `Policy` #### With Pattern Matching Require attribute values to match a regular expression: ```lua -- Only allow hex colors in style local builder, err = policy:allow_attrs("style"):matching("^color:#[0-9a-fA-F]{6}$") if err then return nil, err end builder:on_elements("span") policy:sanitize('Red') -- 'Red' policy:sanitize('Bad') -- 'Bad' ``` | Parameter | Type | Description | |-----------|------|-------------| | `pattern` | string | Go RE2-compatible regular expression | **Returns:** `AttrBuilder, error` #### Standard URLs Enable the standard URL-handling policy. It requires parseable URLs, permits relative URLs plus `mailto`, `http`, and `https`, and adds `rel="nofollow"` to allowed linking elements: ```lua policy:allow_elements("a") policy:allow_attrs("href"):on_elements("a") policy:allow_standard_urls() ``` **Returns:** `Policy` #### URL Schemes Allow specific URL schemes: ```lua policy:allow_url_schemes("https", "mailto") policy:sanitize('OK') -- 'OK' policy:sanitize('XSS') -- 'XSS' ``` | Parameter | Type | Description | |-----------|------|-------------| | `...` | string | Schemes to allow | **Returns:** `Policy` #### Relative URLs Configure whether relative URLs are allowed: ```lua policy:allow_relative_urls(true) policy:sanitize('Link') -- 'Link' ``` | Parameter | Type | Description | |-----------|------|-------------| | `allow` | boolean | Allow relative URLs | **Returns:** `Policy` #### Require Parseable URLs Reject URLs that fail to parse cleanly. With `true`, attribute URLs that the HTML sanitizer cannot parse are stripped instead of passed through. ```lua policy:require_parseable_urls(true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `require` | boolean | Require URLs to be parseable | **Returns:** `Policy` #### Nofollow Links Add `rel="nofollow"` to links: ```lua policy:allow_attrs("href", "rel"):on_elements("a") policy:allow_url_schemes("https") policy:require_parseable_urls(true) policy:require_nofollow_on_links(true) policy:sanitize('Link') -- 'Link' ``` | Parameter | Type | Description | |-----------|------|-------------| | `require` | boolean | Add nofollow | **Returns:** `Policy` #### Noreferrer Links Add `rel="noreferrer"` to links: ```lua policy:require_noreferrer_on_links(true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `require` | boolean | Add noreferrer | **Returns:** `Policy` #### External Links in New Tab Add `target="_blank"` to fully qualified URLs: ```lua policy:allow_attrs("href", "target"):on_elements("a") policy:allow_url_schemes("https") policy:require_parseable_urls(true) policy:add_target_blank_to_fully_qualified_links(true) policy:sanitize('Link') -- 'Link' ``` | Parameter | Type | Description | |-----------|------|-------------| | `add` | boolean | Add target blank | **Returns:** `Policy` When opening untrusted links in a new tab, also enable `require_noreferrer_on_links(true)` to suppress referrer leakage and mitigate opener access. #### Allow Images Permit `` with `align`, `alt`, `height`, `width`, and `src`. This helper also enables the standard URL policy but does not allow data URI images. ```lua policy:allow_images() policy:sanitize('Photo') -- 'Photo' ``` **Returns:** `Policy` #### Allow Data URI Images Permit syntactically valid Base64-encoded `gif`, `jpeg`, `png`, `svg+xml`, or `webp` data URI images. The sanitizer validates the media type and Base64 encoding, not the decoded image contents. Data URIs can carry active content, so enable them only for content whose image data you trust: ```lua policy:allow_elements("img") policy:allow_attrs("src"):on_elements("img") policy:allow_data_uri_images() local input = '' policy:sanitize(input) -- The data URI is preserved. ``` **Returns:** `Policy` #### Allow Lists Permit `ul`, `ol`, `li`, `dl`, `dt`, and `dd`. The helper also allows validated `type` attributes on `ul`, `ol`, and `li`, plus an integer `value` attribute on `li`. ```lua policy:allow_lists() policy:sanitize('
  • Item 1
  • Item 2
') -- '
  • Item 1
  • Item 2
' ``` **Returns:** `Policy` #### Allow Tables Permit table elements: `table`, `caption`, `col`, `colgroup`, `thead`, `tbody`, `tfoot`, `tr`, `td`, `th`. ```lua policy:allow_tables() policy:sanitize('
Cell
') -- '
Cell
' ``` **Returns:** `Policy` #### Allow Standard Attributes Permit common attributes: `id`, `title`, `dir`, `lang`. ```lua policy:allow_elements("p") policy:allow_standard_attributes() policy:sanitize('

Hello

') -- '

Hello

' ``` **Returns:** `Policy` ### Sanitize Apply a policy to an HTML string: ```lua local policy, err = html.sanitize.ugc_policy() if err then return nil, err end policy:require_nofollow_on_links(true) local dirty = '

Hello

' local clean = policy:sanitize(dirty) -- '

Hello

' ``` | Parameter | Type | Description | |-----------|------|-------------| | `html` | string | HTML to sanitize | **Returns:** `string` `sanitize` returns only a string. In runtime `v0.3.32a`, the underlying fragment parser can turn malformed input that it cannot parse into an empty string, and the Lua wrapper cannot distinguish that case from valid input whose content the policy removed. Treat sanitization as output filtering, not input validation; validate required content separately when an empty result matters. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Invalid regex pattern | `errors.INVALID` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "SQL Database" ## SQL Database The `sql` module runs queries against configured PostgreSQL, MySQL, and SQLite databases. It supports parameterized queries, transactions, prepared statements, and query builders. This page is an API reference. Its snippets assume a configured database, permission to acquire it, and any tables named by the query. They illustrate individual calls rather than a standalone application. The combined recipe at the end states its additional schema and driver assumptions. For database configuration, see [Database](system/database.md). ### Loading ```lua local sql = require("sql") ``` ### `sql.get` Acquire a database connection from the resource registry: ```lua local db, err = sql.get("app.db:main") if err then return nil, err end local function finish(value, primary_err) local _, release_err = db:release() if primary_err then return nil, primary_err end if release_err then return nil, release_err end return value end local rows, err = db:query("SELECT * FROM users WHERE active = ?", {1}) if err then return finish(nil, err) end return finish(rows) ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Resource ID (e.g., "app.db:main") | **Returns:** `DB, error` Database leases are released during execution-frame cleanup. Call `db:release()` explicitly when database work finishes, especially in long-running operations. Placeholders are passed to the database driver unchanged; the runtime does not rewrite them. SQLite and MySQL use `?`, PostgreSQL uses `$1, $2` — write them in the form your driver expects. The examples below use `?` (SQLite/MySQL). For queries that target more than one engine, build them with the [query builder](#query-builder): `run_with` rewrites placeholders to `$1, $2` when the handle is PostgreSQL, and `to_sql` uses the builder's `placeholder_format`. #### Database Types ```lua sql.type.POSTGRES -- "postgres" sql.type.MYSQL -- "mysql" sql.type.SQLITE -- "sqlite" sql.type.UNKNOWN -- "unknown" ``` #### Isolation Levels ```lua sql.isolation.DEFAULT -- "default" sql.isolation.READ_UNCOMMITTED -- "read_uncommitted" sql.isolation.READ_COMMITTED -- "read_committed" sql.isolation.WRITE_COMMITTED -- "write_committed" sql.isolation.REPEATABLE_READ -- "repeatable_read" sql.isolation.SERIALIZABLE -- "serializable" ``` #### NULL Value ```lua local insert = sql.builder.insert("users") :columns("name", "email") :values("alice", sql.NULL) ``` #### `sql.as.int` Coerce a value to the SQL integer type. ```lua local value = sql.as.int(42) ``` **Returns:** `userdata` #### `sql.as.float` Coerce a value to the SQL float type. ```lua local value = sql.as.float(19.99) ``` **Returns:** `userdata` #### `sql.as.text` Coerce a value to the SQL text type. ```lua local value = sql.as.text("hello") ``` **Returns:** `userdata` #### `sql.as.binary` Coerce a value to the SQL binary type. ```lua local value = sql.as.binary("binary data") ``` **Returns:** `userdata` #### `sql.as.null` Return the SQL `NULL` marker. ```lua local value = sql.as.null() ``` **Returns:** `userdata` #### `sql.builder.select` Create a `SELECT` query builder. ```lua local query = sql.builder.select("id", "name") :from("users") :where({active = 1}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `columns` | ...string | Column names (optional) | **Returns:** `SelectBuilder` #### `sql.builder.insert` Create an `INSERT` query builder. ```lua local query = sql.builder.insert("users") :columns("name", "email") :values("alice", "alice@example.com") ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name (optional) | **Returns:** `InsertBuilder` #### `sql.builder.update` Create an `UPDATE` query builder. ```lua local query = sql.builder.update("users") :set("status", "active") :where({id = 123}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name (optional) | **Returns:** `UpdateBuilder` #### `sql.builder.delete` Create a `DELETE` query builder. ```lua local query = sql.builder.delete("users") :where({active = 0}) :limit(100) ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name (optional) | **Returns:** `DeleteBuilder` #### `sql.builder.expr` Create a raw SQL expression for use in `WHERE` or `HAVING` clauses. ```lua local expr = sql.builder.expr("score BETWEEN ? AND ?", 80, 90) ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL expression with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `Sqlizer` #### `sql.builder.eq` Create equality conditions from a table. ```lua local cond = sql.builder.eq({active = 1, status = "open"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.not_eq` Create inequality conditions from a table. ```lua local cond = sql.builder.not_eq({status = "closed"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.lt` Create less-than conditions from a table. ```lua local cond = sql.builder.lt({age = 18}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.lte` Create less-than-or-equal conditions from a table. ```lua local cond = sql.builder.lte({price = 100}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.gt` Create greater-than conditions from a table. ```lua local cond = sql.builder.gt({score = 80}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.gte` Create greater-than-or-equal conditions from a table. ```lua local cond = sql.builder.gte({age = 21}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.like` Create `LIKE` conditions from a table. ```lua local cond = sql.builder.like({name = "john%"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.not_like` Create `NOT LIKE` conditions from a table. ```lua local cond = sql.builder.not_like({email = "%@spam.com"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `Sqlizer` #### `sql.builder.and_` Combine multiple conditions with `AND`. ```lua local cond = sql.builder.and_({ sql.builder.eq({active = 1}), sql.builder.gt({score = 80}) }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `conditions` | table | Array of Sqlizer or table conditions | **Returns:** `Sqlizer` #### `sql.builder.or_` Combine multiple conditions with `OR`. ```lua local cond = sql.builder.or_({ sql.builder.eq({status = "pending"}), sql.builder.eq({status = "active"}) }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `conditions` | table | Array of Sqlizer or table conditions | **Returns:** `Sqlizer` ### sqlizer:to_sql Generates the SQL fragment and bind arguments of a condition. ```lua local frag, args = sql.builder.eq({active = 1}):to_sql() ``` **Returns:** `string, table` ### builder.question Use `?` placeholders (default). This format is also available as `sql.builder.default_placeholder`. ```lua local query = sql.builder.select("*") :from("users") :placeholder_format(sql.builder.question) ``` #### `sql.builder.dollar` Use `$1, $2, ...` placeholders. ```lua local query = sql.builder.select("*") :from("users") :placeholder_format(sql.builder.dollar) ``` #### `sql.builder.at` Use `@p1, @p2, ...` placeholders (SQL Server style). Pass this format to `placeholder_format` like the formats above. #### `sql.builder.colon` Use `:1, :2, ...` placeholders. Pass this format to `placeholder_format` like the formats above. ### Connection Methods A connection handle returned by `sql.get()` provides query, transaction, statement, and pool operations. #### `db:type` Return the database type constant. ```lua local dbtype, err = db:type() ``` **Returns:** `string, error` #### `db:query` Run a `SELECT` query and return its rows. ```lua local rows, err = db:query("SELECT id, name FROM users WHERE active = ?", {1}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL query with ? placeholders | | `params` | table | Array of bind parameters (optional) | **Returns:** `table[], error` #### `db:execute` Run an `INSERT`, `UPDATE`, or `DELETE` statement. ```lua local result, err = db:execute("INSERT INTO users (name) VALUES (?)", {"alice"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL statement with ? placeholders | | `params` | table | Array of bind parameters (optional) | **Returns:** `table, error` Returns table with fields: - `last_insert_id` - Last inserted ID - `rows_affected` - Number of rows affected #### `db:prepare` Create a prepared statement for repeated execution. ```lua local stmt, err = db:prepare("SELECT * FROM users WHERE id = ?") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL with ? placeholders | **Returns:** `Statement, error` #### `db:begin` Begin a database transaction. ```lua local tx, err = db:begin({ isolation = sql.isolation.SERIALIZABLE, read_only = false }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `options` | table | Transaction options (optional) | Options table fields: - `isolation` - Isolation level from sql.isolation.* (default: DEFAULT) - `read_only` - Read-only transaction flag (default: false) **Returns:** `Transaction, error` #### `db:release` Release the database resource back to the pool. ```lua local ok, err = db:release() ``` **Returns:** `boolean, error` The operation is idempotent. #### `db:stats` Return connection-pool statistics. ```lua local stats, err = db:stats() ``` **Returns:** `table, error` Returns table with fields: - `max_open_connections` - Max allowed open connections - `open_connections` - Current open connections - `in_use` - Connections currently in use - `idle` - Idle connections in pool - `wait_count` - Total connection wait count - `wait_duration` - Total wait duration - `max_idle_closed` - Connections closed due to max idle - `max_idle_time_closed` - Connections closed due to idle timeout - `max_lifetime_closed` - Connections closed due to max lifetime ### Prepared Statements A prepared statement returned by `db:prepare()` can be queried or executed repeatedly. #### `stmt:query` Run the prepared statement as a `SELECT` query. ```lua local rows, err = stmt:query({123}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `params` | table | Array of bind parameters (optional) | **Returns:** `table[], error` #### `stmt:execute` Run the prepared statement as an `INSERT`, `UPDATE`, or `DELETE` statement. ```lua local result, err = stmt:execute({"alice"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `params` | table | Array of bind parameters (optional) | **Returns:** `table, error` Returns table with fields: - `last_insert_id` - Last inserted ID - `rows_affected` - Number of rows affected #### `stmt:close` Close the prepared statement. ```lua local ok, err = stmt:close() ``` **Returns:** `boolean, error` ### Transactions A transaction returned by `db:begin()` provides query, statement, savepoint, commit, and rollback operations. An active transaction is rolled back automatically during execution-frame cleanup. Commit or roll it back explicitly as soon as its work is complete. #### `tx:db_type` Return the database type constant. ```lua local dbtype, err = tx:db_type() ``` **Returns:** `string, error` #### `tx:query` Run a `SELECT` query within the transaction. ```lua local rows, err = tx:query("SELECT id, name FROM users WHERE active = ?", {1}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL query with ? placeholders | | `params` | table | Array of bind parameters (optional) | **Returns:** `table[], error` #### `tx:execute` Run an `INSERT`, `UPDATE`, or `DELETE` statement within the transaction. ```lua local result, err = tx:execute("INSERT INTO users (name) VALUES (?)", {"alice"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL statement with ? placeholders | | `params` | table | Array of bind parameters (optional) | **Returns:** `table, error` Returns table with fields: - `last_insert_id` - Last inserted ID - `rows_affected` - Number of rows affected #### `tx:prepare` Create a prepared statement within the transaction. ```lua local stmt, err = tx:prepare("SELECT * FROM users WHERE id = ?") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL with ? placeholders | **Returns:** `Statement, error` #### `tx:commit` Commit the transaction. ```lua local ok, err = tx:commit() ``` **Returns:** `boolean, error` #### `tx:rollback` Roll back the transaction. ```lua local ok, err = tx:rollback() ``` **Returns:** `boolean, error` #### `tx:savepoint` Create a named savepoint within the transaction. ```lua local ok, err = tx:savepoint("sp1") ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Savepoint name (alphanumeric and underscore only) | **Returns:** `boolean, error` #### `tx:rollback_to` Roll back to a named savepoint. ```lua local ok, err = tx:rollback_to("sp1") ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Savepoint name | **Returns:** `boolean, error` #### `tx:release` Release a savepoint. ```lua local ok, err = tx:release("sp1") ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Savepoint name | **Returns:** `boolean, error` ### SELECT Builder Build a `SELECT` query one clause at a time. #### `select:from` Set the `FROM` clause. ```lua local query = sql.builder.select("id", "name"):from("users") ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name | **Returns:** `SelectBuilder` #### `select:join` Add a `JOIN` clause. ```lua local query = sql.builder.select("*") :from("users") :join("orders ON orders.user_id = users.id") ``` | Parameter | Type | Description | |-----------|------|-------------| | `join` | string | JOIN clause with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `SelectBuilder` #### `select:left_join` Add a `LEFT JOIN` clause. ```lua local query = sql.builder.select("*") :from("users") :left_join("orders ON orders.user_id = users.id") ``` | Parameter | Type | Description | |-----------|------|-------------| | `join` | string | JOIN clause with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `SelectBuilder` #### `select:right_join` Add a `RIGHT JOIN` clause. ```lua local query = sql.builder.select("*") :from("users") :right_join("orders ON orders.user_id = users.id") ``` | Parameter | Type | Description | |-----------|------|-------------| | `join` | string | JOIN clause with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `SelectBuilder` #### `select:inner_join` Add an `INNER JOIN` clause. ```lua local query = sql.builder.select("*") :from("users") :inner_join("orders ON orders.user_id = users.id") ``` | Parameter | Type | Description | |-----------|------|-------------| | `join` | string | JOIN clause with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `SelectBuilder` #### `select:where` Add a `WHERE` condition. ```lua local query = sql.builder.select("*") :from("users") :where({active = 1}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `condition` | string\|table\|Sqlizer | WHERE condition | | `args` | ...any | Bind arguments (optional, when using string) | The method accepts three formats: - String: `where("status = ?", "active")` - Table: `where({status = "active"})` - Sqlizer: `where(sql.builder.gt({score = 80}))` **Returns:** `SelectBuilder` #### `select:order_by` Add an `ORDER BY` clause. ```lua local query = sql.builder.select("*") :from("users") :order_by("name ASC", "created_at DESC") ``` | Parameter | Type | Description | |-----------|------|-------------| | `columns` | ...string | Column names with optional ASC/DESC | **Returns:** `SelectBuilder` #### `select:group_by` Add a `GROUP BY` clause. ```lua local query = sql.builder.select("status", "COUNT(*)") :from("users") :group_by("status") ``` | Parameter | Type | Description | |-----------|------|-------------| | `columns` | ...string | Column names | **Returns:** `SelectBuilder` #### `select:having` Add a `HAVING` condition. ```lua local query = sql.builder.select("status", "COUNT(*) as cnt") :from("users") :group_by("status") :having(sql.builder.gt({cnt = 10})) ``` | Parameter | Type | Description | |-----------|------|-------------| | `condition` | string\|table\|Sqlizer | HAVING condition | | `args` | ...any | Bind arguments (optional, when using string) | **Returns:** `SelectBuilder` #### `select:limit` Set the `LIMIT` value. ```lua local query = sql.builder.select("*") :from("users") :limit(10) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | Limit value | **Returns:** `SelectBuilder` #### `select:offset` Set the `OFFSET` value. ```lua local query = sql.builder.select("*") :from("users") :offset(20) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | Offset value | **Returns:** `SelectBuilder` #### `select:columns` Add columns to the `SELECT` list. ```lua local query = sql.builder.select():columns("id", "name", "email") ``` | Parameter | Type | Description | |-----------|------|-------------| | `columns` | ...string | Column names | **Returns:** `SelectBuilder` #### `select:distinct` Add the `DISTINCT` modifier. ```lua local query = sql.builder.select("status") :from("users") :distinct() ``` **Returns:** `SelectBuilder` #### `select:suffix` Add an SQL suffix. ```lua local query = sql.builder.select("*") :from("users") :suffix("FOR UPDATE") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL suffix with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `SelectBuilder` #### `select:placeholder_format` Set the placeholder format. ```lua local query = sql.builder.select("*") :from("users") :placeholder_format(sql.builder.dollar) ``` | Parameter | Type | Description | |-----------|------|-------------| | `format` | userdata | Placeholder format (sql.builder.*) | **Returns:** `SelectBuilder` #### `select:to_sql` Generate the SQL string and bind arguments. ```lua local sql_str, args = query:to_sql() ``` **Returns:** `string, table` on success; `nil, error` for an invalid builder state #### `select:run_with` Create an executor for the query. ```lua local executor, err = query:run_with(db) if err then return nil, err end local rows, err = executor:query() ``` | Parameter | Type | Description | |-----------|------|-------------| | `db` | DB\|Transaction | Database or transaction handle | **Returns:** `QueryExecutor, error` ### INSERT Builder Build an `INSERT` query one clause at a time. #### `insert:into` Set the table name. ```lua local query = sql.builder.insert():into("users") ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name | **Returns:** `InsertBuilder` #### `insert:columns` Set the column names. ```lua local query = sql.builder.insert("users"):columns("name", "email") ``` | Parameter | Type | Description | |-----------|------|-------------| | `columns` | ...string | Column names | **Returns:** `InsertBuilder` #### `insert:values` Add row values. ```lua local query = sql.builder.insert("users") :columns("name", "email") :values("alice", "alice@example.com") ``` | Parameter | Type | Description | |-----------|------|-------------| | `values` | ...any | Row values | **Returns:** `InsertBuilder` #### `insert:set_map` Set columns and values from a table. ```lua local query = sql.builder.insert("users") :set_map({name = "alice", email = "alice@example.com"}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs | **Returns:** `InsertBuilder` #### `insert:select` Insert rows from a `SELECT` query. ```lua local select_query = sql.builder.select("name", "email"):from("temp_users") local query = sql.builder.insert("users") :columns("name", "email") :select(select_query) ``` | Parameter | Type | Description | |-----------|------|-------------| | `query` | SelectBuilder | SELECT query | **Returns:** `InsertBuilder` #### `insert:prefix` Add an SQL prefix. ```lua local query = sql.builder.insert("users") :prefix("/* audit import */") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL prefix with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `InsertBuilder` #### `insert:suffix` Add an SQL suffix. ```lua local query = sql.builder.insert("users") :columns("name") :values("alice") :suffix("RETURNING id") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL suffix with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `InsertBuilder` #### `insert:options` Add `INSERT` options. ```lua local query = sql.builder.insert("users") :options("DELAYED", "IGNORE") ``` | Parameter | Type | Description | |-----------|------|-------------| | `options` | ...string | INSERT options | **Returns:** `InsertBuilder` #### `insert:placeholder_format` Set the placeholder format. ```lua local query = sql.builder.insert("users") :placeholder_format(sql.builder.dollar) ``` | Parameter | Type | Description | |-----------|------|-------------| | `format` | userdata | Placeholder format (sql.builder.*) | **Returns:** `InsertBuilder` #### `insert:to_sql` Generate the SQL string and bind arguments. ```lua local sql_str, args = query:to_sql() ``` **Returns:** `string, table` on success; `nil, error` for an invalid builder state #### `insert:run_with` Create an executor for the query. ```lua local executor, err = query:run_with(db) if err then return nil, err end local result, err = executor:exec() ``` | Parameter | Type | Description | |-----------|------|-------------| | `db` | DB\|Transaction | Database or transaction handle | **Returns:** `QueryExecutor, error` ### UPDATE Builder Build an `UPDATE` query one clause at a time. #### `update:table` Set the table name. ```lua local query = sql.builder.update():table("users") ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name | **Returns:** `UpdateBuilder` #### `update:set` Set a column value. ```lua local query = sql.builder.update("users") :set("status", "active") :set("updated_at", sql.builder.expr("NOW()")) ``` | Parameter | Type | Description | |-----------|------|-------------| | `column` | string | Column name | | `value` | any | Column value | **Returns:** `UpdateBuilder` #### `update:set_map` Set multiple columns from a table. ```lua local query = sql.builder.update("users") :set_map({status = "active", login_count = 0}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `map` | table | {column = value} pairs; values are plain values, `sql.NULL`, or `sql.as.*` (use `set` for expressions) | **Returns:** `UpdateBuilder` #### `update:where` Add a `WHERE` condition. ```lua local query = sql.builder.update("users") :set("status", "active") :where({id = 123}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `condition` | string\|table\|Sqlizer | WHERE condition | | `args` | ...any | Bind arguments (optional, when using string) | **Returns:** `UpdateBuilder` #### `update:order_by` Add an `ORDER BY` clause. ```lua local query = sql.builder.update("users") :set("rank", 1) :order_by("score DESC") ``` | Parameter | Type | Description | |-----------|------|-------------| | `columns` | ...string | Column names with optional ASC/DESC | **Returns:** `UpdateBuilder` #### `update:limit` Set the `LIMIT` value. ```lua local query = sql.builder.update("users") :set("status", "active") :limit(10) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | Limit value | **Returns:** `UpdateBuilder` #### `update:offset` Set the `OFFSET` value. ```lua local query = sql.builder.update("users") :set("status", "active") :offset(5) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | Offset value | **Returns:** `UpdateBuilder` #### `update:suffix` Add an SQL suffix. ```lua local query = sql.builder.update("users") :set("status", "active") :suffix("RETURNING id") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL suffix with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `UpdateBuilder` #### `update:from` Add a `FROM` clause. ```lua local query = sql.builder.update("users") :set("status", "active") :from("other_table") ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name | **Returns:** `UpdateBuilder` #### `update:from_select` Update rows from a `SELECT` query. ```lua local select_query = sql.builder.select("*"):from("temp_users") local query = sql.builder.update("users") :set("status", "active") :from_select(select_query, "t") ``` | Parameter | Type | Description | |-----------|------|-------------| | `query` | SelectBuilder | SELECT query | | `alias` | string | Table alias | **Returns:** `UpdateBuilder` #### `update:placeholder_format` Set the placeholder format. ```lua local query = sql.builder.update("users") :placeholder_format(sql.builder.dollar) ``` | Parameter | Type | Description | |-----------|------|-------------| | `format` | userdata | Placeholder format (sql.builder.*) | **Returns:** `UpdateBuilder` #### `update:to_sql` Generate the SQL string and bind arguments. ```lua local sql_str, args = query:to_sql() ``` **Returns:** `string, table` on success; `nil, error` for an invalid builder state #### `update:run_with` Create an executor for the query. ```lua local executor, err = query:run_with(db) if err then return nil, err end local result, err = executor:exec() ``` | Parameter | Type | Description | |-----------|------|-------------| | `db` | DB\|Transaction | Database or transaction handle | **Returns:** `QueryExecutor, error` ### DELETE Builder Build a `DELETE` query one clause at a time. #### `delete:from` Set the table name. ```lua local query = sql.builder.delete():from("users") ``` | Parameter | Type | Description | |-----------|------|-------------| | `table` | string | Table name | **Returns:** `DeleteBuilder` #### `delete:where` Add a `WHERE` condition. ```lua local query = sql.builder.delete("users") :where({active = 0}) ``` | Parameter | Type | Description | |-----------|------|-------------| | `condition` | string\|table\|Sqlizer | WHERE condition | | `args` | ...any | Bind arguments (optional, when using string) | **Returns:** `DeleteBuilder` #### `delete:order_by` Add an `ORDER BY` clause. ```lua local query = sql.builder.delete("users") :where({active = 0}) :order_by("created_at ASC") ``` | Parameter | Type | Description | |-----------|------|-------------| | `columns` | ...string | Column names with optional ASC/DESC | **Returns:** `DeleteBuilder` #### `delete:limit` Set the `LIMIT` value. ```lua local query = sql.builder.delete("users") :where({active = 0}) :limit(100) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | Limit value | **Returns:** `DeleteBuilder` #### `delete:offset` Set the `OFFSET` value. ```lua local query = sql.builder.delete("users") :where({active = 0}) :offset(10) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | Offset value | **Returns:** `DeleteBuilder` #### `delete:suffix` Add an SQL suffix. ```lua local query = sql.builder.delete("users") :where({active = 0}) :suffix("RETURNING id") ``` | Parameter | Type | Description | |-----------|------|-------------| | `sql` | string | SQL suffix with ? placeholders | | `args` | ...any | Bind arguments (optional) | **Returns:** `DeleteBuilder` #### `delete:placeholder_format` Set the placeholder format. ```lua local query = sql.builder.delete("users") :placeholder_format(sql.builder.dollar) ``` | Parameter | Type | Description | |-----------|------|-------------| | `format` | userdata | Placeholder format (sql.builder.*) | **Returns:** `DeleteBuilder` #### `delete:to_sql` Generate the SQL string and bind arguments. ```lua local sql_str, args = query:to_sql() ``` **Returns:** `string, table` on success; `nil, error` for an invalid builder state #### `delete:run_with` Create an executor for the query. ```lua local executor, err = query:run_with(db) if err then return nil, err end local result, err = executor:exec() ``` | Parameter | Type | Description | |-----------|------|-------------| | `db` | DB\|Transaction | Database or transaction handle | **Returns:** `QueryExecutor, error` ### Executing Queries The query executor runs builder-generated queries. #### `executor:query` Run the query and return rows for a `SELECT` statement. ```lua local rows, err = executor:query() ``` **Returns:** `table[], error` #### `executor:exec` Run the query and return the result of an `INSERT`, `UPDATE`, or `DELETE` statement. ```lua local result, err = executor:exec() ``` **Returns:** `table, error` Returns table with fields: - `last_insert_id` - Last inserted ID - `rows_affected` - Number of rows affected #### `executor:to_sql` Return the generated SQL and arguments without executing the query. ```lua local sql_str, args = executor:to_sql() ``` **Returns:** `string, table` ### Permissions Database access is subject to security policy evaluation. | Action | Resource | Description | |--------|----------|-------------| | `db.get` | Database ID | Acquire database connection | ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty resource ID | `errors.INVALID` | no | | Permission denied | `errors.PERMISSION_DENIED` | no | | Resource not found | `errors.NOT_FOUND` | no | | Resource not database | `errors.INVALID` | no | | Invalid parameters | `errors.INVALID` | no | | SQL syntax error | `errors.UNKNOWN` | nil | | Statement closed | `errors.INVALID` | no | | Transaction not active | `errors.INVALID` | no | | Invalid savepoint name | `errors.INVALID` | no | | Query execution error | `errors.UNKNOWN` | nil | See [Error Handling](lua/core/errors.md) for working with errors. ### Combined Partial Recipe This recipe assumes `app.db:main` is a configured SQLite or MySQL database and already contains `users`, `orders`, and `logs` tables with the referenced columns. It uses `?` placeholders; use `$1`, `$2`, and so on for a PostgreSQL resource. Returned rows depend on the application's data. The surrounding application supplies `report_cleanup_error(err)` so rollback or close failures are observable without replacing the initiating operation error. ```lua local sql = require("sql") local db, err = sql.get("app.db:main") if err then return nil, err end local function finish(value, primary_err) local _, release_err = db:release() if primary_err then return nil, primary_err end if release_err then return nil, release_err end return value end -- Direct query local users, err = db:query("SELECT id, name FROM users WHERE active = ?", {1}) if err then return finish(nil, err) end for _, user in ipairs(users) do print(user.id, user.name) end -- Builder pattern local query = sql.builder.select("u.id", "u.name", "COUNT(o.id) as order_count") :from("users u") :left_join("orders o ON o.user_id = u.id") :where(sql.builder.and_({ sql.builder.eq({["u.active"] = 1}), sql.builder.gte({["u.score"] = 80}) })) :group_by("u.id", "u.name") :having(sql.builder.gt({["COUNT(o.id)"] = 0})) :order_by("order_count DESC") :limit(10) local executor, build_err = query:run_with(db) if build_err then return finish(nil, build_err) end local results, err = executor:query() if err then return finish(nil, err) end -- Transaction local tx, err = db:begin({isolation = sql.isolation.SERIALIZABLE}) if err then return finish(nil, err) end local _, err = tx:execute("INSERT INTO users (name) VALUES (?)", {"alice"}) if err then local _, rollback_err = tx:rollback() if rollback_err then report_cleanup_error(rollback_err) end return finish(nil, err) end local _, commit_err = tx:commit() if commit_err then return finish(nil, commit_err) end -- Prepared statements local stmt, err = db:prepare("INSERT INTO logs (message, level) VALUES (?, ?)") if err then return finish(nil, err) end for i = 1, 3 do local _, err = stmt:execute({"log message " .. i, "info"}) if err then local _, close_err = stmt:close() if close_err then report_cleanup_error(close_err) end return finish(nil, err) end end local _, close_err = stmt:close() if close_err then return finish(nil, close_err) end return finish({users = users, ranked_users = results}) ``` --- # "Key-Value Store" ## Key-Value Store The `store` module provides key-value storage with optional TTLs. It can hold cached data, sessions, and other temporary state. This page is an API reference. Its snippets assume a configured store, the permissions listed below, and application-provided values such as `owner` or `new_value`. Snippets after acquisition use an existing live `cache` handle and are not standalone functions. For store configuration, see [Store](system/store.md). ### Loading ```lua local store = require("store") ``` ### Acquiring a Store Acquire a store resource by registry ID: ```lua local cache, err = store.get("app:cache") if err then return nil, err end local _, set_err = cache:set("user:123", {name = "Alice"}, 3600) if set_err then cache:release() return nil, set_err end local user, get_err = cache:get("user:123") cache:release() if get_err then return nil, get_err end return user ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Store resource ID | **Returns:** `Store, error` ### Storing Values Store a value with an optional TTL: ```lua -- Simple set local _, err = cache:set("user:123:name", "Alice") if err then return nil, err end -- Set with TTL (expires in 300 seconds) local ok, ttl_err = cache:set("session:abc", {user_id = 123, role = "admin"}, 300) if ttl_err then return nil, ttl_err end return ok ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Key | | `value` | any | Value (tables, strings, numbers, booleans) | | `ttl` | number | TTL in seconds (optional, 0 = no expiry) | **Returns:** `boolean, error` ### Retrieving Values Retrieve a value by key: ```lua local errors = require("errors") local user, err = cache:get("user:123") if err then if err:kind() == errors.NOT_FOUND then return nil -- key missing or expired end return nil, err end return user ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Key to retrieve | **Returns:** `any, error` The method returns `nil` and an `errors.NOT_FOUND` error when the key does not exist or has expired. ### Checking Existence Check whether a key exists without retrieving its value: ```lua if cache:has("lock:" .. resource_id) then return nil, errors.new({ kind = errors.CONFLICT, message = "Resource is locked" }) end ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Key to check | **Returns:** `boolean, error` ### Deleting Keys Remove a key from the store: ```lua local deleted, err = cache:delete("session:" .. session_id) if err then return nil, err end return deleted ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Key to delete | **Returns:** `boolean, error` The method returns `true` when it deletes the key and `false` when the key does not exist. ### Reading Entry Metadata `entry` returns the value together with its `version` — an opaque string used for optimistic concurrency: ```lua local e, err = cache:entry("user:123") if err then return nil, err end if e then print(e.key, e.value, e.version) end ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Key to read | **Returns:** `Entry, error` — `{key: string, value: any, version: string}` ### Listing Keys List entries in deterministic key order with pagination: ```lua local page, err = cache:list({ prefix = "session:", limit = 100 }) if err then return nil, err end for _, e in ipairs(page.items) do print(e.key, e.value) end -- next page if page.has_more then local next_page, next_err = cache:list({ prefix = "session:", after = page.cursor }) if next_err then return nil, next_err end page = next_page end ``` | Option | Type | Description | |--------|------|-------------| | `prefix` | string | Only keys with this prefix | | `after` | string | Continue after this cursor (from a previous page) | | `limit` | integer | Max items per page | **Returns:** `Page, error` — `{items: Entry[], cursor: string, has_more: boolean}` ### Conditional Writes `put` writes a value and returns its new `Entry`. Options enable optimistic concurrency: ```lua local errors = require("errors") -- create only if the key does not exist local e, err = cache:put("lock:job-1", owner, { only_if_absent = true }) if err and err:kind() == errors.ALREADY_EXISTS then -- someone else holds it elseif err then return nil, err end -- compare-and-set: write only if the version still matches local cur, read_err = cache:entry("config") if read_err then return nil, read_err end local e2, err2 = cache:put("config", new_value, { if_version = cur.version }) if err2 and err2:kind() == errors.CONFLICT then -- a concurrent writer changed it; re-read and retry elseif err2 then return nil, err2 end ``` | Option | Type | Description | |--------|------|-------------| | `ttl` | number | TTL in seconds | | `only_if_absent` | boolean | Write only if the key does not exist | | `if_version` | string | Write only if the current version matches | `only_if_absent` and `if_version` are mutually exclusive. **Returns:** `Entry, error` Conditional writes require a store whose info().conditional_put is true (the memory and store.kv.raft stores). On store.kv.crdt and store.sql they return an errors.INVALID error — use store.kv.raft when you need conditional writes. ### Store Capabilities `info` reports the backend and what it supports, so code can adapt to whichever store is bound: ```lua local info, err = cache:info() if err then return nil, err end -- info.backend -> one of store.backend.* (e.g. "kv.raft") -- info.consistency -> one of store.consistency.* (e.g. "linearizable") -- info.durable / info.list / info.versioned / info.conditional_put / info.ttl (booleans) ``` **Returns:** `Info, error` — `{id, backend, consistency, durable, list, versioned, conditional_put, ttl}` #### Constants | Constant | Values | |----------|--------| | `store.backend` | `MEMORY`, `SQL`, `KV_RAFT`, `KV_CRDT`, `UNKNOWN` | | `store.consistency` | `LINEARIZABLE`, `EVENTUAL`, `LOCAL`, `UNKNOWN` | ```lua local info, err = cache:info() if err then return nil, err end if info.consistency == store.consistency.LINEARIZABLE then -- safe to use compare-and-set end ``` ### Store Methods | Method | Returns | Description | |--------|---------|-------------| | `get(key)` | `any, error` | Retrieve value by key | | `entry(key)` | `Entry, error` | Retrieve value with version metadata | | `set(key, value, ttl?)` | `boolean, error` | Store value with optional TTL | | `put(key, value, opts?)` | `Entry, error` | Conditional/versioned write, returns the new entry | | `list(opts?)` | `Page, error` | Paged listing in key order | | `has(key)` | `boolean, error` | Check if key exists | | `delete(key)` | `boolean, error` | Remove key | | `info()` | `Info, error` | Backend, consistency, and capability flags | | `release()` | `boolean` | Release store back to pool | ### Permissions Security policy evaluation applies to store operations. | Action | Resource | Attributes | Description | |--------|----------|------------|-------------| | `store.get` | Store ID | - | Acquire a store resource | | `store.info` | Store ID | - | Inspect store capabilities | | `store.key.get` | Store ID | `key` | Read a key value (also `entry`) | | `store.key.set` | Store ID | `key` | Write a key value (also `put`) | | `store.key.delete` | Store ID | `key` | Delete a key | | `store.key.has` | Store ID | `key` | Check key existence | | `store.key.list` | Store ID | `prefix` | List entries | Permission denials from `store.get`, `get`, `set`, `delete`, and `has` raise a Lua error. The `info`, `entry`, `list`, and `put` methods instead return an `errors.PERMISSION_DENIED` error. Grant the required actions before calling code that cannot tolerate a raised denial. ### Errors `store.get()` and all methods on the store handle (`get`, `entry`, `set`, `put`, `list`, `has`, `delete`, `info`) return structured errors (use `err:kind()`), except that a permission denial in `store.get`, `get`, `set`, `has` and `delete` raises a Lua error instead. | Condition | Kind | Retryable | |-----------|------|-----------| | Empty resource ID | `errors.INVALID` | no | | Resource not found | `errors.INTERNAL` | no | | Store released | `errors.INVALID` | no | | Permission denied (`entry`, `put`, `list`, `info`) | `errors.PERMISSION_DENIED` | no | | `only_if_absent` and key exists | `errors.ALREADY_EXISTS` | no | | `if_version` mismatch | `errors.CONFLICT` | yes | | Conditional write on a store without support | `errors.INVALID` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Filesystem" ## Filesystem The `fs` module reads, writes, and manages files within configured filesystem volumes. This page is an API reference. Its snippets assume a configured volume and permission to acquire it. Each block is an isolated operation or partial recipe; application values and callbacks such as `config`, `message`, `process`, and `report_cleanup_error` must already exist. `report_cleanup_error(err)` records a close failure without replacing an operation error that already occurred. For filesystem configuration, see [Filesystem](system/filesystem.md). ### Loading ```lua local fs = require("fs") ``` ### Acquiring a Volume Acquire a filesystem volume by registry ID: ```lua local vol, err = fs.get("app:storage") if err then return nil, err end local content, read_err = vol:readfile("/config.json") if read_err then return nil, read_err end return content ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Volume registry ID | **Returns:** `FS, error` Volumes do not require explicit release. The system manages them, and a volume becomes unavailable when its filesystem is detached from the registry. ### Reading Files Read an entire file: ```lua local json = require("json") local vol, get_err = fs.get("app:config") if get_err then return nil, get_err end local data, err = vol:readfile("/settings.json") if err then return nil, err end local config, decode_err = json.decode(data) if decode_err then return nil, decode_err end return config ``` Use `open()` to stream a large file: ```lua local errors = require("errors") local file, err = vol:open("/data/large.csv", "r") if err then return nil, err end while true do local chunk, err = file:read(65536) if err then if err:kind() == errors.NOT_FOUND then break -- EOF end local _, close_err = file:close() if close_err then report_cleanup_error(close_err) end return nil, err end process(chunk) end local _, close_err = file:close() if close_err then return nil, close_err end ``` ### Writing Files Write a string or reader-backed stream to a file: ```lua local json = require("json") local vol, get_err = fs.get("app:data") if get_err then return nil, get_err end -- Overwrite (default) local encoded, encode_err = json.encode(config) if encode_err then return nil, encode_err end local _, write_err = vol:writefile("/config.json", encoded) if write_err then return nil, write_err end -- Append local _, append_err = vol:writefile("/logs/app.log", message .. "\n", "a") if append_err then return nil, append_err end -- Exclusive write (fails if exists) local ok, err = vol:writefile("/lock.pid", tostring(pid), "wx") if err then return nil, err end -- Copy from an open file or another reader-backed value local source, err = vol:open("/incoming/report.csv", "r") if err then return nil, err end local copied, err = vol:writefile("/archive/report.csv", source) local _, close_err = source:close() if err then if close_err then report_cleanup_error(close_err) end return nil, err end if close_err then return nil, close_err end return copied ``` | Mode | Description | |------|-------------| | `"w"` | Overwrite (default) | | `"a"` | Append | | `"wx"` | Exclusive write (fails if file exists) | Use a file handle for streaming writes: ```lua local file, open_err = vol:open("/output/report.txt", "w") if open_err then return nil, open_err end local _, header_err = file:write("Header\n") if header_err then local _, close_err = file:close() if close_err then report_cleanup_error(close_err) end return nil, header_err end local _, data_err = file:write("Data: " .. value .. "\n") if data_err then local _, close_err = file:close() if close_err then report_cleanup_error(close_err) end return nil, data_err end local _, sync_err = file:sync() if sync_err then local _, close_err = file:close() if close_err then report_cleanup_error(close_err) end return nil, sync_err end local _, close_err = file:close() if close_err then return nil, close_err end ``` ### Checking Paths ```lua local vol, get_err = fs.get("app:data") if get_err then return nil, get_err end -- Check existence local exists, exists_err = vol:exists("/cache/results.json") if exists_err then return nil, exists_err end if exists then return vol:readfile("/cache/results.json") end -- Check if directory local is_dir, isdir_err = vol:isdir(path) if isdir_err then return nil, isdir_err end if is_dir then process_directory(path) end -- Get file info local info, stat_err = vol:stat("/documents/report.pdf") if stat_err then return nil, stat_err end print(info.size, info.modified, info.type) ``` **Stat fields:** `name`, `size`, `mode`, `modified`, `is_dir`, `type` ### Directory Operations ```lua local vol, get_err = fs.get("app:data") if get_err then return nil, get_err end -- Create directory local _, mkdir_err = vol:mkdir("/uploads/" .. user_id) if mkdir_err then return nil, mkdir_err end -- List directory contents local iter, state = vol:readdir("/documents") if not iter then return nil, state end for entry in iter, state do print(entry.name, entry.type) end -- Remove file or empty directory local removed, remove_err = vol:remove("/temp/file.txt") if remove_err then return nil, remove_err end return removed ``` Entry fields: `name`, `type` ("file" or "directory") `mkdir` creates one directory and does not create missing parents. `remove` accepts files and empty directories only. ### File Handle Methods When using `vol:open()` for streaming: | Method | Description | |--------|-------------| | `read(size?)` | Read bytes (default: 4096) | | `write(data)` | Write string data | | `seek(whence, offset)` | Set position ("set", "cur", "end") | | `stat()` | Get file info (same fields as `vol:stat`) | | `sync()` | Flush to storage | | `close()` | Release file handle | | `scanner(split?)` | Create line/word scanner | Call `close()` after finishing with a file handle. ### Scanner Use a scanner for line-by-line processing: ```lua local file, err = vol:open("/data/users.csv", "r") if err then return nil, err end local scanner, err = file:scanner("lines") if err then local _, close_err = file:close() if close_err then report_cleanup_error(close_err) end return nil, err end scanner:scan() -- skip header while scanner:scan() do local line = scanner:text() process(line) end local scan_err = scanner:err() if scan_err then local _, close_err = file:close() if close_err then report_cleanup_error(close_err) end return nil, scan_err end local _, close_err = file:close() if close_err then return nil, close_err end ``` Split modes: `"lines"` (default), `"words"`, `"bytes"`, `"runes"` `scanner:scan()` returns only a boolean. When it returns `false`, call `scanner:err()` to distinguish clean EOF from a tokenization or underlying read failure. `scanner:err()` returns a structured `INTERNAL` error or `nil`; unlike a stream scanner, a file scanner has no separate scan-dispatch error return. ### Constants ```lua fs.type.FILE -- "file" fs.type.DIR -- "directory" fs.seek.SET -- from start fs.seek.CUR -- from current fs.seek.END -- from end ``` ### FS Methods | Method | Returns | Description | |--------|---------|-------------| | `readfile(path)` / `read_file(path)` | `string, error` | Read entire file | | `writefile(path, data, mode?)` / `write_file(path, data, mode?)` | `boolean, error` | Write a string or reader-backed value | | `exists(path)` | `boolean, error` | Check if path exists | | `stat(path)` | `table, error` | Get file info | | `isdir(path)` | `boolean, error` | Check if directory | | `mkdir(path)` | `boolean, error` | Create directory | | `remove(path)` | `boolean, error` | Remove file/empty dir | | `readdir(path)` | `iterator, state` | List directory (use in generic `for` loop) | | `open(path, mode)` | `File, error` | Open file handle | | `chdir(path)` | `boolean, error` | Change working dir | | `pwd()` | `string, error` | Get working dir | ### Permissions Security policy evaluation applies when a volume is acquired. | Action | Resource | Description | |--------|----------|-------------| | `fs.get` | Volume ID | Acquire filesystem volume | ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty path | `errors.INVALID` | unspecified | | Path contains a null byte | `errors.INVALID` | no | | Invalid mode | `errors.INVALID` | unspecified | | `scanner()` called on a closed file | `errors.INVALID` | unspecified | | Read, write, seek, stat, or sync called on a closed file | `errors.INTERNAL` | no | | `close()` called on an already closed file | succeeds | not applicable | | File-handle read reached EOF | `errors.NOT_FOUND` | unspecified | | Path not found | `errors.NOT_FOUND` | preserved from the underlying error when available | | Path already exists | `errors.ALREADY_EXISTS` | unspecified | | Permission denied | `errors.PERMISSION_DENIED` | no | | File scanner tokenization or read failed | `errors.INTERNAL` | preserved from the underlying error when available | `unspecified` means `err:retryable()` returns `nil`; it is not equivalent to `false`. See [Error Handling](lua/core/errors.md) for working with errors. --- # "Cloud Storage" ## Cloud Storage Access S3-compatible object storage. Upload, download, list, and manage objects, presign download, upload and multipart-part URLs, and read objects with random access. For storage configuration, see [Cloud Storage](system/cloudstorage.md). ### Loading ```lua local cloudstorage = require("cloudstorage") ``` ### Acquiring Storage Acquire a cloud storage resource by registry ID: ```lua local storage, err = cloudstorage.get("app.infra:files") if err then return nil, err end local uploaded, upload_err = storage:upload_object("data/file.txt", "content") storage:release() if upload_err then return nil, upload_err end return uploaded ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Storage resource ID | **Returns:** `Storage, error` ### Uploading Objects Upload content from a string or file: ```lua local json = require("json") local storage, storage_err = cloudstorage.get("app.infra:files") if storage_err then return nil, storage_err end -- Upload string content local body, encode_err = json.encode({ date = "2024-01-15", total = 1234 }) if encode_err then storage:release() return nil, encode_err end local ok, err = storage:upload_object("reports/daily.json", body) if err then storage:release() return nil, err end -- Upload from file local fs = require("fs") local vol, fs_err = fs.get("app:data") if fs_err then storage:release() return nil, fs_err end local file, open_err = vol:open("/large-file.bin", "r") if open_err then storage:release() return nil, open_err end local uploaded, file_upload_err = storage:upload_object("backups/large-file.bin", file) local _, close_err = file:close() storage:release() if file_upload_err then if close_err then report_cleanup_error(close_err) end return nil, file_upload_err end if close_err then return nil, close_err end return uploaded ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key/path | | `content` | string or Reader | Content as string or file reader | | `options` | table | Optional metadata and conditional write options | **Returns:** `boolean, error` #### Upload Options Attach metadata or guard the write with an options table: ```lua local uploaded, err = storage:upload_object("reports/daily.json", body, { content_type = "application/json", cache_control = "max-age=3600", metadata = { owner = "team-a", run_id = "1234" }, -- stored as x-amz-meta-* only_if_absent = true -- fail if the key already exists }) if err then return nil, err end return uploaded ``` | Option | Type | Description | |--------|------|-------------| | `content_type` | string | MIME type | | `cache_control` | string | Cache-Control header | | `content_disposition` | string | Content-Disposition header | | `content_encoding` | string | Content-Encoding header | | `metadata` | table | User metadata (string keys/values), stored as `x-amz-meta-*` | | `headers` | table | Additional request headers (string keys/values) | | `if_match` | string | Write only if the current object ETag matches | | `if_none_match` | string | Write only if no object matches the ETag (`"*"` means any) | | `only_if_absent` | boolean | Write only if the key does not exist (alias for `if_none_match = "*"`) | A conditional write that fails its precondition returns a `precondition_failed` error. ### Downloading Objects Download an object to a file writer: ```lua local fs = require("fs") local storage, storage_err = cloudstorage.get("app.infra:files") if storage_err then return nil, storage_err end local vol, fs_err = fs.get("app:temp") if fs_err then storage:release() return nil, fs_err end local file, open_err = vol:open("/downloaded.json", "w") if open_err then storage:release() return nil, open_err end local ok, err = storage:download_object("reports/daily.json", file) local _, close_err = file:close() if err then if close_err then report_cleanup_error(close_err) end storage:release() return nil, err end if close_err then storage:release() return nil, close_err end -- Download partial content (first 1KB) local partial, partial_open_err = vol:open("/partial.bin", "w") if partial_open_err then storage:release() return nil, partial_open_err end local partial_ok, partial_err = storage:download_object("backups/large-file.bin", partial, { range = "bytes=0-1023" }) local _, partial_close_err = partial:close() storage:release() if partial_err then if partial_close_err then report_cleanup_error(partial_close_err) end return nil, partial_err end if partial_close_err then return nil, partial_close_err end return partial_ok ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key to download | | `writer` | Writer | Destination file writer | | `options.range` | string | Byte range (e.g., "bytes=0-1023") | | `options.if_match` | string | Download only if the object ETag matches | | `options.if_none_match` | string | Download only if the ETag does not match | **Returns:** `boolean, error` A failed precondition (`if_match`/`if_none_match`) returns a `precondition_failed` error. ### Listing Objects List objects with optional prefix filtering: ```lua local storage, storage_err = cloudstorage.get("app.infra:files") if storage_err then return nil, storage_err end local result, err = storage:list_objects({ prefix = "reports/2024/", max_keys = 100 }) if err then storage:release() return nil, err end for _, obj in ipairs(result.objects) do print(obj.key, obj.size, obj.etag) end -- Paginate through large results local token = nil repeat local page, page_err = storage:list_objects({ prefix = "logs/", max_keys = 1000, continuation_token = token }) if page_err then storage:release() return nil, page_err end for _, obj in ipairs(page.objects) do process(obj) end token = page.next_continuation_token if not page.is_truncated then break end until false storage:release() ``` | Parameter | Type | Description | |-----------|------|-------------| | `options.prefix` | string | Filter by key prefix | | `options.max_keys` | integer | Maximum objects to return | | `options.continuation_token` | string | Pagination token | | `options.include_owner` | boolean | Include each object's `owner` (`id`, `display_name`) | | `options.include_versions` | boolean | List object versions; each item includes `version_id` | **Returns:** `table, error` Result contains `objects`, `is_truncated`, `next_continuation_token`. Each object has `key`, `size`, `etag`, `storage_class`, and optional `last_modified`, `version_id`, and `owner`. In list results content_type is always empty — S3 list operations do not return it. Use head_object to read an object's content type and metadata. ### Object Metadata Fetch a single object's metadata without downloading its body: ```lua local storage, storage_err = cloudstorage.get("app.infra:files") if storage_err then return nil, storage_err end local meta, err = storage:head_object("reports/daily.json") if err then storage:release() return nil, err end print(meta.size, meta.etag, meta.content_type) for k, v in pairs(meta.metadata) do print("meta", k, v) end storage:release() ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key | **Returns:** `table, error` Result fields: | Field | Type | Description | |-------|------|-------------| | `size` | integer | Object size in bytes | | `etag` | string | Entity tag | | `content_type` | string | MIME type | | `cache_control` | string | Cache-Control header | | `content_disposition` | string | Content-Disposition header | | `content_encoding` | string | Content-Encoding header | | `storage_class` | string | Storage class | | `version_id` | string | Version ID (present when versioning is enabled) | | `last_modified` | integer | Last modified time (Unix seconds) | | `metadata` | table | User metadata (`x-amz-meta-*`) | | `headers` | table | Raw response headers (lowercased keys) | A missing object returns a `not_found` error. ### Deleting Objects Remove multiple objects: ```lua local storage, storage_err = cloudstorage.get("app.infra:files") if storage_err then return nil, storage_err end local deleted, err = storage:delete_objects({ "temp/file1.txt", "temp/file2.txt", "temp/file3.txt" }) storage:release() if err then return nil, err end return deleted ``` | Parameter | Type | Description | |-----------|------|-------------| | `keys` | string[] | Array of object keys to delete | **Returns:** `boolean, error` Every key is attempted. Deleting a key that does not exist is not an error. When the provider reports per-key failures, the call returns a single error naming each failed key and its provider error code. ### Download URLs Create a temporary URL that permits downloading an object without storage credentials. A client can use the URL until it expires. ```lua local storage, err = cloudstorage.get("app.infra:files") if err then return nil, err end local url, err = storage:presigned_get_url("reports/quarterly.pdf", { expiration = 3600 }) storage:release() if err then return nil, err end -- Return URL to client for direct download return {download_url = url} ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key | | `options.expiration` | integer | Seconds until URL expires (default: 3600) | **Returns:** `string, error` ### Upload URLs Create a temporary URL that permits uploading an object without storage credentials. A client can upload directly to storage until the URL expires. ```lua local storage, err = cloudstorage.get("app.infra:files") if err then return nil, err end local url, err = storage:presigned_put_url("uploads/user-123/avatar.jpg", { expiration = 600, content_type = "image/jpeg", content_length = 1024 * 1024 }) storage:release() if err then return nil, err end -- Return URL to client for direct upload return {upload_url = url} ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key | | `options.expiration` | integer | Seconds until URL expires (default: 3600) | | `options.content_type` | string | Required content type for upload | | `options.content_length` | integer | Expected upload size in bytes | **Returns:** `string, error` ### Multipart Uploads A single presigned PUT caps an object at 5 GiB. A presigned multipart upload splits a larger object into parts that a client uploads directly, then assembles them server-side. Multipart is a provider capability: S3 implements it, and providers without it return `errors.UNAVAILABLE`. ```lua local storage = cloudstorage.get("app.infra:files") local mp, err = storage:create_multipart_upload("backups/huge.zip", { content_type = "application/zip", metadata = { source = "uploader" }, }) if err then return nil, err end local urls, err = storage:presigned_part_urls("backups/huge.zip", mp.upload_id, { count = 3, expiration = 900, }) if err then storage:abort_multipart_upload("backups/huge.zip", mp.upload_id) return nil, err end -- The client PUTs each url and returns the ETag from the response headers. local done, err = storage:complete_multipart_upload("backups/huge.zip", mp.upload_id, { { part_number = 1, etag = etag1 }, { part_number = 2, etag = etag2 }, { part_number = 3, etag = etag3 }, }) storage:release() ``` #### create_multipart_upload Start a multipart upload for a key. | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key of the final object | | `options` | table | `content_type`, `cache_control`, `content_disposition`, `content_encoding`, `metadata`, `headers` - same semantics as `upload_object` | **Returns:** `table, error` - the table carries `upload_id`, which identifies the upload for every later part, complete and abort call. Conditional writes (`if_match`, `if_none_match`, `only_if_absent`) are not part of the multipart protocol and are not accepted here. #### presigned_part_urls Generate presigned PUT URLs for parts of an in-progress upload. Each URL is uploaded to with a plain HTTP PUT; the uploader must keep the `ETag` response header of each part for `complete_multipart_upload`. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `key` | string | required | Object key | | `upload_id` | string | required | From `create_multipart_upload` | | `options.parts` | int[] | - | Explicit part numbers (1-10000, no duplicates) | | `options.count` | int | - | Presign parts `1..count` | | `options.headers` | table | - | Headers required on each part request; they are signed and must also be sent by the uploader | | `options.expiration` | int | 3600 | Seconds until the URLs expire | Exactly one of `parts` or `count` is required, and a single call presigns at most 1000 URLs - presign in pages for very large objects. **Returns:** `table, error` - an array of `{ part_number, url }`. Every part except the last must be at least 5 MiB; the provider enforces this at completion time. #### complete_multipart_upload Assemble the final object from its uploaded parts. Parts may be reported in any order and are sorted by part number before completion. | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key | | `upload_id` | string | From `create_multipart_upload` | | `parts` | table | Array of `{ part_number = int, etag = string }` | **Returns:** `table, error` - `etag`, plus `version_id` and `location` when the provider reports them. An unknown upload ID returns `errors.NOT_FOUND`. #### abort_multipart_upload Discard an in-progress upload and free its stored parts. | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Object key | | `upload_id` | string | From `create_multipart_upload` | **Returns:** `boolean, error` An upload that is never completed keeps its parts stored, and billed, until it is aborted. Abort on every failure path, and configure a bucket lifecycle rule as a backstop - see [Cloud Storage](system/cloudstorage.md#multipart-uploads). ### Ranged Readers `open_reader` opens random access over an object using ranged GETs - no local staging and no full download. Its main consumer is [`archive.open`](lua/data/archive.md), which reads multi-GB archives straight out of object storage with bounded memory. ```lua local archive = require("archive") local storage = cloudstorage.get("app.infra:files") local reader, err = storage:open_reader("uploads/huge.zip", { block_size = 8 * 1024 * 1024, cache_blocks = 4, }) if err then return nil, err end local r = assert(archive.open(reader)) for e in r:entries() do print(e.name, e.size) end r:close() reader:close() storage:release() ``` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `key` | string | required | Object key | | `options.block_size` | int | 8388608 | Ranged-GET unit in bytes (64 KiB to 128 MiB) | | `options.cache_blocks` | int | 4 | Resident LRU blocks (1 to 64) | `block_size * cache_blocks` may not exceed 256 MiB. A missing object returns `errors.NOT_FOUND`. **Returns:** `Reader, error` The object's ETag is pinned when the reader opens and sent as `If-Match` on every ranged read, so an object overwritten mid-read fails the read with the provider's precondition error instead of serving a mix of two object generations; `archive` surfaces it as `errors.INTERNAL`. A provider that cannot supply an ETag returns `errors.UNAVAILABLE`; the reader never serves an unpinned object. Cache-miss reads perform blocking network IO in the calling task and serialize concurrent readers, so sequential per-entry access - the archive pattern - is the intended shape. #### Reader Methods | Method | Returns | Description | |--------|---------|-------------| | `size()` | `integer` | Object size in bytes, from the open-time stat | | `key()` | `string` | Object key the reader reads from | | `close()` | `boolean, error` | Release the block cache; idempotent | The reader is closed automatically at task scope if it is not closed explicitly. ### Storage Methods | Method | Returns | Description | |--------|---------|-------------| | `upload_object(key, content, opts?)` | `boolean, error` | Upload string or file content | | `download_object(key, writer, opts?)` | `boolean, error` | Download to file writer | | `head_object(key)` | `table, error` | Fetch object metadata | | `list_objects(opts?)` | `table, error` | List objects with prefix filter | | `delete_objects(keys)` | `boolean, error` | Delete multiple objects | | `presigned_get_url(key, opts?)` | `string, error` | Generate temporary download URL | | `presigned_put_url(key, opts?)` | `string, error` | Generate temporary upload URL | | `create_multipart_upload(key, opts?)` | `table, error` | Start a presigned multipart upload | | `presigned_part_urls(key, upload_id, opts)` | `table, error` | Presign PUT URLs for upload parts | | `complete_multipart_upload(key, upload_id, parts)` | `table, error` | Assemble the object from uploaded parts | | `abort_multipart_upload(key, upload_id)` | `boolean, error` | Discard an in-progress multipart upload | | `open_reader(key, opts?)` | `Reader, error` | Open a ranged random-access reader | | `release()` | `boolean` | Release storage resource | ### Permissions Security policy evaluation applies to cloud storage operations. | Action | Resource | Description | |--------|----------|-------------| | `cloudstorage.get` | Storage ID | Acquire a storage resource | ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty resource ID | `errors.INVALID` | no | | Resource not found | `errors.NOT_FOUND` | no | | Not a cloud storage resource | `errors.INVALID` | no | | Storage released | `errors.INVALID` | no | | Empty key | `errors.INVALID` | no | | Content nil | `errors.INVALID` | no | | Writer not valid | `errors.INVALID` | no | | Object not found | `errors.NOT_FOUND` | no | | Unknown upload ID | `errors.NOT_FOUND` | no | | Conditional precondition failed | `errors.CONFLICT` | no | | Object overwritten during a ranged read (surfaced by `archive`) | `errors.INTERNAL` | no | | Provider does not support multipart uploads | `errors.UNAVAILABLE` | no | | Provider supplies no ETag for `open_reader` | `errors.UNAVAILABLE` | no | | Permission denied | raised as a Lua error, not returned | - | | Provider operation failed | `errors.UNKNOWN` | unset | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Message Queue" ## Message Queue The `queue` module publishes messages and processes deliveries from configured distributed queues, including RabbitMQ and other AMQP-compatible brokers. This page is an API reference. Publishing snippets assume the queue entries and permissions already exist. The consumer section is a partial recipe for a handler invoked by `queue.consumer`; it is not a standalone queue deployment. For queue configuration, see [Queue](system/queue.md). ### Loading ```lua local queue = require("queue") ``` ### Publishing Messages Publish a message to a queue by ID: ```lua local ok, err = queue.publish("app:tasks", { action = "send_email", user_id = 456, template = "welcome" }) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `queue_id` | string | Queue identifier (format: "namespace:name") | | `data` | any | Message data (tables, strings, numbers, booleans) | | `headers` | table | Optional message headers | **Returns:** `boolean, error` #### Message Headers Headers carry routing, priority, and tracing metadata. Keys must be strings, and publisher values may be strings, integers, numbers, or booleans: ```lua local ok, err = queue.publish("app:notifications", { type = "order_shipped", order_id = order.id }, { priority = 5, correlation_id = request_id }) if err then return nil, err end ``` Consumers receive every header value as a string. The `x_original_queue`, `x_dead_letter_reason`, `x_dead_letter_time`, and `attempts` keys are reserved for delivery and dead-letter bookkeeping and must not be set by publishers. ### Accessing Delivery Context Access the current delivery from within a queue consumer: ```lua local msg, err = queue.message() if err then return nil, err end local msg_id, id_err = msg:id() if id_err then return nil, id_err end local priority, header_err = msg:header("priority") if header_err then return nil, header_err end local all_headers, headers_err = msg:headers() if headers_err then return nil, headers_err end ``` **Returns:** `Message, error` This function is available only while a queue consumer is processing a message. ### Message Methods | Method | Returns | Description | |--------|---------|-------------| | `id()` | `string, error` | Unique message identifier | | `header(key)` | `string, error` | Single header value as a string (nil if missing) | | `headers()` | `table, error` | All message headers | | `ack()` | `boolean, error` | Acknowledge processing (single-shot) | | `nack()` | `boolean, error` | Signal failure for redelivery or dead-letter (single-shot) | The runtime auto-acks on handler success and auto-nacks on handler error. Call `ack`/`nack` only to settle early. Settlement is single-shot, and a `Message` is invalid after its consumer handler returns. ### Queue Info ```lua local stats, err = queue.info("app:tasks") if err then return nil, err end -- stats may contain: message_count, consumer_count, ready (driver-dependent) ``` **Returns:** `table, error` ### Consumer Pattern A `queue.consumer` entry binds a queue to the handler referenced by `func`. The handler receives the message payload directly: ```yaml entries: - kind: queue.consumer name: email_worker queue: app:emails func: app:email_handler ``` This fragment assumes `app:emails` and the `app:email_handler` function entry already exist. The function source below assumes the application supplies `deliver_email(payload)` and grants any permissions it needs. ```lua local queue = require("queue") local logger = require("logger") local function main(payload) local msg, msg_err = queue.message() if msg_err then return nil, msg_err end local message_id, id_err = msg:id() if id_err then return nil, id_err end logger:info("Processing", { message_id = message_id, to = payload.to }) local ok, send_err = deliver_email(payload) if send_err then return nil, send_err end return ok end return {main = main} ``` Returning an invocation error causes the consumer to nack the unsettled delivery. Redelivery then follows the selected driver's behavior; the built-in dead-letter configuration is not enforced in this release. ### Permissions Security policy evaluation applies to queue operations. | Action | Resource | Description | |--------|----------|-------------| | `queue.publish` | - | General permission to publish messages | | `queue.publish.queue` | Queue ID | Publish to specific queue | The runtime checks the general permission first and the queue-specific permission second. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Queue ID empty | `errors.INVALID` | no | | Message argument missing or an empty table | `errors.INVALID` | no | | No delivery context | `errors.INVALID` | no | | Message released or already settled | `errors.INVALID` | no | | Publish not allowed | `errors.INVALID` | no | | Publish failed | `errors.INTERNAL` | no | | Queue or driver not found for `info` | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Queue Configuration](system/queue.md) - Queue drivers and entry definitions - [Queue Consumers Guide](guides/queue-consumers.md) - Consumer patterns and worker pools - [Process Management](lua/core/process.md) - Process spawning and communication - [Channels](lua/core/channel.md) - Inter-process communication patterns - [Functions](lua/core/funcs.md) - Async function invocation --- # "CDC" ## CDC Subscribe to Change Data Capture streams from [`db.cdc.postgres`](system/cdc.md) and [`db.cdc.sqlite`](system/cdc.md) sources. List configured sources, open a stream, and receive row-level change events over a channel. The API is driver-neutral: both kinds return the same source info and the same change events, and differ only in the [capabilities](system/cdc.md#capabilities) they publish. ### Loading ```lua local cdc = require("cdc") ``` ### `list_sources` List the configured CDC sources the caller is allowed to see: ```lua local sources, err = cdc.list_sources() if err then return nil, err end for _, s in ipairs(sources) do print(s.id, s.kind, s.state, s.capabilities.before_images) end ``` Sources the caller lacks `cdc.source` on are omitted rather than reported as an error. **Returns:** `table, error` ### `source` Retrieve one source by its registry entry ID or replication slot name: ```lua local info, err = cdc.source("app:pg_cdc") if err then return nil, err end if info == nil then -- no such source end ``` **Returns:** `table, error` (source info, or `nil` if not found) ### `stream` Open a change stream on a source. The returned `cdc.Stream` exposes a channel that delivers change events: ```lua local stream, err = cdc.stream("app:pg_cdc", { tables = { "public.users", "public.orders" }, ops = { "insert", "update" }, buffer = 128, }) if err then return nil, err end -- The caller owns stream until close(), release(), or task cleanup. ``` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `name` | string | required | Source name (entry ID) | | `opts.tables` | []string | - | Filter to these tables (omit for all captured tables) | | `opts.ops` | []string | - | Filter to these operations: `insert`, `update`, `delete`, `truncate` | | `opts.buffer` | int | 64 | Backlog item capacity (1-65536) | | `opts.max_bytes` | int | 1048576 | Backlog byte budget for this subscriber (1 MiB) | | `opts.snapshot` | bool | entry default | Request the snapshot/live handoff for this stream | | `opts.after` | string | - | Opaque resume cursor from a previous event's `cursor` | Unknown option keys are rejected with `errors.INVALID`. Table names are matched case-insensitively against both the qualified relation and the bare table name. Snapshot rows are filtered by `tables` only; `ops` applies to live changes. A stream receives a snapshot when either `opts.snapshot` is true or the source entry's `snapshot` field is set; snapshot rows arrive first with `op = "snapshot"`, then the stream continues into live changes with no gap. `opts.after` is reserved for drivers that resume from a cursor — every driver shipped today returns `errors.INVALID` ("cdc operation is not supported by this source") for it, including `db.cdc.postgres` when it reports `capture_resume`. Filters narrow delivery only. Access to a source is granted by the `cdc.subscribe` permission, never by a filter. **Returns:** `Stream, error` The Lua delivery channel has a separate fixed capacity of 64. The `buffer` option controls the PostgreSQL source subscription, not that channel. #### `channel` Return the channel that receives change events. The first call subscribes to the source and yields; subsequent calls return the same channel. The first call can return a subscription error. Channel `:receive()` returns `value, true` for a change or `nil, false` when the stream ends: ```lua local stream, stream_err = cdc.stream("app:pg_cdc") if stream_err then return nil, stream_err end local ch, subscribe_err = stream:channel() if subscribe_err then stream:close() return nil, subscribe_err end while true do local change, ok = ch:receive() if not ok then break end if change.op == "snapshot" then seed_row(change.table, change.after) elseif change.op == "insert" then handle_new_user(change.table, change.after) elseif change.op == "update" then handle_update(change.table, change.before, change.after) elseif change.op == "delete" then handle_delete(change.table, change.before) end end local _, close_err = stream:close() if close_err then return nil, close_err end ``` The stream is lazy: construct it, then call `channel()` before generating the writes it should observe. This is live observation, not replay of changes made before the subscription. When a source terminates a stream with a failure, the channel delivers an error value before it closes. `receive` is an alias for `channel`. #### `close` Stop the subscription and release the stream. The method is idempotent, and the runtime also closes the stream at the end of the task scope. `release` is an alias for `close`. ```lua local _, err = stream:close() if err then return nil, err end ``` ### Change Event Each message received on the channel is a change table: | Field | Description | |-------|-------------| | `op` | Operation: `insert`, `update`, `delete`, `snapshot` or `truncate` | | `schema` | Table schema | | `table` | Table name | | `relation` | Qualified relation name | | `before` | Row state before the change (`update`, `delete`). A full row image is guaranteed only when the source has the `before_images` capability; `db.cdc.postgres` fills it from whatever old tuple the WAL carries, which the table's `REPLICA IDENTITY` controls | | `after` | Row state after the change (`insert`, `update`, `snapshot`; absent for `delete`) | | `source` | Source entry ID | | `source_id` | Source entry ID, as a registry ID | | `generation` | Source generation that produced the event | | `cursor` | Opaque per-event position within the source | | `transaction` | Transaction identifier, when the driver reports one | | `lsn` | Log sequence number of the change (`db.cdc.postgres`) | | `commit_lsn` | LSN of the committing transaction (when applicable) | | `xid` | Transaction ID (when applicable) | | `unchanged` | Columns whose value was not transmitted (unchanged TOAST values) | | `error` | Driver-reported error description carried on the event | `before` and `after` are row maps keyed by column name. ### Source Info `cdc.source` and each entry of `cdc.list_sources` return the same record: | Field | Description | |-------|-------------| | `id` | Entry ID | | `kind` | `db.cdc.postgres` or `db.cdc.sqlite` | | `name` | Source name (the entry ID) | | `state` | `unknown`, `starting`, `running`, `faulted` or `stopped` | | `generation` | Current source generation | | `epoch` | Same value as `generation` | | `engine` | Engine name, when the driver reports one | | `db_resource` | Observed SQL resource entry ID (`db.cdc.sqlite`) | | `slot` | Replication slot name (`db.cdc.postgres`) | | `publication` | Postgres publication, when configured | | `tables` | Captured tables, when configured | | `streaming` | `db.cdc.sqlite`: whether the source is running; `db.cdc.postgres`: the entry's `streaming` protocol setting | | `failover` | Failover slot mode (`db.cdc.postgres`) | | `temporary` | Temporary slot (`db.cdc.postgres`) | | `snapshot` | Entry-level snapshot default | | `faulted` | Whether the source is in the `faulted` state | | `error` | Last source error, when one is recorded | | `admission` | `active`, `snapshots`, `reserved_bytes`, `rejected` | | `capabilities` | `snapshot`, `capture_resume`, `replayable`, `captures_external_writes`, `before_images`, `coalesced` | Branch on `capabilities` rather than on `kind`: ```lua local info = cdc.source("app:changes") if not info.capabilities.before_images then -- before is not a guaranteed full row image; keep your own last-known state end ``` See [CDC sources](system/cdc.md#source-info) for field semantics. ### Permissions | Action | Resource | Description | |--------|----------|-------------| | `cdc.source` | Source entry ID | `cdc.source`; also filters `cdc.list_sources` | | `cdc.subscribe` | Source entry ID | `cdc.stream`, checked again when the subscription is established | A denied action returns `errors.PERMISSION_DENIED`. ### Errors | Condition | Kind | |-----------|------| | No context | `errors.INTERNAL` | | Source name required | `errors.INVALID` | | Invalid or unknown stream option | `errors.INVALID` | | `after` on a source without `capture_resume` | `errors.INVALID` | | Source not registered | `errors.NOT_FOUND` | | Source not started or replacing | `errors.UNAVAILABLE` | | Subscription capacity exhausted | `errors.UNAVAILABLE` | | Permission denied | `errors.PERMISSION_DENIED` | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Change Data Capture](system/cdc.md) - Source configuration and capabilities - [Channel](lua/core/channel.md) - Channel semantics - [Database](system/database.md) - SQL database services --- # "System" ## System The `system` module reports runtime, memory, process, host, supervisor, and cluster state. It also exposes selected runtime controls. This is an API reference. Most snippets show one isolated operation; controls such as shutdown, runtime tuning, and distributed locks require explicit policy authorization and application-specific failure handling. ### Loading ```lua local system = require("system") ``` ### Shutdown Request system shutdown with an exit code. Calling this function from any process or actor terminates the entire system: ```lua local ok, err = system.exit(0) ``` | Parameter | Type | Description | |-----------|------|-------------| | `code` | integer | Exit code (0 = success), defaults to 0 | **Returns:** `boolean, error` ### Listing Modules List the loaded Lua modules and their metadata: ```lua local mods, err = system.modules() ``` **Returns:** `table[], error` Each module table contains: | Field | Type | Description | |-------|------|-------------| | `name` | string | Module name | | `description` | string | Module description | | `class` | string[] | Module classification tags | ### Deployment Sources The `system.source` sub-table reads the normalized deployment baseline: the entry set produced by the sources the application was assembled from, before any registry history is applied. ```lua local loaded, err = system.source.load() ``` **Returns:** `table, error` | Field | Type | Description | |-------|------|-------------| | `owners` | string[] | Source owners authoritative over the baseline entries | | `entries` | table[] | Baseline entries with `id`, `kind`, `meta`, `data` | `owners` is sorted with the application owner first, then the remaining owners alphabetically. The application owner is the string `"application"`. ```lua local loaded, err = system.source.load() if err then return nil, err end for _, owner in ipairs(loaded.owners) do print(owner) end for _, entry in ipairs(loaded.entries) do print(entry.id, entry.kind) end ``` The load is taken from one stable source generation, so entries and owners always describe the same baseline. Filesystem paths behind each source are runtime-private and are not exposed; a failed load reports a generic internal error rather than leaking the backing path. **Permission:** `system.read` on `sources` ### Memory Statistics Read detailed memory statistics: ```lua local stats, err = system.memory.stats() ``` **Returns:** `table, error` The statistics table contains: | Field | Type | Description | |-------|------|-------------| | `alloc` | number | Bytes allocated and in use | | `total_alloc` | number | Cumulative bytes allocated | | `sys` | number | Bytes obtained from system | | `heap_alloc` | number | Bytes allocated on heap | | `heap_sys` | number | Bytes obtained for heap from system | | `heap_idle` | number | Bytes in idle spans | | `heap_in_use` | number | Bytes in non-idle spans | | `heap_released` | number | Bytes released to OS | | `heap_objects` | number | Number of allocated heap objects | | `stack_in_use` | number | Bytes used by stack allocator | | `stack_sys` | number | Bytes obtained for stack from system | | `mspan_in_use` | number | Bytes of mspan structures in use | | `mspan_sys` | number | Bytes obtained for mspan from system | | `num_gc` | number | Number of completed GC cycles | | `next_gc` | number | Target heap size for next GC | ### Current Allocation Read the number of bytes currently allocated: ```lua local bytes, err = system.memory.allocated() ``` **Returns:** `number, error` ### Heap Objects Read the number of allocated heap objects: ```lua local count, err = system.memory.heap_objects() ``` **Returns:** `number, error` ### Memory Limit Set the memory limit and return its previous value: ```lua local prev, err = system.memory.set_limit(1024 * 1024 * 100) ``` | Parameter | Type | Description | |-----------|------|-------------| | `limit` | integer | Memory limit in bytes, -1 for unlimited | **Returns:** `number, error` Read the current memory limit: ```lua local limit, err = system.memory.get_limit() ``` **Returns:** `number, error` ### Force GC Run garbage collection immediately: ```lua local ok, err = system.gc.collect() ``` **Returns:** `boolean, error` ### GC Target Percentage Set the garbage-collection target percentage and return its previous value. A value of 100 triggers collection when the heap doubles: ```lua local prev, err = system.gc.set_percent(200) ``` | Parameter | Type | Description | |-----------|------|-------------| | `percent` | integer | GC target percentage | **Returns:** `number, error` Read the current garbage-collection target percentage: ```lua local percent, err = system.gc.get_percent() ``` **Returns:** `number, error` ### Goroutine Count Read the number of active goroutines: ```lua local count, err = system.runtime.goroutines() ``` **Returns:** `number, error` ### GOMAXPROCS Read or set the `GOMAXPROCS` value: ```lua -- Get current value local current, err = system.runtime.max_procs() -- Set new value local prev, err = system.runtime.max_procs(4) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | If provided, sets GOMAXPROCS (must be > 0) | **Returns:** `number, error` ### CPU Count Read the number of logical CPUs: ```lua local cpus, err = system.runtime.cpu_count() ``` **Returns:** `number, error` ### Process ID Read the current operating-system process ID: ```lua local pid, err = system.process.pid() ``` **Returns:** `number, error` ### Hostname Read the system hostname: ```lua local hostname, err = system.process.hostname() ``` **Returns:** `string, error` ### Working Directory Read the runtime's current working directory: ```lua local dir, err = system.process.cwd() ``` **Returns:** `string, error` ### Process Hosts List process hosts with worker and queue statistics: ```lua local hosts, err = system.hosts.list() ``` **Returns:** `table[], error` Each host table contains: | Field | Type | Description | |-------|------|-------------| | `id` | string | Host registry ID | | `workers` | number | Worker pool size | | `processes` | number | Active processes on this host | | `executed` | number | Total steps executed | | `stolen` | number | Steps stolen from other hosts | | `queue_depth` | number | Pending items in the host queue | List processes running on a specific host: ```lua local procs, err = system.hosts.processes("app:host") ``` | Parameter | Type | Description | |-----------|------|-------------| | `host_id` | string | Host registry ID | **Returns:** `table[], error` Each process table contains: | Field | Type | Description | |-------|------|-------------| | `pid` | string | Process ID | | `host` | string | Host ID | | `source` | string | Source entry ID | | `state` | string | Process state | | `steps` | number | Steps executed | | `started_at` | number | Start timestamp (nanoseconds) | | `parent` | string | Parent PID (omitted if none) | | `actor_id` | string | Actor ID (omitted if none) | | `stats` | table | Process-specific stats (optional) | ### Service State Read the state of a supervised service: ```lua local state, err = system.supervisor.state("namespace:service") ``` | Parameter | Type | Description | |-----------|------|-------------| | `service_id` | string | Service ID (e.g., "namespace:service") | **Returns:** `table, error` The state table contains: | Field | Type | Description | |-------|------|-------------| | `id` | string | Service ID | | `status` | string | Current status | | `desired` | string | Desired status | | `retry_count` | number | Number of retries | | `last_update` | number | Last update timestamp (nanoseconds) | | `started_at` | number | Start timestamp (nanoseconds) | | `details` | string | Optional details (formatted) | ### All Service States List the states of all supervised services: ```lua local states, err = system.supervisor.states() ``` **Returns:** `table[], error` Each state table has the same format as `system.supervisor.state()`. ### Cluster Primitives The `system.node`, `system.cluster`, `system.raft`, and `system.lock` sub-tables expose the clustering layer. They are most useful when [clustering is enabled](guides/cluster.md); on a standalone node they degrade predictably — `system.raft.*` reports "raft not available", `system.cluster` reports just the local node, and `system.lock` requires the Raft-backed KV store that clustering provides. Read calls report this node's local view of committed state and do not block on the network. #### Node Identity `system.node` reports the current node's identity in the cluster. ```lua local id, err = system.node.id() -- this node's ID local addr, err = system.node.addr() -- advertised network address local role, err = system.node.role() -- "leader" | "voter" | "standby" | "non-member" ``` | Function | Returns | Notes | |----------|---------|-------| | `system.node.id()` | `string, error` | Node ID from the relay context | | `system.node.addr()` | `string, error` | Advertised address (e.g. `10.0.0.1:7946`); errors if membership is unavailable | | `system.node.role()` | `string, error` | Raft role of this node; returns `"non-member"` (no error) when Raft is not running | **Permission:** `system.read` on `node`. #### Cluster Membership `system.cluster` reports cluster membership and the current leader. ```lua local members, err = system.cluster.members() -- array of node tables local leader, err = system.cluster.leader() -- leader node ID, or "" if unknown local n, err = system.cluster.size() -- count of visible members ``` `system.cluster.members()` returns an array of node tables. The local node is included once and sorts first. | Field | Type | Description | |-------|------|-------------| | `id` | string | Node ID | | `is_local` | boolean | True for the calling node | | `addr` | string | Advertised address (omitted when unknown) | | `meta` | table | String-to-string gossip metadata (omitted when none) | | Function | Returns | Notes | |----------|---------|-------| | `system.cluster.members()` | `table[], error` | Errors if no membership information is reachable | | `system.cluster.leader()` | `string, error` | Current Raft leader's ID; `""` (no error) when the leader is unknown or Raft is absent | | `system.cluster.size()` | `number, error` | Count of visible members; `0` when no membership info is available | **Permission:** `system.read` on `cluster`. #### Raft State `system.raft` reads this node's local view of the Raft consensus core. Every function returns `nil, error` ("raft not available") when Raft is not running on this node. ```lua local leader, err = system.raft.is_leader() -- boolean local member, err = system.raft.is_member() -- boolean: voter or standby local role, err = system.raft.role() -- same values as system.node.role() local term, err = system.raft.term() -- current Raft term local idx, err = system.raft.commit_index() -- highest committed log index local stats, err = system.raft.stats() -- raw stats map (string -> string) ``` | Function | Returns | Notes | |----------|---------|-------| | `system.raft.is_leader()` | `boolean, error` | True iff this node is the current leader | | `system.raft.is_member()` | `boolean, error` | True iff this node is a voter or standby in the committed configuration | | `system.raft.role()` | `string, error` | `"leader"` / `"voter"` / `"standby"` / `"non-member"` | | `system.raft.term()` | `number, error` | Current term; `0` if unavailable from stats | | `system.raft.commit_index()` | `number, error` | Highest committed log index on this node | | `system.raft.stats()` | `table, error` | Full raw stats map; keys and values are strings | **Permission:** `system.read` on `raft`, except `system.raft.stats()` which requires `system.read` on `raft_stats`. #### Distributed Locks `system.lock` provides cluster-wide mutual exclusion. A lock is a globally unique name owned by the calling process. It is built on the Raft-replicated system KV store, so at most one holder can exist across the cluster, and the lock auto-releases when the holder process exits or its node leaves — there is no stuck lock to clean up. ```lua local ok, err = system.lock.acquire("orders.migration") if not ok then -- err has kind errors.ALREADY_EXISTS when another process holds the lock. -- Apply the caller's retry and backoff policy for that case if needed. return nil, err end -- critical section: only one holder cluster-wide local released, release_err = system.lock.release("orders.migration") if release_err then return nil, release_err end return released ``` Acquisition is fail-fast: when a lock is already held, the call returns `false` immediately instead of blocking. Callers provide any required retry and backoff. Only the current holder can release a lock; a release attempt by another process is a no-op. | Function | Returns | Outcomes | |----------|---------|----------| | `system.lock.acquire(name)` | `boolean, error` | `true, nil` acquired; `false, error` already held (kind `errors.ALREADY_EXISTS`); `nil, error` on failure | | `system.lock.release(name)` | `boolean, error` | `true, nil` released; `false, nil` not held or held by another process; `nil, error` on failure | | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Cluster-wide lock name | **Permission:** `system.lock` on the lock `name` (so policy can restrict which names a caller may lock). ### Permissions Security policy evaluation applies to system operations. | Action | Resource | Description | |--------|----------|-------------| | `system.read` | `memory` | Read memory statistics | | `system.read` | `memory_limit` | Read memory limit | | `system.control` | `memory_limit` | Set memory limit | | `system.read` | `gc_percent` | Read GC percentage | | `system.gc` | `gc` | Force garbage collection | | `system.gc` | `gc_percent` | Set GC percentage | | `system.read` | `goroutines` | Read goroutine count | | `system.read` | `gomaxprocs` | Read GOMAXPROCS | | `system.control` | `gomaxprocs` | Set GOMAXPROCS | | `system.read` | `cpu` | Read CPU count | | `system.read` | `pid` | Read process ID | | `system.read` | `hostname` | Read hostname | | `system.read` | `cwd` | Read working directory | | `system.read` | `hosts` | List hosts / host processes | | `system.read` | `modules` | List loaded modules | | `system.read` | `sources` | Load the deployment source baseline | | `system.read` | `supervisor` | Read supervisor state | | `system.read` | `node` | Read this node's identity | | `system.read` | `cluster` | Read cluster membership and leader | | `system.read` | `raft` | Read Raft state | | `system.read` | `raft_stats` | Read the raw Raft stats map | | `system.lock` | `` | Acquire or release a distributed lock | | `system.exit` | - | Trigger system shutdown | ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Permission denied (`system.source.load`, `system.lock.*`) | `errors.PERMISSION_DENIED` | no | | Permission denied (all other calls) | `errors.INVALID` | no | | Invalid argument | `errors.INVALID` | no | | Missing required argument | `errors.INVALID` | no | | Code manager unavailable | `errors.INTERNAL` | no | | Service info unavailable | `errors.INTERNAL` | no | | OS error (hostname, cwd) | `errors.INTERNAL` | no | | Raft not running on this node | `errors.INTERNAL` | no | | Membership unavailable | `errors.INTERNAL` | no | | Lock already held | `errors.ALREADY_EXISTS` | no | | Lock service unavailable (no Raft on this node) | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Environment Variables" ## Environment Variables The `env` module reads and updates environment variables exposed by the runtime. This is an API reference. Its snippets are isolated operations and assume the named variables and security policies already exist. Variables must be defined in the [Environment System](system/env.md) before they can be accessed. The system controls which storage backends (OS, file, memory) provide values and whether variables are read-only. ### Loading ```lua local env = require("env") ``` ### `get` Retrieve an environment variable. ```lua -- Get database connection string local db_url = env.get("DATABASE_URL") if not db_url then return nil, errors.new({ kind = errors.INVALID, message = "DATABASE_URL not configured" }) end local port, port_err = get_or("PORT", "8080") if port_err then return nil, port_err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Variable name | **Returns:** `string, error` The function returns `nil, error` when the variable does not exist. ### `set` Set an environment variable. ```lua -- Set runtime configuration local updated, set_err = env.set("APP_MODE", "production") if set_err then return nil, set_err end return updated ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | Variable name | | `value` | string | Value to set | **Returns:** `boolean, error` ### `get_all` Retrieve all environment variables accessible to the caller. ```lua local logger = require("logger") local vars, vars_err = env.get_all() if vars_err then return nil, vars_err end -- Log names only. Values such as connection URLs may contain credentials even -- when their keys do not include words like SECRET or KEY. local accessible_keys = {} for key in pairs(vars) do table.insert(accessible_keys, key) end logger:debug("accessible environment variables", {keys = accessible_keys}) -- Check required variables local required = {"DATABASE_URL", "REDIS_URL", "API_KEY"} for _, key in ipairs(required) do if not vars[key] then return nil, errors.new({ kind = errors.INVALID, message = "Missing required env var: " .. key }) end end ``` **Returns:** `table, error` ### Permissions Security policy evaluation applies to environment access. #### Security Actions | Action | Resource | Description | |--------|----------|-------------| | `env.get` | Variable name | Read environment variable | | `env.set` | Variable name | Write environment variable | `get_all` has no dedicated security action: it returns only the variables for which the `env.get` action is permitted, filtering each variable name through `env.get`. #### Checking Access ```lua local security = require("security") if security.can("env.get", "DATABASE_URL") then local url = env.get("DATABASE_URL") end ``` See [Security Model](system/security.md) for policy configuration. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty key | `errors.INVALID` | no | | Variable not found | `errors.NOT_FOUND` | no | | Permission denied | `errors.PERMISSION_DENIED` | no | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Environment System](system/env.md) - Configure storage backends and variable definitions --- # "Logging" ## Logging The `logger` module writes structured messages at debug, info, warn, and error levels. This is an API reference. Each snippet is an isolated logging operation and assumes an execution context with the desired logger configuration. Log calls return no values. When the execution context provides them, each call also adds the process `pid` and the source `location` derived from the current frame. ### Loading ```lua local logger = require("logger") ``` #### `logger:debug` Write a debug-level log message. ```lua logger:debug("message", {key = "value"}) ``` #### `logger:info` Write an info-level log message. ```lua logger:info("message", {key = "value"}) ``` #### `logger:warn` Write a warning-level log message. ```lua logger:warn("message", {key = "value"}) ``` #### `logger:error` Write an error-level log message. ```lua logger:error("message", {key = "value"}) ``` All four log-level methods accept the same parameters: | Parameter | Type | Description | |-----------|------|-------------| | `message` | string | Log message | | `fields` | table? | Contextual key-value pairs | Only string keys become field names. Strings, numbers, integers, booleans, errors, and structured Lua values are converted to log fields; non-string keys are ignored. For `logger:error`, a field named `error` is emitted as an error field and removed from the supplied table before the remaining fields are processed. Do not reuse that table if the `error` entry must remain intact. #### `logger:with` Create a child logger that adds the same fields to every message. ```lua local function request_logger(request_id) return logger:with({request_id = request_id}) end request_logger("req-123"):info("message") ``` | Parameter | Type | Description | |-----------|------|-------------| | `fields` | table | Fields to attach to all logs | **Returns:** `Logger` The original logger is unchanged. Child loggers can be chained with additional `with` and `named` calls. #### `logger:named` Create a child logger with a name. ```lua local named = logger:named("auth") named:info("message") ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Logger name | **Returns:** `Logger` An empty name raises a Lua argument error. It is not returned as a structured `errors.INVALID` value. `logger:named("")` raises a Lua argument error (`name cannot be empty`) instead of returning an error value. Logging methods return nothing. See [Error Handling](lua/core/errors.md) for working with errors. --- # "Terminal I/O" ## Terminal I/O The `io` module reads from standard input and writes to standard output and standard error in terminal applications. This is an API reference. Its snippets are isolated calls; a terminal process should propagate returned structured Lua errors when the result affects control flow. This module is available only to processes running on a Terminal Host, not to regular functions. ### Loading ```lua local io = require("io") ``` ### Writing to Stdout Write values to standard output without a trailing newline: ```lua local ok, err = io.write("text", "more") ``` | Parameter | Type | Description | |-----------|------|-------------| | `...` | any | Variable number of values to write (coerced to string) | **Returns:** `boolean, error` ### Print with Newline Write values to standard output, separated by tabs and followed by a newline: ```lua io.print("value1", "value2", 123) ``` | Parameter | Type | Description | |-----------|------|-------------| | `...` | any | Variable number of values to print | **Returns:** `boolean, error` After terminal context lookup succeeds, output write errors are ignored and the function returns `true`. A missing terminal context returns `nil, "no terminal context"`. ### Writing to Stderr Write values to standard error, separated by tabs and followed by a newline: ```lua io.eprint("Error:", message) ``` | Parameter | Type | Description | |-----------|------|-------------| | `...` | any | Variable number of values to print | **Returns:** `boolean, error` After terminal context lookup succeeds, output write errors are ignored and the function returns `true`. A missing terminal context returns `nil, "no terminal context"`. ### Reading Bytes Read up to `n` bytes from standard input: ```lua local data, err = io.read(1024) ``` | Parameter | Type | Description | |-----------|------|-------------| | `n` | integer | Number of bytes to read (default: 1024, values <= 0 become 1024) | **Returns:** `string, error`. A successful read may return fewer than `n` bytes or an empty string. ### Reading a Line Read one line from standard input: ```lua local line, err = io.readline() ``` **Returns:** `string, error`. The trailing `\n` and `\r` are removed. EOF after partial input returns that partial line; EOF without input returns `nil` and a structured error. ### Raw Mode Enable or disable raw terminal mode, which disables line buffering and echo: ```lua local ok, err = io.raw(true) -- enable local ok, err = io.raw(false) -- disable ``` | Parameter | Type | Description | |-----------|------|-------------| | `enable` | boolean | `true` to enable, `false` to disable (default: `true`) | **Returns:** `boolean, error` Raw mode is reference-counted: each `io.raw(true)` call must be matched by an `io.raw(false)` call. The terminal returns to normal mode automatically when the process exits. ### Flushing Output Flush the standard-output buffer: ```lua local ok, err = io.flush() ``` **Returns:** `boolean, error`. The call is a successful no-op when standard output does not implement `Sync()`. ### Command Line Arguments Retrieve the command-line arguments: ```lua local args = io.args() ``` **Returns:** `string[]` `io.args()` never fails. It returns an empty table when no terminal context is available. ### Errors This module returns structured Lua errors. A missing terminal context uses `errors.UNAVAILABLE`; direct write/flush and invalid yield-response failures use `errors.INTERNAL`. Dispatcher-backed read, readline, and raw-mode failures preserve underlying error metadata when available. `io.args()` has no error return. --- # "Metrics & Telemetry" ## Metrics & Telemetry The `metrics` module records application counters, gauges, and histogram observations. This is an API reference. The snippets show one observation at a time and propagate collector errors. Every function returns `true, nil` after passing the observation to the active collector. If the execution context has no collector, it returns `nil` and a non-retryable `errors.INTERNAL` error. Labels are optional. Only entries with both a string key and a string value are recorded; other entries are silently ignored. A non-table labels argument is treated as if no labels were supplied. Metric names are forwarded without local validation. ### Loading ```lua local metrics = require("metrics") ``` #### `metrics.counter_inc` Increment a counter by one. ```lua local recorded, err = metrics.counter_inc("requests_total", {method = "POST"}) if err then return nil, err end return recorded ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Metric name | | `labels` | table? | Label key-value pairs | **Returns:** `boolean, error` #### `metrics.counter_add` Add a value to a counter. ```lua local recorded, err = metrics.counter_add("bytes_total", 1024, {direction = "out"}) if err then return nil, err end return recorded ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Metric name | | `value` | number | Value to add | | `labels` | table? | Label key-value pairs | **Returns:** `boolean, error` The runtime forwards the value unchanged and does not require it to be positive. #### `metrics.gauge_set` Set a gauge to the current value. ```lua local recorded, err = metrics.gauge_set("queue_depth", 42, {queue = "emails"}) if err then return nil, err end return recorded ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Metric name | | `value` | number | Current value | | `labels` | table? | Label key-value pairs | **Returns:** `boolean, error` #### `metrics.gauge_inc` Increment a gauge by one. ```lua local recorded, err = metrics.gauge_inc("connections", {pool = "db"}) if err then return nil, err end return recorded ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Metric name | | `labels` | table? | Label key-value pairs | **Returns:** `boolean, error` #### `metrics.gauge_dec` Decrement a gauge by one. ```lua local recorded, err = metrics.gauge_dec("connections", {pool = "db"}) if err then return nil, err end return recorded ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Metric name | | `labels` | table? | Label key-value pairs | **Returns:** `boolean, error` #### `metrics.histogram` Record a histogram observation. ```lua local recorded, err = metrics.histogram("duration_seconds", 0.123, {method = "GET"}) if err then return nil, err end return recorded ``` | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Metric name | | `value` | number | Observed value | | `labels` | table? | Label key-value pairs | **Returns:** `boolean, error` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Collector not available | `errors.INTERNAL` | no | Invalid name or value types raise Lua argument errors instead of returning structured errors. See [Error Handling](lua/core/errors.md) for working with errors. --- # "OS Time" ## OS Time The global `os` table provides timestamps, date formatting, elapsed-time measurement, and time-difference calculations. In a workflow, current-time reads use the workflow's time reference; outside a workflow they use the system clock. This is an API reference. Timestamp literals and formatted outputs are illustrative; current values depend on the runtime or workflow clock and timezone. ### Loading The `os` table is global and does not require loading with `require`. ```lua os.time() os.date() os.clock() os.difftime() ``` ### Getting Timestamps Read a Unix timestamp in seconds since January 1, 1970 UTC: ```lua -- Current timestamp local now = os.time() -- 1718462445 -- Specific date/time local t = os.time({ year = 2024, month = 12, day = 25, hour = 10, min = 30, sec = 0 }) ``` **Signature:** `os.time([spec]) -> number` **Parameters:** | Field | Type | Default | Description | |-------|------|---------|-------------| | `year` | number | current year | Four-digit year (e.g., 2024) | | `month` | number | current month | Month 1-12 | | `day` | number | current day | Day of month 1-31 | | `hour` | number | 0 | Hour 0-23 | | `min` | number | 0 | Minute 0-59 | | `sec` | number | 0 | Second 0-59 | With no arguments, `os.time()` returns the current Unix timestamp. When called with a table, missing fields use the defaults shown above. The `year`, `month`, and `day` fields use the current date when omitted. ```lua -- Just date (time defaults to midnight) os.time({year = 2024, month = 6, day = 15}) -- Partial (fills in current year/month) os.time({day = 1}) -- first of current month ``` ### Formatting Dates Format a timestamp as a string or return its date fields in a table: local now = os.time() -- Default format os.date() -- "Sat Jun 15 14:30:45 2024" -- Custom format os.date("%Y-%m-%d", now) -- "2024-06-15" os.date("%H:%M:%S", now) -- "14:30:45" os.date("%Y-%m-%dT%H:%M:%S", now) -- "2024-06-15T14:30:45" -- UTC time (prefix format with !) os.date("!%Y-%m-%d %H:%M:%S", now) -- UTC instead of local -- Date table local t = os.date("*t", now) **Signature:** `os.date([format], [timestamp]) -> string | table` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `format` | string | `"%c"` | Format string, `"*t"` for table | | `timestamp` | number | current time | Unix timestamp to format | #### Format Specifiers | Code | Output | Example | |------|--------|---------| | `%Y` | 4-digit year | 2024 | | `%y` | 2-digit year | 24 | | `%m` | Month (01-12) | 06 | | `%d` | Day (01-31) | 15 | | `%H` | Hour 24h (00-23) | 14 | | `%I` | Hour 12h (01-12) | 02 | | `%M` | Minute (00-59) | 30 | | `%S` | Second (00-59) | 45 | | `%p` | AM/PM | PM | | `%A` | Weekday name | Saturday | | `%a` | Weekday short | Sat | | `%B` | Month name | June | | `%b` | Month short | Jun | | `%w` | Weekday (0-6, Sunday=0) | 6 | | `%j` | Day of year (001-366) | 167 | | `%U` | ISO 8601 week number (01-53, week starts Monday) | 24 | | `%W` | ISO 8601 week number (01-53, week starts Monday) | 24 | | `%z` | Timezone offset | -0700 | | `%Z` | Timezone name | PDT | | `%c` | Full date/time | Sat Jun 15 14:30:45 2024 | | `%x` | Date only | 06/15/24 | | `%X` | Time only | 14:30:45 | | `%%` | Literal % | % | #### Date Table When the format is `"*t"`, `os.date()` returns a table: ```lua local t = os.date("*t") ``` | Field | Type | Description | Example | |-------|------|-------------|---------| | `year` | number | Four-digit year | 2024 | | `month` | number | Month (1-12) | 6 | | `day` | number | Day of month (1-31) | 15 | | `hour` | number | Hour (0-23) | 14 | | `min` | number | Minute (0-59) | 30 | | `sec` | number | Second (0-59) | 45 | | `wday` | number | Weekday (1-7, Sunday=1) | 7 | | `yday` | number | Day of year (1-366) | 167 | | `isdst` | boolean | `true` when the zone's UTC offset is nonzero in this release; not a reliable DST indicator | false | Use `"!*t"` for UTC date table. ### Measuring Elapsed Time Read the seconds between the current runtime time reference and the OS-time module's initialization time: ```lua local start = os.clock() -- do work for i = 1, 1000000 do end local elapsed = os.clock() - start print(string.format("Took %.3f seconds", elapsed)) ``` **Signature:** `os.clock() -> number` Unlike standard Lua's CPU-time definition, this implementation is based on elapsed time. In workflows, it uses the workflow time reference. ### Time Difference Calculate the difference between two timestamps in seconds: ```lua local t1 = os.time({year = 2024, month = 1, day = 1}) local t2 = os.time({year = 2024, month = 12, day = 31}) local diff = os.difftime(t2, t1) -- t2 - t1 local days = diff / 86400 print(days) -- 365 ``` **Signature:** `os.difftime(t2, t1) -> number` | Parameter | Type | Description | |-----------|------|-------------| | `t2` | number | Later timestamp | | `t1` | number | Earlier timestamp | The result is `t2 - t1` in seconds and is negative when `t1 > t2`. ### Platform Constant The `os.platform` constant identifies the runtime: ```lua os.platform -- "wippy" ``` --- # "TTY" ## TTY Terminal input events, styled output, presentation surfaces, and local virtual viewports. Every function resolves the terminal port attached to the calling process frame. A process on a Terminal Host owns the physical terminal; a process.lua on a regular process.host owns a virtual terminal when it is spawned with a viewport grant. Without either attachment the module returns "no terminal context". ### Loading ```lua local tty = require("tty") ``` ### Model A **Surface** is one process's exclusive presentation lease on its terminal port. It publishes complete row snapshots; the backend owns diffing and terminal recovery. Only one surface may be open on a port at a time. A **Canvas** is an in-process styled-cell composition buffer. It clips at cell boundaries and never emits terminal control commands of its own. A **Viewport** is a local, structured terminal boundary that lets one process host another process's surface without sharing byte streams. The shell decides where viewport content appears and translates input into the child's coordinates; the child sees an ordinary terminal port and does not know whether it is full-screen, tiled, tabbed, or hidden. Viewports are local to one runtime node. Grants and handles are opaque local capabilities, not serializable network references. ### Input Loop Start input delivery, subscribe to events, and process them in a loop: ```lua local tty = require("tty") local io = require("io") local function handler() local events = tty.events() tty.start() while true do local ev, open = events:receive() if not open then break end if ev.type == "key" then if ev.key == "q" or (ev.ctrl and ev.key == "c") then break end local _, print_err = io.print("Key: " .. ev.key) if print_err then loop_err = print_err; break end elseif ev.type == "resize" then local _, print_err = io.print("Size: " .. ev.width .. "x" .. ev.height) if print_err then loop_err = print_err; break end end end local _, stop_err = tty.stop() if loop_err then return nil, loop_err end if stop_err then return nil, stop_err end return started end ``` Call `events()` before `start()` so a consumer is ready when the first events arrive. On a virtual port, `start()` opens viewer-to-producer event delivery and `stop()` closes it: a `Viewport:send()` outside that interval fails instead of silently dropping input. Resize delivery is independent of input state. #### `tty.start()` Start input delivery for the current port. A physical terminal switches to raw mode. ```lua local ok, err = tty.start() ``` **Returns:** `boolean, error` #### `tty.stop()` Stop input delivery and restore the terminal to normal mode. ```lua local ok, err = tty.stop() ``` **Returns:** `boolean, error` #### `tty.events()` Subscribe to the port's terminal events and return a channel. Events are delivered as tables with a `type` field. Subscribe once and reuse the channel. ```lua local events, err = tty.events() ``` **Returns:** `EventChannel, error` `EventChannel` has `receive()` and `case_receive()`, so it composes with `channel.select`. #### tty.screen_size() Read the current terminal dimensions. ```lua local width, height, err = tty.screen_size() ``` **Returns:** `number, number, error` #### `tty.mouse(enable)` Enable or disable mouse event tracking. ```lua local ok, err = tty.mouse(true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `enable` | boolean | `true` to enable, `false` to disable | **Returns:** `boolean, error` ### Surface A surface is the port's presentation lease. Acquire one, publish complete frames, and close it when done. #### tty.surface(options?) ```lua local surface, err = tty.surface({ alternate_screen = true, hide_cursor = true, synchronized_output = true, }) ``` | Option | Type | Default | Description | |--------|------|---------|-------------| | `alternate_screen` | boolean | false | Present on the terminal's alternate screen buffer | | `hide_cursor` | boolean | false | Hide the terminal cursor while the surface is open | | `synchronized_output` | boolean | false | Wrap each frame in synchronized-output markers | **Returns:** `Surface, error` Opening a second surface on a port that already has one fails. A virtual port keeps the options as surface metadata; a physical port translates them into terminal modes and restores them on close. #### surface:present(rows, options?) Publish a complete array of row strings. Row `1` is the top line. ```lua local stats, err = surface:present(rows, { cursor = {x = 12, y = 3, visible = true}, images = { {placement_id = "logo", image = logo, x = 2, y = 2, cols = 20, rows = 8, alt = "Logo"}, }, }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `rows` | string[] | Complete frame, at most 16384 rows | | `options.cursor` | table | `{x, y, visible}` in one-based surface coordinates | | `options.images` | table[] | Complete retained-image placement set for the frame | Omitting `cursor` preserves the last explicit cursor state. All three cursor fields are required when `cursor` is present. **Returns:** `stats, error` — an immutable record with `rows`, `changed_rows`, and `bytes_written`. A physical frame identical to the previous one writes nothing. #### surface:invalidate() Forget backend presentation state without erasing the logical frame. The next `present` commits even when its rows are unchanged. Use it after an outer terminal resize or when another owner may have disturbed physical state. **Returns:** `boolean` #### surface:close() Release the lease. Idempotent: later calls return the first close result. A physical backend restores terminal modes. **Returns:** `boolean, error` #### surface:capabilities() Return `{images = "native" | "kitty" | "pending" | "none"}`. Start terminal input before probing. A physical backend may briefly return `pending` while it queries the terminal; virtual surfaces retain images without probing. **Returns:** `table, error` #### surface:clipboard(text) Write an OSC 52 clipboard request on a physical surface. Text must be valid UTF-8 and at most 65,536 bytes. Success means the terminal output accepted the request; terminal policy may still ignore it. Virtual surfaces return an unsupported error, and the API provides no clipboard read or acknowledgement. **Returns:** `boolean, error` ### Retained Images Import a PNG into bounded runtime storage, then place its handle in a complete surface frame: ```lua local image = assert(tty.image(png_bytes)) local info = image:info() -- id, format, width, height, bytes assert(surface:present(rows, {images = {{ placement_id = "preview", image = image, x = 1, y = 1, cols = 40, rows = 12, src = {x = 0, y = 0, width = info.width, height = info.height}, z = 1, alt = "Preview", }}})) ``` `tty.image()` validates PNG bytes asynchronously. `image:read()` explicitly exports the encoded bytes and `image:close()` releases the reference. Source pixel coordinates are zero-based; destination cell coordinates are one-based. Omitting `images` from a later `present` clears prior placements. Unsupported physical terminals display the placement's `alt` text, while virtual surfaces keep the image resource for viewers. ### Canvas A canvas is a bounded styled-cell buffer used to compose a frame before presenting it. #### tty.canvas(width, height) ```lua local canvas = tty.canvas(width, height) ``` Width is capped at 16384 columns, height at 16384 rows, and the area at 262,144 cells. Out-of-range arguments raise an argument error. **Returns:** `Canvas` Drawing accepts styled text, not terminal commands. SGR colors and OSC 8 links are preserved; erase, cursor-motion, and other control-only output is not emitted. Each placement is clipped independently at cell boundaries with grapheme-width awareness, so a clipped escape sequence cannot leak into neighboring content. #### canvas:clear(fill?) Clear every cell. An optional styled `fill` string is repeated across each row. ```lua canvas:clear() canvas:clear(tty.style():background("#1a1a1a"):render(" ")) ``` **Returns:** `boolean` #### canvas:put(x, y, text, width?) Place one styled row at one-based `x`, `y` and clip it to `width` cells (default: the canvas width). Coordinates may be negative or past the edge; the placement is clipped rather than rejected. A newline ends the row, so use `put_rows` for multi-row content. ```lua canvas:put(3, 1, tty.style():bold():render("Title"), 40) ``` **Returns:** `boolean` #### canvas:put_rows(x, y, rows, width?) Place an array of styled rows starting at `x`, `y`, one row per line downward. Every entry is validated before anything is drawn. ```lua canvas:put_rows(2, 2, child_rows, inner_width) ``` **Returns:** `boolean` #### canvas:rows() Render the complete row array, ready for `surface:present`. **Returns:** `string[]` ### Viewport A viewport is a virtual terminal port. The creating process is its first viewer; the process admitted with its grant is its producer. #### tty.viewport(options?) ```lua local view, err = tty.viewport({ width = 80, height = 24, page = {foreground = "#e0def4", background = "#191724"}, }) ``` | Option | Type | Default | Description | |--------|------|---------|-------------| | `width` | number | 80 | Columns, 1 to 65535 | | `height` | number | 24 | Rows, 1 to 65535 | | `page` | table | none | Opaque `#RRGGBB` foreground and background defaults | The area is capped at 262,144 cells. **Returns:** `Viewport, error` #### tty.attach(handle) Add another local viewer to an existing viewport. A handle grants viewing, never presentation ownership, and is not valid on another node. ```lua local view, err = tty.attach(handle) ``` **Returns:** `Viewport, error` #### viewport:grant() Return the one-shot producer capability. Pass it as the `terminal` spawn option: ```lua local grant = assert(view:grant()) local child = assert(process.with_options({terminal = grant}) :spawn_monitored("app:child", "app:workers")) ``` Admission consumes the grant transactionally: a rejected start restores an unresolved grant, while a process that has resolved the port consumes it permanently. A host that does not support terminal attachments rejects the spawn instead of dropping the option. See [Processes](lua/core/process.md#spawner-with-options). **Returns:** `string, error` #### viewport:handle() Return the local viewer handle for `tty.attach`. **Returns:** `string` #### viewport:snapshot(after_revision?) Read the current dimensions, rows, cursor, and revision. With `after_revision`, return `nil` when the revision is unchanged. ```lua local frame = view:snapshot(revision) if frame then revision = frame.revision canvas:put_rows(2, 2, frame.rows, inner_width) end ``` **Returns:** `snapshot` or `nil` | Field | Type | Description | |-------|------|-------------| | `revision` | number | Monotonic revision of this frame | | `width` | number | Viewport columns | | `height` | number | Viewport rows | | `rows` | string[] | Rows last published by the producer | | `cursor` | table | `{x, y, visible}` in one-based coordinates, absent until the producer publishes explicit cursor state | | `images` | table[] | Retained image placement metadata | | `layers` | table[] | Ordered presentation layers | | `images_omitted` | boolean | Image resources exist but are not retained by this plain snapshot | A page resolves terminal-default cells and omitted rows to explicit colors. The creator can change it with `viewport:set_page(page)`; passing `nil` restores the producer's original rows. Page changes advance the revision without requiring the producer to repaint. #### viewport:updates() Return a channel of coalesced revision watermarks. `receive()` yields the revision number; `case_receive()` composes with `channel.select`. ```lua local updates = assert(view:updates()) ``` Updates are bounded hints, not an event log. A slow viewer receives only the newest watermark and must call `snapshot()` for state. Presentation and resize never block on a slow viewer. **Returns:** `ViewportUpdateChannel, error` #### viewport:send(event) Forward a validated event record to the producer. The producer must have called `tty.start()`; otherwise the call fails rather than dropping the event. ```lua assert(view:send(event)) assert(view:send({type = "close"})) ``` **Returns:** `boolean, error` #### viewport:resize(width, height) Update the viewport geometry. When the size changes, viewers get a new revision and the producer receives a `resize` event. **Returns:** `boolean, error` #### viewport:close() Detach this viewer only. Closing the last viewer does not kill a live producer, and closing the producer's port does not destroy state while viewers remain. **Returns:** `boolean, error` #### viewport:mount(recipient_pid, rights) Issue a process-bound reference for a local or remote viewer. Rights are independent and default to false: ```lua local observation = assert(view:mount(agent_pid, {observe = true})) local control = assert(view:mount(agent_pid, {input = true, resize = true})) -- In the exact recipient process, on this node or an authenticated mesh peer: local observer = assert(tty.attach(observation)) local controller = assert(tty.attach(control)) ``` A mount is bound to the recipient's full PID and can be redeemed only once. Mounted viewers cannot create producer grants or delegate further mounts. Remote mounts use a renewable lease; reconnecting requires a fresh mount and cannot replay terminal input. Use `viewport:revoke(reference)` to revoke an issued mount. Closing the owner viewport or ending the owner process revokes its mounts. #### viewport:capture() Atomically pin a viewport revision and its retained image resources: ```lua local capture = assert(view:capture()) local snapshot = capture:snapshot() local image = assert(capture:image(snapshot.images[1].image_id)) assert(capture:close()) ``` A plain `snapshot()` does not retain image bytes. A capture does until closed; image handles already acquired from it remain independently owned. ### Event Types Events are tables with a `type` field that determines which other fields are present. Coordinates are one-based. The same records are accepted by `viewport:send()`. #### Key Event ```lua { type = "key", key = "a", -- printable character or key name key_type = "runes", -- "runes" for printable, or special key name action = "press", -- "press" or "release" alt = false, ctrl = false, shift = false } ``` #### Mouse Event Requires `tty.mouse(true)`. ```lua { type = "mouse", action = "press", -- "press", "release", "motion", "wheel" button = "left", -- button name x = 10, y = 5, alt = false, ctrl = false, shift = false } ``` #### Resize Event ```lua {type = "resize", width = 120, height = 40} ``` #### Start Event Emitted once after `tty.start()` with initial dimensions. ```lua {type = "start", width = 120, height = 40} ``` #### Focus Event Reports keyboard ownership. ```lua {type = "focus", focused = true} ``` #### Visibility Event Reports whether repainting is useful. It does not prescribe application lifecycle or background computation. ```lua {type = "visibility", visible = true} ``` #### Paste Event ```lua {type = "paste", text = "pasted content"} ``` #### Close Event Asks the producer to shut down. A shell sends it through `viewport:send` to request a graceful child exit. ```lua {type = "close"} ``` ### Key Bindings Create reusable key bindings that match against key events: ```lua local quit = tty.bind({ keys = {"q", "ctrl+c"}, help = {key = "q/ctrl+c", desc = "quit"} }) -- In event loop if quit:matches(ev) then break end ``` #### `tty.bind(config)` | Field | Type | Description | |-------|------|-------------| | `keys` | string[] | Required. Key patterns to match (e.g. `"a"`, `"ctrl+c"`, `"enter"`) | | `help` | table | Optional. `{key = "...", desc = "..."}` for help text | **Returns:** `KeyBinding` The type schema requires `keys`. At runtime, an omitted or empty `keys` table creates a binding that never matches. #### KeyBinding Methods | Method | Returns | Description | |--------|---------|-------------| | `matches(event)` | boolean | Test if a key event matches this binding | | `set_enabled(bool)` | self | Enable or disable the binding | | `is_enabled()` | boolean | Check if the binding is enabled | | `help()` | table | Returns `{key, desc}` help info | ### Styles Create styled terminal output. Style values are immutable, so each style method returns a new value. ```lua local tty = require("tty") local io = require("io") local title = tty.style() :bold() :foreground("#FF0000") :padding(0, 1) local box = tty.style() :border(tty.borders.ROUNDED) :border_foreground("#00FF00") :width(40) :padding(1, 2) local _, print_err = io.print(box:render(title:render("Hello"), "World")) if print_err then return nil, print_err end ``` #### `tty.style()` Create an empty style. **Returns:** `Style` #### Style Methods All methods return a new `Style` and can be chained. ##### Text Decoration | Method | Parameter | Description | |--------|-----------|-------------| | `foreground(color)` | string | Text color (hex `"#FF0000"`, ANSI `"9"`, or name) | | `background(color)` | string | Background color | | `bold(enable?)` | boolean | Bold text (default: true) | | `italic(enable?)` | boolean | Italic text | | `underline(enable?)` | boolean | Underline text | | `strikethrough(enable?)` | boolean | Strikethrough text | | `faint(enable?)` | boolean | Dimmed text | | `blink(enable?)` | boolean | Blinking text | | `reverse(enable?)` | boolean | Swap foreground/background | ##### Layout | Method | Parameter | Description | |--------|-----------|-------------| | `width(n)` | number | Fixed width | | `height(n)` | number | Fixed height | | `max_width(n)` | number | Maximum width | | `max_height(n)` | number | Maximum height | | `padding(...)` | numbers | Padding (CSS-style: top, right, bottom, left) | | `margin(...)` | numbers | Margin (CSS-style) | | `align(pos)` | number | Horizontal alignment | | `align_vertical(pos)` | number | Vertical alignment | | `inline(enable?)` | boolean | Inline rendering mode | ##### Borders | Method | Parameter | Description | |--------|-----------|-------------| | `border(name, ...)` | string, booleans | Border style, optional per-side toggles | | `border_foreground(...)` | strings | Border color(s) | | `border_background(...)` | strings | Border background color(s) | ##### Other | Method | Description | |--------|-------------| | `render(...)` | Render strings with this style applied | | `copy()` | Create a copy of this style | #### Border Constants ```lua tty.borders.NORMAL tty.borders.ROUNDED tty.borders.THICK tty.borders.DOUBLE tty.borders.HIDDEN ``` #### Alignment Constants ```lua tty.align.LEFT -- 0 tty.align.CENTER -- 0.5 tty.align.RIGHT -- 1 ``` ### Text Utilities The `tty.text` subtable provides layout and measurement functions for styled text. #### Measurement ```lua local w = tty.text.width("hello") -- printable width (ANSI-aware) local h = tty.text.height("a\nb\nc") -- line count local w, h = tty.text.size("hello\nworld") -- both ``` #### Clipping ```lua -- Truncate to a printable width, with an optional tail local head = tty.text.truncate(line, 40) local head = tty.text.truncate(line, 40, "…") -- Take the printable cell range [left, right) local middle = tty.text.cut(line, 10, 30) ``` Both preserve ANSI state and grapheme boundaries, so styled text can be clipped and spliced without breaking escape sequences. `truncate` returns an empty string for a width of zero or less; `cut` returns an empty string when `right` is not greater than `left`. #### Joining ```lua -- Join side by side, aligned at top local row = tty.text.join_horizontal(tty.text.position.TOP, left, right) -- Stack vertically, centered local col = tty.text.join_vertical(tty.text.position.CENTER, top, bottom) ``` #### Max Dimensions ```lua local w = tty.text.max_width({"short", "a longer string"}) -- widest local h = tty.text.max_height({"one\ntwo", "single"}) -- tallest ``` #### Placement Place a string within a box with the given dimensions: ```lua -- Center in a 80x24 box local out = tty.text.place(80, 24, tty.text.position.CENTER, tty.text.position.CENTER, content) -- Horizontal only local out = tty.text.place_horizontal(80, tty.text.position.RIGHT, content) -- Vertical only local out = tty.text.place_vertical(24, tty.text.position.BOTTOM, content) ``` #### Position Constants ```lua tty.text.position.TOP -- 0 tty.text.position.LEFT -- 0 tty.text.position.CENTER -- 0.5 tty.text.position.BOTTOM -- 1 tty.text.position.RIGHT -- 1 ``` ### Permissions Access to a physical terminal comes from the process frame. Attaching a producer with `process.with_options({terminal = grant})` requires `process.context` on the spawning side. Delegated viewports additionally check: | Action | Resource | Description | |--------|----------|-------------| | `tty.mount` | Owner viewport handle | Issue a process-bound mount | | `tty.observe` | Owner viewport handle | Read snapshots, updates, and captures | | `tty.input` | Owner viewport handle | Forward input events | | `tty.resize` | Owner viewport handle | Resize the viewport | ### See Also - [Terminal UI](tutorials/tty.md) — build a shell that hosts a child in a viewport - [Terminal I/O](lua/system/io.md) — stdin/stdout/stderr operations - [Terminal Host](system/terminal.md) — Terminal host configuration - [Command Execution](lua/dynamic/exec.md) — PTY processes and terminal sessions - [Processes](lua/core/process.md) — spawn options, monitoring, lifecycle events --- # "Hub" ## Hub The `hub` module reads Wippy Hub modules, versions, dependencies, files, artifacts, and READMEs. It also manages the runtime's Hub credential override and can remove unpinned artifacts from the local cache. This is an API reference. Catalog coordinates are illustrative; artifact, authentication, and cache operations require matching network access, credentials, lock state, and security policies. ### Loading ```lua local hub = require("hub") ``` ### Per-call Options Network-backed catalog and artifact calls accept an optional options table with these common keys: | Key | Type | Description | |-----|------|-------------| | `registry` | string | Registry URL override | | `token` | string | API token override | | `timeout` | duration/number | Request timeout (e.g. `"3m"` or seconds) | Pagination-aware calls also accept `page` and `page_size`. Authentication calls take a registry URL directly. Cache calls and package-handle methods use their own options described below. ### Modules ```lua local result, err = hub.modules.list({ org = "wippy", visibility = "public", type = "library", sort_order = "downloads_desc", page = 1, page_size = 20, }) -- result = { items, total, page, page_size } ``` | Function | Description | |----------|-------------| | `hub.modules.list(opts?)` | List modules with filters | | `hub.modules.search(query, opts?)` | Search by query string | | `hub.modules.get(module, opts?)` | Fetch module by `org/name` or module id | | `hub.modules.readme(module, opts?)` | Fetch README; returns `{content, filename, version}` | #### List/Search Options | Option | Values | |--------|--------| | `organization_id` / `org` | string | | `visibility` | `public`, `private`, `internal` | | `type` | `library`, `application`, `agent`, `plugin` | | `sort_order` | `name_asc`, `name_desc`, `created_desc`, `updated_desc`, `downloads_desc` | | `keywords` (search) | array of strings | | `license` (search) | string | | `include_deprecated` (search) | boolean | #### README ```lua local readme, err = hub.modules.readme("wippy/terminal", { version = "1.2.3" }) if err then return nil, err end print(readme.content) ``` The `version` option accepts either a version string or a table like `{id, version, label}`. ### Versions ```lua local versions, err = hub.versions.list("wippy/terminal", { include_yanked = false, page_size = 50, }) local v, err = hub.versions.get("wippy/terminal", "1.0.0") ``` | Function | Description | |----------|-------------| | `hub.versions.list(module, opts?)` | List versions for a module | | `hub.versions.get(module, version, opts?)` | Fetch a specific version | | `hub.versions.inspect(module, version, opts?)` | Inspect a version's artifact (downloads and reads the bundle) | | `hub.versions.open(module, version, opts?)` | Open a version's artifact as a package handle | #### Package Handle `hub.versions.open` downloads an artifact and returns a handle with the fields `version`, `digest`, and `packed`: ```lua local pkg, err = hub.versions.open("wippy/terminal", "1.2.3") if err then return nil, err end local entries, entries_err = pkg:entries({ kind = "function.lua", -- string or string[], omit for all kinds include_data = false, -- default true }) -- each entry: { id = "ns:name", kind = "...", meta = {...}, data = } local _, close_err = pkg:close() if entries_err then return nil, entries_err end if close_err then return nil, close_err end return entries ``` | Method | Description | |--------|-------------| | `pkg:metadata()` | Pack metadata map | | `pkg:entries(opts?)` | Registry entries in the artifact; `opts.kind` filters, `opts.include_data` (default true) controls the `data` field | | `pkg:resources()` | Embedded resources list | | `pkg:fs(resource)` | Filesystem handle for an embedded resource | | `pkg:close()` | Release the handle | Entry `data` is returned without resolving `${env:...}` references. ### Local Artifact Cache ```lua local entries, err = hub.cache.list() local removed, err = hub.cache.remove("wippy/terminal", "1.2.3", { force = false, }) local candidates, err = hub.cache.prune({ dry_run = true, }) ``` | Function | Description | |----------|-------------| | `hub.cache.list()` | List cached artifacts as `{module, version, size, pinned}` records | | `hub.cache.remove(module, version, opts?)` | Remove one cached artifact; `opts.force = true` permits removal when the lock file pins it | | `hub.cache.prune(opts?)` | Remove artifacts not referenced by the lock file; `opts.dry_run = true` only reports candidates | `hub.cache.remove` and `hub.cache.prune` delete files from the lock-resolved vendor directory unless their dry-run or pin protections apply. ### Dependencies ```lua local deps, err = hub.dependencies.get("wippy/terminal", "1.0.0") local users, err = hub.dependents.get("wippy/terminal") ``` | Function | Description | |----------|-------------| | `hub.dependencies.get(module, version?, opts?)` | Dependencies for a module version | | `hub.dependents.get(module, opts?)` | Modules that depend on this one | ### Files ```lua local files, err = hub.files.list("wippy/terminal", "1.0.0") ``` | Function | Description | |----------|-------------| | `hub.files.list(module, version, opts?)` | List files for a version (`version` required); returns `{items, total, page, page_size}` | ### Cache ```lua local cached, err = hub.cache.list() -- each entry: { module, version, size, pinned } local ok, err = hub.cache.remove("wippy/terminal", "1.2.3", { force = true }) local pruned, err = hub.cache.prune({ dry_run = true }) ``` | Function | Description | |----------|-------------| | `hub.cache.list(opts?)` | List cached artifacts under the vendor directory; `pinned` is `true` when the lock file references the artifact | | `hub.cache.remove(module, version, opts?)` | Remove one cached artifact; refuses a lock-pinned artifact (kind `errors.CONFLICT`) unless `opts.force` is `true`; returns `true` | | `hub.cache.prune(opts?)` | Remove every cached artifact the lock file does not reference; with `opts.dry_run = true` nothing is deleted; returns the pruned (or would-be-pruned) entries | **Permissions:** `hub.cache.list`, `hub.cache.remove` (resource: module name), `hub.cache.prune` ### Authentication Install a registry token as a runtime override. Hub consumers use it on subsequent calls without requiring a restart: ```lua local status, err = hub.auth.authenticate("wpy_xxx") -- default registry local status, err = hub.auth.authenticate("wpy_xxx", "https://hub.example.com") local status, err = hub.auth.status() local ok, err = hub.auth.logout() ``` The token strings above are placeholders. Load real credentials from a secret-backed environment entry or another protected source; do not commit them in Lua or registry YAML. | Function | Description | |----------|-------------| | `hub.auth.authenticate(token, registry?)` | Validate the token against the registry and, on success, install it as the runtime override | | `hub.auth.status(registry?)` | Live-validate the current credential | | `hub.auth.logout(registry?)` | Clear the runtime token override | `status` contains `authenticated`, `registry`, and `orgs`. Identity fields (`username`, `user_id`, `scope`, `expires_at`, `expired`) are present only when authenticated. A token that fails validation is not stored; `authenticate` returns `authenticated = false`. The runtime override takes precedence over `WIPPY_TOKEN` and stored credentials. ### Permissions Each top-level `hub.*` operation checks the matching action name, such as `hub.modules.list`, `hub.versions.open`, `hub.dependencies.get`, `hub.files.list`, `hub.auth.status`, or `hub.cache.prune`. Actions that address a module use the supplied module reference as the security resource; authentication actions use the registry URL. Package-handle methods do not perform another permission check after the authorized `hub.versions.open` call. ### See Also - [CLI Reference](guides/cli.md) — `wippy readme`, `wippy search`, `wippy publish` - [Publishing Guide](guides/publishing.md) --- # "Text Processing" ## Text Processing The `text` module provides regular expressions, text comparison and patching, and document splitting. This page is an API reference. Its short blocks are isolated calls; longer splitting blocks are partial recipes whose documents, configured filesystem resources, and downstream processing belong to the surrounding application. ### Loading ```lua local text = require("text") ``` #### `text.regexp.compile` Compile an RE2-compatible regular expression. ```lua local re, err = text.regexp.compile("[0-9]+") if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `pattern` | string | RE2 compatible regex pattern | **Returns:** `Regexp, error` #### `re:match_string` Match a string against the compiled expression. ```lua local ok = re:match_string("abc123") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to match | **Returns:** `boolean` #### `re:find_string` Find the first matching substring. ```lua local match = re:find_string("abc123def") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to search | **Returns:** `string | nil` At this runtime pin, an empty-string match is also represented as `nil`; use a pattern that consumes at least one character when an empty match must be distinguished from no match. #### `re:find_all_string` Find all matching substrings. ```lua local matches = re:find_all_string("a1b2c3") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to search | **Returns:** `string[]` #### `re:find_string_submatch` Find the first match and its capture groups. ```lua local match = re:find_string_submatch("user@example.com") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to search | **Returns:** `string[] | nil` (full match + capture groups) #### `re:find_all_string_submatch` Find all matches and their capture groups. ```lua local matches = re:find_all_string_submatch("a=1 b=2") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to search | **Returns:** `string[][]` #### `re:find_string_index` Find the 1-based bounds of the first match. ```lua local pos = re:find_string_index("abc123") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to search | **Returns:** `table | nil` ({start, end}, 1-based) #### `re:find_all_string_index` Find the bounds of all matches. ```lua local positions = re:find_all_string_index("a1b2c3") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to search | **Returns:** `table[] | nil` (nil when there are no matches) #### `re:replace_all_string` Replace every matching substring. ```lua local result = re:replace_all_string("a1b2", "X") ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | Input string | | `repl` | string | Replacement string | **Returns:** `string` #### `re:split` Split a string at matches of the compiled expression. ```lua local parts = re:split("a,b,c", -1) ``` | Parameter | Type | Description | |-----------|------|-------------| | `s` | string | String to split | | `n` | integer | Max parts, -1 for all | **Returns:** `string[]` #### `re:num_subexp` Return the number of capturing subexpressions. ```lua local count = re:num_subexp() ``` **Returns:** `number` #### `re:subexp_names` Return the names of capturing subexpressions. ```lua local names = re:subexp_names() ``` **Returns:** `string[]` #### `re:string` Return the compiled pattern string. ```lua local pattern = re:string() ``` **Returns:** `string` ### Text Diffing Compare text versions and generate patches with [go-diff](https://github.com/sergi/go-diff), an implementation of Google's diff-match-patch algorithm. #### `text.diff.new` Create a text differ with default or custom options. ```lua local diff, err = text.diff.new() local diff, err = text.diff.new(options) ``` **Returns:** `Differ, error` ##### Options {id="diff-options"} | Field | Type | Default | Description | |-------|------|---------|-------------| | `diff_timeout` | number | 1.0 | Timeout in seconds | | `diff_edit_cost` | integer | 4 | Cost of an empty edit | | `match_threshold` | number | 0.5 | Match tolerance 0-1 | | `match_distance` | integer | 1000 | Distance to search for match | | `patch_delete_threshold` | number | 0.5 | Delete threshold | | `patch_margin` | integer | 4 | Context margin | #### `diff:compare` Compare two strings and return operations that transform `text1` into `text2`. ```lua local diff, diff_err = text.diff.new() if diff_err then return nil, diff_err end local diffs, err = diff:compare("hello world", "hello there") if err then return nil, err end -- diffs contains: -- {operation = "equal", text = "hello "} -- {operation = "delete", text = "world"} -- {operation = "insert", text = "there"} ``` | Parameter | Type | Description | |-----------|------|-------------| | `text1` | string | Original text | | `text2` | string | Modified text | **Returns:** `table, error` (array of {operation, text}) Operations: `"equal"`, `"delete"`, `"insert"` #### `diff:summarize` Count the unchanged, inserted, and deleted UTF-8 bytes. For non-ASCII text, these totals are not Unicode code-point or grapheme counts. ```lua -- `diffs` is the checked result from diff:compare. local summary = diff:summarize(diffs) -- summary.equals = 6 (bytes unchanged) -- summary.deletions = 5 (bytes removed) -- summary.insertions = 5 (bytes added) ``` | Parameter | Type | Description | |-----------|------|-------------| | `diffs` | table | Diff array from compare | **Returns:** `table` ({insertions, deletions, equals}) #### `diff:pretty_text` Format a diff with ANSI colors for terminal output. ```lua local formatted, err = diff:pretty_text(diffs) if err then return nil, err end print(formatted) ``` | Parameter | Type | Description | |-----------|------|-------------| | `diffs` | table | Diff array from compare | **Returns:** `string, error` #### `diff:pretty_html` Format a diff as HTML with `` and `` elements. ```lua local html, err = diff:pretty_html(diffs) if err then return nil, err end -- `html` is an HTML fragment with equal, deleted, and inserted spans. ``` | Parameter | Type | Description | |-----------|------|-------------| | `diffs` | table | Diff array from compare | **Returns:** `string, error` #### `diff:patch_make` Create patches that transform one string into another. The patches can be serialized and applied later. ```lua local text1 = "The quick brown fox jumps over the lazy dog" local text2 = "The quick red fox jumps over the lazy cat" local patches, err = diff:patch_make(text1, text2) if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `text1` | string | Original text | | `text2` | string | Modified text | **Returns:** `table, error` #### `diff:patch_apply` Apply patches to a string and return the result and whether every patch succeeded. ```lua local result, success = diff:patch_apply(patches, text1) -- result = "The quick red fox jumps over the lazy cat" -- success = true ``` Check `success` before treating `result` as the requested transformation. Pass patch tables produced by `patch_make`; at this runtime pin, malformed serialized patch text inside a hand-built table can be skipped rather than reported separately. | Parameter | Type | Description | |-----------|------|-------------| | `patches` | table | Patches from patch_make | | `text` | string | Text to apply patches to | **Returns:** `string, boolean` ### Text Splitting Split documents into chunks while preserving semantic boundaries. The splitters are based on the [langchaingo](https://github.com/tmc/langchaingo) implementation. #### `text.splitter.recursive` The recursive splitter tries double newlines, single newlines, spaces, and then individual characters. It moves to the next separator when a chunk exceeds the size limit. ```lua local splitter, err = text.splitter.recursive({ chunk_size = 1000, chunk_overlap = 100 }) if err then return nil, err end local long_text = "This is a long text that needs splitting..." local chunks, split_err = splitter:split_text(long_text) if split_err then return nil, split_err end ``` **Returns:** `Splitter, error` ##### Options {id="recursive-splitter-options"} | Field | Type | Default | Description | |-------|------|---------|-------------| | `chunk_size` | integer | 4000 | Max characters per chunk | | `chunk_overlap` | integer | 200 | Characters repeated between adjacent chunks | | `keep_separator` | boolean | false | Keep separators in output | | `separators` | string[] | nil | Custom separator list | #### `text.splitter.markdown` The Markdown splitter can keep headings with their content, preserve code blocks, and group table rows. ```lua local splitter, err = text.splitter.markdown({ chunk_size = 2000, code_blocks = true, heading_hierarchy = true }) if err then return nil, err end local readme = fs.get("app:docs"):readfile("README.md") local chunks, err = splitter:split_text(readme) ``` This partial recipe requires the entry to enable both `text` and `fs`, a configured `app:docs` filesystem resource, and a readable `README.md` within that resource. **Returns:** `Splitter, error` ##### Options {id="markdown-splitter-options"} | Field | Type | Default | Description | |-------|------|---------|-------------| | `chunk_size` | integer | 4000 | Max characters per chunk | | `chunk_overlap` | integer | 200 | Characters repeated between adjacent chunks | | `code_blocks` | boolean | false | Keep code blocks together | | `reference_links` | boolean | false | Preserve reference links | | `heading_hierarchy` | boolean | false | Respect heading levels | | `join_table_rows` | boolean | false | Keep table rows together | | `separators` | string[] | nil | Custom separator list | #### `splitter:split_text` Split a single document into an array of chunks. ```lua local chunks, err = splitter:split_text(document) if err then return nil, err end for i, chunk in ipairs(chunks) do -- Process each chunk (e.g., create embedding, send to LLM) process(chunk) end ``` Here, `splitter` is a successfully created splitter, while `document` and `process` are supplied by the application. | Parameter | Type | Description | |-----------|------|-------------| | `text` | string | Text to split | **Returns:** `string[], error` #### `splitter:split_batch` Split multiple documents while preserving their metadata. One input document can produce several chunks, each with the source document's metadata. ```lua -- Input: pages from a PDF with page numbers local pages = { {content = "First page content...", metadata = {page = 1}}, {content = "Second page content...", metadata = {page = 2}} } local chunks, err = splitter:split_batch(pages) if err then return nil, err end -- Output: each chunk knows which page it came from for _, chunk in ipairs(chunks) do print("Page " .. chunk.metadata.page .. ": " .. chunk.content:sub(1, 50)) end ``` | Parameter | Type | Description | |-----------|------|-------------| | `pages` | table | Array of {content, metadata} | **Returns:** `table, error` (array of {content, metadata}) `split_batch` silently skips an item when the item is not a table, its `content` field is missing, empty, or not a string, or splitting that item fails. It still returns the remaining chunks with a `nil` error. Validate every input item before the call and verify any cardinality requirements in application code; do not treat a successful call as proof that every input was represented. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Invalid pattern syntax | `errors.INVALID` | no | | Internal error | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Template Engine" ## Template Engine The `templates` module renders [Jet](https://github.com/CloudyKit/jet) templates from configured sets. Templates can use inheritance and includes. This page is an API reference with isolated rendering examples, not a standalone template deployment. The registry IDs and template sources must already be configured, and the executable entry must enable `templates` and have `template.get` permission for the requested set. For template set configuration, see [Template Engine](system/template.md). ### Loading ```lua local templates = require("templates") ``` ### `templates.get` Acquire a template set by registry ID: ```lua local set, err = templates.get("app.views:emails") if err then return nil, err end -- Use the set... return set:release() ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Template set registry ID | **Returns:** `Set, error` ### `set:render` Render a template by name with data: ```lua local set, get_err = templates.get("app.views:emails") if get_err then return nil, get_err end local html, err = set:render("welcome", { user = {name = "Alice", email = "alice@example.com"}, activation_url = "https://example.invalid/activate" }) set:release() if err then return nil, err end return html ``` The caller owns every acquired set until `release()` is called. Release it after the final render, including checked error paths; repeated releases are safe. Rendering does not make application-provided values safe for every output context. Keep secrets and one-time URLs out of logs, and apply the escaping or sanitization required where the rendered string is consumed. | Parameter | Type | Description | |-----------|------|-------------| | `name` | string | Template name within the set | | `data` | table | Variables to pass to template (optional) | **Returns:** `string, error` ### Set Method Summary The set handle provides these methods: | Method | Returns | Description | |--------|---------|-------------| | `render(name, data?)` | `string, error` | Render template with data | | `release()` | `boolean` | Release set back to pool | ### Jet Syntax Reference Jet uses `{{ }}` for expressions and control structures and `{* *}` for comments. #### Variables ```html {{ user.name }} {{ user.email }} {{ items[0].price }} ``` #### Conditionals ```html {{ if order.shipped }}

Shipped!

{{ else if order.processing }}

Processing...

{{ else }}

Received.

{{ end }} ``` #### Loops ```html {{ range items }}
  • {{ .name }} - ${{ .price }}
  • {{ end }} {{ range i, item := items }}

    {{ i }}. {{ item.name }}

    {{ end }} ``` #### Inheritance ```html {* Parent: layout.jet *} {{ yield title() }} {{ yield body() }} {* Child: page.jet *} {{ extends "layout" }} {{ block title() }}My Page{{ end }} {{ block body() }}

    Content

    {{ end }} ``` #### Includes ```html {{ include "partials/header" }}
    Content
    {{ include "partials/footer" }} ``` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Empty ID | `errors.INVALID` | no | | Empty template name | `errors.INVALID` | no | | Permission denied | `errors.PERMISSION_DENIED` | no | | Template set missing, unavailable, or wrong resource type | `errors.INTERNAL` | no | | Template not found | `errors.NOT_FOUND` | no | | Render error | `errors.INTERNAL` | no | | Render attempted after the set was released | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Tree-sitter Parsing" ## Tree-sitter Parsing The `treesitter` module parses source code into concrete syntax trees with [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) through the [go-tree-sitter](https://github.com/tree-sitter/go-tree-sitter) bindings. This page is an API reference with partial parsing recipes. Source strings and query patterns are application input, and node-level snippets assume a live tree from an earlier checked parse. Parsers, trees, queries, and cursors are owned resources: close every successfully created handle when its last dependent operation is complete. The resulting syntax trees: - Represent the full structure of source code - Update incrementally as code changes - Are robust to syntax errors (partial parsing) - Support pattern-based queries using S-expressions ### Loading ```lua local treesitter = require("treesitter") ``` The `treesitter` module is optional and is present only in builds that include the `treesitter` build tag. Official Wippy binaries include it. Source builds can use `make build-wippy` or `go build -tags treesitter`; without the tag, `require("treesitter")` is unavailable. ### Supported Languages | Language | Aliases | Root Node | |----------|---------|-----------| | Go | `go`, `golang` | `source_file` | | JavaScript | `js`, `javascript` | `program` | | TypeScript | `ts`, `typescript` | `program` | | TSX | `tsx` | `program` | | Python | `python`, `py` | `module` | | Lua | `lua` | `chunk` | | PHP | `php` | `program` | | C# | `csharp`, `cs`, `c#` | `compilation_unit` | | HTML | `html`, `html5` | `document` | | Markdown | `markdown`, `md` | `document` | | SQL | `sql` | - | ```lua local langs = treesitter.supported_languages() -- {go = true, javascript = true, python = true, ...} ``` #### Parse Code ```lua local code = [[ func hello() { return "Hello!" } ]] local tree, err = treesitter.parse("go", code) if err then return nil, err end local root, root_err = tree:root_node() if root_err then tree:close() return nil, root_err end print(root:kind()) -- "source_file" print(root:child_count()) -- number of top-level declarations tree:close() ``` #### Query Syntax Tree ```lua local code = [[ func hello() {} func world() {} ]] local tree, parse_err = treesitter.parse("go", code) if parse_err then return nil, parse_err end local root, root_err = tree:root_node() if root_err then tree:close() return nil, root_err end -- Find all function names local query, query_err = treesitter.query("go", [[ (function_declaration name: (identifier) @func_name) ]]) if query_err then tree:close() return nil, query_err end local captures, captures_err = query:captures(root, code) if captures_err then query:close() tree:close() return nil, captures_err end for _, capture in ipairs(captures) do print(capture.name, capture.text) end -- "func_name" "hello" -- "func_name" "world" query:close() tree:close() ``` #### Simple Parse Parse source code with a temporary internal parser. ```lua local tree, err = treesitter.parse("go", code) if err then return nil, err end -- Use the tree, then call tree:close(). ``` | Parameter | Type | Description | |-----------|------|-------------| | `language` | string | Language name or alias | | `code` | string | Source code | **Returns:** `Tree, error` #### Reusable Parser Create a reusable parser for repeated parsing or incremental updates. ```lua local parser, parser_err = treesitter.parser() if parser_err then return nil, parser_err end local _, language_err = parser:set_language("go") if language_err then parser:close() return nil, language_err end local tree1, first_err = parser:parse("package main") if first_err then parser:close() return nil, first_err end -- Parse another source with the reusable parser. For an incremental update, -- edit the old tree first as shown in the complete recipe below. local tree2, second_err = parser:parse("package main\nfunc foo() {}") if second_err then tree1:close() parser:close() return nil, second_err end tree2:close() tree1:close() parser:close() ``` **Returns:** `Parser, error` #### Parser Methods | Method | Description | |--------|-------------| | `set_language(lang)` | Set parser language, returns `boolean, error` | | `get_language()` | Get current language name | | `parse(code, old_tree?)` | Parse code, optionally with old tree for incremental parsing | | `set_timeout(duration)` | Set parse timeout (string like `"1s"` or nanoseconds) | | `set_ranges(ranges)` | Set byte ranges to parse | | `reset()` | Reset parser state | | `close()` | Release parser resources | Trees created by a reusable parser and the parser itself are independently owned; close each successful handle. Nodes borrow their tree's storage and must not be used after that tree is closed. Cursors borrow a tree as well, so close them before the tree. Queries own separate native resources and also require `close()`; close a query after its captures or matches are no longer needed. Explicit cleanup is deterministic, while the process resource store is only a fallback for handles left open at process teardown. #### Get Root Node ```lua local tree, err = treesitter.parse("go", "package main") if err then return nil, err end local root, root_err = tree:root_node() if root_err then tree:close() return nil, root_err end print(root:kind()) -- "source_file" local source_text, text_err = root:text() if text_err then tree:close() return nil, text_err end print(source_text) -- "package main" tree:close() ``` #### Tree Methods | Method | Description | |--------|-------------| | `root_node()` | Get root node of tree | | `root_node_with_offset(bytes, point)` | Get root with offset applied | | `language()` | Get tree's language object | | `copy()` | Create deep copy of tree | | `walk()` | Create cursor for traversal | | `edit(edit_table)` | Apply incremental edit | | `changed_ranges(other_tree)` | Get ranges that changed | | `included_ranges()` | Get ranges included during parsing | | `dot_graph()` | Get DOT graph representation | | `close()` | Release tree resources | #### Incremental Editing Apply an edit before reparsing changed source code: ```lua local code = "func main() { x := 1 }" local tree, parse_err = treesitter.parse("go", code) if parse_err then return nil, parse_err end -- Mark edit: changed "1" to "100" at byte 19 local _, edit_err = tree:edit({ start_byte = 19, old_end_byte = 20, new_end_byte = 22, start_row = 0, start_column = 19, old_end_row = 0, old_end_column = 20, new_end_row = 0, new_end_column = 22 }) if edit_err then tree:close() return nil, edit_err end -- Re-parse with edited tree (faster than full parse) local parser, parser_err = treesitter.parser() if parser_err then tree:close() return nil, parser_err end local _, language_err = parser:set_language("go") if language_err then parser:close() tree:close() return nil, language_err end local new_tree, new_tree_err = parser:parse("func main() { x := 100 }", tree) if new_tree_err then parser:close() tree:close() return nil, new_tree_err end new_tree:close() parser:close() tree:close() ``` ### Nodes Nodes represent elements in the syntax tree. In the isolated snippets below, `root`, `node`, and `func_decl` are application-selected nodes borrowed from a still-open tree. #### Node Types ```lua local node = root:child(0) -- Type information print(node:kind()) -- "package_clause" print(node:type()) -- same as kind() print(node:is_named()) -- true for significant nodes print(node:grammar_name()) -- grammar rule name ``` #### Navigation ```lua -- Children local child = node:child(0) -- by index (0-based) local named = node:named_child(0) -- named children only local count = node:child_count() local named_count = node:named_child_count() -- Siblings local next = node:next_sibling() local prev = node:prev_sibling() local next_named = node:next_named_sibling() local prev_named = node:prev_named_sibling() -- Parent local parent = node:parent() -- By field name local name_node = func_decl:child_by_field_name("name") local field = node:field_name_for_child(0) ``` #### Position Information ```lua -- Byte offsets local start = node:start_byte() local end_ = node:end_byte() -- Row/column positions (0-based) local start_pt = node:start_point() -- {row = 0, column = 0} local end_pt = node:end_point() -- {row = 0, column = 12} -- Source text local source_text, err = node:text() if err then return nil, err end ``` #### Error Detection ```lua if root:has_error() then -- Tree contains syntax errors end if node:is_error() then -- This specific node is an error end if node:is_missing() then -- Parser inserted this to recover from error end if node:is_extra() then -- Node is an "extra" (e.g. a comment) not required by the grammar end ``` Other node methods: `descendant_count()` and `named_descendant_for_point_range(start_pt, end_pt)`. #### S-Expression ```lua local sexp = node:to_sexp() -- "(source_file (package_clause (package_identifier)))" ``` ### Queries Tree-sitter queries match syntax-tree patterns written as S-expressions. #### Create Query ```lua local query, err = treesitter.query("go", [[ (function_declaration name: (identifier) @func_name parameters: (parameter_list) @params ) ]]) if err then return nil, err end -- The owner calls query:close() after the final query operation. ``` | Parameter | Type | Description | |-----------|------|-------------| | `language` | string | Language name | | `pattern` | string | Query pattern in S-expression syntax | **Returns:** `Query, error` #### Execute Query ```lua -- Get all captures (flattened) local captures, captures_err = query:captures(root, source_code) if captures_err then query:close() tree:close() return nil, captures_err end for _, capture in ipairs(captures) do print(capture.name) -- "func_name" print(capture.text) -- actual text print(capture.index) -- capture index -- capture.node is the Node object end -- Get matches (grouped by pattern) local matches, matches_err = query:matches(root, source_code) if matches_err then query:close() tree:close() return nil, matches_err end for _, match in ipairs(matches) do print(match.id, match.pattern) for _, capture in ipairs(match.captures) do local captured_text, text_err = capture.node:text() if text_err then query:close() tree:close() return nil, text_err end print(capture.name, captured_text) end end query:close() tree:close() ``` Passing userdata of the wrong type instead of a Tree-sitter `Node` returns `nil, error`; passing a primitive or table raises a Lua argument error before that check. `root`, `source_code`, and `query` here must come from a still-open tree and a successfully created query. The snippet uses the owning `tree` handle to close both resources before it returns. #### Query Control ```lua -- Limit query scope query:set_byte_range(0, 1000) query:set_point_range({row = 0, column = 0}, {row = 10, column = 0}) -- Limit matches query:set_match_limit(100) if query:did_exceed_match_limit() then -- More matches exist end -- Timeout (string duration or nanoseconds) query:set_timeout("500ms") query:set_timeout(1000000000) -- 1 second in nanoseconds -- Disable patterns/captures query:disable_pattern(0) query:disable_capture("func_name") ``` #### Query Inspection ```lua local pattern_count = query:pattern_count() local capture_count = query:capture_count() local name = query:capture_name_for_id(0) local id = query:capture_index_for_name("func_name") ``` Further inspection methods: `string_count()`, `start_byte_for_pattern(i)`, `end_byte_for_pattern(i)`, `get_match_limit()`, `get_timeout()`, `is_pattern_rooted(i)`, `is_pattern_non_local(i)`, `is_pattern_guaranteed(i)`, `capture_quantifier(pattern, capture)`, `set_max_start_depth(n)`, `get_property_predicates(i)`, `get_property_settings(i)`, `get_text_predicates(i)`. ### Tree Cursor A tree cursor traverses a tree without creating a node object at every step. #### Basic Traversal ```lua local cursor, err = tree:walk() if err then return nil, err end -- Start at root print(cursor:current_node():kind()) -- "source_file" print(cursor:current_depth()) -- 0 -- Navigate if cursor:goto_first_child() then print(cursor:current_node():kind()) print(cursor:current_depth()) -- 1 end if cursor:goto_next_sibling() then -- moved to next sibling end cursor:goto_parent() -- back to parent cursor:close() ``` #### Cursor Methods | Method | Returns | Description | |--------|---------|-------------| | `current_node()` | `Node` | Node at cursor position | | `current_depth()` | `integer` | Depth (0 = root) | | `current_field_name()` | `string?` | Field name if any | | `current_field_id()` | `integer` | Field ID (0 if none) | | `current_descendant_index()` | `integer` | Descendant index of current node | | `goto_parent()` | `boolean` | Move to parent | | `goto_first_child()` | `boolean` | Move to first child | | `goto_last_child()` | `boolean` | Move to last child | | `goto_next_sibling()` | `boolean` | Move to next sibling | | `goto_previous_sibling()` | `boolean` | Move to previous sibling | | `goto_descendant(index)` | - | Move to descendant by index | | `goto_first_child_for_byte(n)` | `integer?` | Move to child containing byte | | `goto_first_child_for_point(pt)` | `integer?` | Move to child containing point | | `reset(node)` | - | Reset cursor to node | | `reset_to(cursor)` | - | Reset cursor to another cursor's position | | `copy()` | `Cursor` | Create copy of cursor | | `close()` | - | Release resources | ### Language Metadata ```lua local lang, err = treesitter.language("go") if err then return nil, err end print(lang:version()) -- ABI version print(lang:node_kind_count()) -- number of node types print(lang:field_count()) -- number of fields print(lang:parse_state_count()) -- number of parse states -- Node kind lookup local kind = lang:node_kind_for_id(1) local id = lang:id_for_node_kind("identifier", true) local is_named = lang:node_kind_is_named(1) -- Field lookup local field_name = lang:field_name_for_id(1) local field_id = lang:field_id_for_name("name") ``` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Language not supported | `errors.INVALID` | no | | Language has no binding | `errors.INVALID` | no | | Invalid query pattern | `errors.INVALID` | no | | Invalid positions | `errors.INVALID` | no | | Parse failed | `errors.INTERNAL` | no | | No execution context | `errors.INTERNAL` | no | Closing an already closed parser, tree, query, or cursor is safe. Calling any other method on a closed handle raises a Lua argument error. See [Error Handling](lua/core/errors.md) for working with errors. ### Query Syntax Reference Tree-sitter queries use S-expression patterns: ``` ; Match a node type (identifier) ; Match with field names (function_declaration name: (identifier)) ; Capture with @name (function_declaration name: (identifier) @func_name) ; Multiple patterns [ (function_declaration) (method_declaration) ] @declaration ; Wildcards (_) ; any node (identifier)+ ; one or more (identifier)* ; zero or more (identifier)? ; optional ; Predicates ((identifier) @var (#match? @var "^_")) ; regex match ``` See [Tree-sitter Query Syntax](https://tree-sitter.github.io/tree-sitter/using-parsers#query-syntax) for complete documentation. --- # "Security & Access Control" ## Security & Access Control The `security` module exposes authentication actors, authorization scopes, policies, and token stores. This page is an API reference with partial authorization recipes. Registry IDs, actors, request metadata, token values, application objects such as `user` and `doc`, and callbacks such as `show_admin_features` come from the surrounding application; the examples are not a complete authentication deployment. Wippy runs in strict security mode by default. The executable entry must enable `security`, have an actor and scope, and authorize the exact operations it calls. In particular, construction and scope changes need `security.actor.create` or `security.scope.create`; registry lookup needs `security.policy.get` or `security.policy_group.get`; token work needs `security.token_store.get` plus the operation-specific token permission. `new_actor`, `new_scope`, `scope:with`, `scope:without`, and permission-denied `token_store` acquisition raise a Lua error instead of returning a structured `error`. Grant these prerequisites in the entry's security context rather than trying to recover after a denial. See [Security Model](system/security.md) for configuration. ### Loading ```lua local security = require("security") ``` ### `actor` Return the current security actor from the execution context. ```lua local actor = security.actor() if actor then local id = actor:id() local meta = actor:meta() -- Use only the fields required for authorization or application logic. local role = meta.role end ``` Actor metadata can contain identifiers or personal data. Do not log the complete metadata table or copy secrets into it. **Returns:** `Actor|nil` ### `scope` Return the current security scope from the execution context. ```lua local scope = security.scope() if scope then local policies = scope:policies() for _, policy in ipairs(policies) do print("Active policy:", policy:id()) end end ``` **Returns:** `Scope|nil` ### `can` Check whether the current context allows an action on a resource. ```lua -- Check read permission if not security.can("read", "user:" .. user_id) then return nil, errors.new("Cannot read user data"):kind(errors.PERMISSION_DENIED) end -- Check write permission if not security.can("write", "order:" .. order_id) then return nil, errors.new("Cannot modify order"):kind(errors.PERMISSION_DENIED) end -- Check with metadata local allowed = security.can("delete", "document:" .. doc_id, { owner_id = doc.owner_id, department = doc.department }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `action` | string | Action to check | | `resource` | string | Resource identifier | | `meta` | table | Additional metadata (optional) | **Returns:** `boolean` ### `new_actor` Create an actor with an ID and metadata. ```lua -- Create user actor local actor = security.new_actor("user:" .. user.id, { role = user.role, department = user.department, email = user.email }) -- Create service actor local service_actor = security.new_actor("service:payment-processor", { type = "service", version = "1.0.0" }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Unique actor identifier | | `meta` | table | Metadata key-value pairs | **Returns:** `Actor` ### `new_scope` Create a custom scope. ```lua -- Empty scope local scope = security.new_scope() -- Scope with policies local read_policy, read_err = security.policy("app:read-only") if read_err then return nil, read_err end local scope = security.new_scope({read_policy}) -- Build scope incrementally local scope = security.new_scope() local policy1, policy1_err = security.policy("app:read") if policy1_err then return nil, policy1_err end local policy2, policy2_err = security.policy("app:write") if policy2_err then return nil, policy2_err end scope = scope:with(policy1):with(policy2) ``` Each alternative above is an isolated construction pattern. `new_scope` and `scope:with` can raise on missing context or permission denial; they do not return `nil, error` for those checks. **Returns:** `Scope` ### `policy` Retrieve a policy from the registry. ```lua local policy, err = security.policy("app:admin-access") if err then return nil, err end -- Evaluate policy local result = policy:evaluate(actor, "delete", "user:123") if result == "allow" then -- permitted elseif result == "deny" then -- forbidden else -- undefined, check other policies end ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Policy ID "namespace:name" | **Returns:** `Policy, error` ### `named_scope` Retrieve a predefined policy group. ```lua -- Get admin scope local admin_scope, err = security.named_scope("app:admin") if err then return nil, err end -- Use for elevated operations local result = admin_scope:evaluate(actor, "delete", "user:123") ``` Loading a scope does not elevate the current execution context. It produces a value for explicit evaluation or for an API that accepts a scope; the caller still needs permission to perform the protected operation. | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Policy group ID | **Returns:** `Scope, error` ### `token_store` Acquire a token store for managing authentication tokens. ```lua local store, err = security.token_store("app:tokens") if err then return nil, err end -- Use store... return store:close() ``` The caller owns an acquired token store until `close()` is called. Close it after the final operation on every checked success or error path; repeated closes are safe. A permission denial during acquisition raises a Lua error, while lookup and resource failures return `nil, error`. | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Token store ID "namespace:name" | **Returns:** `TokenStore, error` ### `Actor` Methods | Method | Returns | Description | |--------|---------|-------------| | `actor:id()` | string | Actor identifier | | `actor:meta()` | table | Actor metadata | #### `with` / `without` Add or remove policies from scope. ```lua local scope = security.new_scope() -- Add policy local write_policy, err = security.policy("app:write") if err then return nil, err end scope = scope:with(write_policy) -- Remove policy scope = scope:without("app:read-only") ``` `with` and `without` return new immutable scope values and raise when `security.scope.create` is not allowed for the `with` or `without` resource. #### `evaluate` Evaluate all policies in scope. ```lua local result = scope:evaluate(actor, "read", "document:123") -- "allow", "deny", or "undefined" if result ~= "allow" then return nil, errors.new("Access denied"):kind(errors.PERMISSION_DENIED) end ``` #### `contains` Check whether the scope contains a policy. ```lua if scope:contains("app:admin") then show_admin_features() end ``` #### `policies` Return all policies in the scope. ```lua local policies = scope:policies() for _, policy in ipairs(policies) do print(policy:id()) end ``` **Returns:** `Policy[]` ### `Policy` Methods | Method | Returns | Description | |--------|---------|-------------| | `policy:id()` | string | Policy identifier | | `policy:evaluate(actor, action, resource, meta?)` | string | `"allow"`, `"deny"`, or `"undefined"` | #### `create` Create an authentication token. ```lua local actor = security.new_actor("user:123", {role = "user"}) local scope, scope_err = security.named_scope("app:default") if scope_err then return nil, scope_err end local store, store_err = security.token_store("app:tokens") if store_err then return nil, store_err end local token, err = store:create(actor, scope, { expiration = "24h", -- or milliseconds meta = { login_ip = request_ip, user_agent = user_agent } }) store:close() if err then return nil, err end return token ``` `request_ip` and `user_agent` are application-provided request values. Store only metadata needed for security decisions, apply retention limits, and never log or persist the returned bearer token outside the intended credential store. | Parameter | Type | Description | |-----------|------|-------------| | `actor` | Actor | Actor for the token | | `scope` | Scope | Permissions scope | | `options.expiration` | string/number | Duration string or ms | | `options.meta` | table | Token metadata | **Returns:** `string, error` #### `validate` Validate a token and return its actor and scope. ```lua local actor, scope, err = store:validate(token) store:close() if err then return nil, errors.new("Invalid token"):kind(errors.PERMISSION_DENIED) end ``` Here and below, `store` is a live owned handle and `token` is an untrusted bearer credential supplied by the caller. Do not log the token, including on validation or revocation errors. **Returns:** `Actor, Scope, error` #### `revoke` Invalidate a token. ```lua local ok, err = store:revoke(token) store:close() if err then return nil, err end ``` **Returns:** `boolean, error` #### `close` Release the token store resource. ```lua store:close() ``` **Returns:** `boolean` ### Permissions Security policy evaluation applies to security operations. #### Security Actions | Action | Resource | Description | |--------|----------|-------------| | `security.policy.get` | Policy ID | Access policy definitions | | `security.policy_group.get` | Group ID | Access named scopes | | `security.scope.create` | `custom`, `with`, `without` | Create custom scopes (`new_scope`) and add/remove policies (`scope:with`, `scope:without`) | | `security.actor.create` | Actor ID | Create actors | | `security.token_store.get` | Store ID | Access token stores | | `security.token.validate` | Store ID | Validate tokens | | `security.token.create` | Store ID | Create tokens | | `security.token.revoke` | Store ID | Revoke tokens | See [Security Model](system/security.md) for policy configuration. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | No context | `errors.INTERNAL` | no | | Empty token store ID | `errors.INVALID` | no | | Permission denied (`policy`, `named_scope`, token `create`/`validate`/`revoke`) | `errors.INVALID` | no | | Permission denied (`new_scope`, `new_actor`, `token_store`, `scope:with`/`without`) | raised as a Lua error | no | | Policy not found | `errors.INTERNAL` | no | | Token store not found | `errors.INTERNAL` | no | | Token store closed | `errors.INTERNAL` | no | | Invalid expiration format | `errors.INVALID` | no | | Token validation failed | `errors.INTERNAL` | no | ```lua local store, err = security.token_store("app:tokens") if err then if errors.is(err, errors.INVALID) then print("Invalid request:", err:message()) end return nil, err end store:close() ``` See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Security Model](../../system/security.md) - Actors, policies, scopes configuration - [HTTP Middleware](http/middleware.md) - Endpoint and resource firewall --- # "Encryption & Signing" ## Encryption & Signing The `crypto` module generates random values, computes HMACs, encrypts and decrypts data, encodes and verifies JWTs, and derives keys. In deterministic workflows, random generation and encryption (which creates a random nonce) run as recorded side effects; replay returns the recorded bytes. Other operations, including HMAC, decryption, JWT processing, PBKDF2, and comparison, run directly. This page is an API reference. Each code block is an isolated call, not a complete key-management or authentication system. Names such as `data`, `key`, `aad`, `payload`, and `token` are application-provided values. Load keys and passwords through the application's secret-management boundary; do not hard-code, log, or return them in diagnostics. Before consuming any `value, error` result shown here, propagate or handle the error. ### Loading ```lua local crypto = require("crypto") ``` #### Random Bytes ```lua local bytes, err = crypto.random.bytes(32) ``` | Parameter | Type | Description | |-----------|------|-------------| | `length` | integer | Number of bytes (1 to 1,048,576) | **Returns:** `string, error` #### Random String ```lua local str, err = crypto.random.string(32) local str, err = crypto.random.string(32, "0123456789abcdef") ``` | Parameter | Type | Description | |-----------|------|-------------| | `length` | integer | Output length in bytes (1 to 1,048,576) | | `charset` | string? | ASCII byte alphabet to use (default: alphanumeric) | **Returns:** `string, error` The implementation selects bytes from the supplied alphabet. A non-ASCII alphabet can be split into invalid UTF-8, and modulo selection is exactly uniform only when the alphabet's byte length divides 256. For uniformly random secret material, use `crypto.random.bytes` and encode the result for the required transport format. #### Random UUID ```lua local id, err = crypto.random.uuid() ``` **Returns:** `string, error` #### HMAC-SHA256 ```lua local hex, err = crypto.hmac.sha256(key, data) ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | HMAC key | | `data` | string | Data to authenticate | **Returns:** `string, error` #### HMAC-SHA512 ```lua local hex, err = crypto.hmac.sha512(key, data) ``` | Parameter | Type | Description | |-----------|------|-------------| | `key` | string | HMAC key | | `data` | string | Data to authenticate | **Returns:** `string, error` #### AES-GCM {id="encrypt-aes-gcm"} ```lua local encrypted, err = crypto.encrypt.aes(data, key) local encrypted, err = crypto.encrypt.aes(data, key, aad) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Plaintext to encrypt | | `key` | string | 16, 24, or 32 bytes (AES-128/192/256) | | `aad` | string? | Additional authenticated data | **Returns:** `string, error` (nonce prepended) Both encryption functions generate a nonce and prepend it to the ciphertext. Do not remove or reuse it, and use the same AAD during decryption. Ciphertext is not a secret-free log value: it can expose length and correlation information. #### ChaCha20-Poly1305 {id="encrypt-chacha20"} ```lua local encrypted, err = crypto.encrypt.chacha20(data, key) local encrypted, err = crypto.encrypt.chacha20(data, key, aad) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Plaintext to encrypt | | `key` | string | Must be 32 bytes | | `aad` | string? | Additional authenticated data | **Returns:** `string, error` (nonce prepended) #### AES-GCM {id="decrypt-aes-gcm"} ```lua local plaintext, err = crypto.decrypt.aes(encrypted, key) local plaintext, err = crypto.decrypt.aes(encrypted, key, aad) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Encrypted data from encrypt.aes | | `key` | string | Same key used for encryption | | `aad` | string? | Must match AAD used in encryption | **Returns:** `string, error` #### ChaCha20-Poly1305 {id="decrypt-chacha20"} ```lua local plaintext, err = crypto.decrypt.chacha20(encrypted, key) local plaintext, err = crypto.decrypt.chacha20(encrypted, key, aad) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Encrypted data from encrypt.chacha20 | | `key` | string | Same key used for encryption | | `aad` | string? | Must match AAD used in encryption | **Returns:** `string, error` #### Encode ```lua local token, err = crypto.jwt.encode(payload, secret) local token, err = crypto.jwt.encode(payload, secret, "HS256") local token, err = crypto.jwt.encode(payload, private_key_pem, "RS256") ``` | Parameter | Type | Description | |-----------|------|-------------| | `payload` | table | JWT claims (`_header` for custom header) | | `key` | string | Secret (HMAC) or PEM private key (RSA) | | `alg` | string? | HS256, HS384, HS512, RS256 (default: HS256) | **Returns:** `string, error` Pass only one of the documented algorithm names. At this runtime pin, an unsupported value passed to `encode` falls back to HS256 instead of returning an error. Validate any configurable algorithm before this call, and do not copy untrusted fields into `_header`; in particular, do not let input override reserved JWT headers such as `alg`. #### Verify ```lua local claims, err = crypto.jwt.verify(token, secret) local claims, err = crypto.jwt.verify(token, secret, "HS256", false) local claims, err = crypto.jwt.verify(token, public_key_pem, "RS256") ``` | Parameter | Type | Description | |-----------|------|-------------| | `token` | string | JWT token to verify | | `key` | string | Secret (HMAC) or PEM public key (RSA) | | `alg` | string? | Expected algorithm (default: HS256) | | `require_exp` | boolean? | Require an `exp` claim to be present (default: true); an `exp` that is present is always validated | **Returns:** `table, error` Whenever present, `exp` and `nbf` are validated against the JWT library's current wall clock, not the workflow time reference. Setting `require_exp = false` permits a missing `exp` claim; it does not disable validation of a claim that is present. Do not use either time-dependent result for replay-sensitive workflow control; perform the check in an activity or validate time against an explicitly replay-safe value. Always pass the algorithm expected by the issuer; verification restricts the token to that exact method. Treat returned claims as authenticated data, not automatically authorized application input, and still validate issuer, audience, subject, and application-specific constraints. #### PBKDF2 ```lua local key, err = crypto.pbkdf2(password, salt, iterations, key_length) local key, err = crypto.pbkdf2(password, salt, iterations, key_length, "sha512") ``` | Parameter | Type | Description | |-----------|------|-------------| | `password` | string | Password/passphrase | | `salt` | string | Salt value | | `iterations` | integer | Iteration count (max 10,000,000) | | `key_length` | integer | Desired key length in bytes | | `hash` | string? | sha256 or sha512 (default: sha256) | **Returns:** `string, error` The derived key is raw bytes. Use a fresh random salt for each stored password verifier and store the salt and work-factor parameters alongside the verifier; the salt need not be secret. Do not use a fixed example salt for production password storage. #### Constant-Time Compare ```lua local equal = crypto.constant_time_compare(a, b) ``` | Parameter | Type | Description | |-----------|------|-------------| | `a` | string | First string | | `b` | string | Second string | **Returns:** `boolean` The result is `false` when lengths differ. The underlying constant-time comparison guarantee applies to equal-length inputs, so compare fixed-length digests or other same-length secrets. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Invalid length | `errors.INVALID` | no | | Empty key | `errors.INVALID` | no | | Invalid key size | `errors.INVALID` | no | | Decryption failed | `errors.INTERNAL` | no | | Token expired | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Hash Functions" ## Hash Functions The `hash` module computes cryptographic hashes, HMAC values, PBKDF2-derived keys, and non-cryptographic FNV-1 hashes. This page is an API reference of isolated calls. Literal inputs illustrate successful use; when data, secrets, passwords, or salts come from the application, capture and handle the documented second `error` return before consuming the result. A hash is not encryption and does not conceal low-entropy input. Do not log passwords, HMAC keys, derived keys, or raw secret-dependent digests. Use HMAC-SHA256 or HMAC-SHA512 for new message-authentication designs and PBKDF2 with a unique random salt for password verifiers. ### Loading ```lua local hash = require("hash") ``` #### MD5 MD5 is not collision-resistant. Use it only for compatibility with protocols that require MD5, not for security decisions. ```lua local hex = hash.md5("data") local raw = hash.md5("data", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to hash | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### SHA-1 SHA-1 is not collision-resistant. Use it only for compatibility with protocols that require SHA-1, not for security decisions. ```lua local hex = hash.sha1("data") local raw = hash.sha1("data", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to hash | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### SHA-256 ```lua local hex = hash.sha256("data") local raw = hash.sha256("data", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to hash | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### SHA-512 ```lua local hex = hash.sha512("data") local raw = hash.sha512("data", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to hash | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### HMAC-MD5 Use HMAC-MD5 only for compatibility with a protocol that requires it; prefer HMAC-SHA256 or HMAC-SHA512 for new designs. ```lua local hex = hash.hmac_md5("message", "secret") local raw = hash.hmac_md5("message", "secret", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Message to authenticate | | `secret` | string | Secret key | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### HMAC-SHA1 Use HMAC-SHA1 only for compatibility with a protocol that requires it; prefer HMAC-SHA256 or HMAC-SHA512 for new designs. ```lua local hex = hash.hmac_sha1("message", "secret") local raw = hash.hmac_sha1("message", "secret", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Message to authenticate | | `secret` | string | Secret key | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### HMAC-SHA256 ```lua local hex = hash.hmac_sha256("message", "secret") local raw = hash.hmac_sha256("message", "secret", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Message to authenticate | | `secret` | string | Secret key | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### HMAC-SHA512 ```lua local hex = hash.hmac_sha512("message", "secret") local raw = hash.hmac_sha512("message", "secret", true) ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Message to authenticate | | `secret` | string | Secret key | | `raw` | boolean? | Return raw bytes instead of hex | **Returns:** `string, error` #### FNV-1 32-bit Compute a hash for uses such as hash tables and partitioning. ```lua local n = hash.fnv32("data") ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to hash | **Returns:** `number, error` #### FNV-1 64-bit Compute a wider hash for uses such as hash tables and partitioning, reducing collision probability. ```lua local n = hash.fnv64("data") ``` | Parameter | Type | Description | |-----------|------|-------------| | `data` | string | Data to hash | **Returns:** `number, error` #### PBKDF2 ```lua local key, err = hash.pbkdf2(password, salt, iterations, key_length) local key, err = hash.pbkdf2(password, salt, iterations, key_length, "sha512") ``` | Parameter | Type | Description | |-----------|------|-------------| | `password` | string | Password/passphrase (non-empty) | | `salt` | string | Salt value (non-empty) | | `iterations` | integer | Iteration count (1 to 10,000,000) | | `key_length` | integer | Desired key length in bytes | | `hash` | string? | `sha256` or `sha512` (default: `sha256`) | **Returns:** `string, error` (raw key bytes) ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Input not a string | `errors.INVALID` | no | | Secret not a string (HMAC) | `errors.INVALID` | no | | Empty password/salt, non-positive or excessive iterations, unsupported hash (PBKDF2) | `errors.INVALID` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "UUID Generation" ## UUID Generation The `uuid` module generates, validates, inspects, parses, and formats UUIDs. In deterministic workflows, v1, v4, and v7 generation runs as a recorded side effect and returns the recorded value during replay. Namespace-based v3 and v5 generation is deterministic and runs directly. This page is an API reference of isolated calls. Values such as `namespace`, `name`, `input`, and `id` come from the surrounding application. Capture and handle the second `error` return before consuming generated, parsed, inspected, or formatted results. UUIDs are identifiers, not bearer credentials; do not use any UUID version as an authentication token or secret. ### Loading ```lua local uuid = require("uuid") ``` #### Version 1 Time-based UUID with timestamp and node ID. Version 1 exposes its creation time and node identifier. Avoid it where those details are sensitive; prefer v4 when only an opaque identifier is needed. ```lua local id, err = uuid.v1() ``` **Returns:** `string, error` #### Version 4 Random UUID. ```lua local id, err = uuid.v4() ``` **Returns:** `string, error` #### Version 7 A time-ordered UUID that encodes its creation time for chronological indexing. Do not rely on it as a strictly monotonic sequence, especially for values generated within the same timestamp interval. ```lua local id, err = uuid.v7() ``` **Returns:** `string, error` #### Version 3 Deterministic UUID from namespace and name using MD5. ```lua local id, err = uuid.v3(namespace, name) ``` | Parameter | Type | Description | |-----------|------|-------------| | `namespace` | string | Valid UUID string | | `name` | string | Value to hash | **Returns:** `string, error` #### Version 5 Deterministic UUID from namespace and name using SHA-1. ```lua local NS_URL = "6ba7b811-9dad-11d1-80b4-00c04fd430c8" local id, err = uuid.v5(NS_URL, "https://example.com/resource") if err then return nil, err end ``` | Parameter | Type | Description | |-----------|------|-------------| | `namespace` | string | Valid UUID string | | `name` | string | Value to hash | **Returns:** `string, error` #### `validate` ```lua local valid = uuid.validate(input) ``` | Parameter | Type | Description | |-----------|------|-------------| | `input` | any | Value to check | **Returns:** `boolean, nil`. Non-string and malformed inputs return `false`; validation does not raise a structured error. #### `version` ```lua local ver, err = uuid.version(id) ``` | Parameter | Type | Description | |-----------|------|-------------| | `uuid` | string | Valid UUID string | **Returns:** `integer, error` #### `variant` ```lua local var, err = uuid.variant(id) ``` | Parameter | Type | Description | |-----------|------|-------------| | `uuid` | string | Valid UUID string | **Returns:** `string, error` (RFC4122, Reserved, Microsoft, Future, NCS, or Invalid) #### `parse` ```lua local info, err = uuid.parse(id) ``` | Parameter | Type | Description | |-----------|------|-------------| | `uuid` | string | Valid UUID string | **Returns:** `table, error` Returned table fields: - `version` (integer): UUID version (1, 3, 4, 5, or 7) - `variant` (string): RFC4122, Reserved, Microsoft, Future, NCS, or Invalid - `timestamp` (integer): Unix timestamp (v1 and v7 only) - `node` (string): 6 raw node ID bytes (v1 only) #### `format` ```lua local formatted, err = uuid.format(id, "standard") local formatted, err = uuid.format(id, "simple") local formatted, err = uuid.format(id, "urn") ``` | Parameter | Type | Description | |-----------|------|-------------| | `uuid` | string | Valid UUID string | | `format` | string? | standard (default), simple, or urn | **Returns:** `string, error` ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Invalid input type | `errors.INVALID` | no | | Invalid UUID format | `errors.INVALID` | no | | Unsupported format type | `errors.INVALID` | no | | Generation failed | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "Dynamic Evaluation" ## Dynamic Evaluation Execute code dynamically at runtime with sandboxed environments and controlled module access. ### Two Systems Wippy provides two evaluation systems: | System | Purpose | Use Case | |--------|---------|----------| | `expr` | Expression evaluation | Config, templates, simple calculations | | `eval_runner` | Full Lua execution | Plugins, user scripts, dynamic code | ### expr Module Lightweight expression evaluation using the expr-lang syntax. ```lua local expr = require("expr") local result, err = expr.eval("x + y * 2", {x = 10, y = 5}) -- result = 20 ``` #### Compiling Expressions Compile once, run many times: ```lua local program, err = expr.compile("price * quantity") local total1 = program:run({price = 10, quantity = 5}) local total2 = program:run({price = 20, quantity = 3}) ``` #### Supported Syntax ```lua -- Arithmetic expr.eval("1 + 2 * 3") -- 7 expr.eval("10 / 2 - 1") -- 4 expr.eval("10 % 3") -- 1 -- Comparison expr.eval("x > 5", {x = 10}) -- true expr.eval("x == y", {x = 1, y = 1}) -- true -- Boolean expr.eval("a && b", {a = true, b = false}) -- false expr.eval("a || b", {a = true, b = false}) -- true expr.eval("!a", {a = false}) -- true -- Ternary expr.eval("x > 0 ? 'positive' : 'negative'", {x = 5}) -- Functions expr.eval("max(1, 5, 3)") -- 5 expr.eval("min(1, 5, 3)") -- 1 expr.eval("len([1, 2, 3])") -- 3 -- Arrays expr.eval("[1, 2, 3][0]") -- 1 -- String concatenation expr.eval("'hello' + ' ' + 'world'") ``` ### eval_runner Module Full Lua execution with security controls. ```lua local runner = require("eval_runner") local result, err = runner.run({ source = [[ local function double(x) return x * 2 end return { double = double } ]], method = "double", args = {21} }) -- result = 42 ``` #### Configuration | Parameter | Type | Description | |-----------|------|-------------| | `source` | string | Lua source code (required) | | `method` | string | Function to call in returned table | | `args` | any[] | Arguments passed to function | | `modules` | string[] | Allowed builtin modules | | `imports` | table | Registry entries to import | | `context` | table | Values available as `ctx` | | `allow_classes` | string[] | Additional module classes | | `custom_modules` | table | Custom tables as modules | | `limits` | table | Execution limits for this run | #### Step Limit `limits.max_steps` bounds how long one `runner.run` may execute: ```lua local result, err = runner.run({ source = user_source, method = "main", limits = {max_steps = 500} }) ``` A step is one turn of the eval scheduler: the program advances until it yields or finishes, and each resume consumes one step. Pure computation between yields counts as one step no matter how long it runs, so the limit bounds scheduling turns, not CPU time. When the count exceeds the limit the run stops and returns `errors.INTERNAL` with `eval exceeded maximum step limit`. `max_steps = 0` means unlimited. Omitting `limits` inherits the host default: ```yaml ## .wippy.yaml lua: eval: max_steps: 10000 # default budget for runs without limits.max_steps # 0 = unlimited; a negative value fails boot ``` `limits` applies to `runner.run` only; `runner.compile` accepts no limits. `limits` must be a table containing only `max_steps`, and `max_steps` must be a non-negative integer — anything else returns `errors.INVALID` before the program runs. #### Module Access Whitelist allowed modules: ```lua runner.run({ source = [[ local json = require("json") return json.encode({hello = "world"}) ]], modules = {"json"} }) ``` Modules not in the list cannot be required. #### Registry Imports Import entries from the registry: ```lua runner.run({ source = [[ local data = ... local utils = require("utils") return utils.format(data) ]], imports = { utils = "app.lib:utilities" }, args = {{key = "value"}} }) ``` #### Privileged Imports An import can be granted modules the eval'd code itself cannot see. Use the table form with `id` and `modules`: ```lua runner.run({ source = [[ local pricing = require("pricing") return pricing.quote(...) ]], modules = {"json"}, imports = { pricing = { id = "app.lib:pricing", modules = {"funcs"} } }, }) ``` The `pricing` library executes in its own scoped environment where `funcs` is available; the eval'd source cannot require or reach `funcs` directly. Granting a module to an import requires the caller to hold `eval.module` permission for that module — capabilities cannot be delegated beyond what the caller itself is allowed. #### Custom Modules Inject custom tables: ```lua runner.run({ source = [[ return sdk.version ]], custom_modules = { sdk = {version = "1.0.0", api_key = "xxx"} } }) ``` #### Context Values Pass data accessible as `ctx`: ```lua runner.run({ source = [[ return "Hello, " .. ctx.get("user") ]], context = {user = "Alice"} }) ``` #### Compiling Programs `runner.compile` validates source and reports its entrypoint and modules without running it: ```lua local program, err = runner.compile([[ local function process(x) return x * 2 end return { process = process } ]], "process", {modules = {"json"}}) program:method() -- "process" (string) program:modules() -- {"json"} (string[]) ``` The options table accepts the same `modules` and `imports` fields as `runner.run`, and the same `eval.module` and `eval.import` permission checks apply. The compiled program is informational; execute by calling `runner.run` with the source and method. #### Module Classes Modules are categorized by capability: | Class | Description | Default | |-------|-------------|---------| | `deterministic` | Pure functions | Allowed | | `encoding` | Data encoding | Allowed | | `time` | Time operations | Allowed | | `nondeterministic` | Random, etc. | Allowed | | `process` | Spawn, registry | Blocked | | `storage` | File, database | Blocked | | `network` | HTTP, sockets | Blocked | #### Enabling Blocked Classes ```lua runner.run({ source = [[ local http = require("http_client") return http.get("https://api.example.com") ]], modules = {"http_client"}, allow_classes = {"network"} }) ``` #### Permission Checks The system checks permissions for: - `eval.compile` - Before compilation - `eval.run` - Before execution - `eval.module` - For each module in whitelist, and for each module granted to a privileged import - `eval.import` - For each registry import - `eval.class` - For each allowed class Configure in security policies. ### Compile Cache Compiled programs are cached in an LRU keyed by source, method, modules, and allowed classes — repeated runs of identical code skip recompilation. Imports and context are bound at run time and do not affect the cache key. ```yaml ## .wippy.yaml lua: eval: cache_size: 256 # entries; 0 or less disables caching (default: 256) cache_ttl: 0 # expiry; 0 = no expiry (default: 0) ``` ### Error Handling ```lua local result, err = runner.run({...}) if err then if err:kind() == errors.PERMISSION_DENIED then -- Access denied by security policy elseif err:kind() == errors.INVALID then -- Invalid source or configuration elseif err:kind() == errors.INTERNAL then -- Execution or compilation error end end ``` #### Plugin System ```lua local plugins = registry.find({meta = {type = "plugin"}}) for _, plugin in ipairs(plugins) do local source = plugin:data().source runner.run({ source = source, method = "init", modules = {"json", "time"}, context = {config = app_config} }) end ``` #### Template Evaluation ```lua local template = "Hello, {{name}}! You have {{count}} messages." local compiled = expr.compile("name") -- Fast repeated evaluation for _, user in ipairs(users) do local greeting = compiled:run({name = user.name}) end ``` #### User Scripts ```lua local user_code = request:body() local result, err = runner.run({ source = user_code, modules = {"json", "text"}, -- Safe modules only context = {data = input_data} }) ``` ### See Also - [Expression](lua/dynamic/expression.md) - Expression language reference - [Exec](lua/dynamic/exec.md) - System command execution - [Security](lua/security/security.md) - Security policies --- # "Command Execution" ## Command Execution Execute external commands and shell scripts with full control over I/O streams. For executor configuration, see [Executor](system/exec.md). ### Loading ```lua local exec = require("exec") ``` ### Acquiring an Executor Get a process executor resource by ID: ```lua local executor, err = exec.get("app:exec") if err then return nil, err end -- Use executor local proc = executor:exec("ls -la") -- ... -- Release when done executor:release() ``` | Parameter | Type | Description | |-----------|------|-------------| | `id` | string | Resource ID | **Returns:** `Executor, error` ### Creating a Process Create a new process with the specified command: ```lua -- Simple command local proc, err = executor:exec("echo 'Hello, World!'") -- With working directory local proc = executor:exec("npm install", { work_dir = "/app/project" }) -- With environment variables local proc = executor:exec("python script.py", { work_dir = "/scripts", env = { PYTHONPATH = "/app/lib", DEBUG = "true", API_KEY = api_key } }) -- Run shell script local proc = executor:exec("./deploy.sh production", { work_dir = "/app/scripts", env = { DEPLOY_ENV = "production" } }) ``` | Parameter | Type | Description | |-----------|------|-------------| | `cmd` | string | Executable and literal arguments | | `options.work_dir` | string | Working directory | | `options.env` | table | Environment variables | | `options.pty` | table | Allocate a pseudo-terminal for the child | | `options.process_group` | boolean | Start the child in its own process group so signals also reach descendants; unsupported on Windows | **Returns:** `Process, error` The process is created but not started. #### Command Parsing `cmd` is split into an executable and literal arguments using shell-like quoting: single and double quotes group a word, and a backslash escapes the following character. There is no shell, so no variable expansion, globbing, pipes, or redirection happens. An unclosed quote returns `errors.INVALID`. ```lua -- One argument containing a space, passed literally local proc = executor:exec("grep 'hello world' notes.txt") -- $HOME is passed as the five characters $HOME, not expanded local proc = executor:exec("echo $HOME") ``` To use shell features, invoke a shell explicitly: ```lua local proc = executor:exec("/bin/sh -c 'ls *.log | wc -l'") ``` #### PTY Options Allocating a PTY gives the child a real terminal: line editing, job control, and full-screen programs work as they do in a shell. ```lua local proc = executor:exec("/bin/bash --noprofile --norc", { pty = {width = 100, height = 30, term = "xterm-256color"}, }) ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | `width` | number | 80 | Initial PTY columns, 1 to 65535 | | `height` | number | 24 | Initial PTY rows, 1 to 65535 | | `term` | string | none | Child `TERM` value | Width times height may not exceed 262,144 cells. A PTY-backed process merges the child's output into a single terminal stream; drive it with [resize](#resize) and [attach_terminal](#attach_terminal) rather than the stdin/stdout pipe methods. ### start / wait Start the process and wait for completion. ```lua local proc = executor:exec("./build.sh") local ok, err = proc:start() if err then return nil, err end local exit_code, err = proc:wait() if err then return nil, err end if exit_code ~= 0 then return nil, errors.new({ kind = errors.INTERNAL, message = "Build failed with exit code: " .. exit_code }) end ``` ### stdout_stream / stderr_stream Get streams to read process output. ```lua local proc = executor:exec("./process-data.sh") local stdout = proc:stdout_stream() local stderr = proc:stderr_stream() proc:start() -- Read all stdout local output = {} while true do local chunk = stdout:read(4096) if not chunk then break end table.insert(output, chunk) end local result = table.concat(output) -- Check for errors local err_output = {} while true do local chunk = stderr:read(4096) if not chunk then break end table.insert(err_output, chunk) end local exit_code = proc:wait() stdout:close() stderr:close() if exit_code ~= 0 then return nil, errors.new({ kind = errors.INTERNAL, message = table.concat(err_output) }) end return result ``` ### write_stdin Write data to process stdin. ```lua local proc = executor:exec("head -n 3") local stdout = proc:stdout_stream() proc:start() proc:write_stdin("banana\napple\ncherry\n") local lines = stdout:read() proc:wait() stdout:close() ``` Each call writes the given bytes and returns. Call `close_stdin()` when the child must see EOF: ```lua local proc = assert(executor:exec("sort")) local stdout = assert(proc:stdout_stream()) assert(proc:start()) assert(proc:write_stdin("banana\napple\n")) assert(proc:close_stdin()) local sorted = assert(stdout:read()) ``` `close_stdin()` is idempotent. Later writes fail because the input side is closed. PTY-backed processes do not expose this pipe operation. ### done Use `done()` to observe exit without consuming the process handle: ```lua local proc = assert(executor:exec("./worker")) assert(proc:start()) local exits = assert(proc:done()) local status, open = exits:receive() if open then print(status.code, status.signal, status.error) end ``` The returned channel delivers one exit record and then closes. Repeated calls return the same channel. The record contains `code`, optional `signal`, and an `error` only when the runtime could not observe the exit. A signal exit uses `128 + signal` as its code. Unlike `wait()`, `done()` leaves the handle usable, so streams, `signal()`, and `close()` remain available. `wait()` after delivery returns the recorded code. ### signal / close Send signals or release the process. ```lua local proc = executor:exec("./long-running-server.sh") proc:start() -- ... later, need to stop it ... -- Send SIGTERM and release the handle proc:close() -- Send SIGKILL and release the handle proc:close(true) -- Or send a specific signal and keep the handle local SIGINT = 2 proc:signal(SIGINT) ``` `close(force?)` signals a started child with `SIGTERM`, or `SIGKILL` when `force` is true, then reaps it in the background so the call does not block. A child still running after a grace period is killed so the reap always completes. An unstarted handle is simply invalidated, and closing twice is not an error. When `process_group` is enabled, signals target the group and still reach descendants after the leader exits. Streams acquired before reaping remain readable until their last writer closes, including a descendant that inherited the pipe. After `close()`, process methods report `process closed`; use `done()` when the exit matters and the handle must remain usable. ### resize Resize the PTY of a PTY-backed process. A pipe-backed process returns an error. ```lua local ok, err = proc:resize(120, 40) ``` | Parameter | Type | Description | |-----------|------|-------------| | `width` | number | Columns, 1 to 65535 | | `height` | number | Rows, 1 to 65535 | **Returns:** `boolean, error` Use it to set the initial geometry before handing the process to a terminal session. Once a session owns the process, send it a `resize` event instead. ### attach_terminal Attach an unstarted PTY-backed process to the calling process's terminal and return a `TerminalSession`. ```lua local exec = require("exec") local tty = require("tty") local executor = assert(exec.get("app:exec")) local proc = assert(executor:exec("/bin/bash --noprofile --norc", { pty = {term = "xterm-256color"}, })) local session = assert(proc:attach_terminal()) ``` **Returns:** `TerminalSession, error` The call consumes the process: the session becomes its sole lifecycle owner and the original handle can no longer be used. The session opens a surface on the current terminal port and owns PTY emulation, input encoding, resize, graceful and forced termination, and reaping. It needs a terminal port — a [terminal host](system/terminal.md) process, or a process spawned with a [viewport grant](lua/system/tty.md#viewport) — and fails when the port has no input controller or already has an open surface. #### TerminalSession | Method | Returns | Description | |--------|---------|-------------| | `send(event)` | `boolean, error` | Forward one canonical TTY event to the child | | `done()` | channel | Channel that fires once when the child finishes | | `status()` | `string, error` | `"running"` or `"done"`, with the failure error when it failed | | `close()` | `boolean, error` | Request termination of a running child | `send` accepts the key, mouse, resize, focus, and paste records described in [TTY](lua/system/tty.md#event-types). Sending after the child has finished returns an error. ```lua local channel = require("channel") local events = assert(tty.events()) assert(tty.start()) local done = session:done() while true do local selected = channel.select({ events:case_receive(), done:case_receive(), }) if not selected.ok or selected.channel == done then break end if selected.value.type == "close" then break end assert(session:send(selected.value)) end assert(session:close()) ``` ### Permissions Exec operations are subject to security policy evaluation. | Action | Resource | Description | |--------|----------|-------------| | `exec.get` | Executor ID | Acquire an executor resource | | `exec.run` | Command | Execute a specific command | `exec.run` is evaluated against the raw command string, with the requested options as metadata: | Key | Type | Description | |-----|------|-------------| | `work_dir` | string | Requested working directory, empty when unset | | `env_names` | string[] | Names of the environment variables passed, sorted; values are not exposed | | `pty.requested` | boolean | Whether a PTY was requested | | `pty.width` | number | Resolved PTY columns, present when requested | | `pty.height` | number | Resolved PTY rows, present when requested | | `pty.term` | string | Requested `TERM` value, present when requested | A policy can therefore allow plain commands while restricting the ones that ask for a terminal or a particular working directory. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Invalid ID | `errors.INVALID` | no | | Permission denied | `errors.INVALID` | no | | Process closed | `errors.INVALID` | no | | Process not started | `errors.INVALID` | no | | Already started | `errors.INVALID` | no | | Unclosed quote in command | `errors.INVALID` | no | | No PTY on the process | `errors.INVALID` | no | | Terminal port unavailable | `errors.UNAVAILABLE` | no | See [Error Handling](lua/core/errors.md) for working with errors. ### See Also - [Executor](system/exec.md) — executor configuration - [TTY](lua/system/tty.md) — terminal events, surfaces, and viewports - [Terminal UI](tutorials/tty.md) — a shell that hosts a PTY child in a viewport --- # "Expression Language" ## Expression Language The `expr` module compiles and evaluates [expr-lang](https://expr-lang.org/) expressions for filtering, validation, calculations, and rule evaluation without running Lua source code. This page is the canonical Lua API reference; its examples run inside an existing Wippy Lua process whose entry declares the `expr` module, but they are not standalone Wippy applications. See [Dynamic Evaluation](./eval.md) when choosing between expressions and capability-restricted Lua. ### Loading ```lua local expr = require("expr") ``` ### Caching `expr.eval` keeps an internal LRU cache of compiled expressions (default capacity 1000). The cache is built into the module and requires no configuration. ### Evaluating Expressions Evaluate an expression string and return its result. The function uses the internal cache of compiled expressions: ```lua -- Simple math local result, err = expr.eval("1 + 2 * 3") if err then return nil, err end -- result == 7 -- With variables local total, total_err = expr.eval("price * quantity", { price = 29.99, quantity = 3 }) if total_err then return nil, total_err end -- total == 89.97 -- Ternary operator local label, label_err = expr.eval('score > 90 ? "A" : score > 80 ? "B" : "C"', { score = 85 }) if label_err then return nil, label_err end -- label == "B" ``` | Parameter | Type | Description | |-----------|------|-------------| | `expression` | string | expr-lang syntax expression | | `env` | `any` | Variable environment for the expression (optional; normally a table) | **Returns:** `any, error` ### Compiling Expressions Compile an expression into a reusable `Program` for repeated evaluation: ```lua -- Compile once for repeated use local discount_calc, err = expr.compile("price * (1 - discount_rate)") if err then return nil, err end -- Reuse with different inputs local price1, run_err = discount_calc:run({price = 100, discount_rate = 0.1}) if run_err then return nil, run_err end local price2, second_run_err = discount_calc:run({price = 50, discount_rate = 0.2}) if second_run_err then return nil, second_run_err end -- price1 == 90 and price2 == 40 ``` | Parameter | Type | Description | |-----------|------|-------------| | `expression` | string | expr-lang syntax expression | | `env` | `any` | Type-hint environment for compilation (optional; normally a table) | **Returns:** `Program, error` ### Running Compiled Programs Run a compiled expression with the provided environment: ```lua -- Validation rule local validator, compile_err = expr.compile("len(password) >= 8 and len(password) <= 128") if compile_err then return nil, compile_err end local valid, run_err = validator:run({password = "securepass123"}) if run_err then return nil, run_err end -- valid == true -- Pricing rule local pricer, pricing_compile_err = expr.compile([[ base_price * quantity * (1 - bulk_discount) + shipping ]]) if pricing_compile_err then return nil, pricing_compile_err end local order_total, pricing_run_err = pricer:run({ base_price = 25.00, quantity = 10, bulk_discount = 0.15, shipping = 12.50 }) if pricing_run_err then return nil, pricing_run_err end -- order_total == 225.00 ``` | Parameter | Type | Description | |-----------|------|-------------| | `env` | `any` | Variable environment for the expression (optional; normally a table) | **Returns:** `any, error` ### Built-in Functions Expr-lang includes built-in functions for common operations: ```lua local maximum, max_err = expr.eval("max(1, 5, 3)") if max_err then return nil, max_err end local uppercase, upper_err = expr.eval('upper("hello")') if upper_err then return nil, upper_err end local total, sum_err = expr.eval("sum(values)", {values = {1, 2, 3, 4}}) if sum_err then return nil, sum_err end -- maximum == 5, uppercase == "HELLO", and total == 10 ``` Other built-ins include `min`, `abs`, `ceil`, `floor`, `len`, `lower`, and `trim`. Expr-lang also provides operators such as `contains` for strings and `in` for membership tests. ### Errors | Condition | Kind | Retryable | |-----------|------|-----------| | Expression is empty | `errors.INVALID` | no | | Expression syntax invalid | `errors.INTERNAL` | no | | Expression evaluation fails | `errors.INTERNAL` | no | | Result conversion fails | `errors.INTERNAL` | no | See [Error Handling](lua/core/errors.md) for working with errors. --- # "WebAssembly Runtime" ## WebAssembly Runtime > The WASM runtime is an experimental extension. Configuration is stable, but runtime internals may change between releases. Wippy registers WebAssembly modules alongside Lua code. Function entries join the function registry and run through function pools. Process entries register factories for persistent WASM actors: each PID owns one isolated module instance and bounded mailbox until it exits. Both use the runtime scheduler and security model. **Classification: conceptual overview.** The Lua block contains independent call patterns and assumes the named WASM entries and their WIT contracts are already registered. See the Rust/WASM tutorial for a project with a compiled component. ### Entry Kinds | Kind | Description | |------|-------------| | `function.wat` | Inline WebAssembly Text format function defined in YAML | | `function.wasm` | Precompiled WASM binary loaded from a filesystem entry | | `process.wasm` | Stateful WASM actor with one module instance per PID | ### How It Works 1. WASM modules are declared as registry entries in `_index.yaml` 2. At boot, `function.wat` and `function.wasm` entries are compiled, registered as functions, and placed into their configured function pools 3. Lua calls those function entries through `funcs.call()` 4. `process.wasm` entries register actor factories and are spawned under a process host; each PID keeps its module state across messages 5. Function arguments and return values are mapped between Lua tables and WIT types 6. Supported dispatcher-bridged operations, including clock polling and outgoing HTTP, yield so the scheduler can run other work ### Component Model Wippy supports the WebAssembly Component Model with WIT (WebAssembly Interface Types). Component modules map these types between the host and guest: - Records map to Lua tables with named fields - Lists map to Lua arrays - Results map to `(value, error)` return tuples - Primitives (`s32`, `f64`, `string`, etc.) map directly Raw/core WASM modules are also supported with explicit WIT signatures. ### Calling WASM from Lua Call a WASM function by its registry ID through `funcs.call()`: ```lua local funcs = require("funcs") -- No arguments local result, err = funcs.call("myns:answer_wat") if err then return nil, err end -- With arguments local computed, compute_err = funcs.call("myns:compute", 6, 7) if compute_err then return nil, compute_err end -- With complex data local users = { {id = 1, name = "Alice", tags = {"admin"}, active = true}, {id = 2, name = "Bob", tags = {"user"}, active = false}, } local transformed, err = funcs.call("myns:transform_users", users) if err then return nil, err end ``` ### Security WASM executions inherit the caller's security context by default: - Actor identity is inherited - Scope is inherited - Request context is inherited Host capabilities are opt-in through explicit imports. Each entry declares the host profiles it needs, such as `funcs`, `wippy:actor`, `wasi1`, `wasi:cli`, or `wasi:filesystem`, limiting the module's access surface. Enabling a profile does not bypass runtime security checks on operations such as process messaging, function calls, sockets, or outgoing HTTP. A guest that imports `funcs` can call back into the registry. Each call is policy-checked as `funcs.call` against the target ID, so the reachable set is exactly what the inherited scope already permits. Socket dials are authorized the same way, by the [network service](system/network.md), against the `socket.*` permissions. ### See Also - [Functions](wasm/functions.md) - WASM function entry configuration - [Host Functions](wasm/hosts.md) - Available WASI and Wippy host interfaces - [Processes](wasm/processes.md) - Running WASM as long-lived processes - [Rust/WASM Tutorial](../tutorials/rust-wasm.md) - Build and register a component --- # "WASM Functions" ## WASM Functions Use `function.wat` for inline WebAssembly Text source and `function.wasm` for precompiled binaries. **Classification: function configuration reference.** WAT blocks are small registry examples. Precompiled examples assume an external component build, a filesystem entry, exported methods matching the guest WIT, and a SHA-256 digest calculated from the exact binary. Real-looking sample hashes are illustrative. ### Inline WAT Functions Define a WAT function directly in `_index.yaml`: ```yaml entries: - name: answer kind: function.wat source: | (module (func (export "answer") (result i32) i32.const 42 ) ) wit: | answer: func() -> s32; method: answer pool: type: inline ``` For larger WAT sources, use a file reference: ```yaml - name: answer kind: function.wat source: file://answer.wat wit: | answer: func() -> s32; method: answer pool: type: inline ``` #### WAT Configuration Fields | Field | Required | Description | |-------|----------|-------------| | `source` | Yes | Inline WAT source or `file://` reference | | `method` | Yes | Exported function name to call | | `wit` | No | WIT signature for raw/core modules | | `pool` | No | Worker pool configuration | | `transport` | No | Input/output mapping (default: `payload`) | | `imports` | No | Host imports to enable (e.g., `wasi:cli`, `wasi:io`) | | `wasi` | No | WASI configuration (args, env, mounts) | | `options.limits` | No | Execution limits (`limits` remains a deprecated compatibility spelling) | ### Precompiled WASM Functions Load compiled `.wasm` binaries from a filesystem entry: ```yaml entries: - name: assets kind: fs.directory directory: ./wasm - name: compute kind: function.wasm fs: myns:assets path: /compute.wasm hash: sha256:292b796376f8b4cc360acf2ea6b82d1084871c3607a079f30b446da8e5c984a4 method: compute pool: type: lazy max_size: 4 ``` #### WASM Configuration Fields | Field | Required | Description | |-------|----------|-------------| | `fs` | Yes | Filesystem entry ID containing the binary | | `path` | Yes | Path to `.wasm` file within the filesystem | | `hash` | Yes | SHA-256 hash for integrity verification (`sha256:...`) | | `method` | Yes | Exported function name to call | | `wit` | No | WIT signature for raw/core modules | | `pool` | No | Worker pool configuration | | `transport` | No | Input/output mapping (default: `payload`) | | `imports` | No | Host imports to enable | | `wasi` | No | WASI configuration | | `options.limits` | No | Execution limits (`limits` remains a deprecated compatibility spelling) | ### Worker Pools Each WASM function uses a pool of pre-compiled instances. The pool type controls concurrency and resource usage. | Type | Description | |------|-------------| | `inline` | Mutex-serialized. Synchronous and asyncified calls reuse one warm instance; retained-memory policy or an execution failure can trigger replacement. | | `lazy` | Zero idle workers. Scales on demand up to `max_size`. | | `static` | Fixed number of workers with request queue. | | `adaptive` | Auto-scaling elastic pool. | #### Pool Configuration ```yaml pool: type: static size: 4 # Total pool size workers: 2 # Worker threads buffer: 16 # Request queue buffer (default: workers * 64) ``` ```yaml pool: type: lazy max_size: 8 # Maximum concurrent instances ``` ```yaml pool: type: adaptive max_size: 16 # Upper scaling bound ``` The 100-worker default applies only to the implicitly selected pool (when no `type` is set). When you explicitly set `type: lazy` or `type: adaptive` without `max_size`, the default maximum is 16 workers. #### Worker Classes and Core Affinity Setting `pool.worker_class` routes the function to a dedicated pool of OS-thread-pinned workers instead of the shared pool types above (`type` is ignored when set; conventional name: `wasm`): ```yaml pool: worker_class: wasm workers: 8 # optional; defaults to reserved cores, else min(NumCPU, 4) ``` Core isolation is opted into per runtime in `.wippy.yaml`: ```yaml scheduler: wasm_isolation: enabled: true # default: false reserved_cores: 2 # cores reserved for WASM pools (default: 1) ``` With isolation enabled, the actor scheduler and the pinned WASM pools run on disjoint CPU sets (`sched_setaffinity`, Linux only — other platforms size the pools but do not bind threads). Long-running WASM calls then cannot starve actor scheduling. ### Transports Transports control how input and output are mapped between the runtime and the WASM module. | Transport | Description | |-----------|-------------| | `payload` | Maps runtime payloads directly to WASM call arguments (default) | | `wasi-http` | Maps HTTP request/response context to WASM arguments and results | #### Payload Transport The default transport passes arguments directly. Lua values are transcoded to Go types, then lowered to WIT types: ```yaml - name: compute kind: function.wasm fs: myns:assets path: /compute.wasm hash: sha256:... method: compute pool: type: inline ``` ```lua -- Arguments passed directly as WASM function parameters local result, err = funcs.call("myns:compute", 6, 7) if err then return nil, err end -- result: 42 ``` #### WASI HTTP Transport The `wasi-http` transport maps HTTP requests to WASM and writes results back to the HTTP response. Use this to expose WASM functions as HTTP endpoints: ```yaml - name: greet_wasm kind: function.wasm fs: myns:assets path: /greet.wasm hash: sha256:... method: greet transport: wasi-http pool: type: inline - name: greet_endpoint kind: http.endpoint meta: router: myns:api method: POST path: /api/greet func: greet_wasm ``` ### Execution Limits The `options.limits` block bounds a function's execution time, its warm-worker memory, and the sockets it may open: ```yaml options: limits: max_execution_ms: 5000 max_retained_memory_bytes: 134217728 retained_memory_check_interval: 32 max_open_sockets: 8 socket_timeout_ms: 5000 ``` | Field | Default | Description | |-------|---------|-------------| | `max_execution_ms` | unlimited | Wall-clock budget for one call. When exceeded, the execution is cancelled and an error is returned. | | `max_retained_memory_bytes` | `67108864` (64 MiB) | Post-call recycling trigger. A warm worker whose linear memory exceeds this is retired after the call instead of being reused. An explicit `0` disables retained-memory recycling. | | `retained_memory_check_interval` | `16` with the built-in limit, every call with an explicit limit | Number of calls between post-call memory inspections. | | `max_open_sockets` | `16` | Concurrently open connections per instance for the `socket` host. | | `socket_timeout_ms` | `30000` | Deadline for a `socket` dial and for each send/receive. | Negative values are rejected at boot. The root `limits` and `meta.options.limits` spellings are accepted temporarily with a deprecation warning. Keep `pool` at the entry root; it has not moved under `options`. ### WASI Configuration Configure WASI capabilities for the guest module: ```yaml wasi: args: ["--verbose"] cwd: "/app" env: - id: myns:api_key name: API_KEY required: true - id: myns:debug_mode name: DEBUG mounts: - fs: myns:data_files guest: /data read_only: true - fs: myns:output guest: /output ``` | Field | Description | |-------|-------------| | `args` | Command-line arguments passed to the guest | | `cwd` | Working directory inside the guest (must be absolute) | | `env` | Environment variables mapped from registry env entries | | `mounts` | Filesystem mounts from registry filesystem entries | Environment variables are resolved from the environment registry at call time. Required variables cause an error if not found. Mount paths must be absolute and unique. Each mount maps a runtime filesystem entry to a guest directory path. #### Data Transformation Pipeline ```yaml entries: - name: wasm_binaries kind: fs.directory directory: ./wasm - name: transform_users kind: function.wasm fs: myns:wasm_binaries path: /mapper.wasm hash: sha256:7304fc7d19778605458ae5804dae9a7343dcd3f5fc22bcc9415e98b5047192dd method: transform-users pool: type: lazy max_size: 4 - name: filter_active kind: function.wasm fs: myns:wasm_binaries path: /mapper.wasm hash: sha256:7304fc7d19778605458ae5804dae9a7343dcd3f5fc22bcc9415e98b5047192dd method: filter-active pool: type: lazy max_size: 4 ``` ```lua local funcs = require("funcs") local users = { {id = 1, name = "Alice", tags = {"admin", "dev"}, active = true}, {id = 2, name = "Bob", tags = {"user"}, active = false}, {id = 3, name = "Carol", tags = {"dev"}, active = true}, } -- Transform: adds display field and tag count local transformed, err = funcs.call("myns:transform_users", users) if err then return nil, err end -- Filter: returns only active users local active, filter_err = funcs.call("myns:filter_active", users) if filter_err then return nil, filter_err end ``` #### Async Sleep with WASI Clocks WASM components that import `wasi:clocks`, `wasi:io` and `wasi:poll` can use clocks and polling. The async yield mechanism integrates with the Wippy dispatcher: ```yaml - name: sleep_ms kind: function.wasm fs: myns:wasm_binaries path: /sleep_test.wasm hash: sha256:... method: "test-sleep#sleep-ms" imports: - wasi:io - wasi:poll - wasi:clocks pool: type: inline ``` The `#` separator in the method field references an interface method: `test-sleep#sleep-ms` calls the `sleep-ms` function from the `test-sleep` interface. ### See Also - [Overview](wasm/overview.md) - WebAssembly runtime overview - [Host Functions](wasm/hosts.md) - Available host interfaces - [Processes](wasm/processes.md) - Running WASM as processes - [Entry Kinds](guides/entry-kinds.md) - All registry entry kinds --- # "Host Functions" ## Host Functions WASM modules access runtime capabilities through host function imports. Each import is declared explicitly per entry in the `imports` list. ### Import Types | Import | Namespace | Module kind | Description | |--------|-----------|-------------|-------------| | `wasi:cli` | `wasi:cli/*` | component | Environment, exit, stdin/stdout/stderr, terminal | | `wasi:io` | `wasi:io/error`, `wasi:io/streams` | component | Streams and error handling | | `wasi:poll` | `wasi:io/poll` | component | Async polling / cooperative yielding | | `wasi:clocks` | `wasi:clocks/*` | component | Wall clock and monotonic clock | | `wasi:filesystem` | `wasi:filesystem/*` | component | File system access through mounted directories | | `wasi:random` | `wasi:random/*` | component | Cryptographically secure and insecure random numbers | | `wasi:sockets` | `wasi:sockets/*` | component | TCP/UDP networking and DNS resolution | | `wasi:http` | `wasi:http/*` | component | Outgoing HTTP client requests | | `funcs` | `wippy:runtime/funcs@0.1.0` | component | Calling registry functions from the guest | | `wippy:actor` | `wippy:actor/process@0.1.0` | component | PID identity and bounded actor mailbox messaging | | `wasi1` | `wasi_snapshot_preview1` | core | WASI Preview 1 compatibility imports | | `socket` | `wippy:runtime/socket@0.1.0` | core | Instance-owned outbound TCP through integer-only imports | The eight `wasi:*` profiles, `funcs`, and `wippy:actor` are component-only: declaring one on a core module fails the entry. `wasi1` and `socket` expose core imports. Each profile resolves under its short name, under any of the interface namespaces it provides, and under a versioned namespace. The version suffix is stripped before lookup, so `wasi:io/poll`, `wasi:io/poll@0.2.3` and `wasi:poll` all select the same profile. An import that resolves to no profile fails the entry with `unsupported wasm host import: `; a component-only profile on a core module fails with `wasm host import requires component module: `. Enable imports in your entry configuration: ```yaml - name: my_function kind: function.wasm fs: myns:assets path: /module.wasm hash: sha256:... method: run imports: - wasi:cli - wasi:io - wasi:clocks - wasi:filesystem pool: type: inline ``` Only declare the imports your module actually needs. ### WASI Imports Each `wasi:*` import enables a group of related WASI Preview 2 interfaces. #### wasi:clocks **Interfaces:** `wasi:clocks/wall-clock`, `wasi:clocks/monotonic-clock` Wall clock and monotonic clock for time operations. Monotonic clock integrates with the Wippy dispatcher for async sleep. #### wasi:io **Interfaces:** `wasi:io/error`, `wasi:io/streams` Stream read/write operations and error handling. The `wasi:io/poll` interface is provided separately by the `wasi:poll` import. #### wasi:poll **Interfaces:** `wasi:io/poll` Async polling. The poll interface enables cooperative yielding through the dispatcher. #### wasi:cli **Interfaces:** `wasi:cli/environment`, `wasi:cli/exit`, `wasi:cli/stdin`, `wasi:cli/stdout`, `wasi:cli/stderr`, `wasi:cli/terminal-stdin`, `wasi:cli/terminal-stdout`, `wasi:cli/terminal-stderr` Access to environment variables, process exit codes, and standard I/O streams. Environment variables are mapped from the Wippy environment registry through WASI configuration. #### wasi:filesystem **Interfaces:** `wasi:filesystem/types`, `wasi:filesystem/preopens` File system access through mounted directories. Mounts are configured per-entry and map Wippy filesystem entries to guest paths. ```yaml wasi: mounts: - fs: myns:data guest: /data read_only: true ``` #### wasi:random **Interfaces:** `wasi:random/random`, `wasi:random/insecure`, `wasi:random/insecure-seed` Cryptographically secure and insecure random number generation. #### wasi:sockets **Interfaces:** `wasi:sockets/instance-network`, `wasi:sockets/ip-name-lookup`, `wasi:sockets/tcp`, `wasi:sockets/tcp-create-socket`, `wasi:sockets/udp`, `wasi:sockets/udp-create-socket` TCP and UDP networking with DNS resolution. Socket operations suspend the guest and run through the dispatcher, which performs every dial, bind and lookup on the [network service](system/network.md). #### wasi:http **Interfaces:** `wasi:http/types`, `wasi:http/outgoing-handler` Outgoing HTTP client requests from within WASM modules. Supports request/response types defined by the WASI HTTP specification. ### funcs **Namespace:** `wippy:runtime/funcs@0.1.0` Calls registry functions from a component guest. Two entry points are exposed: ```wit interface funcs { call-string: func(target: string, input: string) -> result; call-bytes: func(target: string, input: list) -> result, string>; } ``` `target` is a registry ID in `namespace:name` form. Every call is policy-checked as `funcs.call` against that target, so a guest can only reach functions the caller's scope already permits. ### wippy:actor **Namespace:** `wippy:actor/process@0.1.0` Provides messaging for a component running as `process.wasm`: ```wit interface process { use wasi:io/poll@0.2.8.{pollable}; type pid = string; record payload { format: string, data: list } record message { %from: pid, topic: string, payloads: list } self: func() -> pid; send: func(target: pid, topic: string, payloads: list) -> result; try-receive: func() -> result, string>; receive: func() -> result; subscribe: func() -> pollable; } ``` `receive` suspends the actor until its mailbox has a message; `try-receive` never waits. `subscribe` lets a guest wait on mailbox readiness together with other pollables. `send` accepts `bytes`, `text`, and `json` payload formats and is policy-checked as `process.send` against the target PID. The profile requires an actor context and is intended for `process.wasm` entries. ### wasi1 **Namespace:** `wasi_snapshot_preview1` Declares that a core module links against WASI Preview 1. The profile also resolves under `preview1` and `wasi-preview1`. It registers no hosts of its own; Preview 1 imports are satisfied by the underlying WASM runtime. ### socket **Namespace:** `wippy:runtime/socket@0.1.0` Outbound TCP for core (non-component) modules. The host exports four integer-only functions, so a guest needs no component tooling to use it: | Function | Signature | Result | |----------|-----------|--------| | `connect` | `(host_ptr: i32, host_len: i32, port: i32, timeout_ms: i32) -> i64` | `status << 32 \| handle` | | `send` | `(handle: i32, buf_ptr: i32, buf_len: i32) -> i64` | `status << 32 \| written` | | `recv` | `(handle: i32, out_ptr: i32, out_cap: i32) -> i64` | `status << 32 \| read` | | `close` | `(handle: i32) -> i32` | `status` | The high 32 bits of the 64-bit result carry the status; the low 32 bits carry the value. | Status | Value | Meaning | |--------|-------|---------| | `OK` | 0 | Operation succeeded | | `Invalid` | 1 | Bad arguments or an out-of-range memory region | | `Denied` | 2 | The network service denied the dial | | `Failed` | 3 | The operation failed | | `UnknownHandle` | 4 | The handle is not an open connection of this instance | | `Limit` | 5 | `max_open_sockets` reached | | `Timeout` | 6 | The dial or the read/write deadline expired | `connect` reads the host name from guest memory; `host_len` must be between 1 and 253 bytes and `port` between 1 and 65535. `timeout_ms` narrows the dial deadline: the effective deadline is the smaller of `timeout_ms` and the entry's `socket_timeout_ms`. `send` and `recv` are bounded by `socket_timeout_ms`. `recv` reports a clean end of stream as `OK` with a read count of 0. Connections are owned by the instance that opened them. A handle is meaningless to another instance, the open-socket count is counted per instance, and every connection is closed when the instance is closed or the warm worker is recycled. ### Network Authorization Neither socket host decides access itself. Every dial, bind and lookup goes through the runtime network service, which checks the `socket.connect`, `socket.listen` and `socket.resolve` permissions, applies the private-IP policy, and routes through an [overlay network](system/network.md) when one is selected. `wasi:sockets` additionally pre-checks `socket.resolve` before a DNS lookup and `socket.listen` before a UDP bind. ### See Also - [Overview](wasm/overview.md) - WebAssembly runtime overview - [Functions](wasm/functions.md) - WASM function configuration - [Processes](wasm/processes.md) - Running WASM as processes - [Network Overlays](system/network.md) - Overlay selection and socket permissions --- # "WASM Processes" ## WASM Processes A `process.wasm` entry creates a persistent, isolated WASM actor under a Wippy process host. One module instance lives for the PID lifetime, keeps its guest state between messages, and participates in spawning, monitoring, messaging, and supervised shutdown. **Classification: process configuration and lifecycle reference.** Binary-backed blocks assume an external component build and application-owned filesystem, process host, environment, and policy entries. Placeholder hashes must be replaced with the exact binary digest. ### Entry Configuration ```yaml entries: - name: wasm_binaries kind: fs.directory directory: ./wasm - name: compute_worker kind: process.wasm fs: myns:wasm_binaries path: /worker.wasm hash: sha256:292b796376f8b4cc360acf2ea6b82d1084871c3607a079f30b446da8e5c984a4 method: run imports: - wippy:actor - wasi:io - wasi:poll options: limits: memory_bytes: 67108864 mailbox: capacity: 128 bytes: 8388608 message_bytes: 1048576 ``` #### Configuration Fields | Field | Required | Description | |-------|----------|-------------| | `fs` | Yes | Filesystem entry ID containing the binary | | `path` | Yes | Path to `.wasm` file within the filesystem | | `hash` | Yes | SHA-256 hash for integrity verification | | `method` | Yes | Exported function name to execute | | `transport` | No | Invocation transport: `payload` (default) or `wasi-http` | | `wit` | No | WIT signature for raw/core modules | | `imports` | No | Host imports to enable | | `wasi` | No | WASI configuration (`args`, `cwd`, `env`, and `mounts`) | | `options` | No | Actor controls: `worker_class`, `limits`, and `mailbox` | `process.wasm` actors own one instance for their whole PID lifetime, so function pooling does not apply. A root `pool` block is rejected. Put actor limits under `options.limits`; the old root `limits` and `meta.options` spellings are accepted temporarily with a deprecation warning. ### Stateful Actors and Messaging Import `wippy:actor` in a component guest to access the current PID and its bounded mailbox. The `wippy:actor/process@0.1.0` interface provides: | Function | Behavior | |----------|----------| | `self()` | Return the current actor PID as a string | | `send(target, topic, payloads)` | Send a policy-checked message to another PID | | `try-receive()` | Return the next message immediately, or `none` | | `receive()` | Suspend until a message is available | | `subscribe()` | Return a `wasi:io/poll` pollable for mailbox readiness | Messages contain the sender PID, a topic, and up to 16 payloads. Payload formats are `bytes`, UTF-8 `text`, and UTF-8 `json`. Sending is authorized as `process.send` against the target PID. Mailbox admission rejects malformed, oversized, and over-capacity messages before the guest receives them. The guest normally exports a long-running `run` function. For example: ```wit package example:worker; world worker { import wippy:actor/process@0.1.0; import wasi:io/poll@0.2.8; export run: func() -> result<_, string>; } ``` Inside `run`, call `receive()` in a loop, update guest state, and use `send()` to reply to `message.from`. Returning from `run` exits the process. ### Actor Controls Configure persistent resource and mailbox budgets under `options`: ```yaml options: worker_class: wasm limits: memory_bytes: 67108864 host_buffer_bytes: 8388608 asyncify_stack_bytes: 65536 max_execution_ms: 0 max_open_sockets: 16 socket_timeout_ms: 30000 mailbox: capacity: 128 bytes: 8388608 message_bytes: 1048576 ``` | Field | Default | Description | |-------|---------|-------------| | `worker_class` | `wasm` | Dedicated scheduler worker class; `wasm` is currently the only supported value | | `limits.memory_bytes` | 64 MiB | Guest linear-memory ceiling; a positive 64 KiB multiple, at most 4 GiB | | `limits.host_buffer_bytes` | unlimited | Accounted resident host-buffer ceiling; `0` disables this byte ceiling | | `limits.asyncify_stack_bytes` | runtime default (64 KiB) | Owned suspension storage for a core module | | `limits.max_execution_ms` | unlimited | Wall-clock lifetime for the actor; `0` means no deadline | | `limits.max_open_sockets` | 16 | Concurrent open sockets owned by the actor | | `limits.socket_timeout_ms` | 30000 | Socket operation timeout in milliseconds | | `mailbox.capacity` | 128 | Maximum queued messages | | `mailbox.bytes` | 8 MiB | Aggregate queued-message budget | | `mailbox.message_bytes` | 1 MiB | Per-message budget, including framing overhead | `mailbox.message_bytes` cannot exceed `mailbox.bytes`. The capacity must also fit the byte budget's minimum 256-byte accounting per queued message. Unknown fields and invalid values fail entry admission. ### CLI Commands Register a WASM process as a named command with `meta.command`: ```yaml - name: greet kind: process.wasm meta: command: name: greet short: Greet someone via WASM fs: myns:wasm_binaries path: /component.wasm hash: sha256:... method: greet ``` Run it with: ```bash wippy run greet ``` List available commands: ```bash wippy run list ``` | Field | Required | Description | |-------|----------|-------------| | `name` | Yes | Command name used with `wippy run ` | | `short` | No | Short description shown in `wippy run list` | | `main` | No | Mark the entry as the default command for a pack or hub module | | `use_case` | No | Entrypoint category; defaults to `run` | | `security` | No | Security context applied only when the trusted terminal launcher starts this command | A `terminal.host` must be present for CLI commands to work; it is the process host that runs the command. ### Process Lifecycle WASM processes follow the Init/Step/Close lifecycle model: 1. **Init** - Call context, method, and input arguments are captured 2. **Step** - The first step instantiates and starts the module. Later steps advance dispatcher-bridged operations; a synchronous execution can complete in the first step. 3. **Close** - Instance resources are released ### Spawning from Lua Spawn a WASM process and monitor it for completion: ```lua local errors = require("errors") -- Spawn with monitoring local pid, err = process.spawn_monitored( "myns:compute_worker", -- entry ID "myns:processes", -- process host 6, 7 -- arguments passed to the WASM function ) if err then return nil, err end -- Wait for the process to complete local events = process.events() while true do local event, open = events:receive() if not open then return nil, errors.new("process event channel closed") end if event.kind == process.event.EXIT and event.from == pid then local result = event.result.value -- return value from the WASM function return result, event.result.error end end ``` ### Async Execution WASM actors yield for host operations that the runtime bridges through the dispatcher, including mailbox receive/send, polling, clocks, sockets, DNS, filesystem streams, and outgoing HTTP. The scheduler suspends the process until the pending operation completes, then resumes the same guest instance: ```yaml - name: http_worker kind: process.wasm fs: myns:wasm_binaries path: /http_worker.wasm hash: sha256:... method: run imports: - wasi:io - wasi:cli - wasi:http wasi: env: - id: myns:api_url name: API_URL required: true ``` The yield/resume mechanism is transparent to an asyncified core module or a component using the supported pollable interfaces. ### WASI Configuration Processes support the same WASI configuration as functions: ```yaml - name: file_processor kind: process.wasm fs: myns:wasm_binaries path: /processor.wasm hash: sha256:... method: process imports: - wasi:cli - wasi:io - wasi:clocks - wasi:filesystem wasi: args: ["--input", "/data/input.csv"] cwd: "/app" env: - id: myns:output_format name: OUTPUT_FORMAT mounts: - fs: myns:input_data guest: /data read_only: true - fs: myns:output_dir guest: /output ``` ### See Also - [Overview](wasm/overview.md) - WebAssembly runtime overview - [Functions](wasm/functions.md) - WASM function configuration - [Host Functions](wasm/hosts.md) - Available host interfaces - [Process Model](concepts/process-model.md) - Process lifecycle - [Supervision](guides/supervision.md) - Process supervision trees --- # "Framework" ## Framework Official framework modules are published through the Wippy Hub under the `wippy` organization. This page is a module-management reference for an existing Wippy project. The commands are runnable from the project root; the YAML and import blocks are independent reference snippets rather than a complete application. ### Adding Framework Modules ```bash wippy add wippy/test wippy install ``` This adds the module to your lock file and downloads it to `.wippy/vendor/`. ### Declaring Dependencies in Source Framework modules can also be declared as dependencies in your `_index.yaml`: ```yaml version: "1.0" namespace: app entries: - name: dependency.test kind: ns.dependency component: wippy/test version: "*" ``` Then resolve and install: ```bash wippy update ``` ### Importing Framework Libraries Once installed, import framework libraries into your entries: ```yaml entries: - name: my_test kind: function.lua meta: type: test suite: my-suite source: file://my_test.lua method: run imports: test: wippy.test:test ``` The import maps `wippy.test:test` (the `test` entry from the `wippy.test` namespace) to the local name `test`, which you then `require("test")` in Lua. ### Available Modules | Module | Description | |--------|-------------| | `wippy/llm` | Unified LLM interface with generation, streaming, tool calling, structured output | | `wippy/agent` | Agent framework with tools, delegates, traits, and memory | | `wippy/embeddings` | Vector embeddings storage and similarity search | | `wippy/test` | BDD-style testing framework with assertions and mocking | | `wippy/dataflow` | Workflow orchestration with DAG-based node execution | | `wippy/relay` | WebSocket relay with per-user hubs and plugin routing | | `wippy/views` | Virtual page/component system with template rendering | | `wippy/facade` | Frontend host configuration, theming, and config endpoint | | `wippy/terminal` | Terminal UI components | | `wippy/migration` | Database schema migrations | | `wippy/security` | Actor scopes, policy bundles, and security helpers | | `wippy/usage` | Token and cost usage accounting for LLM calls | Search the Hub for the current module catalog: ```bash wippy search wippy ``` ### See Also - [Dependency Management](guides/dependency-management.md) — Lock files and version constraints - [Publishing](guides/publishing.md) — Publish a module - [CLI Reference](guides/cli.md) — Module-management commands --- # "LLM" ## LLM The `wippy/llm` module provides one interface for language models from OpenAI, Anthropic, Google, and local providers. It supports text generation, tool calling, structured output, embeddings, and streaming. This page is an API primer with composable reference snippets, not a standalone tutorial. The snippets assume an existing Wippy project, a registered model and provider, and any credentials required by that provider. Replace example model names with one your registry exposes; remote generation and embedding calls may incur provider charges. For a complete runnable project, follow [Build an LLM Agent](tutorials/llm-agent.md). ### Setup Add the module to your project: ```bash wippy add wippy/llm wippy install ``` Declare the dependency in your `_index.yaml`: ```yaml version: "1.0" namespace: app entries: - name: dep.llm kind: ns.dependency component: wippy/llm version: "*" ``` The module supplies an OS environment storage and defaults its background process host to `wippy.terminal:host`. Override the `env_storage` or `process_host` dependency parameter only when the application needs a different entry. Set provider API keys through variables such as `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`. ### Text Generation Import the `llm` library into your entry and call `generate()`: ```yaml entries: - name: ask kind: function.lua source: file://ask.lua method: handler imports: llm: wippy.llm:llm ``` ```lua local llm = require("llm") local function handler() local response, err = llm.generate("What are the three laws of robotics?", { model = "gpt-4o" }) if err then return nil, err end return response.result end return { handler = handler } ``` The first argument to `generate()` can be a string prompt, a prompt builder, or a table of messages. The second argument is an options table. #### Generate Options | Option | Type | Description | |--------|------|-------------| | `model` | string | Model name or class (required) | | `temperature` | number | Randomness control, 0-2 (provider support may vary) | | `max_tokens` | number | Maximum tokens to generate | | `top_p` | number | Nucleus sampling parameter | | `top_k` | number | Top-k filtering | | `thinking_effort` | number | Thinking depth 0-100 (models with thinking capability) | | `tools` | table | Array of tool definitions | | `tool_choice` | string | `"auto"`, `"none"`, `"any"`, or tool name | | `stream` | table | Streaming config: `{ reply_to, topic, buffer_size }` | | `timeout` | number | Request timeout in seconds (default 600) | #### Response Structure | Field | Type | Description | |-------|------|-------------| | `result` | string | Generated text content | | `tokens` | table | Token usage: `prompt_tokens`, `completion_tokens`, `thinking_tokens`, `total_tokens`, plus optional `cache_read_input_tokens`, `cache_read_tokens`, `cache_creation_input_tokens`, `cache_write_tokens` | | `finish_reason` | string | Why generation stopped: `"stop"`, `"length"`, `"tool_call"`, `"filtered"`, `"error"` | | `tool_calls` | table? | Array of tool calls (if model invoked tools) | | `metadata` | table | Provider-specific metadata | | `usage_record` | table? | Usage tracking record | ### Prompt Builder Use the prompt builder to construct multi-turn conversations and structured messages: ```yaml imports: llm: wippy.llm:llm prompt: wippy.llm:prompt ``` ```lua local llm = require("llm") local prompt = require("prompt") local conversation = prompt.new() conversation:add_system("You are a helpful assistant.") conversation:add_user("What is the capital of France?") local response, err = llm.generate(conversation, { model = "gpt-4o", temperature = 0.7, max_tokens = 500 }) ``` #### Builder Methods | Method | Description | |--------|-------------| | `prompt.new()` | Create empty builder | | `prompt.with_system(content)` | Create builder with system message | | `:add_system(content, meta?)` | Add system message | | `:add_user(content, meta?)` | Add user message | | `:add_assistant(content, meta?)` | Add assistant message | | `:add_developer(content, meta?)` | Add developer message | | `:add_message(role, content_parts, name?, meta?)` | Add message with role and content parts | | `:add_function_call(name, arguments, id?, options?)` | Add tool call from assistant (`arguments` is the raw JSON string) | | `:add_function_result(name, result, id?)` | Add tool execution result | | `:add_cache_marker(id?)` | Mark cache boundary (Claude models) | | `:get_messages()` | Get message array | | `:build()` | Get `{ messages = ... }` table for `llm.generate()` | | `:clone()` | Deep copy the builder | | `:clear()` | Remove all messages | All `add_*` methods return the builder for chaining. #### Multi-Turn Conversations Build up context across turns by appending messages: ```lua local conversation = prompt.new() conversation:add_system("You are a helpful assistant.") -- first turn conversation:add_user("What is Lua?") local r1 = llm.generate(conversation, { model = "gpt-4o" }) conversation:add_assistant(r1.result) -- second turn with full context conversation:add_user("What makes it different from Python?") local r2 = llm.generate(conversation, { model = "gpt-4o" }) ``` #### Multimodal Content Combine text and images in a single message: ```lua local conversation = prompt.new() conversation:add_message(prompt.ROLE.USER, { prompt.text("What's in this image?"), prompt.image("https://example.com/photo.jpg") }) ``` | Function | Description | |----------|-------------| | `prompt.text(content)` | Text content part | | `prompt.image(url, mime_type?)` | Image from URL | | `prompt.image_base64(mime_type, data)` | Base64-encoded image | #### Role Constants | Constant | Value | |----------|-------| | `prompt.ROLE.SYSTEM` | `"system"` | | `prompt.ROLE.USER` | `"user"` | | `prompt.ROLE.ASSISTANT` | `"assistant"` | | `prompt.ROLE.DEVELOPER` | `"developer"` | | `prompt.ROLE.FUNCTION_CALL` | `"function_call"` | | `prompt.ROLE.FUNCTION_RESULT` | `"function_result"` | | `prompt.ROLE.CACHE_MARKER` | `"cache_marker"` | #### Cloning Clone a builder to create independent variations: ```lua local base = prompt.new() base:add_system("You are a helpful assistant.") local conv1 = base:clone() conv1:add_user("What is AI?") local conv2 = base:clone() conv2:add_user("What is ML?") ``` ### Streaming Stream responses through process communication. Streaming requires a `process.lua` entry: ```lua local llm = require("llm") local TOPIC = "llm_stream" local function main() local stream_ch, listen_err = process.listen(TOPIC) if listen_err then return nil, listen_err end local function finish(text, response, err) local ok, cleanup_err = process.unlisten(stream_ch) if not ok then cleanup_err = cleanup_err or "Failed to remove LLM stream listener" if err then return nil, tostring(err) .. "; cleanup failed: " .. tostring(cleanup_err) end return nil, cleanup_err end if err then return nil, err end return text, response end local self_pid, pid_err = process.pid() if pid_err then return finish(nil, nil, pid_err) end local done_ch = channel.new(1) coroutine.spawn(function() local response, err = llm.generate("Write a short story", { model = "gpt-4o", stream = { reply_to = self_pid, topic = TOPIC, }, }) done_ch:send({ response = response, err = err }) end) local full_text = "" local generation_result = nil local stream_done = false local stream_err = nil while true do local cases = {} if not stream_done then table.insert(cases, stream_ch:case_receive()) end if not generation_result then table.insert(cases, done_ch:case_receive()) end local result = channel.select(cases) if not result.ok then return finish(nil, nil, "LLM stream closed before completion") end if result.channel == done_ch then generation_result = result.value if generation_result.err then return finish(nil, nil, generation_result.err) end if stream_done then return finish(full_text, generation_result.response, stream_err) end else local chunk = result.value if chunk.type == "chunk" then local content = chunk.content or "" print(content) full_text = full_text .. content elseif chunk.type == "thinking" then print(chunk.content or "") elseif chunk.type == "error" then stream_done = true stream_err = chunk.error and chunk.error.message or "LLM stream failed" elseif chunk.type == "done" then stream_done = true end if stream_done and generation_result then return finish(full_text, generation_result.response, stream_err) end end end end ``` #### Chunk Types | Type | Fields | Description | |------|--------|-------------| | `"chunk"` | `content` | Text content fragment | | `"thinking"` | `content` | Model thinking process | | `"tool_call"` | `name`, `arguments`, `id` | Tool invocation | | `"error"` | `error.message`, `error.type` | Stream error | | `"done"` | `meta` | Stream complete | Streaming requires a process.lua entry because it uses Wippy's process communication system (process.pid(), process.listen()). Run generation in a separate coroutine so the listener drains chunks concurrently, and remove the listener on every return path. ### Tool Calling Define tools with inline schemas and pass them to `generate()`: ```lua local llm = require("llm") local prompt = require("prompt") local json = require("json") local tools = { { name = "get_weather", description = "Get current weather for a location", schema = { type = "object", properties = { location = { type = "string", description = "City name" }, }, required = { "location" }, }, }, } local conversation = prompt.new() conversation:add_user("What's the weather in Tokyo?") local response = llm.generate(conversation, { model = "gpt-4o", tools = tools, tool_choice = "auto", }) if response.tool_calls and #response.tool_calls > 0 then for _, tc in ipairs(response.tool_calls) do -- execute the tool and get a result local result = { temperature = 22, condition = "sunny" } -- add the exchange to the conversation conversation:add_function_call(tc.name, json.encode(tc.arguments), tc.id) conversation:add_function_result(tc.name, json.encode(result), tc.id) end -- continue generation with tool results local final = llm.generate(conversation, { model = "gpt-4o" }) print(final.result) end ``` #### Tool Call Fields | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique call identifier | | `name` | string | Tool name | | `arguments` | table | Parsed arguments matching the schema | #### Tool Choice | Value | Behavior | |-------|----------| | `"auto"` | Model decides when to use tools (default) | | `"none"` | Never use tools | | `"any"` | Must use at least one tool | | `"tool_name"` | Must use the specified tool | ### Structured Output Generate JSON validated against a schema: ```lua local llm = require("llm") local schema = { type = "object", properties = { name = { type = "string" }, age = { type = "number" }, hobbies = { type = "array", items = { type = "string" }, }, }, required = { "name", "age", "hobbies" }, additionalProperties = false, } local response, err = llm.structured_output(schema, "Describe a fictional character", { model = "gpt-4o", }) if not err then print(response.result.name) print(response.result.age) end ``` For OpenAI models, all properties must be in the required array. Use union types for optional fields: type = {"string", "null"}. Set additionalProperties = false. ### Model Configuration Define models as registry entries with `meta.type: llm.model`: ```yaml entries: - name: gpt-4o kind: registry.entry meta: name: gpt-4o type: llm.model title: GPT-4o comment: OpenAI's flagship model capabilities: - generate - tool_use - structured_output - vision class: - balanced priority: 100 max_tokens: 128000 output_tokens: 16384 pricing: input: 2.5 output: 10 providers: - id: wippy.llm.openai:provider provider_model: gpt-4o ``` #### Model Entry Fields | Field | Description | |-------|-------------| | `meta.name` | Model identifier used in API calls | | `meta.type` | Must be `llm.model` | | `meta.capabilities` | Feature list: `generate`, `tool_use`, `structured_output`, `embed`, `thinking`, `vision`, `caching` | | `meta.class` | Class membership: `fast`, `balanced`, `reasoning`, etc. | | `meta.priority` | Numeric priority for class-based resolution (higher wins) | | `max_tokens` | Maximum context window | | `output_tokens` | Maximum output tokens | | `pricing` | Cost per million tokens: `input`, `output` | | `providers` | Array with `id` (provider entry) and `provider_model` (provider-specific model name) | #### Local Models For locally hosted models (LM Studio, Ollama), define a separate provider entry with a custom `base_url`: ```yaml - name: local_provider kind: registry.entry meta: name: ollama type: llm.provider title: Ollama Local driver: id: wippy.llm.openai:driver options: api_key_env: none base_url: http://127.0.0.1:11434/v1 - name: local-llama kind: registry.entry meta: name: local-llama type: llm.model title: Local Llama capabilities: - generate max_tokens: 4096 output_tokens: 4096 pricing: input: 0 output: 0 providers: - id: app:local_provider provider_model: llama-3.2 ``` ### Model Resolution Models can be referenced by exact name, class, or explicit class prefix: ```lua -- exact model name llm.generate("Hello", { model = "gpt-4o" }) -- model class (picks highest priority in that class) llm.generate("Hello", { model = "fast" }) -- explicit class syntax llm.generate("Hello", { model = "class:reasoning" }) ``` Resolution order: 1. Match by exact `meta.name` 2. Match by class name (highest `meta.priority` wins) 3. With `class:` prefix, search only in that class ### Model Discovery Query available models and their capabilities at runtime: ```lua local llm = require("llm") -- all models local models = llm.available_models() -- filter by capability local tool_models = llm.available_models("tool_use") local embed_models = llm.available_models("embed") -- list model classes local classes = llm.get_classes() for _, c in ipairs(classes) do print(c.name .. ": " .. c.title) end ``` ### Embeddings Generate vector embeddings for semantic search: ```lua local llm = require("llm") -- A single input still returns an array of vectors. local single_response, single_err = llm.embed("The quick brown fox", { model = "text-embedding-3-small", dimensions = 512, }) if single_err then error("Embedding failed: " .. tostring(single_err)) end local vector = single_response.result[1] -- Multiple inputs return one vector per input. local batch_response, batch_err = llm.embed({ "First document", "Second document", }, { model = "text-embedding-3-small" }) if batch_err then error("Batch embedding failed: " .. tostring(batch_err)) end local vectors = batch_response.result ``` ### Provider Status Probe a provider before sending work, such as during readiness checks: ```lua local status, err = llm.status({ model = "gpt-4o", }) ``` | Option | Description | |--------|-------------| | `model` | Required. Model to check. | | `provider_id` | Optional. Skip model resolution and target a specific provider. | Returns the provider's `StatusResponse` (contents are provider-dependent). ### Error Handling Errors are returned as the second return value. On error, the first return value is `nil`: ```lua local response, err = llm.generate("Hello", { model = "gpt-4o" }) if err then print("Error: " .. tostring(err)) return end print(response.result) ``` #### Error Types | Constant | Description | |----------|-------------| | `llm.ERROR_TYPE.INVALID_REQUEST` | Malformed request | | `llm.ERROR_TYPE.AUTHENTICATION` | Invalid API key | | `llm.ERROR_TYPE.RATE_LIMIT` | Provider rate limit exceeded | | `llm.ERROR_TYPE.SERVER_ERROR` | Provider server error | | `llm.ERROR_TYPE.CONTEXT_LENGTH` | Input exceeds context window | | `llm.ERROR_TYPE.CONTENT_FILTER` | Content filtered by safety systems | | `llm.ERROR_TYPE.TIMEOUT` | Request timed out | | `llm.ERROR_TYPE.MODEL_ERROR` | Invalid or unavailable model | #### Finish Reasons | Constant | Description | |----------|-------------| | `llm.FINISH_REASON.STOP` | Normal completion | | `llm.FINISH_REASON.LENGTH` | Reached max tokens | | `llm.FINISH_REASON.CONTENT_FILTER` | Content filtered | | `llm.FINISH_REASON.TOOL_CALL` | Model made a tool call | | `llm.FINISH_REASON.ERROR` | Error during generation | ### Capabilities | Constant | Description | |----------|-------------| | `llm.CAPABILITY.GENERATE` | Text generation | | `llm.CAPABILITY.TOOL_USE` | Tool/function calling | | `llm.CAPABILITY.STRUCTURED_OUTPUT` | JSON structured output | | `llm.CAPABILITY.EMBED` | Vector embeddings | | `llm.CAPABILITY.THINKING` | Extended thinking | | `llm.CAPABILITY.VISION` | Image understanding | | `llm.CAPABILITY.CACHING` | Prompt caching | ### See Also - [Agents](framework/agents.md) — Agent framework with tools, delegates, and memory - [Building an LLM Agent](../tutorials/llm-agent.md) — Build an agent step by step - [Framework Overview](framework/overview.md) — Install and import framework modules --- # "Agents" ## Agents The `wippy/agent` module defines agents declaratively and runs them through a context and runner. Agents can use tools, stream responses, delegate work, apply traits, and recall memory. This page is an API primer with composable reference snippets, not a standalone tutorial. The snippets assume an existing Wippy project, a registered LLM model and provider, configured provider credentials, and the agent, tool, or resolver entries referenced by each example. Later snippets build on variables such as `ctx`, `runner`, and `conversation` created in earlier sections. For a complete runnable project, follow [Build an LLM Agent](tutorials/llm-agent.md). ### Setup Add the module to your project: ```bash wippy add wippy/agent wippy install ``` The agent module declares its `wippy/llm` dependency itself. Add the agent dependency to source when it is not already present: ```yaml version: "1.0" namespace: app entries: - name: dep.agent kind: ns.dependency component: wippy/agent version: "*" ``` ### Agent Definitions Agents are registry entries with `meta.type: agent.gen1`: ```yaml entries: - name: assistant kind: registry.entry meta: type: agent.gen1 name: assistant title: Assistant comment: A helpful chat assistant prompt: | You are a helpful assistant. Be concise and direct. Answer questions clearly. model: gpt-4o max_tokens: 1024 temperature: 0.7 ``` #### Agent Fields | Field | Type | Description | |-------|------|-------------| | `meta.type` | string | Must be `agent.gen1` | | `meta.name` | string | Agent identifier | | `prompt` | string | System prompt | | `model` | string | Model name or class | | `max_tokens` | number | Maximum output tokens (default `512`) | | `temperature` | number | Optional sampling temperature; omitted by default, with range and support determined by the provider | | `thinking_effort` | number | Forwarded to the model only when `> 0` (provider-defined scale) | | `tools` | array | Tool registry IDs | | `traits` | array | Trait references | | `delegates` | array | Delegate agent references | | `memory` | array | Static memory items (strings) | | `memory_contract` | table | Dynamic memory configuration | ### Agent Context Create an agent context, configure it as needed, and then load an agent: ```yaml imports: agent_context: wippy.agent:context prompt: wippy.llm:prompt ``` ```lua local agent_context = require("agent_context") local ctx = agent_context.new() local runner, err = ctx:load_agent("app:assistant") if err then error("Failed to load agent: " .. tostring(err)) end ``` #### Context Methods | Method | Description | |--------|-------------| | `agent_context.new(options?)` | Create new context | | `:add_tools(specs)` | Add tools at runtime | | `:add_delegates(specs)` | Add delegate agents | | `:configure_delegate_tools(config)` | Configure how delegates expose themselves as tools | | `:set_memory_contract(config)` | Configure dynamic memory | | `:set_context_merger(fn)` | Provide a function to merge runtime context updates | | `:update_context(updates)` | Update runtime context | | `:load_agent(spec_or_id, options?)` | Load and compile agent, returns runner | | `:switch_to_agent(id, options?)` | Switch to different agent, returns `(boolean, string?)` | | `:switch_to_model(name)` | Change model on current agent, returns `(boolean, string?)` | | `:get_current_agent()` | Get current runner | | `:get_config()` | Return a summary of the context configuration | #### Context Options ```lua local ctx = agent_context.new({ context = { session_id = "abc", user_id = "u1" }, delegate_tools = { enabled = true }, enable_cache = true, }) ``` | Option | Description | |--------|-------------| | `context` | Base runtime context forwarded to tools and delegates | | `delegate_tools` | Default delegate-tool configuration (overridden by `configure_delegate_tools`) | | `enable_cache` | Prompt cache marker setting for Claude models. The current implementation always enables markers, including when this option is `false`. | #### Loading by Inline Spec Load an agent without a registry entry: ```lua local runner, err = ctx:load_agent({ id = "inline-agent", name = "helper", prompt = "You are a helpful assistant.", model = "gpt-4o", max_tokens = 1024, tools = { "app.tools:search" }, }) ``` ### Running Steps The runner executes one agent step from a prompt-builder conversation: ```lua local prompt = require("prompt") local conversation = prompt.new() conversation:add_user("What is the capital of France?") local response, err = runner:step(conversation) if err then error(tostring(err)) end print(response.result) ``` #### Step Options ```lua local self_pid, pid_err = process.pid() if pid_err then error("Failed to get process PID: " .. tostring(pid_err)) end local response, err = runner:step(conversation, { context = { session_id = "abc" }, stream_target = { reply_to = self_pid, topic = "stream" }, tool_call = "auto", }) if err then error("Agent step failed: " .. tostring(err)) end ``` | Option | Type | Description | |--------|------|-------------| | `context` | table | Runtime context merged with agent context | | `stream_target` | table | Streaming: `{ reply_to, topic }` | | `tool_call` | string | `"auto"`, `"any"`, `"none"`, or a tool name | #### Step Response | Field | Type | Description | |-------|------|-------------| | `result` | string | Generated text | | `tokens` | table | Token usage | | `finish_reason` | string | Stop reason | | `tool_calls` | table? | Tool calls to execute | | `delegate_calls` | table? | Delegate invocations | #### Runner Stats ```lua local stats = runner:get_stats() -- stats.id, stats.name, stats.total_tokens ``` ### Tool Definitions Tools are `function.lua` entries with `meta.type: tool`. Define them in a separate `_index.yaml`: ```yaml version: "1.0" namespace: app.tools entries: - name: calculate kind: function.lua meta: type: tool title: Calculate input_schema: | { "type": "object", "properties": { "expression": { "type": "string", "description": "Math expression to evaluate" } }, "required": ["expression"], "additionalProperties": false } llm_alias: calculate llm_description: Evaluate a mathematical expression. source: file://calculate.lua modules: [expr] method: handler ``` ```lua local expr = require("expr") local function handler(args) local result, err = expr.eval(args.expression) if err then return { error = tostring(err) } end return { result = result } end return { handler = handler } ``` #### Tool Metadata | Field | Type | Description | |-------|------|-------------| | `meta.type` | string | Must be `tool` | | `meta.input_schema` | string/table | JSON Schema for tool arguments | | `meta.llm_alias` | string | Name exposed to the LLM | | `meta.llm_description` | string | Description exposed to the LLM | | `meta.exclusive` | boolean | If true, cancels concurrent tool calls | #### Referencing Tools in Agents List tool registry IDs in the agent definition: ```yaml - name: assistant kind: registry.entry meta: type: agent.gen1 name: assistant prompt: You are a helpful assistant with tools. model: gpt-4o max_tokens: 1024 tools: - app.tools:calculate - app.tools:search - app.tools:* # wildcard: all tools in namespace ``` Tools can also be referenced with custom aliases and context: ```yaml tools: - id: app.tools:search alias: web_search context: api_key: "${SEARCH_API_KEY}" ``` ### Tool Execution When an agent step returns `tool_calls`, execute the calls and add their results to the conversation: ```lua local json = require("json") local funcs = require("funcs") local function execute_and_continue(runner, conversation) while true do local response, err = runner:step(conversation) if err then return nil, err end local tool_calls = response.tool_calls if not tool_calls or #tool_calls == 0 then return response.result, nil end for _, tc in ipairs(tool_calls) do local result, call_err = funcs.call(tc.registry_id, tc.arguments) local result_str if call_err then result_str = json.encode({ error = tostring(call_err) }) else result_str = json.encode(result) end conversation:add_function_call(tc.name, json.encode(tc.arguments), tc.id) conversation:add_function_result(tc.name, result_str, tc.id) end end end ``` #### Tool Call Fields | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique call identifier | | `name` | string | Tool name (alias or llm_alias) | | `arguments` | table | Parsed arguments | | `registry_id` | string | Full registry ID for `funcs.call()` | Use funcs.call(tc.registry_id, tc.arguments) to execute tools. The registry_id field maps directly to the tool's entry in the registry. For how agent tool access and observability are secured, see the [Security Model](concepts/security-model.md). ### Streaming Stream agent responses through `stream_target`: ```lua local TOPIC = "agent_stream" local function stream_step(runner, conversation) local stream_ch, listen_err = process.listen(TOPIC) if listen_err then return nil, nil, listen_err end local function finish(text, response, err) local ok, cleanup_err = process.unlisten(stream_ch) if not ok then cleanup_err = cleanup_err or "Failed to remove agent stream listener" if err then return text, nil, tostring(err) .. "; cleanup failed: " .. tostring(cleanup_err) end return text, nil, cleanup_err end if err then return text, nil, err end return text, response, nil end local self_pid, pid_err = process.pid() if pid_err then return finish("", nil, pid_err) end local done_ch = channel.new(1) coroutine.spawn(function() local response, err = runner:step(conversation, { stream_target = { reply_to = self_pid, topic = TOPIC, }, }) done_ch:send({ response = response, err = err }) end) local full_text = "" local step_result = nil local stream_done = false local stream_err = nil while true do local cases = {} if not stream_done then table.insert(cases, stream_ch:case_receive()) end if not step_result then table.insert(cases, done_ch:case_receive()) end local result = channel.select(cases) if not result.ok then return finish(full_text, nil, "Agent stream closed before completion") end if result.channel == done_ch then step_result = result.value if step_result.err then return finish(full_text, nil, step_result.err) end if stream_done then return finish(full_text, step_result.response, stream_err) end else local chunk = result.value if chunk.type == "chunk" then local content = chunk.content or "" print(content) full_text = full_text .. content elseif chunk.type == "error" then stream_done = true stream_err = chunk.error and chunk.error.message or "Agent stream failed" elseif chunk.type == "done" then stream_done = true end if stream_done and step_result then return finish(full_text, step_result.response, stream_err) end end end end ``` The stream uses the same chunk types as direct LLM streaming: `"chunk"`, `"thinking"`, `"tool_call"`, `"error"`, `"done"`. Use coroutine.spawn to run runner:step() in a separate coroutine so you can receive stream chunks concurrently. Use channel.select to multiplex the stream and completion channels. ### Delegates Agents can delegate to other agents. Delegates appear as tools to the parent agent: ```yaml - name: coordinator kind: registry.entry meta: type: agent.gen1 name: coordinator prompt: Route questions to the right specialist. model: gpt-4o max_tokens: 1024 delegates: - id: app:code_agent name: ask_coder rule: for programming questions - id: app:math_agent name: ask_mathematician rule: for math problems ``` Delegate calls appear in `response.delegate_calls`: ```lua local response, err = runner:step(conversation) if err then error("Delegate step failed: " .. tostring(err)) end if response.delegate_calls then for _, dc in ipairs(response.delegate_calls) do -- dc.agent_id - target agent registry ID -- dc.name - delegate tool name -- dc.arguments - forwarded message end end ``` Delegates can also be added at runtime: ```lua ctx:add_delegates({ { id = "app:specialist", name = "ask_specialist", rule = "for domain questions" }, }) ``` ### Traits Traits are reusable definitions that contribute prompts, tools, and behavior to agents: ```yaml - name: assistant kind: registry.entry meta: type: agent.gen1 name: assistant prompt: You are a helpful assistant. model: gpt-4o traits: - time_aware - id: custom_trait context: key: value ``` #### Built-in Traits | Trait | Description | |-------|-------------| | `time_aware` | Injects current date and time into the prompt | The `time_aware` trait accepts context options: ```yaml traits: - id: time_aware context: timezone: America/New_York time_interval: 15 ``` #### Custom Traits Traits are registry entries with `meta.type: agent.trait`. They can contribute: - **prompt** - static text appended to the system prompt - **build_func_id** - function called at compile time to contribute tools, prompts, delegates - **prompt_func_id** - function called at each step to inject dynamic content - **step_func_id** - function called at each step for side effects #### Static Memory Static memory items are appended to the system prompt: ```yaml - name: assistant kind: registry.entry meta: type: agent.gen1 name: assistant prompt: You are a helpful assistant. model: gpt-4o memory: - "User prefers concise answers" - "Always cite sources when possible" ``` #### Dynamic Memory Contract Configure dynamic memory recall through an external implementation: ```yaml memory_contract: implementation_id: app:memory_store context: user_id: "${user_id}" options: max_items: 3 max_length: 1000 recall_cooldown: 1 min_conversation_length: 2 ``` The memory contract is called during `runner:step()` to recall relevant items based on the conversation context. Results are injected as developer messages. | Option | Default | Description | |--------|---------|-------------| | `max_items` | `3` | Maximum memory items per recall | | `max_length` | `1000` | Maximum total character length | | `recall_cooldown` | `1` | Minimum steps between recalls | | `min_conversation_length` | `2` | Minimum conversation turns before first recall | ### Resolver Contract When `load_agent()` receives a string identifier, it first tries to resolve it through the `wippy.agent:resolver` contract. If no resolver is bound or the resolver returns nil, it falls back to the registry lookup. This allows applications to implement custom agent resolution, such as loading agent definitions from a database. #### Binding a Resolver Define a resolver function and bind it to the contract: ```yaml entries: - name: agent_resolver.resolve kind: function.lua source: file://agent_resolver.lua method: resolve modules: - logger imports: agent_registry: wippy.agent.discovery:registry - name: agent_resolver_binding kind: contract.binding contracts: - contract: wippy.agent:resolver default: true methods: resolve: app:agent_resolver.resolve ``` #### Resolver Implementation The resolver receives `{ agent_id = "..." }` and returns an agent spec table or nil: ```lua local agent_registry = require("agent_registry") local CUSTOM_PREFIX = "custom:" function resolve(args) local agent_id = args.agent_id if not agent_id then return nil, "agent_id is required" end if agent_id:sub(1, #CUSTOM_PREFIX) == CUSTOM_PREFIX then local id = agent_id:sub(#CUSTOM_PREFIX + 1) -- load from database, config file, or any other source return { id = agent_id, name = "custom-agent", prompt = "You are a custom agent.", model = "class:balanced", max_tokens = 1024, tools = {}, } end -- fall back to registry local spec, err = agent_registry.get_by_id(agent_id) if not spec then spec, err = agent_registry.get_by_name(agent_id) end return spec, err end return { resolve = resolve, } ``` #### Resolution Order 1. Try `wippy.agent:resolver` contract (if bound) 2. Try registry lookup by ID 3. Try registry lookup by name 4. Return error if not found Custom resolution can load agent definitions outside the framework registry, including definitions scoped by user or workspace. ### See Also - [LLM](framework/llm.md) — Underlying model interface - [Building an LLM Agent](../tutorials/llm-agent.md) — Build an agent step by step - [Framework Overview](framework/overview.md) — Install and import framework modules --- # "Test Framework" ## Test Framework The `wippy/test` module provides BDD suites, assertions, lifecycle hooks, mocks, and a runner for test entries. This page is an API primer. Its Lua, YAML, output, and project-layout blocks are reference snippets that can be combined in an existing Wippy project; they are not one copy-and-run project. Names such as `validate`, `format_name`, `db`, `connect`, and `notify_user` stand for application functions or modules supplied by the test subject. For a complete runnable example, follow [Testing a Wippy Application](../tutorials/testing.md). ### Setup Add the dependency: ```bash wippy add wippy/test wippy install ``` The module registers the test entrypoint automatically (a command with `use_case: test`). Once installed, `wippy test` discovers and runs all test entries in your project. ### Defining Tests Tests are `function.lua` entries with `meta.type: test`: ```yaml version: "1.0" namespace: app.test entries: - name: math kind: function.lua meta: type: test suite: math name: Math operations source: file://math_test.lua method: main imports: test: wippy.test:test ``` #### Test Metadata | Field | Required | Description | |-------|----------|-------------| | `type` | Yes | Must be `"test"` for the runner to discover it | | `suite` | No | Groups tests in the runner output | | `name` | No | Display name shown in runner output | | `order` | No | Sort order within a suite (lower runs first) | #### BDD Style Use `describe` and `it` blocks to structure tests: ```lua local test = require("test") local function define_tests() test.describe("calculator", function() test.it("adds numbers", function() test.eq(1 + 1, 2) end) test.it("multiplies numbers", function() test.eq(3 * 4, 12) end) end) end local run_cases = test.run_cases(define_tests) local function run(options) local result = run_cases(options) if result.failed_tests > 0 then error("tests failed: " .. result.failed_tests) end return result end return { run = run } ``` #### Nested Suites Suites can be nested to group related behavior: ```lua test.describe("user", function() test.describe("validation", function() test.it("requires name", function() test.ok(validate({}).error) end) test.it("accepts valid input", function() test.is_nil(validate({name = "Alice"}).error) end) end) test.describe("formatting", function() test.it("formats display name", function() test.eq(format_name("alice"), "Alice") end) end) end) ``` #### Skipping Tests ```lua test.it_skip("not implemented yet", function() test.fail("TODO") end) ``` Skipped tests appear in the output but do not count as failures. #### Suite Aliases `test.spec` and `test.context` are aliases for `test.describe`: ```lua test.spec("feature", function() test.context("when valid input", function() test.it("succeeds", function() test.ok(true) end) end) end) ``` #### Equality ```lua test.eq(actual, expected, msg?) -- actual == expected test.neq(actual, expected, msg?) -- actual ~= expected ``` #### Truthiness ```lua test.ok(val, msg?) -- val is truthy test.fail(msg?) -- unconditional failure ``` #### Nil Checks ```lua test.is_nil(val, msg?) -- val == nil test.not_nil(val, msg?) -- val ~= nil ``` #### Type Checks ```lua test.is_true(val, msg?) -- val == true test.is_false(val, msg?) -- val == false test.is_string(val, msg?) test.is_number(val, msg?) test.is_table(val, msg?) test.is_function(val, msg?) test.is_boolean(val, msg?) ``` #### Strings and Collections ```lua test.contains(str, substr, msg?) -- substring match test.matches(str, pattern, msg?) -- Lua pattern match test.has_key(tbl, key, msg?) -- table key exists test.len(val, expected, msg?) -- #val == expected ``` #### Numeric Comparisons ```lua test.gt(a, b, msg?) -- a > b test.gte(a, b, msg?) -- a >= b test.lt(a, b, msg?) -- a < b test.lte(a, b, msg?) -- a <= b ``` #### Error Handling ```lua test.throws(fn, msg?) -- fn() raises error, returns it test.has_error(val, err, msg?) -- val is nil, err is not nil test.no_error(val, err, msg?) -- err is nil ``` All assertions accept an optional message as the last argument. On failure, the message is included in the error output. ### Lifecycle Hooks ```lua test.describe("database", function() test.before_all(function() -- runs once before the suite db = connect() end) test.after_all(function() -- runs once after the suite db:close() end) test.before_each(function() -- runs before each test db:begin_transaction() end) test.after_each(function() -- runs after each test db:rollback() end) test.it("inserts a record", function() db:exec("INSERT INTO users (name) VALUES ('Alice')") local count = db:query_row("SELECT COUNT(*) FROM users") test.eq(count, 1) end) end) ``` Hooks in nested suites execute in order: parent `before_each` runs before child `before_each`, and child `after_each` runs before parent `after_each`. ### Mocking The mock system replaces global object fields and restores them after each test. #### Basic Mocking ```lua test.describe("notifications", function() test.it("sends message", function() local sent = false test.mock("process.send", function(pid, topic, payload) sent = true end) notify_user("hello") test.is_true(sent) -- mock is auto-restored after this test end) end) ``` #### Mock API ```lua test.mock("object.field", replacement) -- replace a global field test.mock_process("field", replacement) -- shorthand for process fields test.restore_mock("object.field") -- restore one mock test.restore_all_mocks() -- restore all mocks ``` Mock paths use dot notation: `"process.send"` replaces `_G.process.send`. Mocks for `process.send` automatically proxy test framework messages through the original function, so test event reporting continues to work when process.send is mocked. All mocks are automatically restored after each test via the `after_each` hook. #### Run All Tests ```bash wippy test ``` #### Filter by Pattern ```bash wippy test math wippy test user validation ``` Filters match literal substrings of entry IDs. When you provide multiple patterns, an entry runs if its ID matches any of them. #### Example Output ``` 3 tests in 1 suites calculator + adds numbers 0ms + multiplies numbers 0ms - divides by zero 1ms Error: expected error, got nil 1 suite | 2 passed | 1 failed | 0 skipped | 3ms ``` ### Simple Tests For tests that do not need BDD suites, define a function that returns `true` or raises an error: ```lua local funcs = require("funcs") local function main() local result, err = funcs.call("app:my_function", "input") if err then error("call failed: " .. tostring(err)) end if result ~= "expected" then error("expected 'expected', got: " .. tostring(result)) end return true end return { main = main } ``` ```yaml - name: integration kind: function.lua meta: type: test suite: integration source: file://integration_test.lua method: main modules: - funcs ``` The runner detects whether a test uses BDD case events or returns a simple value. Both patterns work with `wippy test`. ### Project Structure A typical test layout: ``` src/ _index.yaml app.lua test/ _index.yaml # test entries math_test.lua user_test.lua integration_test.lua ``` The test `_index.yaml` defines the test namespace and entries: ```yaml version: "1.0" namespace: app.test entries: - name: math kind: function.lua meta: type: test suite: math source: file://math_test.lua method: main imports: test: wippy.test:test - name: user kind: function.lua meta: type: test suite: user source: file://user_test.lua method: main imports: test: wippy.test:test ``` ### Terminal Host `wippy/test` depends on `wippy/terminal`, which supplies the auto-starting `wippy.terminal:host` used by the CLI runner. Applications do not need to declare a separate process or terminal host solely to run `wippy test`. ### See Also - [Framework Overview](framework/overview.md) — Install and import framework modules - [CLI Reference](guides/cli.md) — Test command and flags - [Functions](concepts/functions.md) — Function entries and invocation --- # "Dataflow" ## Dataflow The `wippy/dataflow` module provides a workflow orchestration engine based on directed acyclic graphs (DAGs). Workflows are composed of nodes — functions, agents, cycles, and parallel processors — connected by typed data routes. The orchestrator manages execution, state persistence, and recovery. ### Setup Add the module to your project: ```bash wippy add wippy/dataflow wippy install ``` Declare the dependency: ```yaml version: "1.0" namespace: app entries: - name: dep.dataflow kind: ns.dependency component: wippy/dataflow version: "*" ``` The dataflow module depends on `wippy/agent`, `wippy/llm`, and `wippy/session` — these are resolved automatically when you run `wippy install`. The module requires a database resource at `app:db` for workflow persistence and runs migrations automatically via `wippy/migration`. The module publishes an `env.variable` entry `userspace.dataflow.env:web_host_origin` (default `https://front.wippy.ai`) that downstream flows can read for building public URLs. Override it through the env router or a requirement. ### Flow Builder The flow builder provides a fluent interface for composing workflows. Import it into your entry: ```yaml imports: flow: userspace.dataflow.flow:flow ``` ```lua local flow = require("flow") ``` #### Core API ```lua flow.create() :with_title(title) :with_metadata(metadata) :with_input(data) :with_data(data) :[operation](config) :as(name) :to(target, input_key, transform) :error_to(target, input_key, transform) :when(condition) :run() -- synchronous :start() -- asynchronous flow.template() :[operations]... ``` #### Linear Pipeline Nodes chain automatically when no explicit routing is defined. Output of each node flows to the next: ```lua local result, err = flow.create() :with_input({ text = "Hello world" }) :func("app:tokenize") :func("app:translate", { args = { target_lang = "fr" } }) :func("app:format_output") :run() ``` #### Named Routing Use `:as()` to name nodes and `:to()` to route data between them. Only use `:as()` when the node needs to be referenced: ```lua local result, err = flow.create() :with_input(task) :to("router") :func("app:router"):as("router") :to("context", "routing") :to("dev", "routing") :agent("app:context_agent"):as("context") :to("dev", "gathered_context") :agent("app:dev_agent"):as("dev") :to("@success") :run() ``` The second parameter to `:to()` is the **discriminator** — the input key at the receiving node. When a node receives multiple inputs, they are collected as a table keyed by discriminator. #### Workflow Input and Static Data `:with_input()` is the single primary input to the workflow. `:with_data()` creates independent static data sources: ```lua flow.create() :with_input(task) :to("router") :with_data(config):as("cfg") :to("dev", "config") :to("logger", "config") :with_data(branch):as("branch_data") :to("checker", "branch") :func("app:router"):as("router") :to("dev", "task") :func("app:dev"):as("dev") :to("@success") :error_to("@fail") :run() ``` Use `:with_input()` for external data entering the workflow. Use `:with_data()` for config, constants, and reference data shared across multiple nodes. Static data uses reference optimization — the first route creates actual data, subsequent routes create lightweight references. #### Conditional Routing Use `:when()` after `:to()` to add conditions. Conditions evaluate against the node's output using `expr` syntax: ```lua flow.create() :with_input(data) :func("app:classify"):as("classify") :to("handler_a"):when("output.category == 'a'") :to("handler_b"):when("output.category == 'b'") :to("fallback") :func("app:handler_a"):as("handler_a"):to("@success") :func("app:handler_b"):as("handler_b"):to("@success") :func("app:fallback"):as("fallback"):to("@success") :run() ``` Conditions can combine with inline transforms for more complex routing: ```lua :func("app:decompose"):as("decompose") :to("@success", nil, "{passed: true, feedback: nil}"):when("len(output.items) == 0") :to("processor", "items", "output.items") ``` Conditional expressions support: comparisons (`output.score > 0.8`), logical operators (`output.valid && output.count > 5`), array functions (`len(output.items) > 0`, `any(output.errors, {.critical})`), string operations (`output.status contains 'success'`), and optional chaining (`output.data?.nested?.value`). #### Workflow Terminals Route to `@success` or `@fail` to terminate the workflow explicitly. In nested contexts (cycles, parallel), terminals create node outputs instead of workflow outputs: ```lua :func("app:final_step"):to("@success") :func("app:handler"):error_to("@fail") ``` #### Error Routing Use `:error_to()` to route node errors to a handler. Errors can be routed as normal inputs to recovery nodes: ```lua :agent("app:gpt_planner", { model = "gpt-5" }):as("gpt_planner") :to("consolidator", "gpt_plan") :error_to("consolidator", "gpt_plan") :agent("app:claude_planner", { model = "claude-4-5-sonnet" }):as("claude_planner") :to("consolidator", "claude_plan") :error_to("consolidator", "claude_plan") :agent("app:consolidator", { inputs = { required = { "gpt_plan", "claude_plan" } } }):as("consolidator") ``` This pattern runs both planners in parallel — if one fails, its error becomes the input for the consolidator, which proceeds with whatever results are available. ### Input Merging How nodes receive inputs depends on discriminators and whether `args` is configured. **Without args — single default input:** ```lua :func("source"):to("target") -- target receives: raw content (unwrapped) ``` **Without args — single named input:** ```lua :func("source"):to("target", "task") -- target receives: { task = content } ``` **Without args — multiple inputs:** ```lua :func("source1"):to("target", "data") :func("source2"):to("target", "config") -- target receives: { data = content1, config = content2 } ``` **With args — inputs merge into base:** ```lua :func("app:api_client", { args = { base_url = "https://api.com", timeout = 5000 } }) -- with :to("api_client", "body") from upstream -- api_client receives: { base_url = "https://api.com", timeout = 5000, body = content } ``` Nodes with args cannot receive inputs with the "default" discriminator. Use named discriminators with :to(target, "input_key") instead. ### Input Transforms Transform data before it reaches a node: ```lua -- String transform: single expression :func("app:step", { input_transform = "input.nested.field" }) -- Table transform: named expressions :func("app:step", { input_transform = { task = "inputs.task", config = "inputs.settings", priority = "output.score > 0.8 ? 'high' : 'normal'" } }) ``` Context variables available in transforms: `input` (workflow input), `inputs` (all incoming node inputs), `output` (current node's output when routing). #### Inline Route Transforms The third parameter to `:to()` is an inline transform expression: ```lua :func("source"):as("source") :to("target", nil, "output.data") :to("other", nil, "{passed: true, value: output.x}") :to("list", nil, "map(output.items, {.id})") ``` #### Function Node Executes a registered `function.lua` entry: ```lua :func("app:my_function", { args = { key = "value" }, inputs = { required = { "task", "config" } }, context = { session_id = "abc" }, input_transform = { task = "inputs.prompt" }, metadata = { title = "Process Data" } }) ``` | Option | Type | Description | |--------|------|-------------| | `args` | table | Base arguments merged with node inputs | | `inputs` | table | Input requirements: `{ required = {...}, optional = {...} }` | | `context` | table | Execution context passed to function | | `input_transform` | string/table | Expression to transform inputs | | `metadata` | table | Node metadata (e.g., `{ title = "..." }`) | If the function returns `{ _control = { commands = [...] } }`, the orchestrator spawns a child workflow. This is how nested flows work. #### Agent Node Executes an agent with tool calling and optional structured exit: ```lua :agent("app:content_writer", { model = "gpt-5", inputs = { required = { "context", "content_plan", "analysis" } }, arena = { prompt = "Write content based on the provided context.", max_iterations = 12, tool_calling = "any", exit_schema = { type = "object", properties = { content = { type = "string" }, title = { type = "string" } }, required = { "content", "title" } } }, show_tool_calls = true, metadata = { title = "Content Writer" } }) ``` | Option | Type | Description | |--------|------|-------------| | `model` | string | Override model | | `arena.prompt` | string | System prompt | | `arena.max_iterations` | number | Max reasoning loops (default: 32) | | `arena.min_iterations` | number | Min iterations before exit (default: 1) | | `arena.tool_calling` | string | `"auto"`, `"any"` (requires `exit_schema`), `"none"` (rejects `exit_schema`) | | `arena.tools` | array | Tool registry IDs | | `arena.exit_schema` | table | JSON schema for structured exit | | `arena.exit_func_id` | string | Function to validate exit output | | `arena.context` | table | Additional context | | `inputs` | table | Input requirements | | `show_tool_calls` | boolean | Include tool calls in output | | `input_transform` | string/table | Transform inputs | | `metadata` | table | Node metadata | **Dynamic agent selection:** Pass an empty string as agent ID and resolve it via `input_transform`: ```lua :agent("", { inputs = { required = { "spec", "task" } }, input_transform = { agent_id = "inputs.spec.agent_id", task = "inputs.task" }, arena = { prompt = "Process according to spec", max_iterations = 25 } }) ``` **Exit validation:** When `exit_func_id` is set, the function validates the agent's exit output. On validation failure, the agent receives the error as observation and continues (up to `max_iterations`). #### Cycle Node Iterates a function or template repeatedly with persistent state: ```lua :cycle({ func_id = "app:content_cycle", max_iterations = 3, initial_state = { entry_id = entry_id, content_prompt = prompt, min_score = 8.0, feedback_history = {} } }) ``` The cycle function receives on each iteration: ```lua { input = , -- only on the first iteration (iteration == 1); nil thereafter state = , last_result = , iteration = } ``` `input` carries the workflow input only on the first iteration and is `nil` thereafter; persist anything needed across iterations into `state`. The function controls continuation: ```lua function my_cycle(cycle_context) -- stop if approved if cycle_context.last_result and cycle_context.last_result.approved then return { state = cycle_context.state, result = cycle_context.last_result, continue = false } end -- spawn child workflow for this iteration -- task is read from state since cycle_context.input is nil after iteration 1 return flow.create() :with_input({ task = cycle_context.state.task }) :agent("app:worker") :agent("app:qa") :run() end ``` | Option | Type | Description | |--------|------|-------------| | `func_id` | string | Iteration function (mutually exclusive with `template`) | | `template` | FlowBuilder | Template for each iteration (mutually exclusive with `func_id`) | | `max_iterations` | number | Maximum iterations | | `initial_state` | table | Starting state | | `continue_condition` | string | Expression: continue while true | **Template-based cycle:** ```lua :cycle({ template = flow.template() :agent("app:worker") :func("app:validator"), max_iterations = 5 }) ``` #### Parallel Node Map-reduce pattern over arrays: ```lua :parallel({ inputs = { required = { "specs", "task" } }, source_array_key = "specs", iteration_input_key = "spec", passthrough_keys = { "task" }, batch_size = 10, on_error = "collect_errors", filter = "successes", unwrap = true, template = flow.template() :agent("app:processor", { inputs = { required = { "spec", "task" } }, input_transform = { agent_id = "inputs.spec.agent_id", task = "inputs.task" }, arena = { prompt = "Process according to spec", max_iterations = 25 } }) :to("@success"), metadata = { title = "Process Specs" } }) ``` | Option | Type | Description | |--------|------|-------------| | `source_array_key` | string | Input key containing the array (required) | | `template` | FlowBuilder | Template for each item (required, must route to `@success`) | | `iteration_input_key` | string | Input key for current item (default: `"default"`) | | `batch_size` | number | Items per parallel batch (default: 1 = sequential) | | `on_error` | string | `"collect_errors"` (default) or `"fail_fast"` | | `filter` | string | `"all"` (default), `"successes"`, `"failures"` | | `unwrap` | boolean | Return raw results instead of wrapped metadata (default: false) | | `passthrough_keys` | array | Input keys forwarded to every iteration | **Passthrough keys** provide shared context (config, task description) to every iteration without duplicating data in the source array: ```lua :with_data(file_list):as("files"):to("processor", "files") :with_data("secret"):as("api_key"):to("processor", "api_key") :parallel({ inputs = { required = { "files", "api_key" } }, source_array_key = "files", iteration_input_key = "filename", passthrough_keys = { "api_key" }, template = flow.template() :func("app:upload", { inputs = { required = { "filename", "api_key" } } }) :to("@success") }):as("processor") ``` #### Signal Node Pauses execution until an external signal arrives. Use for human approvals, external events, or staged workflows: ```lua :signal({ signal_id = "approval", inputs = { required = { "draft" } }, metadata = { title = "Wait for approval" } }) ``` | Option | Type | Description | |--------|------|-------------| | `signal_id` | string | Signal name matched against `client:signal()`. If empty or omitted, a UUID v7 is generated at runtime | | `inputs` | table | Input requirements | | `input_transform` | string/table | Transform inputs before the node receives them | | `metadata` | table | Node metadata | Send the signal from outside the workflow using the client API (see `client:signal()` below). ##### Behavior The node yields with `wait_for_signal = true` and persists that yield in the workflow state. The orchestrator resumes the node when a matching `NODE_SIGNAL` commit arrives. - The signal is satisfied by any non-`nil` payload. `false`, `0`, `""`, and `{}` all satisfy the yield; only `nil` keeps it pending. - A signal yield blocks `COMPLETE_WORKFLOW` but does not block other pending nodes — parallel branches continue to execute while one branch waits. - Signals can be pre-queued before `:start()`: if a matching `NODE_SIGNAL` commit arrives before the signal node reaches the yield, it is delivered the moment the yield is tracked. - Only one signal satisfies each yield. If a second signal with the same `signal_id` arrives before the yield is satisfied, it overwrites the first. - When multiple signal yields share the same `signal_id`, the first matching yield receives the data. - If the `signal_id` field is absent, matching falls back to the node's discriminator. - Delivered signal data is passed to the node's output as the signal payload. ##### Durability and recovery The signal yield is part of the workflow state, persisted through the same outbox mechanism as every other command. If the orchestrator process is killed while waiting: - The pending yield is restored on restart. - Signals delivered during the outage are queued and applied when the state reloads. - Compound pipelines (`func → signal → signal → func`) recover step-by-step — each signal can be delivered across a separate restart. Orphaned signal yields (yields whose parent process exited without completion) are cleaned up by the workflow state's process exit handler. ##### Pipeline patterns Signal nodes participate in any topology: ```lua -- Human-in-the-loop approval between two functions flow.create() :func("app:draft") :signal({ signal_id = "approve_draft" }) :func("app:publish") :run() -- Two parallel approvals that must both arrive before release flow.create() :with_input({ doc = "release-notes" }) :as("trigger") :to("legal", "doc") :to("finance", "doc") :signal({ signal_id = "legal_ok", inputs = { required = { "doc" } } }) :as("legal") :to("gate", "legal") :signal({ signal_id = "finance_ok", inputs = { required = { "doc" } } }) :as("finance") :to("gate", "finance") :join({ inputs = { required = { "legal", "finance" } } }) :as("gate") :to("release") :func("app:release"):as("release"):to("@success") :run() ``` Signal data is exposed as the node output, so downstream nodes receive whatever was passed to `client:signal()`. #### Join Node Collects multiple inputs before proceeding: ```lua :join({ inputs = { required = { "source1", "source2" } }, output_mode = "object", ignored_keys = { "triggered" } }) ``` | Option | Type | Description | |--------|------|-------------| | `output_mode` | string | `"object"` (default) or `"array"` (arrival order) | | `ignored_keys` | array | Input keys excluded from output | | `inputs` | table | Input requirements | ### Templates Templates define reusable sub-workflows. Use `flow.template()` to create, `:use()` to inline: ```lua local preprocessor = flow.template() :func("app:clean") :func("app:tokenize") flow.create() :with_input(data) :use(preprocessor) :func("app:process") :run() ``` Templates inline their operations into the parent flow at compile time. ### Nested Workflows Functions used in cycles and parallel nodes can spawn child workflows by returning `flow.create():run()`: ```lua function my_processor(input) return flow.create() :with_input(input) :func("app:step_a") :func("app:step_b") :run() end ``` When `:run()` executes inside an existing dataflow context, it returns `{ _control = { commands = [...] } }` instead of executing directly. The orchestrator handles the child workflow through the yield mechanism. Functions that participate in dataflow composition must return flow.create():run(). Functions returning anything else cannot spawn child workflows. ### Synchronous vs Asynchronous `:run()` blocks until the workflow completes and returns output: ```lua local result, err = flow.create() :with_input({ text = "hello" }) :func("app:process") :run() ``` `:start()` returns immediately with a workflow ID: ```lua local dataflow_id, err = flow.create() :with_input({ text = "hello" }) :func("app:process") :start() ``` `:start()` cannot be used in nested contexts. ### Client API For programmatic workflow management: ```yaml imports: client: userspace.dataflow:client ``` ```lua local client = require("client") local c, err = client.new() ``` | Method | Description | |--------|-------------| | `client.new()` | Create client (requires security actor) | | `:create_workflow(commands, options?)` | Create workflow, returns `dataflow_id` | | `:execute(dataflow_id, options?)` | Run synchronously, returns result | | `:start(dataflow_id, options?)` | Run asynchronously, returns `dataflow_id` | | `:output(dataflow_id)` | Fetch workflow outputs | | `:get_status(dataflow_id)` | Get current status | | `:cancel(dataflow_id, timeout?)` | Gracefully cancel (default: 30s) | | `:terminate(dataflow_id)` | Force terminate | | `:signal(dataflow_id, signal_id, data?)` | Deliver an external signal to a waiting signal node | ### Workflow Status | Status | Description | |--------|-------------| | `template` | Node is a template instance | | `pending` | Waiting for inputs | | `ready` | Inputs collected, ready to execute | | `running` | Actively executing | | `paused` | Yielded, waiting for child workflow | | `completed` | Finished successfully | | `failed` | Failed | | `cancelled` | User cancelled | | `skipped` | Conditional branch not taken | | `terminated` | Force terminated | ### Metadata ```lua flow.create() :with_title("Document Processing Pipeline") :with_metadata({ source = "api", priority = "high" }) :func("app:process", { metadata = { title = "Process Document" } }) :run() ``` Title defaults to "Flow Builder Workflow" if not provided. ### Validation Rules The compiler validates workflows at compile time: - All `:as(name)` names must be unique - All `:to()` and `:error_to()` targets must reference existing names (except `@success`, `@fail`) - Graph must be acyclic - All nodes must have incoming routes (from another node, workflow input, or static data) - `:cycle()` requires `func_id` or `template` (not both) - `:parallel()` requires `source_array_key` and `template` - At least one path must lead to `@success` or have auto-output - `:when()` only follows `:to()` or `:error_to()` from nodes (not static data) - Nodes with `args` cannot receive inputs with `"default"` discriminator ### Expression Reference Expressions use the `expr` module syntax, available in `:when()` conditions and `input_transform` values. **Operators:** `+`, `-`, `*`, `/`, `%`, `**`, `==`, `!=`, `<`, `<=`, `>`, `>=`, `&&`, `||`, `!`, `contains`, `startsWith`, `endsWith` **Array functions:** `all()`, `any()`, `none()`, `one()`, `filter()`, `map()`, `count()`, `len()`, `first()`, `last()` **Math functions:** `max()`, `min()`, `abs()`, `ceil()`, `floor()`, `round()`, `sqrt()`, `pow()` **String functions:** `len()`, `upper()`, `lower()`, `trim()`, `split()`, `join()` **Type functions:** `type()`, `int()`, `float()`, `string()` **Literals:** numbers, strings, booleans (`true`/`false`), null (`nil`), arrays (`[1, 2, 3]`), objects (`{key: value}`) **Ternary:** `output.age >= 18 ? output.verified : false` **Optional chaining:** `output.data?.nested?.value` ### Error Handling Both `:run()` and `:start()` follow standard Lua error conventions: - Success: `data, nil` (run) or `dataflow_id, nil` (start) - Failure: `nil, error_message` Error categories: compilation errors, client errors, workflow creation errors, execution errors, and workflow failures. ### See Also - [Agents](framework/agents.md) - Agent framework used by agent nodes - [LLM](framework/llm.md) - LLM module - [Framework Overview](framework/overview.md) - Framework module usage --- # "Relay" ## Relay The `wippy/relay` module routes WebSocket connections through a central hub and per-user hubs. User hubs manage client connections and dispatch messages to prefixed plugins. This page is a partial integration recipe and protocol reference, not a standalone WebSocket application. The setup and plugin blocks assume an existing Wippy project, a real security scope at the configured `user_security_scope`, and an HTTP WebSocket endpoint connected to the relay as described in [WebSocket Relay](http/websocket-relay.md). Protocol payloads and lifecycle blocks are reference shapes. ### Architecture ``` Central Hub ├── User Hub (alice) │ ├── Plugin: session_ │ ├── Plugin: ai_ │ ├── WebSocket Client 1 │ └── WebSocket Client 2 ├── User Hub (bob) │ ├── Plugin: session_ │ └── WebSocket Client 1 └── ... ``` The central hub runs as a service. When a WebSocket client connects, it finds or creates a hub for that user. The user hub manages the connection lifecycle and routes messages by command prefix. ### Setup Add the module to your project: ```bash wippy add wippy/relay wippy install ``` Declare the dependency with required parameters: ```yaml version: "1.0" namespace: app entries: - name: os_env kind: env.storage.os - name: processes kind: process.host lifecycle: auto_start: true - name: dep.relay kind: ns.dependency component: wippy/relay version: "*" parameters: - name: application_host value: app:processes - name: env_storage value: app:os_env - name: user_security_scope value: app.security:user_scope ``` #### Configuration Parameters | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `application_host` | yes | — | Process host for relay processes | | `env_storage` | no | internal | Environment variable storage | | `user_security_scope` | yes | — | Security scope for user hubs | | `max_connections_per_user` | no | `5` | WebSocket connections per user | | `queue_multiplier` | no | `100` | Message queue = connections × multiplier | | `user_hub_inactivity_timeout` | no | `7200s` | Idle time before hub cleanup | ### Client Connection Flow 1. WebSocket client connects with `user_id` in metadata 2. Central hub validates the connection and checks per-user limits 3. Central hub creates or reuses a user hub for the user 4. User hub sends a `welcome` message to the client: ```json { "user_id": "alice", "client_count": 1, "plugins": [ { "prefix": "session_", "process_id": "...", "status": "running" }, { "prefix": "ai_", "process_id": "...", "status": "pending" } ] } ``` Plugin `status` can be `"not_started"` (registered but never spawned), `"pending"` (spawn in progress), `"running"`, `"failed"`, or `"stopped"`. ### Message Routing Clients send JSON messages with a `type` field. The user hub matches the type prefix against registered plugins and routes the message: ```json { "type": "session_get_state", "data": { "key": "value" } } ``` The `session_` prefix selects the session plugin. The hub removes the prefix and sends the message to the plugin process, using the remaining type as the topic: ```lua -- process topic: "get_state" -- payload: { conn_pid = client_pid, type = "session_get_state", -- original full type preserved data = { key = "value" }, request_id = "...", session_id = "..." } ``` Plugins respond by sending messages back to `conn_pid`. ### Plugins Plugins are `process.lua` entries with `meta.type: relay.plugin`: ```yaml entries: - name: session_plugin kind: process.lua meta: type: relay.plugin command_prefix: session_ auto_start: true source: file://session_plugin.lua modules: [json, time, logger] method: run ``` #### Plugin Metadata | Field | Type | Description | |-------|------|-------------| | `meta.type` | string | Must be `relay.plugin` | | `meta.command_prefix` | string | Message type prefix this plugin handles | | `meta.auto_start` | boolean | Start when user hub initializes | | `meta.default_host` | string | Override process host | #### Plugin Lifecycle The user hub spawns each plugin with these startup arguments: ```lua function run(args) local user_id = args.user_id local user_metadata = args.user_metadata local user_hub_pid = args.user_hub_pid local config = args.config end ``` The `session_` plugin receives lifecycle messages: | Message | When | |---------|------| | `"resume"` | First client connects to user hub | | `"shutdown"` | Last client disconnects from user hub | Plugins get 1 automatic restart on crash. After a second crash, the plugin is marked as `"failed"` and not restarted. #### Plugin Implementation Plugins receive messages through their process inbox. Each message has a topic derived from the command type and a payload containing the original message data and `conn_pid` for responses. ```lua local json = require("json") local function handle_message(topic, payload) if topic == "get_state" then if not payload.conn_pid then return nil, "Relay message is missing conn_pid" end local encoded, encode_err = json.encode({ type = "session_state", data = { status = "active" } }) if encode_err then return nil, encode_err end local sent, send_err = process.send(payload.conn_pid, "ws.message", encoded) if not sent then return nil, send_err or "Relay response was not sent" end end return true end local function run(args) local user_id = args.user_id local inbox = process.inbox() local events = process.events() while true do local result = channel.select({ inbox:case_receive(), events:case_receive() }) if not result.ok then break end if result.channel == inbox then local msg = result.value local topic = msg:topic() local payload = msg:payload():data() if topic == "resume" then -- first client connected elseif topic == "shutdown" then -- last client disconnected else local ok, err = handle_message(topic, payload) if not ok then error("Failed to handle relay message: " .. tostring(err)) end end elseif result.channel == events then local event = result.value if event.kind == process.event.CANCEL then break end end end end return { run = run } ``` ### Error Handling The relay reports client errors using these codes: | Error Code | Description | |------------|-------------| | `max_connections_reached` | User at connection limit | | `missing_user_id` | No user_id in connection metadata | | `hub_creation_failed` | Failed to spawn user hub | | `invalid_json` | Message decode error | | `unknown_command` | Message missing type field | | `plugin_not_found` | No plugin matches the command prefix | | `plugin_failed` | Plugin unavailable or crashed | #### User Hub Creation The first client connection for a user creates that user's hub. The hub runs with the user's security actor and scope. #### Garbage Collection The central hub periodically checks for inactive user hubs. A hub with no connected clients for longer than `user_hub_inactivity_timeout` (default 2 hours) is gracefully terminated with a 10-second cancel timeout. The GC check interval is automatically derived: `inactivity_timeout / 2.5`. #### Security The central hub runs under its own security group (`wippy.relay.security:root`) with full access. Each user hub spawns with the configured `user_security_scope`, isolating user-level operations. ### Internal Topics | Topic | Direction | Description | |-------|-----------|-------------| | `ws.join` | Client → Central/User Hub | Connection request | | `ws.leave` | Client → Central/User Hub | Disconnection | | `ws.message` | Client → User Hub | WebSocket message | | `ws.cancel` | Central → User Hub | Graceful shutdown | | `ws.control` | Central → Client | Redirects the client connection's target PID to its user hub | | `hub.activity_update` | User Hub → Central | Client count update | ### See Also - [WebSocket Relay](../http/websocket-relay.md) — HTTP WebSocket endpoint configuration - [Process Model](concepts/process-model.md) — Process lifecycle and messaging - [Security](system/security.md) — Security actors and scopes - [Framework Overview](framework/overview.md) — Install and import framework modules --- # "Views" ## Views The `wippy/views` module defines pages and components, manages their resources, and maps environment variables into rendered output. It supports two page models: - **Jet template pages** (`kind: template.jet`) render HTML on the server after assembling the page's data and resources. See [Template Pages](#template-pages). - **Registry-entry frontends** (`kind: registry.entry`) describe micro frontend applications (`view.page`) and reusable web components (`view.component`) served from a CDN or static mount. The registry entry contains routing and deployment policy. Frontend-owned metadata comes from the package's generated `wippy-meta.json`, with explicit registry fields taking precedence. See [Component Pages](#component-pages) and [View Components](#view-components). This page is a registry and HTTP API reference. Its YAML, HTML, and JSON blocks are independent reference snippets, not one runnable project. Before adapting them, provide the `http.router`, environment storage, and HTTP service referenced by the dependency, plus any template sets, functions, resources, or frontend bundles named by the selected example. ### Setup Add the module to your project: ```bash wippy add wippy/views wippy install ``` Declare the dependency: ```yaml version: "1.0" namespace: app entries: - name: dep.views kind: ns.dependency component: wippy/views version: "*" parameters: - name: api_router value: app:api.public - name: env_storage value: app:env.storage ``` | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `api_router` | yes | — | HTTP router for view API endpoints | | `env_storage` | yes | — | Environment storage backing the `PUBLIC_API_URL` variable | | `server` | no | `app:gateway` | HTTP service the self-mounted [Web Fragments gateway](#web-fragments-gateway) router (`/@fragment`) binds to. Override only if your `http.service` id differs from `app:gateway`. | ### Template Pages > **Server-rendered model.** `wippy/views` assembles template data and resources on the server, then renders the final HTML with Jet. The response is plain HTML and does not use an iframe proxy or client-side micro frontend. For external SPAs and components, see [Component Pages](#component-pages). Template pages render server-side using Jet templates. Data is injected via `data.set`, `data.data_func`, and `data.resources` (server-side resource injection): ```yaml entries: - name: contact_page kind: template.jet meta: type: view.page name: contact title: Contact Us icon: mail order: 5 group: main group_icon: layout-grid group_order: 1 announced: true secure: false data: set: app.templates:default data_func: app:contact_data resources: - contact_styles ``` #### Page Metadata | Field | Type | Default | Description | |-------|------|---------|-------------| | `meta.type` | string | — | Must be `view.page` | | `meta.name` | string | entry name | Page identifier | | `meta.title` | string | — | Display title | | `meta.icon` | string | — | Icon identifier | | `meta.order` | number | `9999` | Sort order within group | | `meta.group` | string | — | Group category | | `meta.group_icon` | string | — | Group icon | | `meta.group_order` | number | `9999` | Group sort order | | `meta.group_placement` | string | `"default"` | Placement: `"default"`, `"sidebar"` | | `meta.secure` | boolean | `false` | Requires authentication | | `meta.public` | boolean | `false` | Makes the page announced when true; it does not bypass `meta.secure` access control | | `meta.announced` | boolean | `false` | Show in navigation. The current resolver uses `announced or public`, so `public: true` overrides an explicit `announced: false` | | `meta.inline` | boolean | `false` | Returned by `/pages/list` as the numeric `hidden` marker | | `meta.content_type` | string | `text/html` | Response MIME type | | `meta.parent` | string | — | Parent page ID | #### Template Data | Field | Description | |-------|-------------| | `data.set` | Required template set registry ID | | `data.data_func` | Function ID that returns page data | | `data.resources` | Array of resource registry IDs | The `data_func` receives `{ params, query }` and returns a table that becomes the `data` context in the template. Omitting `data.data_func`, or returning `nil` from it, produces an empty table. A configured function that cannot be resolved, or a function that returns an error, aborts rendering. #### Rendering Pipeline 1. Load page from registry 2. Check access (security) 3. Call `data_func` if defined 4. Collect resources: globals + template set resources + page-specific resources 5. Load environment variables (mapping failures are logged and produce an empty `env` table) 6. Render Jet template with context: `{ data, resources, query_params, route_params, env }` ### Component Pages Component pages point to external single-page applications (SPAs or micro frontends) that the Web Host loads with its configured page engine: an iframe by default, or a Web Fragment when enabled. Their registry entries define URL serving, access control, the mount route, and per-page configuration overrides: > **Required registry shape:** component pages are `kind: registry.entry` with `meta.type: view.page`. `view.page` is never a `kind` value. Proxy deployment overrides live at `meta.proxy`, not `data.proxy`. ```yaml entries: - name: dashboard kind: registry.entry meta: type: view.page name: dashboard title: Dashboard icon: chart-bar url: /app base_path: app/dashboard entry_point: index.html mountRoute: /dashboard/:part(.*)* secure: true announced: true config_overrides: customization: cssVariables: "--p-primary": "#7c9ed9" ``` The API returns a component descriptor with the resolved base URL. The Web Host then renders the SPA with the selected iframe or Web Fragment engine. Iframe pages apply the proxy injections requested by the frontend package; the Fragment gateway uses its own fixed transformation and Host-CSS injection path. #### Component Page Fields | Field | Type | Default | Description | |-------|------|---------|-------------| | `meta.name` | string | — | Page name. Keep it in registry YAML because `/pages/list` does not load bundled metadata | | `meta.title` | string | — | Display title. Keep it in registry YAML because `/pages/list` sorts raw registry titles | | `meta.url` | string | — | Base URL prefix where the bundle is mounted (CDN origin or `http.static` path) | | `meta.base_path` | string | — | Subdirectory within the static mount | | `meta.entry_point` | string | bundled `wippy.path`, then `index.html` | HTML entry file; combined as `//` | | `meta.mountRoute` | string | — | Claims a URL path in the host router; only the catch-all form `/:part(.*)*` (root) or `//:part(.*)*` is allowed — arbitrary Vue Router patterns are rejected (HTTP 500). See [view-page.md](../frontend/frontend-registry/view-page.md) / [dynamic-routing.md](../frontend/frontend-registry/dynamic-routing.md) | | `meta.announced` | boolean | `announced or public or false` | Show in navigation and `/pages/list`; `public: true` wins over an explicit `announced: false` | | `meta.secure` | boolean | `false` | Requires authentication | | `meta.render_engine` | string | bundled `wippy.renderEngine` | Per-page engine preference: `auto`, `iframe`, or `fragment` | | `meta.config_overrides` | object | — | Per-page AppConfig overrides (camelCase), deep-merged over the bundled defaults | For component pages, `wippy/views` requests `wippy-meta.json` from the resolved bundle root when building the content descriptor. Registry YAML wins field by field; bundled metadata fills omitted frontend-owned fields such as package version, entry path, proxy settings, render engine, and config overrides. If the metadata file cannot be used, the module falls back to the legacy YAML descriptor. Keep `meta.name` and `meta.title` in registry YAML: `/pages/list` consumes raw registry fields without fetching the bundle metadata, and missing titles can break same-order sorting. `config_overrides` supports `customization`, `axiosDefaults`, `routePrefix`, `apiRoutes`, and `themeMode`. #### Proxy Injection For SPA pages, configure proxy injection in the frontend package's camelCase `wippy.proxy.injections` block. The build records this configuration in `wippy-meta.json`. A deployment can override it with a camelCase `proxy:` block under the registry entry's `meta:` field, using the same shape and `injections` wrapper as the package's `wippy.proxy` block. The host deep-merges the deployment value over the bundled configuration, with YAML values taking precedence at each nested key. There is no snake_case form or casing normalization. `config_overrides` deep-merges only `customization`, `axiosDefaults`, `routePrefix`, `apiRoutes`, and `themeMode`; it does not affect `proxy.injections`. See [Micro Frontend Apps (view.page)](../frontend/frontend-registry/view-page.md) and [CSS Injection](../frontend/web-host/css-injection.md). Deployment override example: ```yaml entries: - name: dashboard kind: registry.entry meta: type: view.page proxy: injections: css: themeConfig: true customCss: true customVariables: true tailwindConfig: false ``` ### View Components View components are reusable custom elements (web components or micro frontends) that the Web Host discovers and registers. They are not pages and do not have navigation entries. As with component pages, their registry entries define routing and deployment policy: ```yaml entries: - name: reaction-bar kind: registry.entry meta: type: view.component name: reaction-bar tag_name: example-reaction-bar announced: true auto_register: true secure: false url: /app/wc/reaction-bar entry_point: index.js ``` Components use `meta.type: view.component` instead of `view.page`. YAML can override `tag_name`, `entry_point`, `props`, and `events`; otherwise those frontend-owned fields come from `wippy-meta.json`, with `index.js` as the final entry-point fallback. Components do not use the page iframe's proxy-injection block. Shadow-DOM platform CSS is requested by the component implementation through `hostCssKeys`. See [Web Components (view.component)](../frontend/frontend-registry/view-component.md) and [CSS Injection](../frontend/web-host/css-injection.md). ### Resources Resources are CSS, JS, and font files associated with pages: ```yaml entries: - name: global_styles kind: registry.entry meta: type: view.resource name: Global Styles resource_type: style global: true order: 1 url: https://cdn.example.com/global.css - name: app_script kind: registry.entry meta: type: view.resource name: App Script resource_type: script template_set: app.templates:default order: 10 url: https://cdn.example.com/app.js defer: true ``` #### Resource Fields | Field | Type | Description | |-------|------|-------------| | `meta.type` | string | Must be `view.resource` | | `meta.resource_type` | string | Free-form (defaults to `"other"`); common values are `"style"`, `"script"`, `"font"` | | `meta.order` | number | Sort order within type | | `meta.global` | boolean | Applied to all pages | | `meta.template_set` | string | Specific to a template set | | `meta.url` | string | Resource URL | | `meta.integrity` | string | SRI hash | | `meta.crossorigin` | string | `"anonymous"` or `"use-credentials"` | | `meta.media` | string | CSS media query | | `meta.defer` | boolean | Deferred script loading | | `meta.async` | boolean | Async script loading | #### Resource Collection Resources are selected cumulatively from three sources: 1. **Global resources** — `global: true`, applied to all pages 2. **Template set resources** — matched by `template_set` ID 3. **Page resources** — listed in `data.resources` array After collection, resources are grouped by `resource_type` and each group is sorted by `order`. The three source layers do not establish a separate output order. ### Environment Variable Mapping The env loader maps environment variables to template context keys through a priority-based system. #### Defining Mappings ```yaml entries: - name: app_env kind: registry.entry meta: type: view.env_mapping priority: 20 data: mappings: api_endpoint: API_BASE_URL app_title: APP_NAME debug_mode: DEBUG_ENABLED ``` Each mapping entry associates context keys (used in templates as `env.api_endpoint`) with environment variable names. #### Priority System | Range | Category | Description | |-------|----------|-------------| | 0–9 | Framework defaults | Built-in framework mappings | | 10–19 | System overrides | System-level configuration | | 20–29 | Application mappings | Application-specific mappings | | 30–100 | Environment overrides | Runtime overrides | Higher priority wins when multiple mappings define the same context key. Do not define the same key more than once at a single priority: equal-priority ordering is not defined. #### Using in Templates Resolved environment values are available in the `env` context object: ```html ``` ### HTTP API Endpoints The views module registers these endpoints on the configured router: | Method | Path | Description | |--------|------|-------------| | GET | `/pages/list` | List accessible, announced pages | | GET | `/components/list` | List accessible, announced view components | | GET | `/pages/content/{id}` | Render page or return component descriptor | | GET | `/pages/public/{id}` | Get component base URL | | GET | `/components/by-tag/{tag}` | Resolve a custom-element tag name to its `view.component` descriptor (used by host `loadByTagName`) | | GET | `/pages/routes` | Return the `mountRoute` → `pageId` map; HTTP 500 on invalid or duplicate `mountRoute`. Not filtered by `announced` (hidden pages still need URL resolution); access control applies to secure pages | #### Render Response For template pages, returns rendered HTML with the page's `content_type`. For component pages, returns a descriptor: ```json { "name": "dashboard", "version": "1.0.0", "specification": "wippy-component-1.0", "title": "Dashboard", "baseUrl": "https://cdn.example.com/dashboard/", "wippy": { "type": "page", "path": "index.html", "proxy": { "enabled": true, "injections": { "css": { "themeConfig": true, "iframe": true }, "tailwindConfig": false, "resizeObserver": true, "preventLinkClicks": true } } } } ``` The `css` injection flags are `themeConfig`, `iframe`, `primevue`, `markdown`, `customCss`, and `customVariables`. There is no `fonts` flag — Google Fonts are delivered via `theming.global.customCSS` (an `@import` rule), injected by `customCss`. ### Web Fragments Gateway When the Web Host renders a page with the [fragment render engine](../frontend/web-host/render-engines.md), the page is mounted as ``. `wippy/views` serves that reframing contract through a dedicated gateway endpoint at **`/@fragment/{id}/{path...}`**. Unlike the view API, which mounts on the consumer's `api_router`, the gateway declares its own top-level `/@fragment` `http.router`, making it CDN-cache-routable and independent of `token_auth`. Authentication is handled client-side through the injected fragment proxy's handshake with the host. Consumers do not need a router entry or `fragment_router` parameter, and applications using the iframe engine do not require fragment configuration. The self-mounted router binds to a `server` requirement that defaults to `app:gateway`. If the application's `http.service` entry has another ID, set the `wippy/views` `server` parameter to that entry: ```yaml entries: - name: dep.views kind: ns.dependency component: wippy/views version: "*" parameters: - name: api_router value: app:api.public - name: env_storage value: app:env.storage - name: server # optional — only if your http.service id ≠ app:gateway value: app:my_http_service ``` > **Fragment availability.** A page that sets `wippy.renderEngine: "fragment"` in an otherwise iframe-based deployment uses a runtime capability probe. If the gateway or `proxy-fragment.js` is unavailable, the page remains on the iframe engine without reporting an error. The global `render_engine: fragment` setting does not perform this probe. #### Reframing Contract The gateway answers the same `/@fragment/{id}/` URL three ways, discriminated by the request's `Sec-Fetch-Dest` header and subpath: | Request | Response | |---------|----------| | Realm iframe load (`Sec-Fetch-Dest: iframe`) | A tiny **reframed stub** carrying the host import map + `loading.js` + `proxy-fragment.js`. | | Document fetch (empty subpath) | The page's app HTML, transformed for the realm: remove the first import map and development placeholder, rewrite relative `href="./…"` and `src="./…"` attributes, inject Host CSS links, and rename ``/``/`` to ``. The gateway does not inject ``. | | Asset (non-empty subpath) | Proxied to the page's real `base_url` + subpath. | Responses carry `Cache-Control`: the stub is shared-cacheable (`public, max-age=300`); the access-gated document and assets are `private` (they pass a per-user `can_access` check, so a shared cache would leak across users). Runtime errors are explicit HTTP responses — `400 Missing fragment id`, `404 Fragment page not found`, `401 Access denied`, `502 Fragment document fetch failed: … (url: …)`. The FE selects the engine and mounts the fragment — see [Render Engines](../frontend/web-host/render-engines.md). ### Access Control Pages with `secure: true` require authentication. The page registry checks `security.can("view", "page:")` against the current actor and scope. Non-secure pages are always accessible. The `announced` flag controls visibility in navigation listings without affecting access. ### ID Qualification Relative IDs in page definitions are qualified with the entry's namespace: ```yaml ## In namespace "app" data: data_func: my_data_func # resolves to app:my_data_func set: templates:default # stays as templates:default (already qualified) resources: - page_styles # resolves to app:page_styles ``` ### See Also - [Facade](./facade.md) — Frontend facade and navigation sidebar - [Template](../system/template.md) — Jet template engine - [Security](../system/security.md) — Security actors and access control - [Environment](../system/env.md) — Environment variable storage - [Framework Overview](./overview.md) — Framework module usage - [Micro Frontend Apps (`view.page`)](../frontend/frontend-registry/view-page.md) — Full `view.page` metadata and proxy injection reference - [Web Components (`view.component`)](../frontend/frontend-registry/view-component.md) — Full `view.component` autoload and props reference - [Render Engines](../frontend/web-host/render-engines.md) — Iframe and Web Fragment page rendering --- # "Facade" ## Facade The `wippy/facade` module serves a page that loads and configures the Wippy Web Host from a CDN. The page loads `module.js` for the default compatibility shell or `managed-layout.js` for managed mode, handles authentication, and passes backend configuration to the frontend. The loaded module controls the page and its browser history. For isolated or partial-page integrations, the host can still be embedded manually through `iframe.html` and a `SetConfig` postMessage handshake. The facade itself does not use this delivery mode. This page is a partial deployment recipe and configuration reference. The setup block can be adapted to an existing Wippy project, while the theming, config-response, navigation, and publishing blocks are independent reference snippets. Provide any login page, filesystem entries, static assets, and frontend view entries that an adapted snippet names. For a complete runnable facade project, follow [Serve the Web Host with Facade](../tutorials/facade.md). ### Setup Add the module to your project: ```bash wippy add wippy/facade wippy install ``` Declare the dependency: ```yaml version: "1.0" namespace: app entries: - name: gateway kind: http.service addr: :8090 lifecycle: auto_start: true - name: api kind: http.router meta: server: app:gateway prefix: /api/public - name: dep.facade kind: ns.dependency component: wippy/facade version: "*" parameters: - name: server value: app:gateway - name: router value: app:api ``` #### Configuration Parameters | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `server` | yes | — | HTTP server for static and page serving | | `router` | yes | — | Public API router for config endpoint | | `fe_facade_url` | no | `https://web-host.wippy.ai/webcomponents-1.0.56` | Base CDN URL for the frontend bundle | | `fe_entry_path` | no | `/iframe.html` | Path to the **iframe** entry on the bundle, used by the iframe embedding mode. The current facade's page loads the JS-module entry (`module.js`/`managed-layout.js`) instead; this iframe path remains available for manual, facade-less iframe embeddings. | | `fe_mode` | no | `compat` | Which shell the facade page loads: `compat` loads `module.js` (the default chat shell); `managed` loads `managed-layout.js` (opt-in declarative multi-panel layout). Surfaced on `/facade/config` as `mode`/`module_file`. | | `host_config_layout` | no | `{}` | JSON layout config emitted as `hostConfig.layout`; consumed by the **managed** shell only. | | `render_engine` | no | `iframe` | Page render engine, emitted as `hostConfig.renderEngine`. See [Render engine](#render-engine). | | `login_path` | no | `/login.html` | Path on the page's origin to redirect unauthenticated users to; works with `login_redirect_param`. | | `login_redirect_param` | no | `""` (off) | Query-parameter name to append the post-login return URL to when redirecting to `login_path`. Empty disables the return-URL append. | | `extra_scripts` | no | `[]` | JSON array of extra script URLs the facade page loads; emitted on `/facade/config` as `extraScripts`. | #### Render Engine `render_engine` selects the [page render engine](../frontend/web-host/render-engines.md) for the whole deployment. It is emitted as `hostConfig.renderEngine` and read by the Web Host at its single page-render fork. | Value | Effect | |-------|--------| | `iframe` _(default)_ | Pages render as srcdoc iframes — the main (default) engine. | | `fragment` | Pages render as [Web Fragments](../frontend/web-host/render-engines.md) (a `reframed` realm reflected into a shadow root). | Only the exact string `fragment` opts in; **any other value — including a typo like `fragmnet` — is clamped to `iframe`** (fail-safe, but silent). Enabling the fragment engine also requires the [`/@fragment` gateway](./views.md#web-fragments-gateway), which is self-provided by `wippy/views` (≥ 0.5.9) — no consumer wiring. A page can override the deployment default per-page with [`wippy.renderEngine`](../frontend/frontend-registry/view-page.md#render-engine). #### App Identity | Parameter | Default | Description | |-----------|---------|-------------| | `app_title` | `Wippy` | Title shown in sidebar | | `app_name` | `Wippy AI` | Full application name | | `app_icon` | `wippy:logo` | Iconify icon reference | #### Feature Flags | Parameter | Default | Description | |-----------|---------|-------------| | `hide_nav_bar` | `false` | Hide the left navigation sidebar | | `disable_right_panel` | `false` | Disable the right sidebar panel | | `start_nav_open` | `false` | Navigation drawer open by default | | `show_admin` | `true` | Show admin panel toggle | | `allow_select_model` | `false` | Allow user to select LLM model | | `session_type` | `non-persistent` | Web Host session policy: `cookie` stores a secondary token cookie; any other value is normalized to `non-persistent` and does not use that cookie. | | `history_mode` | `hash` | Browser history mode: `hash` or `browser`. The Web Host treats any value other than `browser` as `hash`. | | `hide_session_selector` | `false` | Hide the session picker UI | The facade shell's bootstrap token is separate from `session_type`. The shell always reads `localStorage["@wippy_token_info"]`, parses its JSON `token` field, and redirects to `login_path` when the value is missing or invalid. It passes that token to the Web Host. In `cookie` mode the Web Host also stores the token in its `@wippy-gen2/token` cookie; in `non-persistent` mode it does not use that secondary cookie. #### Theming Three scopes apply: **global** (everywhere), **host** (the Web Host chrome — sidebar, chat, page area), and **children** (child `view.page` render contexts and `view.component` web components). For which surface each knob reaches, see the [CSS Delivery Matrix](../frontend/web-host/css-injection.md#css-delivery-matrix). | Parameter | Scope | Default | Description | |-----------|-------|---------|-------------| | `custom_css` | global | Google Fonts import | Global CSS — reaches host chrome, `view.page` render contexts, and `view.component` shadow roots (1.0.43+). | | `css_variables` | global | `{}` | JSON map of arbitrary CSS custom properties; compiled for Auto and forced modes and bridged into component shadow roots. | | `icon_sets` | global | `{}` | Iconify icon sets keyed by prefix (inline JSON only — no `fs://`) | | `host_custom_css` | host | `""` | CSS for the host chrome only — not children. Scope class-based rules to `.wippy-host-app`. | | `host_css_variables` | host | `{}` | CSS custom properties for the host chrome only | | `host_icon_sets` | host | `{}` | Icon sets keyed by prefix for host only (inline JSON only) | | `children_custom_css` | children | `""` | CSS for children only — injected into `view.page` render contexts and `view.component` shadow roots (1.0.43+), not host chrome | | `children_css_variables` | children | `{}` | CSS custom properties for children only | Put shared brand styling in the global `custom_css` and `css_variables` parameters so it reaches every surface. Use `host_custom_css` and `host_css_variables` for host-only elements such as the sidebar, chat panel, and splitters. A `view.component` can opt out of shadow-root `*_custom_css` with `customCss: false`. ##### Theme Mode and Persistence | Parameter | Default | Description | |-----------|---------|-------------| | `theme_mode` | `auto` | Forced theme for host + children: `auto` (follow OS), `light`, or `dark`. Emitted on `/facade/config` as `themeMode`. | | `theme_persist` | `none` | Persist the user's chosen theme across reloads: `none`, `cookie`, or `localStorage`. In `cookie` mode the Jet-rendered shell reads the cookie server-side and applies the `w-theme-*` class before the first paint (no flash). Emitted as `themePersist`. | | `theme_storage_key` | `@wippy-theme-mode` | Cookie / localStorage key the mode is stored under. Emitted as `themeStorageKey` and baked into the generated `/facade/theme-persist.js`. | Theme persistence is **opt-in**: `theme_persist` defaults to `none`, so nothing is stored until a deployment sets it to `cookie` or `localStorage`. When enabled the facade serves a ready-made script at **`GET /facade/theme-persist.js`** with the key and mode baked in; include it on any page that should share the theme. See [Theme Persistence](../frontend/web-host/theme-persistence.md) for the full model, the `themeChanged` host event, and non-Wippy-page integration. ##### Reusing Facade Theming on Non-Web-Host Pages A page served outside the Web Host, such as `login.html`, an error page, or an email confirmation page, can reuse the facade theme. This keeps brand tokens and custom rules in one place. First, keep `custom_css` and `css_variables` in standalone files rather than inlining them, and point the parameters at those files with `fs://` plus a `content_fs` filesystem: ```yaml custom_css: fs://custom-css.facade.css css_variables: fs://css-variables.facade.json content_fs: app:app_fs ``` Use `fs://` (resolved by `content_fs` at runtime), **not** `file://` — `file://` is inlined by the wippy loader relative to the YAML at load time. Keep the files in the same static folder your `login_path` page is served from (in `app`, `static/` served at `/app`). `fs://` resolution applies to exactly the **six theming parameters** — `custom_css`, `css_variables`, `host_custom_css`, `host_css_variables`, `children_custom_css`, `children_css_variables` (CSS strings are read verbatim; JSON `*_css_variables` files are parsed as the variable map). `icon_sets` / `host_icon_sets` and every other JSON parameter (`api_routes`, `chat`, `tanstack`, …) are **inline-only**; `fs://` is not resolved there. A standalone page then links both: - **`custom_css`** — already a `.css` file, so link it directly from where it is served. - **`css_variables`** — JSON, so it is not linkable as-is. The facade renders it at **`GET /facade/variables.css`** as base plus effective Auto-light, Auto-dark, forced Light, and forced Dark blocks. Top-level values apply everywhere; `@light` / `@dark` replace selected names. The sheet is cached for 1h and registered on the same public router as `/facade/config`, so it carries the router prefix. ```html ``` To also share the **theme mode** (so a `login.html` honours and persists the same light/dark choice as the host), add the generated theme-persist script and call its `write()` from your switcher: ```html ``` See [Theme Persistence → Non-Wippy-hosted pages](../frontend/web-host/theme-persistence.md) for a complete switcher example. #### Optional JSON Parameters Each of the following is a JSON-encoded string parameter; defaults are empty (`{}` or `[]`). These four are surfaced verbatim under `hostConfig` for the frontend: | Parameter | Default | Description | |-----------|---------|-------------| | `additional_nav_items` | `[]` | Extra sidebar entries | | `state_cache` | `{}` | Frontend state cache configuration | | `allow_additional_tags` | `{}` | HTML sanitizer tag whitelist (`Record`, tag → allowed attributes) | | `chat` | `{}` | Chat UI overrides | These three are emitted as **top-level** `AppConfig` fields (siblings of `hostConfig`), not under `hostConfig`: | Parameter | Emitted as | Default | Description | |-----------|------------|---------|-------------| | `api_routes` | `apiRoutes` | `{}` | Route overrides for the frontend | | `axios_defaults` | `axiosDefaults` | `{}` | Frontend axios HTTP client defaults | | `tanstack` | `tanstack` | `{}` | TanStack Query defaults: `{ default?, content?, lists? }`. `default` applies to all queries; `content` targets single-resource renders, `lists` targets navigation/index queries. Host default is `refetchOnWindowFocus:false` | ### Config Endpoint The facade registers `GET /facade/config` on the configured public router, so the effective URL includes that router's prefix. With the `/api/public` prefix from [Setup](#setup), the page fetches `/api/public/facade/config`. The same router exposes `GET /facade/variables.css`, which renders `css_variables` as a `text/css` stylesheet for pages outside the Web Host. See [Reusing Facade Theming on Non-Web-Host Pages](#reusing-facade-theming-on-non-web-host-pages). The frontend fetches the configuration on load: ```json { "facade_url": "https://web-host.wippy.ai/webcomponents-1.0.56", "iframe_origin": "https://web-host.wippy.ai", "iframe_url": "https://web-host.wippy.ai/webcomponents-1.0.56/iframe.html?waitForCustomConfig", "login_path": "/login.html", "login_redirect_param": null, "mode": "compat", "module_file": "/module.js", "extraScripts": null, "env": { "APP_API_URL": "https://api.example.com", "APP_AUTH_API_URL": "https://api.example.com", "APP_WEBSOCKET_URL": "wss://api.example.com" }, "routePrefix": "https://api.example.com", "themeMode": "auto", "themePersist": "none", "themeStorageKey": "@wippy-theme-mode", "apiRoutes": { "...": "..." }, "axiosDefaults": { "...": "..." }, "tanstack": { "lists": { "refetchOnWindowFocus": true } }, "theming": { "global": { "customCSS": "...", "cssVariables": {}, "iconSets": {} }, "host": { "customCSS": "...", "cssVariables": {}, "iconSets": {}, "i18n": { "app": { "title": "Wippy", "icon": "wippy:logo", "appName": "Wippy AI" } } }, "children": { "customCSS": "...", "cssVariables": {} } }, "hostConfig": { "session": { "type": "non-persistent" }, "history": "hash", "renderEngine": "iframe", "showAdmin": true, "allowSelectModel": false, "startNavOpen": false, "hideNavBar": false, "disableRightPanel": false, "hideSessionSelector": false, "additionalNavItems": [], "stateCache": { "...": "..." }, "allowAdditionalTags": { "w-chart": ["data", "type"] }, "chat": { "...": "..." } } } ``` The API URL is read from the `PUBLIC_API_URL` environment variable; `APP_WEBSOCKET_URL` is derived by replacing `http://` with `ws://` or `https://` with `wss://`. Theming has three scopes (`global`, `host`, `children`) — `host.i18n` carries app branding. `hostConfig` keys are camelCased and assembled from facade parameters: `session_type`, `history_mode`, `render_engine`, `show_admin`, `allow_select_model`, `start_nav_open`, `hide_nav_bar`, `disable_right_panel`, `hide_session_selector`, plus optional `additional_nav_items`, `state_cache`, `allow_additional_tags`, and `chat`. `render_engine` becomes `renderEngine` (see [Render engine](#render-engine)). The `api_routes`, `axios_defaults`, and `tanstack` parameters are emitted as top-level `AppConfig` fields (`apiRoutes`, `axiosDefaults`, `tanstack`), siblings of `hostConfig`, not inside it. The `facade_url`, `iframe_origin`, `iframe_url`, `login_path`, `mode`, and `module_file` fields are **shell-level** fields used by the embedding page to build itself — they are not part of the child `AppConfig` that the host initializes with. The `iframe_origin`/`iframe_url` fields are consumed only by manual, facade-less iframe embeddings (see [Facade Entry Point](../frontend/web-host/entry-point.md)). The `mode` field is the normalized `fe_mode` (`compat` or `managed`), and `module_file` is the JS-module entry the facade page loads — `/module.js` for compat, `/managed-layout.js` for managed. ### Navigation Sidebar Pages registered via `wippy/views` appear in the sidebar automatically based on their metadata: ```yaml entries: - name: dashboard kind: registry.entry meta: type: view.page name: dashboard title: Dashboard icon: tabler:chart-bar group: Analytics group_icon: tabler:chart-dots group_order: 10 order: 1 announced: true secure: true url: https://cdn.example.com/dashboard/ ``` #### Sidebar Groups Pages with the same `group` value are collected into collapsible sections. Groups are sorted by `group_order` (lower first), pages within groups by `order`. | Field | Description | |-------|-------------| | `group` | Category name displayed in sidebar | | `group_icon` | Icon for the category header | | `group_order` | Sort position of the group (lower = higher) | | `group_placement` | `"sidebar"` (in sidebar) or `"default"` (main area only) | Pages without a `group` appear as top-level items. #### Controlling Visibility | Field | Effect | |-------|--------| | `announced: true` | Page appears in sidebar navigation | | `announced: false` | Page hidden from navigation but still accessible via URL | | `inline: true` | Internal page, hidden from all UI listings | | `hide_nav_bar: true` | Facade parameter — hides the entire left sidebar | ### Publishing with Embedded Assets When publishing a component that includes static files (like the facade's `public/` directory), use `--embed` to include `fs.directory` entries in the package: ```bash wippy publish --embed facade:public_files ``` Without `--embed`, `fs.directory` entries are excluded from the published package. The `--embed` flag accepts entry IDs or names matching `fs.directory` entries. ### See Also - [Views](./views.md) — Page and component system - [HTTP Server](../http/server.md) — HTTP service configuration - [Framework Overview](./overview.md) — Framework module usage - [Facade Entry Point](../frontend/web-host/entry-point.md) — How the facade starts the Web Host - [CSS Injection](../frontend/web-host/css-injection.md) — How facade theming reaches child iframes - [Render Engines](../frontend/web-host/render-engines.md) — Iframe and Web Fragment page rendering --- # "Embeddings" ## Embeddings The `wippy/embeddings` module generates embeddings through `wippy/llm`, stores them in an application database, and performs vector similarity searches. It supports PostgreSQL with pgvector and SQLite with sqlite-vec. This page is an API primer with reference snippets, not a standalone tutorial. The snippets assume an existing Wippy project, a configured database, and the embedding model, provider, and credentials described below. Remote embedding calls may incur provider charges. For a complete application that indexes and searches content, follow [Build a RAG Pipeline](../tutorials/rag.md). ### Setup Add the module to your project: ```bash wippy add wippy/embeddings wippy install ``` #### Required Model and Provider Before calling the embeddings API, register an `llm.model` whose `meta.name` is `text-embedding-3-small`, whose capabilities include `embed`, and whose provider mapping resolves to an embedding provider. Configure that provider's credentials, such as `OPENAI_API_KEY`, through the environment storage used by `wippy/llm`. See [LLM model configuration](./llm.md#model-configuration). #### Database Dependency Declare the dependency and set its `target_db` parameter to the application database: ```yaml version: "1.0" namespace: app entries: - name: app_db kind: db.sql.sqlite file: ./data/app.db - name: dep.embeddings kind: ns.dependency component: wippy/embeddings version: "*" parameters: - name: target_db value: app:app_db ``` On startup, `wippy/migration` picks up the `01_create_embeddings_table` migration and creates the `embeddings_512` table for the configured database driver. If you use the relative SQLite path shown above, create the `data` directory before starting the application. ### Current Fixed Constants The module currently defines these private constants; they are not dependency parameters: | Constant | Default | Description | |----------|---------|-------------| | `EMBEDDING_MODEL` | `text-embedding-3-small` | LLM model used to generate vectors | | `EMBEDDING_DIMENSIONS` | `512` | Vector size passed to the model | | `MAX_TOKENS_PER_REQUEST` | `8000` | Per-call token budget; large batches are split | | `DEFAULT_SEARCH_LIMIT` | `10` | Default number of hits returned by `search` | Tokens are estimated as `ceil(#text / 4)`. Oversized batches are split between items. An individual item larger than the budget is not split and causes that sub-batch to fail before the LLM call. ### Import ```yaml entries: - name: my_app kind: library.lua source: file://my_app.lua imports: embeddings: wippy.embeddings:embeddings ``` ```lua local embeddings = require("embeddings") ``` #### add ```lua local result, err = embeddings.add(content, content_type, origin_id, context_id, meta) ``` Generates an embedding for `content` and persists it. | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `content` | string | yes | Text to embed | | `content_type` | string | yes | Label such as `"document_chunk"` or `"question"`; PostgreSQL limits it to 32 characters | | `origin_id` | string | yes | Identifier for the source document or record; must be a UUID when `target_db` is PostgreSQL | | `context_id` | string | no | Additional scoping key (section, chat, tenant) | | `meta` | table | no | Arbitrary JSON-serialisable metadata | Returns `{ entry_id, origin_id, content_type, context_id }` or `nil, err`. At the pinned framework baseline, the single-item helper passes the nested result from `llm.embed()` to the repository instead of its first vector, so `embeddings.add()` cannot persist successfully. Use `embeddings.add_batch()` with one item, or call `llm.embed()` and pass `response.result[1]` to `embedding_repo.add()`, until the framework implementation is corrected. #### add_batch The following uses SQLite-compatible application IDs. For PostgreSQL, replace `doc-1` with a UUID because the PostgreSQL schema stores `origin_id` as `UUID`. ```lua local result, err = embeddings.add_batch({ { content = "...", content_type = "chunk", origin_id = "doc-1" }, { content = "...", content_type = "chunk", origin_id = "doc-1", context_id = "s1" }, }) ``` Embeds and stores multiple items in one call. If the total estimated token count exceeds `MAX_TOKENS_PER_REQUEST`, the method splits the batch into chunks. Each repository chunk is transactional, but a split high-level batch is not atomic across chunks: earlier chunks remain stored if a later chunk fails. Returns `{ count, items = { ... } }`. To remove records created while testing, use the repository API's `delete_by_origin(origin_id)` method for each sample origin. #### search ```lua local hits, err = embeddings.search("how do migrations work?", { content_type = "document_chunk", origin_id = "doc-1", context_id = "section-2", limit = 10, }) ``` Embeds the query string and performs a similarity search against stored vectors. All filters are optional; matching records are ordered by similarity. `origin_id` may be a string or a non-empty array of strings. Each hit contains `entry_id`, `origin_id`, `content_type`, `context_id`, `content`, decoded `meta`, timestamps, and `similarity`. #### find_by_type ```lua local hits, err = embeddings.find_by_type( "how do migrations work?", "document_chunk", { limit = 10 } ) ``` Calls `search` with a single `content_type`. The default limit is `10`. #### find_by_origin ```lua local hits, err = embeddings.find_by_origin("how do migrations work?", "doc-1", { content_type = "document_chunk", context_id = "section-2", limit = 5, }) ``` Calls `search` with a single `origin_id` and optional `content_type` and `context_id` filters. The default limit is `5`. ### Repository API (`wippy.embeddings:embedding_repo`) Use the repository directly when you already have a vector and want to skip embedding generation. Raw embeddings must contain exactly 512 numeric values: | Function | Description | |----------|-------------| | `embedding_repo.add(content, content_type, origin_id, context_id, meta, embedding)` | Insert a precomputed vector | | `embedding_repo.add_batch(batch)` | Insert many precomputed vectors in one transaction | | `embedding_repo.get_by_origin(origin_id)` | List all records for a given origin | | `embedding_repo.delete_by_origin(origin_id)` | Remove all records for a given origin | | `embedding_repo.delete_by_entry(entry_id)` | Remove a single record by its row id | | `embedding_repo.search_by_embedding(vector, options)` | Similarity search against a raw vector | `search_by_embedding` accepts `{ content_type, origin_id, context_id, limit }`. ### Database Support The migration creates the schema appropriate for the database driver at `target_db`: - **PostgreSQL** — `embeddings_512` table with a `vector(512)` column and an IVFFlat cosine index. The migration attempts to install the `vector` extension, so the database role must either be allowed to create it or the extension must already exist. PostgreSQL stores `origin_id` as `UUID`. - **SQLite** — `embeddings_512` `vec0` virtual table holding the `embedding float[512]` vector column alongside the metadata and content columns for KNN search. ### See Also - [LLM](framework/llm.md) — `llm.embed(...)` for raw embedding generation - [Migrations](framework/migration.md) — Migration runner that provisions the table - [Framework Overview](framework/overview.md) — Framework module usage --- # "Bootloader" ## Bootloader The `wippy/bootloader` module discovers and runs application initialization functions in a defined order at startup. Framework modules use bootloaders for tasks such as encryption-key setup and database migrations. This page is a partial integration recipe and API reference, not a standalone application. The definition below is structurally complete, but `apply_seed()` represents application code that must implement the actual seed operation and its idempotency check. Any persistent cleanup or reversal depends on that application-specific operation. ### Setup Add the module to your project: ```bash wippy add wippy/bootloader wippy install ``` Declare the dependency and the required application host: ```yaml version: "1.0" namespace: app entries: - name: processes kind: process.host lifecycle: auto_start: true - name: os_env kind: env.storage.os - name: dep.bootloader kind: ns.dependency component: wippy/bootloader version: "*" parameters: - name: application_host value: app:processes - name: env_storage value: app:os_env ``` The dependency activates `wippy.bootloader:bootloader.service`, a `process.service` with `auto_start: true`. ### How It Works At startup the bootloader: 1. Discovers every entry with `meta.type: bootloader` from the registry. 2. Sorts them by `meta.order` ascending (lowest first). 3. Executes each one sequentially as a Lua function. 4. Stops the remaining bootloader sequence on the first result with `status = "error"`. 5. Reports total, successful, failed, and skipped counts when finished. Each bootloader checks its own conditions, performs its work, and reports a structured result. ### Defining a Bootloader A bootloader is any `function.*` entry with `meta.type: bootloader`. Most application bootloaders use `function.lua`: ```yaml - name: seed_defaults kind: function.lua meta: type: bootloader order: 50 description: Seed default rows for a new install source: file://seed_defaults.lua method: run modules: - logger imports: sql: :sql ``` | Field | Required | Description | |-------|----------|-------------| | `meta.type` | Yes | Must be `bootloader` | | `meta.order` | No | Execution order (default `999`); lower runs first | | `meta.description` | No | Human-readable summary | | `meta.requires` | No | One ID or an array of bootloader/service IDs. Earlier bootloaders must have returned `success` or `skipped`; service requirements must exist in the registry. An unmet requirement stops the remaining sequence. | Dependency type is determined from the referenced registry entry: `meta.type: bootloader` identifies a bootloader, while other resolved entries are treated as services. If an ID cannot be resolved, the fallback treats a dotted namespace as a bootloader ID and another colon-qualified ID as a service ID. A service check waits up to 20 attempts at 500 ms intervals, but it checks registry presence, not runtime health. #### Return Contract The `method` returns a table describing the outcome: ```lua local function run() local ok, err = apply_seed() if err then return { status = "error", message = "seed failed: " .. tostring(err) } end if not ok then return { status = "skipped", message = "already seeded" } end return { status = "success", message = "seeded default rows" } end return { run = run } ``` | Status | Meaning | |--------|---------| | `success` | Work completed | | `skipped` | No-op (already done, precondition unmet) | | `error` | Failure — stops the remaining bootloader sequence | A bootloader that raises a Lua error, returns an execution error, or returns a non-table value is converted to an `error` result. The orchestrator measures and overwrites `duration`; a returned `details` value is preserved for logging. Use the three status strings exactly. Another value is logged as `UNKNOWN`, is not included in a status counter, and does not currently stop later bootloaders. ### Execution Order Lower `order` values run first. Reserve low orders for infrastructure: | Order | Typical Use | |-------|-------------| | `10` | Secrets and encryption keys (provided by the module) | | `20` | Schema migrations (provided by `wippy/migration`) | | `50` | Data seeding, search index warmup | | `100` | Application-level tasks (convention) | When two bootloaders share an order, they run in alphabetical order by their fully-qualified entry ID. #### Encryption Key (order `10`) Generates 32 random bytes, encodes them as a 64-character hexadecimal `ENCRYPTION_KEY`, and stores the value through the configured `env_storage` if no value is present. Skipped when the variable already exists. #### Migration Bootloader (order `20`) Provided by `wippy/migration`. Discovers every entry with `meta.type: migration`, groups them by `meta.target_db`, and applies the pending ones. See [Migrations](framework/migration.md). ### Observing Boot Status The service logs the discovery count, then one result line per executed bootloader (`SUCCESS`, `FAILED`, `SKIPPED`) with the entry ID, order, and duration. The final summary reports executed and per-status counts. A failed bootloader stops later bootloaders and makes the orchestrator return `false` with its statistics; it does not raise a Lua process error by itself. Keep bootloaders idempotent. They run again whenever `bootloader.service` is started again, so check preconditions (row exists, file present, env var set) before doing work. ### See Also - [Migrations](framework/migration.md) — Migration bootloader and DSL - [Supervision](guides/supervision.md) — Service lifecycle and restart policy - [Framework Overview](framework/overview.md) — Framework module usage --- # "Migrations" ## Migrations The `wippy/migration` module provides a database migration framework with a small DSL for defining schema changes, a runner that discovers and executes them, and a bootloader that runs pending migrations for every `target_db` registered in the project. Migrations support SQLite, PostgreSQL, and MySQL, with per-driver `up`/`down` implementations defined side by side. ### Setup Add the module to your project: ```bash wippy add wippy/migration wippy install ``` Declare the dependency and the application database the migrations target: ```yaml version: "1.0" namespace: app entries: - name: app_db kind: db.sql.sqlite path: ./data/app.db - name: dep.migration kind: ns.dependency component: wippy/migration version: "*" ``` The migration bootloader registers with `wippy/bootloader` at order `20`. When the application starts, it discovers every migration entry in the registry, groups them by `meta.target_db`, and runs pending migrations against each database. ### Defining a Migration A migration is a `function.lua` entry with `meta.type: migration`. The entry returns a function produced by `migration.define(...)`. ```yaml entries: - name: 01_create_users_table kind: function.lua meta: type: migration target_db: app:app_db timestamp: "2025-01-15T10:00:00Z" source: file://01_create_users_table.lua imports: migration: wippy.migration:migration ``` ```lua return require("migration").define(function() migration("Create users table", function() database("sqlite", function() up(function(db) local ok, err = db:execute([[ CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL UNIQUE ) ]]) if err then error(err) end end) down(function(db) db:execute("DROP TABLE IF EXISTS users") end) end) database("postgres", function() up(function(db) db:execute([[ CREATE TABLE users ( id SERIAL PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL UNIQUE ) ]]) end) down(function(db) db:execute("DROP TABLE IF EXISTS users") end) end) end) end) ``` #### Required Metadata | Field | Required | Description | |-------|----------|-------------| | `meta.type` | yes | Must be `"migration"` for discovery | | `meta.target_db` | yes | Registry ID of the database to run against | | `meta.timestamp` | no | ISO-8601 timestamp used for ordering when multiple migrations target the same database | | `meta.tags` | no | Array of tags; the runner can filter migrations by tag | Migrations for a database run in ascending `meta.timestamp` order. `meta.timestamp` is optional; the full entry id is the tie-breaker, so migrations with equal or absent timestamps still run in a stable, deterministic order. ### DSL Inside the function passed to `migration.define`, the following nested functions are available: | Function | Description | |----------|-------------| | `migration(description, fn)` | Open a new migration with a human-readable description | | `database(type, fn)` | Declare an implementation for `"sqlite"`, `"postgres"`, or `"mysql"` | | `up(fn)` / `down(fn)` | Define forward and rollback functions | | `after(fn)` | Optional post-migration hook (same transaction) | Each `up`/`down`/`after` function receives a transaction object, not a raw connection. All three operations run in a single transaction that rolls back on error. #### Transaction Methods ```lua local rows, err = db:query(sql, params) -- SELECT, returns array of rows local result, err = db:execute(sql, params) -- INSERT/UPDATE/DDL, returns { rows_affected, last_insert_id } local stmt, err = db:prepare(sql) -- prepared statement ``` Always use parameterised queries: ```lua db:execute("INSERT INTO users (name, email) VALUES (?, ?)", { "Alice", "alice@example.com" }) ``` #### Error Handling Calling `error(...)` aborts the migration and rolls back the transaction. Wrap every statement that may fail: ```lua up(function(db) local _, err = db:execute("CREATE TABLE ...") if err then error(err) end end) ``` ### Runner API The runner is exposed as a library for programmatic use: ```yaml imports: runner: wippy.migration:runner ``` ```lua local runner = require("runner").setup("app:app_db") local result = runner:run() -- apply all pending migrations local result = runner:run_next() -- apply the next pending migration local result = runner:rollback() -- roll back the most recently applied migration local status = runner:status() -- list applied + pending migrations ``` #### `runner:run(options)` Applies every pending migration for the configured database. Returns a summary: ```lua { status = "complete", -- "complete" or "error" migrations_found = 3, migrations_applied = 2, migrations_skipped = 1, migrations_failed = 0, duration = 0.123, migrations = { ... }, -- per-migration status skipped_details = { ... }, } ``` Options: | Option | Description | |--------|-------------| | `tags` | Array of tags; only migrations whose `meta.tags` intersect are considered | #### `runner:rollback(options)` Rolls back applied migrations in reverse order of application. With no options it reverts the single most recently applied migration: ```lua runner:rollback() -- roll back the last migration runner:rollback({ count = 3 }) -- roll back the last 3 runner:rollback({ allowed_ids = { "app:01_create_users_table" } }) -- restrict to specific ids ``` Options: | Option | Description | |--------|-------------| | `count` | Number of migrations to roll back; defaults to `1` | | `allowed_ids` | Array of migration ids; only these are eligible for rollback | #### `runner:status(options)` Returns a status report describing every migration for the database: ```lua { database_id = "app:app_db", db_type = "sqlite", total_migrations = 3, applied_migrations = 2, pending_migrations = 1, migrations = { { id = "app:01_...", description = "...", timestamp = "...", tags = {}, status = "applied", applied_at = ... }, -- ... }, } ``` Applied migrations are listed first (ordered by `applied_at`), followed by pending ones (ordered by `meta.timestamp`, then by id). ### Registry API `wippy.migration:registry` offers direct registry queries: | Function | Description | |----------|-------------| | `registry.find({ target_db, tags })` | Return all migration entries matching the criteria | | `registry.get(id)` | Return a single migration entry by id | | `registry.get_target_dbs()` | Return every unique `meta.target_db` present in migrations | | `registry.get_tags()` | Return every unique tag present on migrations | The bootloader uses these to discover the full set of target databases at startup. ### Migration Tracking The runner creates a `_migrations` table in each target database on first run. Applied migrations are recorded by id so subsequent runs skip them. The tracking table is created automatically; do not write your own migration to create it. ### Best Practices - **One logical change per migration** - create one table, add one column, create one index. - **Write a real `down`** - if rollback is impossible (data loss), document that and raise an error rather than silently succeeding. - **Prefer idempotency** - `CREATE TABLE IF NOT EXISTS` and `DROP TABLE IF EXISTS` survive reruns without special handling. - **Keep DDL and DML separate** - don't seed data in the same migration that creates a table when you can avoid it. - **Test both directions** - apply the migration, roll it back, and verify the schema matches the starting state. ### See Also - [SQL Driver](system/database.md) - Database resource configuration - [Bootloader](framework/bootloader.md) - Bootloader ordering and hooks - [Framework Overview](framework/overview.md) - Framework module usage --- # "Usage Tracking" ## Usage Tracking The `wippy/usage` module records LLM token consumption and provides aggregate queries by time interval, model, or user. It is the default implementation of the `wippy.llm:usage_tracker` contract, so calls made through the LLM module produce usage records automatically. This page is an API primer with reference snippets, not a standalone tutorial. The snippets assume an existing Wippy project, a configured SQL database, and `wippy/llm` when automatic tracking is required. Usage rows persist in the selected database; remove sample rows through your normal database-maintenance workflow when testing is complete. ### Setup Add the module to your project: ```bash wippy add wippy/usage wippy install ``` Declare the dependency and set `target_db` to the database that will store usage records: ```yaml version: "1.0" namespace: app entries: - name: app_db kind: db.sql.sqlite file: ./data/app.db - name: dep.usage kind: ns.dependency component: wippy/usage version: "*" parameters: - name: target_db value: app:app_db ``` When the application starts, `wippy/migration` runs the module's `01_create_token_usage_table` migration, which creates the `token_usage` table along with indexes on `user_id`, `context_id`, `model_id`, and `timestamp`. If you use the relative SQLite path shown above, create the `data` directory before starting the application. ### Schema ``` token_usage ├── usage_id text primary key (uuid v7) ├── user_id text not null ├── context_id text ├── model_id text not null ├── prompt_tokens integer ├── completion_tokens integer ├── thinking_tokens integer default 0 ├── cache_read_tokens integer default 0 ├── cache_write_tokens integer default 0 ├── timestamp timestamp └── meta text (JSON) ``` ### Automatic Tracking `wippy/llm` resolves the `wippy.llm:usage_tracker` contract before each generation. `wippy/usage` binds its implementation as the default: ```yaml contracts: - contract: wippy.llm:usage_tracker default: true methods: track_usage: wippy.usage:usage_tracker ``` Every successful LLM call invokes `track_usage` with the model id, token counts, and an optional `context_id`. The `user_id` is taken from the active security actor; calls outside of a user context are recorded as `"system"`. ### Tracker API Import the tracker directly to record usage outside the LLM flow: ```yaml imports: usage_tracker: wippy.usage:usage_tracker ``` ```lua local tracker = require("usage_tracker") -- Numeric counts supplied by the caller or model provider. local prompt_tokens, completion_tokens = 120, 40 local thinking_tokens = 0 local cache_read_tokens, cache_write_tokens = 0, 0 local usage_id, err = tracker.track_usage( "openai:gpt-4o", prompt_tokens, completion_tokens, thinking_tokens, cache_read_tokens, cache_write_tokens, { context_id = "chat-42", metadata = { feature = "summary" } } ) if err then error("Failed to record usage: " .. tostring(err)) end ``` | Parameter | Type | Description | |-----------|------|-------------| | `model_id` | string | Canonical model id | | `prompt_tokens` | number | Input tokens | | `completion_tokens` | number | Output tokens | | `thinking_tokens` | number | Reasoning tokens (0 when not reported) | | `cache_read_tokens` | number | Prompt-cache hits | | `cache_write_tokens` | number | Prompt-cache writes | | `options.context_id` | string | Free-form tag; falls back to `ctx.get("context_id")` | | `options.timestamp` | number | Unix timestamp; defaults to now (UTC) | | `options.metadata` | table | Arbitrary JSON metadata stored alongside the record | Returns `usage_id` or `nil, err`. ### Repository API `wippy.usage:token_usage_repo` offers aggregate queries: ```yaml modules: - time imports: usage: wippy.usage:token_usage_repo ``` ```lua local usage = require("usage") local time = require("time") -- Inclusive query bounds expressed as UNIX timestamps. local end_unix = time.now():unix() local start_unix = end_unix - (24 * 60 * 60) local function require_result(value, err) if err then error("Usage query failed: " .. tostring(err)) end return value end local summary = require_result(usage.get_summary(start_unix, end_unix)) local by_time = require_result(usage.get_usage_by_time(start_unix, end_unix, usage.INTERVAL.DAY)) local by_model = require_result(usage.get_usage_by_model(start_unix, end_unix)) local by_user = require_result(usage.get_usage_by_user(start_unix, end_unix)) ``` #### Functions | Function | Returns | |----------|---------| | `get_summary(start, end)` | Totals across the range: prompt/completion/thinking/cache tokens, request count, `total_tokens` (prompt + completion + thinking) | | `get_usage_by_time(start, end, interval)` | Array of buckets, one per interval; missing buckets return zeroes | | `get_usage_by_model(start, end)` | Per-model totals, ordered by `total_tokens` descending | | `get_usage_by_user(start, end)` | Per-user totals, ordered by `total_tokens` descending | | `create(user_id, model_id, prompt, completion, options)` | Low-level insert used by the tracker | #### Intervals ```lua usage.INTERVAL.HOUR -- "hour" usage.INTERVAL.DAY -- "day" usage.INTERVAL.WEEK -- "week" usage.INTERVAL.MONTH -- "month" ``` `get_usage_by_time` aligns buckets to the configured interval. On PostgreSQL it uses `generate_series` with interval arithmetic; on SQLite it uses a recursive CTE over UNIX timestamps. `total_tokens` in each bucket excludes cache tokens. #### Time Ranges Both the tracker and the repository accept UNIX timestamps at the public API boundary. Internally the repository converts to RFC3339 strings for storage and querying. Pass `os.time()` or `time.now():unix()` values, not formatted strings. ### Metadata and Context The `meta` column stores free-form JSON for correlating records with application events: ```lua local usage_id, err = tracker.track_usage("openai:gpt-4o", 120, 40, 0, 0, 0, { context_id = "chat-42", metadata = { session_id = "s-7", route = "/api/summarise", agent_id = "writer", }, }) if err then error("Failed to record usage metadata: " .. tostring(err)) end ``` `context_id` is a top-level column and can be indexed; `metadata` is stored as text and is intended for display, not filtering. ### See Also - [LLM](framework/llm.md) — LLM generation and the `usage_tracker` contract - [Migrations](framework/migration.md) — Migration runner that creates the schema - [Framework Overview](framework/overview.md) — Framework module usage --- # "Frontend Contract: Start Here" ## Frontend Contract: Start Here This page is an orientation guide and navigation reference. It identifies the contracts a frontend module must follow; it is not a build tutorial or a complete application example. Wippy frontend modules are portable by default. A module must continue to work when it is imported into another Wippy project whose facade supplies a different compliant PrimeVue theme and no project-private CSS. ### Choose the correct path 1. Use a `view.page` for an application rendered by the configured page engine: a legacy `about:srcdoc` iframe or a Web Fragment. 2. Use a `view.component` for a custom element rendered in the host document, normally with a shadow root. 3. If the UI renders a button, input, form field, menu, overlay, or another PrimeVue-like control, use PrimeVue unless it cannot provide the required semantics and affordance. 4. A content-only component, such as a Chart.js visualization with no controls, may omit PrimeVue and Tailwind. 5. If a custom control is necessary, follow the [Portable UI Contract](./portable-ui-contract.md) and [Custom Composites](./micro-frontends/custom-composites.md). PrimeVue is the shared component vocabulary. The Wippy Tailwind preset is a supported build-time vocabulary. Only utilities documented as runtime-backed remain responsive to facade theme changes after compilation. ### Ownership map ```text module source -> build command -> emitted artifact -> registry owner -> served URL -> Web Host -> page surface (srcdoc iframe or Web Fragment) or component shadow root -> AppConfig / router / theme delivery ``` Do not infer one stage from another. Before debugging a missing asset, identify the source package, build target, emitted file, registry entry, filesystem mount, and served URL. ### Contract pages - [Platform Topology](./platform-topology.md): runtime boundaries, routing, CSS delivery, overlays, and ownership. - [Portable UI Contract](./portable-ui-contract.md): normative component and styling rules. - [Theme Authoring](./micro-frontends/theming.md): what belongs in facade `custom_css`, PrimeVue theme CSS, or a module. - [Tailwind Contract](./micro-frontends/tailwind-contract.md): runtime-backed utilities versus compiled constants. - [Token Catalogue](./micro-frontends/token-catalogue.md): generated token reference and provenance. - [The Design Layer](./design-layer.md): where something belongs when several of your own modules need it and the theme has no component for it. - [Page Recipe](./micro-frontends/micro-frontend-app.md) and [Web Component Recipe](./micro-frontends/web-component.md). - [Build and Dependency Contract](./micro-frontends/build-system.md). - [Configuration and Casing](./micro-frontends/configuration-casing.md). - [Compliance Rule Index](./micro-frontends/compliance-checklist.md). ### Non-negotiable checks - Never invent a PrimeVue prop, component API, CSS variable, or Tailwind semantic utility. Verify it in the selected package source and generated catalogue. - Never construct a `--p-*` token name by analogy. - Never require an arbitrary facade class from a portable module. - Never infer host route context from browser location. Pages receive host context through AppConfig and use `@wippy-fe/router`. - Rebuild the exact owning package into the served output before browser verification. - Verify the browser console after navigation and material interaction. Project-bound modules are outside the portable contract. They are documented only on the [Unsupported Project-Bound Modules](./micro-frontends/unsupported-project-bound.md) page; standard compliance returns `UNSUPPORTED` and standard CI fails. --- # "Platform Topology" ## Platform Topology This page is an architecture and diagnostic reference. The delivery chain and diagrams describe system boundaries; they do not provide a runnable project. ### Delivery chain | Stage | Owner | Verification | |---|---|---| | Source and package build | Frontend module | The package build emits the expected entry file. | | Artifact location | Deployment build target | The build command receives `--outDir`; Vite does not hardcode it. | | Registry entry | Backend module | `view.page` or `view.component` points at the emitted entry. | | Served URL | Filesystem and HTTP registry entries | A direct asset request returns the built JavaScript or HTML. | | Runtime container | Web Host | A page uses the configured page engine: a legacy `about:srcdoc` iframe or a Web Fragment. A component uses a custom element, normally with shadow DOM. | | Context | AppConfig and Wippy packages | Routing, API access, and theme data arrive through supported packages. | The presence of source, a successful build, or a valid registry entry does not prove the next stage. Verify each boundary. ### Pages A `view.page` runs through one of two engines: a legacy `about:srcdoc` iframe or a Web Fragment. The global `hostConfig.renderEngine` setting selects the baseline; a page's `wippy.renderEngine` can follow it, opt out to `iframe`, or request `fragment` when the deployment supports it. Application code stays engine-agnostic. In neither engine is browser location the supported child-route contract. Use AppConfig and `@wippy-fe/router`; the package handles Wippy route integration. The `iframe` CSS injection currently provides default themed scrollbar styling. Its name is historical and broader than its present purpose. Keep it enabled for scrollbar consistency; do not describe it as a layout reset. ### Web components A `view.component` runs in the host document and normally owns a shadow root. CSS selectors do not cascade through a shadow boundary. The Web Host may deliver approved stylesheets and facade CSS into that root according to component configuration. CSS variable inheritance and stylesheet injection are different mechanisms: - Public inherited variables can cross the host-to-shadow boundary. - Selector rules affect a shadow root only when delivered into that root. - Delivery does not make an arbitrary selector a portable API. ### Theme and overlays The facade supplies the PrimeVue theme. Shared `.p-*` rules in facade `custom_css` are valid theme implementation and may be global when intended for host and children. Use `.wippy-host-app` only for host-specific chrome. Theme mode is AppConfig state, not a CSS-class API. Applications, components, fixtures, and browser tests switch mode with `host.setThemeMode('auto' | 'light' | 'dark')` from `@wippy-fe/proxy`, then wait for `@theme` and verify `host.getThemeMode()`. AppConfig carries the change through the host-to-child transport. The host updates its document, re-broadcasts AppConfig to live iframe and Web Fragment page realms, and mirrors the mode into web-component roots. Never force `w-theme-dark` or `w-theme-light` classes directly. PrimeVue overlays may be teleported. Verify the actual overlay root in the top document, iframe documents, and recursively discovered shadow roots. Do not assume generic PrimeVue placement. ### Runtime debugging order 1. Confirm the backend is listening. 2. Inspect backend logs for unexpected 5xx responses. 3. Confirm the registry owner and served asset URL. 4. Confirm the exact package build emitted that asset. 5. Load the host root before navigating through the SPA when direct deep links are unsupported. 6. Inspect console and network errors after navigation and interaction. 7. For theme scenarios, call the public proxy theme method, observe `@theme`, and verify `host.getThemeMode()` before accepting a screenshot. --- # "Portable UI Contract" ## Portable UI Contract This page is a normative contract reference. Its rule IDs define review and acceptance requirements rather than an implementation tutorial. The following IDs are the canonical owners of their rules. #### FE-PORT-001: Portable is the default A compliant module works with another compliant facade theme without module edits and without project-private facade classes. #### FE-STYLE-001: No private facade dependency Portable modules cannot require arbitrary classes or selectors defined only by one facade. Shared PrimeVue `.p-*` theme rules are not private classes. Non-PrimeVue styling required by one module belongs in that module, but should be minimized by conforming to shared components and semantics. When *several* of your own modules need the same non-PrimeVue styling, it belongs in neither the facade nor each module: see [The Design Layer](./design-layer.md). #### FE-UI-001: Use PrimeVue when it satisfies the control If PrimeVue provides the required semantics, interaction, and intended affordance, the module must use it. #### FE-UI-002: Data shape is not affordance The ability to represent the same values does not make two controls equivalent. A `SelectButton` is not automatically a substitute for a sliding three-position toggle when the intended affordance is visibly and behaviorally a toggle. #### FE-UI-003: Same semantics and affordance means same appearance Equivalent controls must share sizes, spacing, colors, typography, borders, shadows, focus, hover, disabled, invalid, and motion behavior. A custom composite names its PrimeVue visual sibling and inherits every applicable shared runtime property. #### FE-UI-004: PrimeVue omission is narrow PrimeVue may be omitted only when the module renders nothing that is physically or semantically PrimeVue-like. A chart-only component qualifies; a chart with a button or form field does not. #### FE-UI-005: Never invent component APIs An undocumented prop or behavior is not a shortcut. PrimeVue `ToggleSwitch` does not become a three-position control by inventing a new positions prop. When no PrimeVue component or composition supplies the required affordance, use the reviewed custom-sibling process. #### FE-TW-001: Wippy Tailwind is supported The shared Wippy preset is a supported build-time contract. Modules may use its documented utilities and extend it for domain layout, application-specific breakpoints, decoration, and novel visualization. #### FE-TW-002: Compiled values are not runtime tokens Utilities such as `px-3`, `rounded-md`, and `duration-200` normally compile to constants. They provide a consistent baseline but do not change when a facade swaps runtime theme variables. #### FE-TW-003: Shared sibling appearance tracks runtime semantics When an appearance property must track a PrimeVue sibling across themes, use a documented runtime-backed semantic utility or a direct public token. A fixed utility is allowed only when the property is explicitly classified `platform-invariant`. #### FE-TW-004: Protected mappings keep their meaning Modules may extend the preset but cannot redefine protected primary, surface, severity, text, content, highlight, or portable-control semantics incompatibly. #### FE-TOKEN-001: Every token must exist Every `--p-*` reference must be present in the selected generated manifest. #### FE-TOKEN-002: Token names are not guessable APIs Never construct a token by analogy. Search the [Token Catalogue](./micro-frontends/token-catalogue.md) or the selected package manifest. #### FE-A11Y-001: Custom is not an accessibility waiver A custom-control exception must preserve valid HTML, keyboard interaction, focus, accessible name, state, and disabled behavior. Interactive elements must not be nested. --- # "The Design Layer" ## The Design Layer This page is a design-ownership decision guide. Its CSS and component snippets are partial patterns that assume an existing Wippy frontend package and build. A Wippy frontend can contain many independently published modules in one application. The **theme** reaches every surface, while each **module** owns its local presentation. A **shared design layer** covers the narrower case where several modules share a concept that the theme does not provide. ### The layers | Layer | Reaches | Owns | |---|---|---| | **Theme** | *Every* surface, including modules you do not own | PrimeVue components, the shared semantic tokens, documented classes | | **Shared design layer** | Only the modules that opt in | Vocabulary those modules share that has no themed component behind it | | **Module** | Itself | What is genuinely specific to one surface | #### The theme is universal, and that is the constraint The theme styles markup **you do not own**. Any module — including a third-party plugin written by someone who has never seen your app — renders into the same host and is painted by the same theme. That is what makes the theme the universal layer, and it cuts both ways: **Nothing app-specific may go into the theme**, because it would be imposed on every module that never asked for it. **A module may not depend on anything app-specific being in the theme.** The contract is *PrimeVue components + the shared Wippy semantic tokens + documented classes* — nothing an application added on top. Note that PrimeVue's own presets are not the contract either: Wippy runs PrimeVue with `theme: 'none'`, so it is the Wippy semantic tokens you rely on. ```css /* GOOD — shared Wippy semantic tokens, present for every module */ .my-panel { color: var(--p-text-color); background: var(--p-content-background); border: 1px solid var(--p-content-border-color); } /* BAD — an application-specific token. Your module now only works inside one app, and silently loses the declaration anywhere else: an undefined custom property makes the declaration invalid at computed-value time, so it drops and the element quietly inherits instead. */ .my-panel { background: var(--kx-surface-2); } ``` This is also the answer to *"can I put our shared vocabulary in the facade?"* Only if it must genuinely reach arbitrary, unowned markup. If it is scoped to *your* set of modules, it does not belong in the theme — it belongs in the layer below. #### The backbone, and when a component may opt out PrimeVue and Tailwind, as shipped by the host, are the recommended backbone for any component. A component **may** opt out — but the opt-out narrows the moment it renders anything conventional, and the ladder only goes one way: | The component… | Then it must load | |---|---| | is presentation-neutral — canvas, SVG, a chart with no controls, no tokens, no utilities, no scrolling | nothing: `hostCssKeys: []` | | consumes semantic tokens or dark mode | `themeConfigUrl` | | can scroll | `iframeCssUrl` | | renders markdown | `markdownCssUrl` | | chooses Tailwind utilities for routine layout or spacing | `primeVueCssUrl` (the Host bundles Tailwind with this asset) | | renders anything **PrimeVue** ships a component for — button, input, form, table, dialog, menu, tag, tooltip, any feedback control | `primeVueCssUrl` **and** `PrimeVuePlugin` | A chart on a canvas is the archetypal legitimate opt-out: it has no classic UI, so it needs none of the backbone. Give that same chart a toolbar and it is no longer presentation-neutral — the button is a PrimeVue button, and the whole integration comes with it. Note the coupling: **Tailwind utilities are delivered with `primeVueCssUrl`.** There is no separate Tailwind host CSS key, so in practice a component that chooses Tailwind is loading the PrimeVue asset too. Prefer utilities for ordinary layout and spacing when they keep the component clear, but portable module-owned CSS remains valid when a utility is not the best expression of the design. (`preflightCssUrl` is not part of the key union; if Tailwind preflight is genuinely required inside the shadow root, load it imperatively — rarely needed.) The practical consequence for this page: **most of what a module wants already exists in the backbone.** The shared design layer is a narrow band above it, not a place to re-do what PrimeVue and Tailwind already cover. See [CSS Injection](./web-host/css-injection.md) for the mechanics. #### The shared design layer Some ideas recur across a known set of modules and have no application-level contract in the theme: a domain-specific match summary, a surface header row, an empty state, or a project-specific tag-sizing vocabulary. These concepts belong in the shared design layer. They ship as a **published package**, materialized into each consumer at build time. It must be a package rather than a path alias because consumers live in different repositories. A module in another repository, with no path access to the producer, must be able to consume the vocabulary and build. The producing module declares the package as a **build-time artifact** and each consumer materializes it into its own tree. See [Build-time Artifacts](../guides/artifacts.md) for the declaration, the `node-package` format, what the runtime reconciles for you, and the glue a build still has to supply itself. #### The module Everything else, plus every deliberate divergence from the shared vocabulary. ### Deciding where something belongs Ask in order. First yes wins. 1. **Is it a value?** Colour, radius, spacing, elevation, severity. → **Theme.** Read a semantic token. Never a literal. 2. **Does the theme already ship a component for this?** Button, Dialog, Select, Tag. → **Theme.** Use the component. Style it by putting a class *on* it — never rebuild it. 3. **Do two or more of your modules need this same concept, with no themed component behind it?** → **Shared design layer.** 4. Otherwise → **Module.** ### Worked examples The examples use the `kx-` prefix for application-specific classes and stylesheet names. The placement rules apply to any Wippy application. #### Never rebuild a themed component PrimeVue ships `Button`. Replacing it with `.kx-btn` on a native `

    disconnected

    SymbolPrice
    ``` ### Running Initialize the lock, run the migration to completion, then start the long-running services. Running the migration as a separate command prevents the token endpoint from racing the table creation. ```bash mkdir -p data wippy init wippy run -x app:migrate wippy run ``` Open `http://127.0.0.1:8081` and enter the demo API key from the migration log. The page should show `Connected as demo`, followed by BTC, ETH, and SOL prices that update once per second. You can also verify the exchange before opening the browser: ```bash curl -X POST http://127.0.0.1:8081/auth/token \ -H "Content-Type: application/json" \ -d '{"api_key":""}' ``` In PowerShell: ```powershell Invoke-RestMethod -Method Post ` -Uri http://127.0.0.1:8081/auth/token ` -ContentType 'application/json' ` -Body '{"api_key":""}' ``` A successful response contains `token`, `user_id: "demo"`, `role: "user"`, and `expires_in: 3600`. An invalid key returns HTTP 401. ### Troubleshooting and Cleanup - `no such table: api_keys` means the migration command was skipped or failed. Stop the runtime and rerun `wippy run -x app:migrate` before starting it again. - A 401 from `/auth/token` means the API key does not match the row in `data/auth.db`. Reset the database if the one-time log value was lost. - A 401 or immediate close on the WebSocket usually means the query parameter was removed or the in-memory token store was reset by a runtime restart. Exchange the API key again after every restart. - An origin rejection means the browser URL does not exactly match `http://127.0.0.1:8081`; use that URL or update both origin options together. - Stop the runtime with Ctrl+C. Delete `data/auth.db` to remove the demo API key. ### Next Steps - [WebSocket Relay](http/websocket-relay.md) — Middleware configuration - [Security Module](lua/security/security.md) — Actors, policies, and token stores - [Process Management](lua/core/process.md) — Process spawning and messaging --- # "Frontend Facade" ## Frontend Facade Use `wippy/facade` to serve the Wippy Web Host from a backend application. The facade loads the frontend bundle from a CDN and configures it through a JSON endpoint served by the application, without requiring a frontend build step. Dependency parameters control branding, theming, and feature flags. **Classification:** Partial integration recipe. It completely configures and verifies the facade shell and config endpoint, but it does not invent an authentication system or the application APIs consumed by the Web Host. ### What You'll Build A backend app that serves the Wippy UI: 1. An HTTP server and public router. 2. A `wippy/facade` dependency connected to the server and router, with custom branding. 3. The facade shell at `/` and its configuration at `/api/public/facade/config`. ### Prerequisites - Wippy runtime `v0.3.32a` and a project created with `wippy init` or the [Wippy application template](https://github.com/wippyai/app). - For browser rendering, a same-origin login flow that obtains a real backend token and stores `{"token":"..."}` under the localStorage key `@wippy_token_info`. The facade does not issue or validate that token. - The facade installed: ```bash wippy add wippy/facade@0.6.37 wippy install ``` ### How It Works 1. The shell is rendered from the facade's template and served at `/` by your HTTP server; its assets and the deep-link fallback come from a static mount on the same server. 2. On load it fetches `GET /api/public/facade/config`. 3. It reads `@wippy_token_info` from `localStorage`, redirecting to `login_path` only when the item is absent or cannot be parsed as JSON. 4. It imports the Web Host bundle from the CDN (`facade_url + '/module.js'`) and calls `initWippyApp(...)` with the config. The application serves the shell and its configuration; the UI bundle comes from the CDN. ### Dependencies The facade requires an `http.service` for the shell and an `http.router` for its configuration endpoint. Other parameters customize branding and behavior. ```yaml version: "1.0" namespace: app entries: - name: gateway kind: http.service addr: ":8087" lifecycle: auto_start: true - name: api.public kind: http.router meta: server: app:gateway prefix: /api/public - name: facade kind: ns.dependency component: wippy/facade version: "*" parameters: - name: server value: app:gateway - name: router value: app:api.public - name: app_title value: Verify App ``` The shell requests its config, theme script, and CSS variables under `/api/public/facade/`, so the public router's prefix must be `/api/public`. ### Run It ```bash wippy run ``` The shell is served at the server root, and the config endpoint returns the runtime configuration: ```bash curl http://localhost:8087/api/public/facade/config ``` Selected fields from the response are shown below: ```json { "facade_url": "https://web-host.wippy.ai/webcomponents-1.0.58", "iframe_origin": "https://web-host.wippy.ai", "iframe_url": "https://web-host.wippy.ai/webcomponents-1.0.58/iframe.html?waitForCustomConfig", "module_file": "/module.js", "mode": "compat", "login_path": "/login.html", "themeMode": "auto", "themePersist": "none", "themeStorageKey": "@wippy-theme-mode", "env": { "APP_API_URL": "", "APP_AUTH_API_URL": "", "APP_WEBSOCKET_URL": "" }, "themeMode": "auto", "themePersist": "none", "themeStorageKey": "@wippy-theme-mode", "theming": { "host": { "i18n": { "app": { "title": "Verify App", "icon": "wippy:logo", "appName": "Wippy AI" } } } }, "hostConfig": { "showAdmin": true, "allowSelectModel": false, "hideNavBar": false, "disableRightPanel": false, "startNavOpen": false, "hideSessionSelector": false, "renderEngine": "iframe", "session": { "type": "non-persistent" }, "history": "hash" } } ``` The `app_title` parameter appears as `theming.host.i18n.app.title` in the response. Also fetch the root document: ```bash curl http://localhost:8087/ ``` It should return an HTML shell that fetches the config endpoint and checks `@wippy_token_info`. These two HTTP checks verify the recipe without bypassing auth. ### Browser Authentication and Rendering The facade's localStorage contract is origin-scoped. A login page on another port or hostname cannot populate the token for `http://localhost:8087`. After a successful same-origin token exchange, the login page writes the real token and returns to the shell: ```js localStorage.setItem('@wippy_token_info', JSON.stringify({token: result.token})); window.location.assign('/'); ``` The shell reads the token, imports `https://web-host.wippy.ai/webcomponents-1.0.56/module.js`, and passes the token to the Host. Rendering is complete only when the browser shows the Host without redirecting and its API requests authenticate successfully. Do not use a placeholder token merely to suppress the redirect: the shell does not validate it, so the failure only moves to the first protected API request. ### Configuration Parameters are passed as dependency `parameters` (values are strings; JSON values are JSON-encoded strings). Common ones: | Parameter | Purpose | |---|---| | `server` / `router` | _(required)_ HTTP server and public router | | `app_title` / `app_name` / `app_icon` | Branding (icon is an Iconify ref) | | `show_admin` / `hide_nav_bar` | Feature flags (`"true"` / `"false"`) | | `login_path` | Where the shell redirects when no auth token is present | | `session_type` | `non-persistent` or `cookie` | | `history_mode` | `hash` or `browser` | | `css_variables` | JSON string of CSS custom properties, e.g. `'{"--p-primary":"#6366f1"}'` | | `fe_facade_url` | CDN bundle URL (pinned per facade release; leave default unless overriding) | Two values are derived at runtime from `PUBLIC_API_URL` rather than parameters: the API base URL and the WebSocket URL (`http`→`ws`, `https`→`wss`). The facade reads it through the env registry, so declare it as an `env.variable` in your app. If unset, the browser falls back to `window.location.origin`. ### Limitations - The facade does not provide authentication. It expects an auth flow that writes a token to `localStorage`; without one it redirects to `login_path`. Pair it with `userspace/users` or your own auth. - The UI bundle loads from the CDN (`fe_facade_url`), so the user's browser must be able to reach that URL. ### Troubleshooting - A redirect loop to `/login.html` means the current origin has no parseable `@wippy_token_info` item. Complete the real login flow on the same origin. A parseable object with a missing or empty `token` suppresses this redirect but still fails when the Host reaches a protected API. - HTTP 404 from `/api/public/facade/config` means the router prefix is not `/api/public` or the `router` dependency parameter points at another entry. - A config response with the right values but a blank shell usually means the browser cannot load `facade_url + module_file`; check the browser network panel and CDN policy. - Authenticated API errors after the Host renders belong to the application's API and token validation layer, not to the facade shell. ### Next Steps - [Hello World](tutorials/hello-world.md) — Minimal project layout - [Authentication](tutorials/auth.md) — Add the login flow expected by the shell - [HTTP Endpoints](http/endpoint.md) — Routers, static files, and handlers --- # "Network Overlays" ## Network Overlays Configure a SOCKS5 overlay for outbound HTTP calls, then review inheritance, inbound listeners, application defaults, and permissions. **Classification:** Runnable SOCKS5 tutorial with a partial Tailscale recipe. The direct/Tor probe is complete once an external Tor listener is available. The Tailscale section explains Wippy wiring but intentionally defers account provisioning to Tailscale. For I2P configuration, use the network-system reference linked below. ### Overview Wippy represents overlay networks as registry entries. Code can select an overlay for a call, and that selection propagates to nested calls until a descendant overrides it. Wippy supports three overlay entry kinds: - `network.socks5` — generic SOCKS5 proxy (also Tor's SOCKS5 listener) - `network.tailscale` — tsnet overlay node - `network.i2p` — I2P SAM v3 bridge ### Prerequisites - Wippy runtime `v0.3.32a`. - `curl` and outbound HTTPS access to `api.ipify.org`. - A Tor daemon exposing SOCKS5 on `127.0.0.1:9050`. Install a supported package from the [Tor Project download page](https://www.torproject.org/download/tor/), start it, and verify the listener before running Wippy: ```bash curl --socks5-hostname 127.0.0.1:9050 https://api.ipify.org?format=json ``` A successful check returns JSON containing an IP address. Tor Browser commonly uses port 9150 instead; if that is the listener you are intentionally using, change the registry entry and the verification command together. - An empty working directory: ```bash mkdir netdemo cd netdemo mkdir src ``` ### Project Structure ``` netdemo/ ├── wippy.lock └── src/ ├── _index.yaml └── probe.lua ``` ### Step 1: Define an Overlay Create `src/_index.yaml`: ```yaml version: "1.0" namespace: app entries: - name: net_policy kind: security.policy policy: actions: - http_client.request - network.select resources: "*" effect: allow - name: processes kind: process.host lifecycle: auto_start: true - name: terminal kind: terminal.host lifecycle: auto_start: true # SOCKS5 proxy entry (Tor exposes one at 127.0.0.1:9050 by default) - name: tor kind: network.socks5 host: 127.0.0.1 port: 9050 isolate_streams: true - name: probe kind: process.lua meta: command: name: probe short: Check outbound IP through overlays security: actor: id: system.probe policies: - app:net_policy source: file://probe.lua method: main modules: - io - http_client - json ``` With `isolate_streams: true`, the SOCKS5 driver creates random credentials for each connection so Tor can open a fresh circuit for each dial. Security is strict by default, so the command carries the actor and policy its launch runs under. `http_client.request` covers the outbound call and `network.select` covers the explicit overlay choice; without them every check fails closed. ### Step 2: Route Outbound Calls Create `src/probe.lua`: ```lua local io = require("io") local http_client = require("http_client") local json = require("json") local function fetch_ip(overlay) local options = { timeout = "15s" } if overlay then options.overlay_network = overlay end local resp, err = http_client.get("https://api.ipify.org?format=json", options) if err then return nil, tostring(err) end if resp.status_code ~= 200 then return nil, "HTTP " .. resp.status_code end local body = json.decode(resp.body or "") return body and body.ip, nil end local function main() local direct, d_err = fetch_ip(nil) if d_err then io.print("direct failed: " .. d_err) else io.print("direct IP: " .. direct) end local routed, r_err = fetch_ip("app:tor") if r_err then io.print("tor failed: " .. r_err) else io.print("tor IP: " .. routed) end return 0 end return { main = main } ``` The `overlay_network` option selects the overlay for that HTTP call. Without it, the dial uses the process default: `network_service.default_network` from `.wippy.yaml`, or a direct connection when no default is set. ### Step 3: Run It ```bash wippy init wippy run probe ``` With Tor running locally: ``` direct IP: tor IP: ``` Both lines must contain valid IP addresses. They should normally differ; the important proof is that the routed request succeeds only through the configured SOCKS listener. If Tor is not running, the `tor IP` line will report a dial error — the SOCKS5 overlay does not silently fall back to a direct connection. ### Inheritance Overlay selection propagates through nested calls. Selecting an overlay at a `funcs.call` or `process.spawn` boundary applies it to nested HTTP calls, function calls, and process spawns until one explicitly overrides it: ```lua local funcs = require("funcs") local result, err = funcs.new() :with_options({ network = "app:tor" }) :call("app:scrape_site", url) ``` ```lua local pid, err = process.with_options({ network = "app:tor" }) :spawn_monitored("app.workers:probe", "app:processes") ``` The nested function or spawned process sees the overlay on every outgoing dial without passing it explicitly. ### Binding a Listener Tailscale can also accept HTTP listeners. Attach the overlay to the `http.service` instead of the client: ```yaml - name: bind_policy kind: security.policy policy: actions: "network.bind" resources: "*" effect: allow - name: tailnet kind: network.tailscale hostname: wippy-node auth_key_env: TS_AUTHKEY ephemeral: true - name: gateway kind: http.service addr: ":8080" network: app:tailnet lifecycle: auto_start: true security: actor: id: system.gateway policies: - app:bind_policy ``` `auth_key` resolves through the [env registry](system/env.md), so `TS_AUTHKEY` is a registered variable — an OS value needs an `env.variable` backed by `env.storage.os`. Binding through an overlay is gated by `network.bind`, checked when the listener starts, so the service declares a scope that allows it. The server binds on the tailnet interface; clients reach it via the Tailscale address. SOCKS5 is outbound-only — assigning it to `http.service` fails the listener with `inbound listeners are not exposed over SOCKS5`. ### App-wide Default Set a default overlay in `.wippy.yaml` so every call uses it unless overridden: ```yaml network_service: state_dir: .wippy/net default_network: app:tor ``` ### Permissions The `network.select` action gates explicit overlay selection. Deny it on a scope to stop code from choosing an overlay: ```yaml - name: deny_network kind: security.policy policy: actions: "network.select" resources: "*" effect: deny groups: - untrusted ``` Inherited overlays bypass this check — they were authorized at the caller's edge. Only explicit re-selection at a Lua boundary is gated. ### Troubleshooting and Cleanup - `connection refused` on `127.0.0.1:9050` means Tor is not listening on the configured port. Verify Tor with the prerequisite `curl` command before debugging Wippy. - A direct request failure and a routed success usually indicate local DNS, proxy, or firewall rules affecting the direct path. The two calls are independent. - `access denied` for the routed call means the command security context lacks `network.select` for `app:tor`; keep `app:probe_policy` attached under `meta.command.security`. - The SOCKS5 driver never falls back to a direct connection. Do not remove the error merely to make the demo continue. - Stop the Wippy command when it exits and stop the Tor daemon only if you started it solely for this tutorial. The SOCKS5 example creates no persistent network state. A Tailscale entry can persist node state under `.wippy/net/tailscale/`; remove the `.wippy/net` state directory only after stopping Wippy and only when you intend to discard that local tailnet identity. ### Next Steps - [Network System](system/network.md) — Entry-kind reference - [HTTP Client](lua/http/client.md) — Per-call overlay options - [Security Model](system/security.md) — Policies and scopes - [Authentication](tutorials/auth.md) — Token-based security --- # "Running Rust on Wippy" ## Running Rust on Wippy Build a Rust WebAssembly component, register it with Wippy, and expose it through function, CLI, and HTTP entries. **Classification:** Runnable tutorial with an external Rust component toolchain. The page supplies the WIT, Rust implementation, Wippy registry, integrity-hash workflow, commands, expected results, and failure checks. ### What We're Building A Rust component with four exported functions: - **greet** — Accepts a name and returns a greeting - **add** — Adds two integers - **fibonacci** — Computes the nth Fibonacci number - **list-files** — Lists files in a mounted directory The Wippy application registers these exports as callable functions, CLI commands, and an HTTP endpoint. ### Prerequisites - Wippy runtime `v0.3.32a`. - [Rust toolchain](https://rustup.rs/) with the `wasm32-wasip1` target. - A working C toolchain. On Linux, `cargo-component` also requires OpenSSL development libraries. - [cargo-component 0.21.1](https://github.com/bytecodealliance/cargo-component/releases/tag/v0.21.1), the release used by this tutorial. ```bash rustup target add wasm32-wasip1 cargo install cargo-component --version 0.21.1 --locked ``` Create the generated component scaffold and Wippy directories: ```bash mkdir rust-wasm-demo cd rust-wasm-demo cargo component new --lib demo mkdir -p app/src/demo/wasm ``` In PowerShell: ```powershell New-Item -ItemType Directory -Path rust-wasm-demo Set-Location rust-wasm-demo cargo component new --lib demo New-Item -ItemType Directory -Path app\src\demo\wasm -Force ``` `cargo component new` writes a compatible `Cargo.toml`, `src/lib.rs`, WIT file, and later regenerates `src/bindings.rs`. Keep the generated `wit-bindgen-rt` version paired with the installed `cargo-component`; the tool describes that interface as experimental and does not guarantee generated-code compatibility across versions. ### Project Structure ``` rust-wasm-demo/ ├── demo/ # Rust component │ ├── Cargo.toml │ ├── wit/ │ │ └── world.wit # WIT interface │ └── src/ │ ├── bindings.rs # generated by cargo-component │ └── lib.rs # implementation └── app/ # Wippy application ├── wippy.lock └── src/ ├── _index.yaml # Infrastructure └── demo/ ├── _index.yaml # CLI processes └── wasm/ ├── _index.yaml # WASM entries └── demo_component.wasm # Compiled binary ``` ### Step 1: Create the WIT Interface WebAssembly Interface Types (WIT) defines the contract between the host and guest component. Create `demo/wit/world.wit`: ```wit package component:demo; world demo { export greet: func(name: string) -> string; export add: func(a: s32, b: s32) -> s32; export fibonacci: func(n: u32) -> u64; export list-files: func(path: string) -> string; } ``` Each export becomes a function that Wippy can call. ### Step 2: Implement in Rust Keep the generated `demo/Cargo.toml`. Its package metadata must target `component:demo`, matching the WIT package, and its library crate type must remain `cdylib`. Create `demo/src/lib.rs`: ```rust #[allow(warnings)] mod bindings; use bindings::Guest; struct Component; impl Guest for Component { fn greet(name: String) -> String { format!("Hello, {}!", name) } fn add(a: i32, b: i32) -> i32 { a + b } fn fibonacci(n: u32) -> u64 { if n <= 1 { return n as u64; } let (mut a, mut b) = (0u64, 1u64); for _ in 2..=n { let next = a + b; a = b; b = next; } b } fn list_files(path: String) -> String { let mut result = String::new(); match std::fs::read_dir(&path) { Ok(entries) => { for entry in entries { match entry { Ok(e) => { let name = e.file_name().to_string_lossy().to_string(); let meta = e.metadata(); let (kind, size) = match meta { Ok(m) => { let kind = if m.is_dir() { "dir" } else { "file" }; (kind, m.len()) } Err(_) => ("?", 0), }; let line = format!("{:<6} {:>8} {}", kind, size, name); println!("{}", line); result.push_str(&line); result.push('\n'); } Err(e) => { let line = format!("error: {}", e); eprintln!("{}", line); result.push_str(&line); result.push('\n'); } } } } Err(e) => { let line = format!("cannot read {}: {}", path, e); eprintln!("{}", line); result.push_str(&line); result.push('\n'); } } result } } bindings::export!(Component with_types_in bindings); ``` The `bindings` module is generated by `cargo-component` from the WIT definition. ### Step 3: Build the Component ```bash cd demo cargo component build --release ``` This produces `target/wasm32-wasip1/release/demo.wasm`. Copy it to your Wippy app: ```bash mkdir -p ../app/src/demo/wasm cp target/wasm32-wasip1/release/demo.wasm ../app/src/demo/wasm/demo_component.wasm ``` In PowerShell: ```powershell New-Item -ItemType Directory -Path ..\app\src\demo\wasm -Force Copy-Item -LiteralPath target\wasm32-wasip1\release\demo.wasm ` -Destination ..\app\src\demo\wasm\demo_component.wasm ``` Get the SHA-256 hash for integrity verification: ```bash sha256sum ../app/src/demo/wasm/demo_component.wasm ``` On PowerShell, use: ```powershell (Get-FileHash ..\app\src\demo\wasm\demo_component.wasm -Algorithm SHA256).Hash.ToLowerInvariant() ``` Copy the 64 lowercase hexadecimal characters into every `YOUR_HASH_HERE` below. The final field must have the form `sha256:<64-hex-characters>`; it is the hash of the copied binary, not the Rust source or the original build path. #### Infrastructure Create `app/src/_index.yaml`: ```yaml version: "1.0" namespace: demo entries: - name: gateway kind: http.service meta: comment: HTTP server addr: ":8090" lifecycle: auto_start: true - name: api kind: http.router meta: comment: Public API router server: demo:gateway prefix: / - name: processes kind: process.host lifecycle: auto_start: true - name: terminal kind: terminal.host lifecycle: auto_start: true - name: policy kind: security.policy meta: comment: Grants access to mounted filesystems and WASM functions policy: actions: - fs.get - funcs.call resources: "*" effect: allow ``` Mounting a filesystem into a WASM module and calling a WASM function are both guarded actions. The policy grants them; entries that need them reference it. #### WASM Functions Create `app/src/demo/wasm/_index.yaml`: ```yaml version: "1.0" namespace: demo.wasm entries: - name: assets kind: fs.directory meta: comment: Filesystem with WASM binaries directory: ./src/demo/wasm - name: greet_function kind: function.wasm meta: comment: Greet function via payload transport fs: demo.wasm:assets path: /demo_component.wasm hash: sha256:YOUR_HASH_HERE method: greet pool: type: inline - name: add_function kind: function.wasm meta: comment: Add function via payload transport fs: demo.wasm:assets path: /demo_component.wasm hash: sha256:YOUR_HASH_HERE method: add pool: type: inline - name: fibonacci_function kind: function.wasm meta: comment: Fibonacci function via payload transport fs: demo.wasm:assets path: /demo_component.wasm hash: sha256:YOUR_HASH_HERE method: fibonacci pool: type: inline ``` Key points: - A single `fs.directory` entry provides the WASM binary. - Multiple functions reference the same binary with different `method` values. - The `hash` field verifies binary integrity at load time. - The `inline` pool serializes calls through one warm instance. It resets per-call execution state between synchronous calls; use another pool type when you need concurrent workers. #### Functions with WASI The `list-files` function accesses the filesystem, so it needs WASI imports: ```yaml - name: list_files_function kind: function.wasm meta: comment: Filesystem listing with WASI mounts fs: demo.wasm:assets path: /demo_component.wasm hash: sha256:YOUR_HASH_HERE method: list-files imports: - wasi:cli - wasi:io - wasi:clocks - wasi:filesystem wasi: mounts: - fs: demo.wasm:assets guest: /data pool: type: inline ``` The `wasi.mounts` section maps a Wippy filesystem entry to a guest path. Inside the WASM module, `/data` points to the `demo.wasm:assets` directory. #### CLI Commands Create `app/src/demo/_index.yaml`: ```yaml version: "1.0" namespace: demo.cli entries: - name: wasm_cli_policy kind: security.policy policy: actions: - fs.get resources: - demo.wasm:assets effect: allow - name: ls kind: process.wasm meta: comment: List files from mounted WASI filesystem command: name: ls short: List files from mounted directory security: actor: {id: demo.cli:ls} policies: [demo:policy] fs: demo.wasm:assets path: /demo_component.wasm hash: sha256:YOUR_HASH_HERE method: list-files imports: - wasi:cli - wasi:io - wasi:clocks - wasi:filesystem wasi: mounts: - fs: demo.wasm:assets guest: /data ``` The `meta.command` block registers the process as a named CLI command. The `greet` command needs no WASI imports since it only uses string operations. The `ls` command needs filesystem access, so it also carries the security context that grants the mount. #### HTTP Endpoint Add to `app/src/demo/wasm/_index.yaml`: ```yaml - name: http_greet kind: function.wasm meta: comment: Greet exposed via wasi-http transport fs: demo.wasm:assets path: /demo_component.wasm hash: sha256:YOUR_HASH_HERE method: greet transport: wasi-http pool: type: inline - name: http_greet_endpoint kind: http.endpoint meta: comment: HTTP POST endpoint for WASM greet router: demo:api method: POST path: /greet func: http_greet ``` The `wasi-http` transport maps HTTP request/response context to WASM arguments and results. ### Step 5: Initialize and Run ```bash cd app wippy init ``` #### Run CLI Commands ```bash ## List available commands wippy run list ``` ``` Available commands: greet Greet someone via WASM (demo.cli:greet) ls List files from mounted directory (demo.cli:ls) Run with: wippy run ``` Arguments after the command name are passed to the exported function as string parameters, so each command takes exactly the arguments its WIT signature declares: ```bash ## Run greet wippy run greet World ``` ``` Hello, World! ``` ```bash ## Run ls to list mounted directory wippy run ls /data ``` The command should print at least `demo_component.wasm` with its file size and exit with status 0. Wippy does not print arbitrary `process.wasm` return payloads, which is why the CLI example uses the Rust function that writes to WASI stdout. #### Run as a Service ```bash wippy run ``` This starts the HTTP server on port 8090. The `wasi-http` transport passes the request body as the function's single string argument: ```bash curl -X POST http://localhost:8090/greet -d 'World' ``` ``` Hello, World! ``` #### Call from Lua WASM functions are called the same way as Lua functions. The calling process needs `funcs.call` on the target, which `demo:policy` grants: ```lua local funcs = require("funcs") local greeting, err = funcs.call("demo.wasm:greet_function", "World") -- greeting: "Hello, World!" local sum, err = funcs.call("demo.wasm:add_function", 6, 7) -- sum: 13 local fib, err = funcs.call("demo.wasm:fibonacci_function", 10) -- fib: 55 ``` ### Troubleshooting and Cleanup - If `cargo component` is unknown, install it and rerun `cargo component build`; plain `cargo build` does not generate the same bindings/component output for this setup. - A missing `src/bindings.rs` before the first build is expected. A missing file after `cargo component build` indicates the WIT package or component metadata could not be resolved; fix that build error before copying a binary. - `WASM hash mismatch` means the binary changed after the documented digest was calculated or one placeholder remains. Recopy the release binary, recompute the digest, and update every entry that references it. - An import-instantiation error means the component imports a host profile omitted by the entry. Keep the documented `wasi:cli`, `wasi:io`, `wasi:clocks`, and `wasi:filesystem` imports on the filesystem examples. - `cannot read /data` means the `wasi.mounts` guest path or its filesystem entry does not match the registry. - Stop the HTTP runtime with Ctrl+C. Rust build output remains under `demo/target/`; remove that directory and the copied `.wasm` file to clean generated artifacts. ### Next Steps - [WASM Overview](wasm/overview.md) — WebAssembly runtime overview - [WASM Functions](wasm/functions.md) — Function configuration reference - [WASM Processes](wasm/processes.md) — Process configuration reference - [Host Functions](wasm/hosts.md) — Available WASI imports - [CLI Reference](guides/cli.md) — CLI command documentation --- # "LLM Agent" ## LLM Agent Build a terminal chat agent in five phases, from a single LLM call to streaming responses and tool execution. **Classification: runnable tutorial with an external provider.** Each phase is a cumulative edit to the same project and is runnable before you continue. The Wippy contracts and local control flow are testable without credentials; generation requires network access and a valid `OPENAI_API_KEY`. ### What We're Building A terminal chat agent that: - Generates text with an LLM. - Maintains multi-turn conversations. - Streams responses incrementally. - Calls registered tools. ### Project Structure ``` llm-agent/ ├── wippy.lock └── src/ ├── _index.yaml ├── ask.lua ├── chat.lua └── tools/ ├── _index.yaml ├── current_time.lua └── calculate.lua ``` ### Phase 1: Simple Generation Start with a basic function that calls `llm.generate()` with a string prompt. Start in a Wippy project whose source directory is `./src`. Set `OPENAI_API_KEY` in the environment that starts Wippy. This tutorial declares its model explicitly; do not also copy a second entry with the same model name from another application. #### Entry Definitions Create `src/_index.yaml`: ```yaml version: "1.0" namespace: app entries: - name: policy kind: security.policy policy: actions: "*" resources: "*" effect: allow - name: os_env kind: env.storage.os - name: processes kind: process.host lifecycle: auto_start: true - name: dep.llm kind: ns.dependency component: wippy/llm version: "*" parameters: - name: env_storage value: app:os_env - name: process_host value: app:processes - name: dep.terminal kind: ns.dependency component: wippy/terminal version: "*" - name: ask kind: process.lua meta: command: name: ask short: Ask a single question security: actor: id: app:ask policies: - app:policy source: file://ask.lua method: main modules: - io imports: llm: wippy.llm:llm ``` The LLM module needs two infrastructure entries: - `env.storage.os` provides API keys from environment variables. - `process.host` provides the process runtime used internally by the LLM module. The `wippy/terminal` dependency provides the `terminal.host` that commands execute on and where `io.print` writes. `meta.command` gives the process a name so `wippy run ask` launches it with the remaining arguments as string payloads. Its `security` block installs the actor and policy scope for that launch: the LLM module resolves models from the registry, and a command launched without a scope reads nothing from it. #### Generation Code Create `src/ask.lua`: ```lua local io = require("io") local llm = require("llm") local function main(input) local response, err = llm.generate(input, { model = "gpt-4.1-nano", temperature = 0.7, max_tokens = 512, }) if err then io.print("Error: " .. tostring(err)) return 1 end io.print(response.result) return 0 end return { main = main } ``` #### Model Definition The LLM module resolves models from the registry. Add a model entry to `_index.yaml`: ```yaml - name: gpt-4o-mini kind: registry.entry meta: name: gpt-4o-mini type: llm.model title: GPT-4o mini comment: Fast, affordable model capabilities: - generate - tool_use - structured_output class: - fast priority: 100 max_tokens: 128000 output_tokens: 16384 pricing: input: 0.15 output: 0.6 providers: - id: wippy.llm.openai:provider provider_model: gpt-4o-mini ``` #### Initialize and Test ```bash wippy init wippy run ask "What is the capital of France?" ``` This runs the `ask` process on the terminal host with the question as its argument and prints the result. The model definition tells the LLM module which provider to use and what model name to send to the API. ### Phase 2: Conversations Upgrade from a single call to a multi-turn conversation using the prompt builder. Register the process as a named command. #### Update Entry Definitions Replace the `ask` entry with a `chat` process: ```yaml - name: chat kind: process.lua meta: command: name: chat short: Start a terminal chat security: actor: id: app:chat policies: - app:policy source: file://chat.lua method: main modules: - io imports: llm: wippy.llm:llm prompt: wippy.llm:prompt ``` Executable Lua entries receive `process` as an ambient runtime module, so it is used directly in the code below and does not belong in the entry's `modules` list. #### Chat Process Create `src/chat.lua`: ```lua local io = require("io") local llm = require("llm") local prompt = require("prompt") local function main() io.print("Chat (type 'quit' to exit)") io.print("") local conversation = prompt.new() conversation:add_system("You are a helpful assistant. Be concise and direct.") while true do io.write("> ") io.flush() local input = io.readline() if not input or input == "quit" or input == "exit" then break end if input == "" then goto continue end conversation:add_user(input) local response, err = llm.generate(conversation, { model = "gpt-4o-mini", temperature = 0.7, max_tokens = 1024, }) if err then io.print("Error: " .. tostring(err)) goto continue end io.print(response.result) io.print("") conversation:add_assistant(response.result) ::continue:: end io.print("Bye!") end return { main = main } ``` #### Run It ```bash wippy update wippy install wippy run chat ``` The prompt builder maintains the full conversation history. Each turn appends the user message and assistant response, giving the model context of prior exchanges. ### Phase 3: Agent Framework The agent module defines prompts, models, and tools declaratively, then loads and executes the resulting agent through a context and runner. #### Add Agent Dependency Add to `_index.yaml`: ```yaml - name: dep.agent kind: ns.dependency component: wippy/agent version: "*" parameters: - name: process_host value: app:processes ``` #### Define an Agent Add an agent entry: ```yaml - name: assistant kind: registry.entry meta: type: agent.gen1 name: assistant title: Assistant comment: Terminal chat agent prompt: | You are a helpful terminal assistant. Be concise and direct. Answer questions clearly. If you don't know something, say so. Do not use emoji in responses. model: gpt-4o-mini max_tokens: 1024 temperature: 0.7 ``` #### Update the Chat Process Switch to the agent framework. Update the entry imports: ```yaml - name: chat kind: process.lua meta: command: name: chat short: Start a terminal chat security: actor: id: app:chat policies: - app:policy source: file://chat.lua method: main modules: - io imports: prompt: wippy.llm:prompt agent_context: wippy.agent:context ``` Update `src/chat.lua`: ```lua local io = require("io") local prompt = require("prompt") local agent_context = require("agent_context") local function main() io.print("Chat (type 'quit' to exit)") io.print("") local ctx = agent_context.new() local runner, err = ctx:load_agent("app:assistant") if err then io.print("Failed to load agent: " .. tostring(err)) return end local conversation = prompt.new() while true do io.write("> ") io.flush() local input = io.readline() if not input or input == "quit" or input == "exit" then break end if input == "" then goto continue end conversation:add_user(input) local response, gen_err = runner:step(conversation) if gen_err then io.print("Error: " .. tostring(gen_err)) goto continue end io.print(response.result) io.print("") conversation:add_assistant(response.result) ::continue:: end io.print("Bye!") end return { main = main } ``` The agent definition contains the prompt, model, and parameters, while the process controls execution. A context can add tools or override the model at runtime. Resolve the newly added agent dependency, then run this phase: ```bash wippy update wippy install wippy run chat ``` ### Phase 4: Streaming Process response chunks as they arrive instead of waiting for the full response. #### Streaming Implementation Update `src/chat.lua`: ```lua local io = require("io") local prompt = require("prompt") local agent_context = require("agent_context") local STREAM_TOPIC = "stream" local stream_sequence = 0 local function stream_response(runner, conversation) stream_sequence = stream_sequence + 1 local topic = STREAM_TOPIC .. ":" .. tostring(stream_sequence) local stream_ch = process.listen(topic) local done_ch = channel.new(1) coroutine.spawn(function() local response, err = runner:step(conversation, { stream_target = { reply_to = process.pid(), topic = topic, }, }) done_ch:send({ response = response, err = err }) end) local full_text = "" local response_result = nil local stream_done = false local function finish(text, response, err) process.unlisten(stream_ch) return text, response, err end while true do local result = channel.select({ stream_ch:case_receive(), done_ch:case_receive(), }) if not result.ok then break end if result.channel == done_ch then response_result = result.value else local chunk = result.value if chunk.type == "chunk" then io.write(chunk.content or "") full_text = full_text .. (chunk.content or "") elseif chunk.type == "done" then stream_done = true elseif chunk.type == "error" then return finish(nil, nil, chunk.error and chunk.error.message or "stream error") end end if response_result and response_result.err then return finish(full_text, response_result.response, response_result.err) end if response_result and stream_done then return finish(full_text, response_result.response, response_result.err) end end return finish(full_text, nil, nil) end local function main() io.print("Chat (type 'quit' to exit)") io.print("") local ctx = agent_context.new() local runner, err = ctx:load_agent("app:assistant") if err then io.print("Failed to load agent: " .. tostring(err)) return end local conversation = prompt.new() while true do io.write("> ") io.flush() local input = io.readline() if not input or input == "quit" or input == "exit" then break end if input == "" then goto continue end conversation:add_user(input) local text, _, gen_err = stream_response(runner, conversation) if gen_err then io.print("Error: " .. tostring(gen_err)) goto continue end io.print("") if text and text ~= "" then conversation:add_assistant(text) end ::continue:: end io.print("Bye!") end return { main = main } ``` Key patterns: - `coroutine.spawn` runs `runner:step()` separately so the main coroutine can process stream chunks. - `channel.select` waits on both the stream channel and completion channel. - Each turn uses a unique topic and removes its listener after both the runner and that turn's stream report completion. - The process accumulates streamed text for the conversation history. Run the streaming phase with the same command: ```bash wippy run chat ``` ### Phase 5: Tools Give the agent tools it can call to access external capabilities. #### Define Tools Create `src/tools/_index.yaml`: ```yaml version: "1.0" namespace: app.tools entries: - name: current_time kind: function.lua meta: type: tool title: Current Time input_schema: | { "type": "object", "properties": {}, "additionalProperties": false } llm_alias: get_current_time llm_description: Get the current date and time in UTC. source: file://current_time.lua modules: [time] method: handler - name: calculate kind: function.lua meta: type: tool title: Calculate input_schema: | { "type": "object", "properties": { "expression": { "type": "string", "description": "Math expression to evaluate" } }, "required": ["expression"], "additionalProperties": false } llm_alias: calculate llm_description: Evaluate a mathematical expression and return the result. source: file://calculate.lua modules: [expr] method: handler ``` Tool metadata describes the callable interface to the LLM: - `input_schema` defines the arguments with JSON Schema. - `llm_alias` is the function name presented to the LLM. - `llm_description` explains when to use the tool. #### Implement Tools Create `src/tools/current_time.lua`: ```lua local time = require("time") local function handler() local now = time.now() return { utc = now:format("2006-01-02T15:04:05Z"), unix = now:unix(), } end return { handler = handler } ``` Create `src/tools/calculate.lua`: ```lua local expr = require("expr") local function handler(args) local result, err = expr.eval(args.expression) if err then return { error = tostring(err) } end return { result = result } end return { handler = handler } ``` #### Register Tools with the Agent Update the agent entry in `src/_index.yaml` to reference the tools: ```yaml - name: assistant kind: registry.entry meta: type: agent.gen1 name: assistant title: Assistant comment: Terminal chat agent prompt: | You are a helpful terminal assistant. Be concise and direct. Answer questions clearly. If you don't know something, say so. Use tools when they help answer the question. Do not use emoji in responses. model: gpt-4o-mini max_tokens: 1024 temperature: 0.7 tools: - app.tools:current_time - app.tools:calculate ``` #### Add Tool Execution Update the chat process modules to include `json` and `funcs`: ```yaml modules: - io - json - funcs ``` Update `src/chat.lua` with tool execution: ```lua local io = require("io") local json = require("json") local funcs = require("funcs") local prompt = require("prompt") local agent_context = require("agent_context") local STREAM_TOPIC = "stream" local stream_sequence = 0 local function stream_response(runner, conversation) stream_sequence = stream_sequence + 1 local topic = STREAM_TOPIC .. ":" .. tostring(stream_sequence) local stream_ch = process.listen(topic) local done_ch = channel.new(1) coroutine.spawn(function() local response, err = runner:step(conversation, { stream_target = { reply_to = process.pid(), topic = topic, }, }) done_ch:send({ response = response, err = err }) end) local full_text = "" local response_result = nil local stream_done = false local function finish(text, response, err) process.unlisten(stream_ch) return text, response, err end while true do local result = channel.select({ stream_ch:case_receive(), done_ch:case_receive(), }) if not result.ok then break end if result.channel == done_ch then response_result = result.value else local chunk = result.value if chunk.type == "chunk" then io.write(chunk.content or "") full_text = full_text .. (chunk.content or "") elseif chunk.type == "done" then stream_done = true elseif chunk.type == "error" then return finish(nil, nil, chunk.error and chunk.error.message or "stream error") end end if response_result and response_result.err then return finish(full_text, response_result.response, response_result.err) end if response_result and stream_done then return finish(full_text, response_result.response, response_result.err) end end return finish(full_text, nil, nil) end local function execute_tools(tool_calls) local results = {} for _, tc in ipairs(tool_calls) do local args = tc.arguments if type(args) == "string" then args = json.decode(args) or {} end io.write("[" .. tc.name .. "] ") io.flush() local result, err = funcs.call(tc.registry_id, args) if err then results[tc.id] = { error = tostring(err) } io.print("error") else results[tc.id] = result io.print("done") end end return results end local function run_turn(runner, conversation) while true do local text, response, err = stream_response(runner, conversation) if err then io.print("") return nil, err end if text and text ~= "" then io.print("") end local tool_calls = response and response.tool_calls if not tool_calls or #tool_calls == 0 then return text, nil end if text and text ~= "" then conversation:add_assistant(text) end local results = execute_tools(tool_calls) for _, tc in ipairs(tool_calls) do local result = results[tc.id] local result_str = json.encode(result) or "{}" conversation:add_function_call(tc.name, tc.arguments, tc.id) conversation:add_function_result(tc.name, result_str, tc.id) end end end local function main() io.print("Terminal Agent (type 'quit' to exit)") io.print("") local ctx = agent_context.new() local runner, err = ctx:load_agent("app:assistant") if err then io.print("Failed to load agent: " .. tostring(err)) return end local conversation = prompt.new() while true do io.write("> ") io.flush() local input = io.readline() if not input or input == "quit" or input == "exit" then break end if input == "" then goto continue end conversation:add_user(input) local text, gen_err = run_turn(runner, conversation) if gen_err then io.print("Error: " .. tostring(gen_err)) goto continue end if text and text ~= "" then conversation:add_assistant(text) end ::continue:: end io.print("Bye!") end return { main = main } ``` The tool-execution loop: 1. Call `runner:step()` with streaming. 2. If the response contains `tool_calls`, execute each tool with `funcs.call()`. 3. Add the tool calls and results to the conversation. 4. Call the runner again so it can incorporate the results. 5. Return the final text when the response contains no more tool calls. #### Run the Agent ```bash wippy update wippy install wippy run chat ``` ``` Terminal Agent (type 'quit' to exit) > what time is it? [get_current_time] done The current time is 17:20 UTC on February 12, 2026. > what is 125 * 16? [calculate] done 125 * 16 = 2000. > quit Bye! ``` ### Completeness and Limits - The page contains every authored Lua file and registry entry needed by the five phases. `wippy.lock` and installed modules are generated by the commands above. - Model output, token usage, tool-choice order, and wording are provider-dependent; the displayed interaction is illustrative rather than an assertion of exact text. - The calculator is intentionally a small arithmetic parser, not a general expression evaluator. Treat every real tool as an authority boundary and attach narrow security policies before exposing side effects. ### Next Steps - [LLM Module](framework/llm.md) — LLM API reference - [Agent Module](framework/agents.md) — Agent framework reference - [CLI Applications](tutorials/cli.md) — Terminal I/O patterns - [Processes](tutorials/processes.md) — Process model and communication --- # "Micro AGI" ## Micro AGI Study an agent that reads documentation, generates Lua tools, registers them at runtime, and loads them into its active session. **Classification: reference implementation walkthrough.** The snippets explain the published `wippy/micro-agi` module but are intentionally not a complete source tree. Run the Hub module to exercise the implementation; use the LLM Agent tutorial when you need a self-contained build. ### What the Package Demonstrates A terminal agent that: - Streams answers from an LLM. - Searches Wippy documentation for APIs. - Inspects the registry for existing capabilities. - Creates and loads tools when a capability is missing. - Compresses conversation history when it approaches the context limit. ```mermaid flowchart LR User -->|prompt| Agent Agent -->|step| LLM[Configured model] LLM -->|tool_calls| Agent Agent -->|funcs.call| Tools Tools -->|result| Agent Agent -->|text| User subgraph Tools doc_search registry_list registry_read create_tool load_tool end ``` ### Architecture The agent runs as a Wippy process with access to the registry. When the LLM decides it needs a capability it doesn't have, it uses the self-modification loop: ```mermaid sequenceDiagram participant U as User participant A as Agent participant L as LLM participant R as Registry U->>A: "what time is it?" A->>L: step(conversation) L->>A: tool_call: doc_search("lua/core/time") A->>A: execute doc_search A->>L: step(conversation + tool result) L->>A: tool_call: create_tool(name, source, schema) A->>R: apply namespace denylist + changeset create R->>A: ok A->>L: step(conversation + tool result) L->>A: tool_call: load_tool("app.generated:current_time") A->>A: ctx:add_tools() + reload agent A->>L: step(conversation + tool result) L->>A: tool_call: current_time() A->>A: execute new tool A->>L: step(conversation + tool result) L->>A: text: "The current time is..." A->>U: stream response ``` Tools are registry entries. To create one, the agent writes a `function.lua` entry with inline Lua source in `data.source`; the runtime then compiles and loads that entry. ### Published Package Structure The package owns all of these files. This page reproduces `doc_search.lua` and the contracts that matter to the architecture, but abbreviates the registry helpers, changeset plumbing, dynamic-loader helpers, and the agent loop. In particular, the `create_tool`, `load_tool`, and `agent.lua` sections are excerpts, not files that can be copied verbatim. The complete registry definitions for `registry_list` and `registry_read` also remain in the published module. ``` micro-agi/ ├── .wippy.yaml ├── wippy.lock └── src/ ├── _index.yaml ├── README.md ├── agent.lua └── tools/ ├── _index.yaml ├── doc_search.lua ├── registry_list.lua ├── registry_read.lua ├── create_tool.lua └── load_tool.lua ``` ### Infrastructure The package uses this `.wippy.yaml` configuration: ```yaml version: "1.0" logger: encoding: console ``` ### Entry Definitions The following selected `src/_index.yaml` entries show the infrastructure, security policies, models, agent, and process: ```yaml version: "1.0" namespace: app entries: - name: definition kind: ns.definition readme: file://README.md meta: title: Micro AGI description: Self-modifying development agent that builds its own tools at runtime depends_on: [wippy/llm, wippy/agent] - name: os_env kind: env.storage.os - name: processes kind: process.host lifecycle: auto_start: true - name: __dep.llm kind: ns.dependency component: wippy/llm version: "*" parameters: - name: env_storage value: app:os_env - name: process_host value: app:processes - name: __dep.agent kind: ns.dependency component: wippy/agent version: "*" parameters: - name: process_host value: app:processes - name: __dep.security kind: ns.dependency component: wippy/security version: "*" ``` `wippy/security` provides the `wippy.security:process` policy group that the LLM module's background services run under; without it they fail to start. #### Security Policies Two `security.policy` entries form an application-level namespace denylist: ```yaml - name: deny_core_ns kind: security.policy policy: actions: "*" resources: "app:*" effect: deny groups: - agent_security - name: deny_tools_ns kind: security.policy policy: actions: "*" resources: "app.tools:*" effect: deny groups: - agent_security ``` These policies are loaded as a named scope (`app:agent_security`) by `create_tool`. The helper rejects an explicit `deny` for `app:*` (core entries, models, and the agent definition) or `app.tools:*` (built-in tools), but treats the unmatched `undefined` result for `app.generated:*` as passing its bespoke filter. This is not Wippy runtime authorization: guarded operations require an explicit `allow` from the execution context, including the security-module operations shown below and `registry.apply` inside `changes:apply()`. A third policy grants the process itself access to the registry. A process launched without a security context is denied every registry read, so the `agent` command carries this policy as its own scope: ```yaml - name: agent_policy kind: security.policy policy: actions: "*" resources: "*" effect: allow ``` See [Security Model](system/security.md) for details on policy evaluation. #### Models Two models serve different purposes: ```yaml - name: gpt-5.1 kind: registry.entry meta: name: gpt-5.1 type: llm.model title: GPT-5.1 comment: Reasoning model capabilities: [generate, tool_use, structured_output, vision, thinking] class: [reasoning] priority: 210 max_tokens: 400000 output_tokens: 128000 pricing: input: 1.25 output: 10 providers: - id: wippy.llm.openai:provider options: reasoning_model_request: true provider_model: gpt-5.1 - name: gpt-4.1-nano kind: registry.entry meta: name: gpt-4.1-nano type: llm.model title: GPT-4.1 Nano comment: Compression model capabilities: [generate, tool_use, structured_output] class: [fast] priority: 100 max_tokens: 1047576 output_tokens: 32768 pricing: input: 0.1 output: 0.4 providers: - id: wippy.llm.openai:provider provider_model: gpt-4.1-nano ``` GPT-5.1 handles reasoning and tool use. GPT-4.1 Nano handles context compression. #### Agent Definition ```yaml - name: dev_assistant kind: registry.entry meta: type: agent.gen1 name: dev_assistant title: Dev Assistant comment: Wippy development assistant prompt: | Self-modifying Wippy development agent. You run inside Wippy runtime with access to docs, registry, and dynamic tool creation. Rules: - NEVER fabricate, guess, or hallucinate facts. If you need real data, use or build a tool to get it. Only state what a tool actually returned. - Maximum 2-3 sentences per response. No bullet lists. No disclaimers. - Never say "I can't" or "I don't have". Build the tool and do it. - Act first, explain only if asked. To gain new capabilities: doc_search the API, create_tool with Lua source, load_tool, call it. All in one turn. model: gpt-5.1 thinking_effort: 10 max_tokens: 2048 tools: - "app.tools:*" ``` The prompt gives the agent three operating rules: - **Use retrieved data** — use tools for external facts. - **Create missing capabilities** — build a tool when an allowed capability is absent. - **Prioritize actions** — perform the requested operation before explaining it. #### Process ```yaml - name: agent kind: process.lua meta: command: name: agent short: Start dev assistant security: actor: id: app:agent policies: - app:agent_policy source: file://agent.lua method: main modules: [io, json, funcs, registry, time, security] imports: prompt: wippy.llm:prompt agent_context: wippy.agent:context compress: wippy.llm.util:compress ``` The process runs as a terminal command. `meta.command.security` gives it the actor and scope it runs under — without it `registry.get` fails with `not allowed to access entry` and the agent never loads. Security enforcement for writes happens inside `create_tool`, which loads the `agent_security` policy group and evaluates it before writing. Imports: - `prompt` — Conversation builder - `agent_context` — Agent loading and dynamic tool management - `compress` — LLM-based text compression for context management ### Tools Create `src/tools/_index.yaml` with five tools: #### doc_search Fetches Wippy documentation via the `wippy.ai/llm` API. Supports two modes: fetch a page by path, or search by query. ```lua local http_client = require("http_client") local json = require("json") local BASE_URL = "https://wippy.ai/llm" local MAX_CHARS = 8000 local function fetch_page(path) local url = BASE_URL .. "/path/en/" .. path local resp, err = http_client.get(url, { headers = { ["User-Agent"] = "wippy-agent/1.0" }, }) if err then return nil, tostring(err) end if resp.status_code ~= 200 then return nil, "HTTP " .. resp.status_code end local body = resp.body or "" if #body > MAX_CHARS then body = body:sub(1, MAX_CHARS) .. "\n... (truncated)" end return body, nil end local function search_docs(query) local url = BASE_URL .. "/search?q=" .. http_client.encode_uri(query) local resp, err = http_client.get(url, { headers = { ["User-Agent"] = "wippy-agent/1.0" }, }) if err then return { error = tostring(err) } end if resp.status_code ~= 200 then return { error = "HTTP " .. resp.status_code } end local body = resp.body or "" if #body > MAX_CHARS then body = body:sub(1, MAX_CHARS) .. "\n... (truncated)" end return { results = body } end local function handler(input) if input.path then local content, err = fetch_page(input.path) if err then return { error = err } end return { path = input.path, content = content } end if input.query then return search_docs(input.query) end return { error = "provide either 'path' or 'query'" } end return { handler = handler } ``` #### create_tool This tool evaluates the package's namespace denylist and creates a `function.lua` registry entry with inline Lua source. The `modules` field on the generated entry controls which non-ambient runtime modules the tool can require. The `process` module is ambient for every executable Lua entry, so omitting it is not a security boundary; process operations still rely on runtime security policies. ```lua local registry = require("registry") local json = require("json") local security = require("security") local NAMESPACE = "app.generated" local MAX_SOURCE_LEN = 16000 local MAX_NAME_LEN = 64 local ALLOWED_MODULES = { time = true, json = true, http_client = true, expr = true, text = true, base64 = true, yaml = true, crypto = true, hash = true, uuid = true, } ``` **Denylist evaluation** — `create_tool` loads the `agent_security` named scope. Writes to `app:*` or `app.tools:*` are rejected when the scope returns `deny`; an unmatched `app.generated:*` target returns `undefined` and passes this application filter: ```lua local actor = security.new_actor("service:agent", { role = "agent" }) local scope, scope_err = security.named_scope("app:agent_security") if scope_err then return { error = "failed to load security scope: " .. tostring(scope_err) } end local result = scope:evaluate(actor, action, id) if result == "deny" then return { error = "policy denied: " .. action .. " on " .. id } end ``` This check does not authorize the registry mutation. The current command also needs a runtime actor and scope that explicitly allow the security-module calls and `registry.apply`. **Registry write** — the entry is written with source in `data.source` and only the allowed modules: ```lua local entry = { id = id, kind = "function.lua", meta = { type = "tool", title = input.name, comment = input.description, input_schema = schema, llm_alias = input.name, llm_description = input.description, }, data = { source = input.source, modules = modules, method = "handler", }, } local snap = registry.snapshot() local changes = snap:changes() if existing then changes:update(entry) else changes:create(entry) end local _, apply_err = changes:apply() if apply_err then return { error = "failed to apply registry change: " .. tostring(apply_err) } end ``` The generated tool is stored in the registry rather than written to a source file. #### load_tool Validates the entry is a tool and signals the agent loop to reload: ```lua local function handler(input) local entry, err = registry.get(input.id) if err then return { error = tostring(err) } end if not entry then return { error = "not found: " .. input.id } end if not entry.meta or entry.meta.type ~= "tool" then return { error = "not a tool (meta.type != 'tool'): " .. input.id } end return { loaded = true, id = entry.id, alias = entry.meta.llm_alias or input.id, description = entry.meta.llm_description or "", } end ``` The agent loop detects `loaded = true` in the result and calls `ctx:add_tools(id)` followed by `ctx:load_agent()` to recompile the agent with the new tool. ### Agent Loop The agent loop in `src/agent.lua` handles streaming, tool execution, dynamic loading, and context compression. #### Streaming Uses the same coroutine + channel pattern from the [LLM Agent tutorial](tutorials/llm-agent.md): ```lua coroutine.spawn(function() local response, err = session.runner:step(session.conversation, { stream_target = { reply_to = process.pid(), topic = STREAM_TOPIC, }, }) done_ch:send({ response = response, err = err }) end) ``` #### Tool Execution Tools are called via `funcs.call()`. `pcall` catches raised Lua errors, while the normal second return from `funcs.call()` carries invocation errors: ```lua local ok, result, call_err = pcall(funcs.call, tc.registry_id, args) if not ok then results[tc.id] = { error = tostring(result) } elseif call_err then results[tc.id] = { error = tostring(call_err) } else results[tc.id] = result end ``` #### Dynamic Tool Loading When `load_tool` returns `loaded = true`, the agent reloads itself: ```mermaid flowchart TD A[load_tool returns loaded=true] --> B[ctx:add_tools id] B --> C[ctx:load_agent] C --> D[New runner with added tool] D --> E[Conversation preserved] E --> F[Next LLM step sees new tool] ``` ```lua local function handle_tool_loading(tool_calls, results) local reload_needed = false for _, tc in ipairs(tool_calls) do if tc.name == "load_tool" then local result = results[tc.id] if result and result.loaded then session.ctx:add_tools(result.id) reload_needed = true end end end if reload_needed then reload_agent() end end ``` The conversation is preserved across reloads because it lives in the prompt builder, not in the runner. #### Context Compression When prompt tokens exceed 300K (75% of the 400K context window), the conversation is compressed using GPT-4.1 Nano: ```lua if response.tokens and response.tokens.prompt_tokens and response.tokens.prompt_tokens > PROMPT_TOKEN_LIMIT then try_compress() end ``` Compression extracts message content, calls `compress.to_size()` targeting 4000 characters, and replaces the conversation with a summary: ```lua local summary, compress_err = compress.to_size(COMPRESS_MODEL, full_text, COMPRESS_TARGET) if compress_err then return nil, compress_err end session.conversation = prompt.new() session.conversation:add_system("Conversation summary:\n\n" .. summary) ``` ### Security Model An application denylist and module-level access controls constrain generated tools, but they do not replace runtime authorization. ```mermaid flowchart TD LLM[LLM generates tool] --> P{Application Namespace Denylist} P -->|scope:evaluate| Check{Target namespace?} Check -->|app.generated:*| OK[No deny match] Check -->|app:* or app.tools:*| Deny[Policy Denied] OK --> M{Non-ambient Module Allowlist} M -->|only listed non-ambient modules| R[Registry write] M -->|unknown module requested| Err[Rejected] R --> A[Ambient process API remains available] ``` #### Namespace Denylist | Policy | Resources | Effect | |--------|-----------|--------| | `deny_core_ns` | `app:*` | deny | | `deny_tools_ns` | `app.tools:*` | deny | `create_tool` loads the `agent_security` policy group and evaluates the target entry ID. It deliberately treats `undefined` as "not denied" for this application-level filter. Wippy's guarded authorization does not: it permits an operation only on explicit `allow`. The context that runs this code must still carry the required runtime permissions. This prevents the agent from: - Modifying its own prompt or agent definition (`app:dev_assistant`) - Overwriting its built-in tools (`app.tools:*`) - Changing infrastructure entries (`app:processes`, etc.) #### Module Access Control Generated tools declare non-ambient capabilities in `data.modules`, and `create_tool` accepts only names from `ALLOWED_MODULES`. An undeclared non-ambient module cannot be required. The runtime still injects `process` into every executable Lua entry, including a generated tool, so process operations must be constrained with security policies rather than by omitting `process` from `data.modules`. This tutorial does not define policies for `process.spawn` or `process.exec`. Its generated tools are therefore not a complete sandbox: add runtime policies for ambient process operations before allowing untrusted tool source. ### Run and Current Package Limitation The published artifact is the Hub module. Start in a fresh empty directory that does not contain `wippy.lock`; Hub bootstrap rejects an unrelated or multi-root lock. The first run creates the deployment lock, and later runs from the same directory reuse that matching lock. ```bash mkdir micro-agi-deploy cd micro-agi-deploy wippy run wippy/micro-agi agent ``` The command downloads the selected module version, resolves its declared dependencies, and invokes its `agent` command. It still requires the provider credentials and model configuration expected by that module, plus registry/network access for Hub download and documentation search. This page does not provide a local clone or lockfile, so it does not claim a reproducible source build. At the reviewed release, `wippy/micro-agi` v0.3.1 declares no `meta.command.security` context for `agent`. With default strict mode, the guarded tool paths—including `funcs.call`, registry reads and writes, and the documentation search HTTP request—do not receive the explicit allows they require. The tool and self-modification flows above are therefore reference designs, not successful default-strict-mode runs. Do not disable strict mode to make an untrusted code generator work; the package should first add a least- privilege command scope for its required actions. ### Next Steps - [LLM Agent](tutorials/llm-agent.md) — Build a basic agent from scratch - [Agent Module](framework/agents.md) — Agent framework reference - [Registry](concepts/registry.md) — Registry concepts - [Security Model](system/security.md) — Declarative security policies - [Entry Kinds](guides/entry-kinds.md) — Available entry types --- # "Keeper over MCP" ## Keeper over MCP Wippy Keeper is the control plane for a running Wippy app — a registry workbench, filesystem↔registry governance, agent/task orchestration, Hub install, knowledge base, logs and process inspection, and a Git review/push flow, all behind a built-in UI. Its defining feature is that it exposes those operator capabilities to AI clients (Claude, Codex, …) over **MCP (Model Context Protocol)**. This page adds Keeper to an app and connects an MCP client to it. ### What You'll Build 1. Keeper added to an app scaffolded from `app-template`. 2. The Keeper UI at `/app/keeper` and the MCP endpoint at `/keeper-mcp/`. 3. A scoped MCP token, and an MCP client configured to drive the app through Keeper. ### Prerequisites - An app from [app-template](https://github.com/wippyai/app-template). It already provides everything Keeper binds to: `app:gateway`, `app:api`, `app:db`, `app:processes`, `app.security:admin`, and `app.env:store`. - The Keeper module installed: ```bash wippy add keeper/keeper wippy install ``` ### Add Keeper Declare the dependency and bind it to the app's resources. Only `admin_scope` is required (no default); the rest default to the names `app-template` already uses, shown here explicitly for clarity: ```yaml ## src/app/deps/_index.yaml - name: keeper kind: ns.dependency component: keeper/keeper version: '>=v0.5.18' parameters: - { name: app_db, value: app:db } - { name: admin_scope, value: app.security:admin } - { name: env_storage, value: app.env:store } - { name: public_gateway, value: app:gateway } # hosts /keeper-mcp/ - { name: mcp_route, value: /keeper-mcp/ } - { name: ui_server, value: app:gateway } - { name: process_host, value: app:processes } ``` Start the app: ```bash wippy run ``` Keeper auto-mounts three surfaces: - **UI** — `/app/keeper` - **MCP transport** — `/keeper-mcp/` on the public gateway - **Token API** — on `app:api` (`/keeper/mcp/tokens`, `/keeper/mcp/scopes`) The MCP transport is gated by the `MCP_ENABLED` environment variable (default `true`); set it to `false` to close the endpoint. ### Mint an MCP Token Tokens are issued by an admin user, scoped, and shown exactly once. Create one via the token API (or the MCP page in the Keeper UI): ```bash curl -X POST http://localhost:8080/api/v1/keeper/mcp/tokens \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{"label": "claude-dev", "preset": "developer"}' ## -> { "success": true, "token": { "token": "wkmcp_<64 hex>", ... } } ``` `preset` bundles a set of scopes. Available presets: `root`, `developer`, `wippy_operator`, `observer`, `knowledge_manager`, `explorer_tools_only`. For finer control, pass an explicit `scopes` array instead (e.g. `registry.read`, `state.write`, `git.pr`, `tasks.run`, `knowledge.read`). The raw `wkmcp_...` token is returned once and stored only as a hash — copy it immediately. ### Connect a Client Point an MCP client at the endpoint with the token as a bearer header. For Claude Code / Codex, an `.mcp.json` in the project root: ```json { "mcpServers": { "keeper": { "type": "http", "url": "http://localhost:8080/keeper-mcp/", "headers": { "Authorization": "Bearer wkmcp_" } } } } ``` Use the app's public base URL in place of `http://localhost:8080` in a deployed environment. ### How the MCP Surface Works Keeper does not expose a flat, fixed tool list. It presents a few **meta-tools** plus **traits** that activate concrete tools on demand, so the surface stays small until you opt into a capability: - `session_info` — always available; reports the session's scopes and active traits. - `list_traits` / `describe_trait` — discover what's available. - `use_trait` / `drop_trait` (and `set_traits`) — activate or remove a trait; this emits an MCP `notifications/tools/list_changed`, so the visible tools change live. - `list_tools` — enumerate the tools a trait materialized, with their schemas. - `call_tool` — invoke any registry tool by id; visible only to a token holding `mcp.root`. What a token can activate is bounded by its **scopes** — roughly `registry.*`, `state.*`, `hub.*`, `knowledge.*`, `git.*`, `components.*`, `tasks.*`, `agents.*`, `tests.run`, `logger.*`, `env.*`, `functions.call`, `app.ui` (plus `mcp.root` for full admin bypass). The token's `access_mode` (`any` / `traits` / `tools_only`) further constrains how it may call tools. ### Notes - **Governance scope** — set `GOV_MANAGED_NAMESPACES=app` so Keeper's filesystem↔registry sync only governs your app's namespace. Do not add `keeper`, `wippy`, or `userspace` unless you are developing those modules. - **Security** — tokens are bound to the issuing admin identity and a scope set, stored as SHA-256, and revocable via `POST /keeper/mcp/tokens/revoke`. The `/keeper-mcp/` route runs no auth middleware; the handler enforces the bearer token itself. - **Reference app** — `app-keeper` is the worked example that wires Keeper into an app shell; copy its `src/app/deps/_index.yaml` block if you want a known-good setup. ### Next Steps - [Hello World](tutorials/hello-world.md) — the minimal project layout - [Authentication](tutorials/auth.md) — the admin identity that issues tokens - [Agents](framework/agents.md) — the agents and tools Keeper traits expose --- # "Dataflow: Local Knowledge Base" ## Dataflow: Local Knowledge Base Build a knowledge base on your own machine — create the vector store, then chunk and ingest documents into it. This is the data-creation companion to the [RAG tutorial](tutorials/rag.md): here you stand up and fill a local KB; there you retrieve from it and generate answers. Both use the `wippy/embeddings` module backed by a local SQLite vector store. ### What You'll Build 1. A local app whose database holds a 512-dimension vector store. 2. The migration that creates the `embeddings_512` table on startup. 3. An ingest function that chunks markdown and writes embeddings into the store. ### Prerequisites - A Wippy project (clone [app-template](https://github.com/wippyai/app-template), or `wippy init`). - An LLM provider with an embedding model configured (e.g. `text-embedding-3-small`) — see [LLM Framework](framework/llm.md). The vector store is created locally without it, but ingesting (which calls `llm.embed`) needs a configured provider. Install the dependencies: ```bash wippy add wippy/embeddings wippy add wippy/migration wippy add wippy/bootloader wippy add wippy/security wippy add wippy/llm wippy install ``` ### Create the Store The KB lives in a local SQLite database. `wippy/embeddings` ships a migration that creates the vector table; the bootloader runs it at startup. Wire the pieces together: ```yaml ## src/_index.yaml version: "1.0" namespace: app entries: - name: db kind: db.sql.sqlite file: ./data/app.db lifecycle: auto_start: true - name: processes kind: process.host host: workers: 8 - name: embeddings kind: ns.dependency component: wippy/embeddings version: "*" parameters: - name: target_db value: app:db - name: migration kind: ns.dependency component: wippy/migration version: "*" parameters: - name: app_db value: app:db - name: bootloader kind: ns.dependency component: wippy/bootloader version: "*" parameters: - name: application_host value: app:processes - name: app_db value: app:db - name: env_storage value: app.env:store - name: security kind: ns.dependency component: wippy/security version: "*" - name: process_access kind: security.policy groups: - wippy.security:process policy: resources: '*' actions: '*' effect: allow ``` The bootloader and the provider services run under the `wippy.security:process` policy group, so `wippy/security` and a policy in that group are part of the wiring. The bootloader needs an environment store; add the standard one in its own namespace: ```yaml ## src/env/_index.yaml version: "1.0" namespace: app.env entries: - name: file kind: env.storage.file auto_create: true file_path: .env lifecycle: auto_start: true - name: os kind: env.storage.os lifecycle: auto_start: true - name: store kind: env.storage.router lifecycle: auto_start: true storages: - app.env:file - app.env:os ``` Create the data directory and start the app: ```bash mkdir -p data wippy run ``` On boot the migration runs and the store appears in `data/app.db`: ``` $ sqlite3 data/app.db ".tables" _migrations embeddings_512 embeddings_512_chunks embeddings_512_info embeddings_512_rowids embeddings_512_vector_chunks00 ... ``` `embeddings_512` is a SQLite `vec0` virtual table; the `embeddings_512_*` shadow tables hold its chunks, row ids, and metadata. (On PostgreSQL the same migration uses `pgvector` instead.) ### Ingest Documents Ingestion is two steps: split text into chunks with the `text` module, then write them with `embeddings.add_batch`, which embeds and persists each chunk. ```lua -- src/ingest.lua local text = require("text") local embeddings = require("embeddings") local function ingest(doc_id, title, markdown) local splitter, err = text.splitter.markdown({ chunk_size = 800, chunk_overlap = 100, heading_hierarchy = true, code_blocks = true, }) if err then return nil, err end local chunks, split_err = splitter:split_text(markdown) if split_err then return nil, split_err end local batch = {} for i, chunk in ipairs(chunks) do table.insert(batch, { content = chunk, content_type = "doc_chunk", origin_id = doc_id, context_id = tostring(i), meta = { title = title, chunk = i }, }) end return embeddings.add_batch(batch) end return { ingest = ingest } ``` Register the function: ```yaml - name: ingest kind: function.lua source: file://ingest.lua method: ingest modules: - text imports: embeddings: wippy.embeddings:embeddings ``` Key points: - `origin_id` groups all chunks from one source document — delete and re-ingest per document with `embedding_repo.delete_by_origin(doc_id)`. - `content_type` lets you keep distinct corpora (`doc_chunk`, `faq`, `code_snippet`) in one store and filter at query time. - `add_batch` auto-splits when the batch exceeds the 8000-token request limit. ### Verify the Contents Once documents are ingested, confirm rows landed and run a similarity search: ```lua local embeddings = require("embeddings") local results, err = embeddings.search("how do I configure TLS?", { content_type = "doc_chunk", limit = 5, }) -- results[i].content, .similarity, .meta, .origin_id, .context_id ``` From there, the [RAG tutorial](tutorials/rag.md) shows how to feed these results to an LLM for grounded answers. ### Operational Notes - **Chunk size**: `chunk_size` and `chunk_overlap` count characters, not tokens; 2000–4000 characters is a good default. Use `chunk_overlap` (~10–20% of chunk size) so sentences aren't cut across boundaries. - **Dimensions**: `text-embedding-3-small` at 512 dimensions is cost-efficient and matches the `embeddings_512` table. Larger vectors mean larger storage and slower search. - **Local vs. shared**: SQLite (`vec0`) keeps the whole KB in one local file — ideal for development and single-node apps. Point `target_db` at a `db.sql.postgres` with `pgvector` for a shared, production store; the ingest code is unchanged. ### Next Steps - [RAG](tutorials/rag.md) — retrieve from this store and generate grounded answers - [LLM Framework](framework/llm.md) — `llm.embed`, embedding models, providers - [Text Module](lua/text/text.md) — splitters and tokenization --- # "Retrieval-Augmented Generation (RAG)" ## Retrieval-Augmented Generation (RAG) Build a knowledge base that answers questions from your own documents. This tutorial uses the `wippy/embeddings` module for vector search and the LLM framework for generation. ### What You'll Build A minimal RAG pipeline: 1. Ingest markdown documents — split into chunks, embed, persist. 2. Retrieve — vector search returns the most relevant chunks for a query. 3. Generate — an LLM call uses the retrieved chunks as grounding context. ### Prerequisites - A database: `db.sql.sqlite` (includes `vec0` support) or `db.sql.postgres` with the `pgvector` extension. - `OPENAI_API_KEY` in the environment — the embedding and generation calls go through it. Create the project and install the modules: ```bash mkdir rag && cd rag mkdir -p src/app data wippy init wippy add wippy/embeddings wippy add wippy/migration wippy add wippy/bootloader wippy add wippy/security wippy install ``` ``` rag/ ├── wippy.lock ├── data/ └── src/ ├── _index.yaml ├── env/ │ └── _index.yaml └── app/ ├── ingest.lua ├── answer.lua ├── answer_http.lua └── seed.lua ``` ### Dependencies Declare the `wippy/embeddings` dependency and point it at your database. The `target_db` parameter is the registry ID of the database entry the embeddings table will live in. `wippy/embeddings` pulls in `wippy/llm` and the migration that creates the `embeddings_512` table, so `wippy/migration` and `wippy/bootloader` need wiring too — the bootloader runs the migration at startup, and both it and the LLM module run processes under the `wippy.security:process` policy group shipped by `wippy/security`: ```yaml ## src/_index.yaml version: "1.0" namespace: app entries: - name: db kind: db.sql.sqlite file: ./data/app.db lifecycle: auto_start: true - name: processes kind: process.host lifecycle: auto_start: true - name: embeddings kind: ns.dependency component: wippy/embeddings version: "*" parameters: - name: target_db value: app:db - name: migration kind: ns.dependency component: wippy/migration version: "*" parameters: - name: app_db value: app:db - name: bootloader kind: ns.dependency component: wippy/bootloader version: "*" parameters: - name: application_host value: app:processes - name: env_storage value: app.env:store - name: security kind: ns.dependency component: wippy/security version: "*" ``` The bootloader persists a generated `ENCRYPTION_KEY`, so it needs a writable environment store: ```yaml ## src/env/_index.yaml version: "1.0" namespace: app.env entries: - name: file kind: env.storage.file auto_create: true file_path: .env lifecycle: auto_start: true - name: os kind: env.storage.os lifecycle: auto_start: true - name: store kind: env.storage.router lifecycle: auto_start: true storages: - app.env:file - app.env:os ``` ### Models `wippy/embeddings` calls `llm.embed` with `text-embedding-3-small`, and generation below uses `gpt-4o-mini`. Both are resolved from the registry, so declare them in `src/_index.yaml` as well: ```yaml - name: text-embedding-3-small kind: registry.entry meta: name: text-embedding-3-small type: llm.model title: Text Embedding 3 Small capabilities: - embed dimensions: 512 max_tokens: 8191 pricing: input: 0.02 output: 0 providers: - id: wippy.llm.openai:provider provider_model: text-embedding-3-small - name: gpt-4o-mini kind: registry.entry meta: name: gpt-4o-mini type: llm.model title: GPT-4o mini capabilities: - generate max_tokens: 128000 output_tokens: 16384 pricing: input: 0.15 output: 0.6 providers: - id: wippy.llm.openai:provider provider_model: gpt-4o-mini ``` The OpenAI provider reads `OPENAI_API_KEY` from the OS environment by default. See [LLM Framework](framework/llm.md) for other providers and model fields. ### Ingest Documents Splitting is handled by the `text` module; embedding and persistence by the `embeddings` library. ```lua -- src/app/ingest.lua local text = require("text") local embeddings = require("embeddings") local function ingest(doc_id: string, title: string, markdown: string) local splitter, err = text.splitter.markdown({ chunk_size = 800, chunk_overlap = 100, heading_hierarchy = true, code_blocks = true, }) if err then return nil, err end local chunks, split_err = splitter:split_text(markdown) if split_err then return nil, split_err end local batch = {} for i, chunk in ipairs(chunks) do table.insert(batch, { content = chunk, content_type = "doc_chunk", origin_id = doc_id, context_id = tostring(i), meta = { title = title, chunk = i }, }) end return embeddings.add_batch(batch) end return { ingest = ingest } ``` Register the function and its imports: ```yaml - name: ingest kind: function.lua source: file://app/ingest.lua method: ingest modules: - text imports: embeddings: wippy.embeddings:embeddings ``` Key points: - `origin_id` groups chunks that belong to the same source document. - `context_id` is an optional sub-key (section, page, chunk index). - `add_batch` auto-splits if total tokens exceed the 8000-token request limit. ### Retrieve Vector search returns the most similar chunks to the query, along with similarity scores: ```lua local embeddings = require("embeddings") local results, err = embeddings.search("how do I configure TLS?", { content_type = "doc_chunk", limit = 5, }) -- results[i].content, .similarity, .meta, .origin_id, .context_id ``` Filter by origin when you want to ground the answer in a specific document: ```lua local hits = embeddings.find_by_origin("refund policy", "doc-42", { limit = 3 }) ``` ### Generate an Answer Compose the retrieved chunks into a prompt and call the LLM. Here the retrieved text is appended to the system prompt; the user's question becomes the user turn: ```lua -- src/app/answer.lua local embeddings = require("embeddings") local llm = require("llm") local prompt = require("prompt") local SYSTEM = [[ Answer using only the provided context. If the context does not contain the answer, say you don't know. Cite the chunk title for each claim. ]] local function format_context(hits) local parts = {} for i, h in ipairs(hits) do local title = h.meta and h.meta.title or h.origin_id table.insert(parts, string.format("[%d] %s\n%s", i, title, h.content)) end return table.concat(parts, "\n\n") end local function answer(question: string) local hits, err = embeddings.search(question, { limit = 4 }) if err then return nil, err end local p = prompt.new() p:add_system(SYSTEM) p:add_system("Context:\n\n" .. format_context(hits)) p:add_user(question) local response, gen_err = llm.generate(p, { model = "gpt-4o-mini" }) if gen_err then return nil, gen_err end return { answer = response.result, sources = hits, } end return { answer = answer } ``` ```yaml - name: answer kind: function.lua source: file://app/answer.lua method: answer imports: embeddings: wippy.embeddings:embeddings llm: wippy.llm:llm prompt: wippy.llm:prompt ``` ### End-to-End Example Putting it together behind an HTTP endpoint. Append these entries to `src/_index.yaml`: ```yaml - name: ingest kind: function.lua source: file://app/ingest.lua method: ingest modules: - text imports: embeddings: wippy.embeddings:embeddings - name: answer kind: function.lua source: file://app/answer.lua method: answer imports: embeddings: wippy.embeddings:embeddings llm: wippy.llm:llm prompt: wippy.llm:prompt - name: seed kind: process.lua meta: command: name: seed short: Ingest the sample document security: groups: - wippy.security:process source: file://app/seed.lua method: main modules: - funcs - io - name: gateway kind: http.service addr: ":8080" lifecycle: auto_start: true security: actor: id: gateway groups: - wippy.security:process - name: api kind: http.router meta: server: app:gateway prefix: /api - name: ask kind: http.endpoint meta: router: app:api method: POST path: /ask func: app:answer_http - name: answer_http kind: function.lua source: file://app/answer_http.lua method: handler modules: - http imports: answer: app:answer ``` The server declares a security context because retrieval resolves the embedding model from the registry, and a request without an actor and scope reads no entries at all — model resolution then fails with `Model or class not found`. ```lua -- src/app/answer_http.lua local http = require("http") local answer = require("answer") local function handler() local req = http.request() local res = http.response() local body, err = req:body_json() if err or not body or not body.question then res:set_status(http.STATUS.BAD_REQUEST) res:write_json({ error = "question is required" }) return end local result, ans_err = answer.answer(tostring(body.question)) if ans_err then res:set_status(http.STATUS.INTERNAL_ERROR) res:write_json({ error = ans_err }) return end res:write_json(result) end return { handler = handler } ``` Seed the index from a CLI command. `meta.command` makes the process runnable as `wippy run seed`, and its `security` block gives it the scope needed to call `app:ingest`: ```lua -- src/app/seed.lua local funcs = require("funcs") local io = require("io") local DOC = [[ ## TLS Configuration Wippy servers terminate TLS when the `tls` block is present on the `http.service` entry. Set `cert_file` and `key_file` to PEM paths. ### Refund Policy Refunds are issued within 14 days of purchase. ]] local function main() local res, err = funcs.call("app:ingest", "doc-42", "Handbook", DOC) if err then io.print("ingest failed: " .. tostring(err)) return end io.print("ingested " .. tostring(res.count) .. " chunks") end return { main = main } ``` The first `wippy run` creates `data/app.db` and applies the embeddings migration. Seed the index, then start the server and query it: ```bash wippy run seed ## ingested 2 chunks wippy run ``` ```bash curl -X POST http://localhost:8080/api/ask \ -H 'Content-Type: application/json' \ -d '{"question":"how do I configure TLS?"}' ``` ```json { "answer": "You can configure TLS by adding a `tls` block to the `http.service` entry. Set `cert_file` and `key_file` to the paths of your PEM files. (See: Handbook, TLS Configuration)", "sources": [ { "entry_id": "52fafcc0-2d18-40d9-8a6e-7662ef9d9bea", "origin_id": "doc-42", "context_id": "1", "content_type": "doc_chunk", "content": "# TLS Configuration\nWippy servers terminate TLS when the `tls` block is present on the\n`http.service` entry. Set `cert_file` and `key_file` to PEM paths.", "meta": { "title": "Handbook", "chunk": 1 }, "similarity": 0.0736 } ] } ``` ### Operational Notes - **Chunk size**: `chunk_size` and `chunk_overlap` count characters, not tokens (the splitter measures length with `utf8.RuneCountInString`). Roughly 2000–4000 characters is a good starting point. Too small loses local context; too large dilutes similarity scores. Use `chunk_overlap` (~10–20% of chunk size) to preserve sentences across boundaries. - **Content types**: Use distinct `content_type` values (`doc_chunk`, `faq`, `code_snippet`) so search can filter by type. - **Re-indexing**: Delete and re-ingest per document via `embedding_repo.delete_by_origin(doc_id)` before adding new chunks. The repository is a separate library — import it as `embedding_repo: wippy.embeddings:embedding_repo`. - **Hybrid search**: For exact-term recall (names, IDs), combine vector search with full-text search over your source table and re-rank. - **Model choice**: `wippy/embeddings` is fixed to `text-embedding-3-small` at 512 dimensions, and the `embeddings_512` table stores `vector(512)`/`float[512]`. A different model or vector size means changing the library constants and the migration table. ### Next Steps - [LLM Framework](framework/llm.md) — `llm.generate`, `llm.embed`, prompt construction - [Agents](framework/agents.md) — wrap the retriever as an agent tool - [SQL Module](lua/storage/sql.md) — underlying database access - [Text Module](lua/text/text.md) — splitters and tokenization --- # "Architecture" ## Architecture Wippy is a layered system built on Go. Components initialize in dependency order, communicate through an event bus, and execute Lua processes via a work-stealing scheduler. This is an implementation reference. The diagrams and Go types describe runtime internals rather than application registry entries or extension APIs. ### Layers | Layer | Components | |-------|------------| | Application | Lua processes, functions, workflows | | Runtime | Lua engine (wippyai/go-lua), 40+ modules | | Services | HTTP, Queue, Storage, Temporal | | System | Topology, Factory, Functions, Contracts | | Core | Scheduler, Registry, Dispatcher, EventBus, Relay | | Infrastructure | AppContext, Logger, Transcoder | Each layer depends only on layers below it. The Core layer provides fundamental primitives, while Services build higher-level abstractions on top. ### Boot Sequence Application startup proceeds through four phases. #### Phase 1: Infrastructure Creates core infrastructure before any components load: | Component | Purpose | |-----------|---------| | AppContext | Sealed dictionary for component references | | EventBus | Pub/sub for inter-component communication | | Transcoder | Payload serialization (JSON, YAML, Lua) | | Logger | Structured logging with event streaming | | Relay | Message routing (Node, Router, Mailbox) | #### Phase 2: Component Loading The Loader resolves dependencies via topological sort and loads components level by level, one component at a time. Dependency edges determine the levels; package groups such as Core and System do not impose a separate global order. Components with no dependency edge may therefore load in the same level regardless of package group. Each component attaches itself to context during Load, making services available to dependent components. #### Phase 3: Activation After all components load: 1. **Start runtime services** - Calls `StartRuntimeServices(ctx)` 2. **Freeze Dispatcher** - Locks command handler registry for lock-free lookups 3. **Seal AppContext** - No more writes allowed, enables lock-free reads 4. **Start Components** - Calls `Start()` on each component with `Starter` interface #### Phase 4: Entry Loading Registry entries from `_index.json`, `_index.yaml`, and `_index.yml` project manifests are loaded and validated: 1. Entries parsed from project files 2. Pipeline stages transform entries (override, link, bytecode) 3. Services marked `auto_start: true` begin running 4. Supervisor monitors registered services ### Components Components are Go services that participate in application lifecycle. #### Lifecycle Phases | Phase | Method | Purpose | |-------|--------|---------| | Load | `Load(ctx) (ctx, error)` | Initialize and attach to context | | Start | `Start(ctx) error` | Begin active operation | | Stop | `Stop(ctx) error` | Graceful shutdown | Components declare dependencies. The loader builds a directed acyclic graph and executes in topological order. Shutdown occurs in reverse order. #### Standard Components | Component | Dependencies | Purpose | |-----------|--------------|---------| | PIDGen | none | Process ID generation | | Dispatcher | none | Command handler dispatch | | Registry | Artifact | Entry storage and versioning | | Finder | Registry | Entry lookup and search | | Supervisor | Registry | Service restart policies | | Topology | none | Process parent/child tree | | Lifecycle | Topology | Service lifecycle management | | Factory | none | Process spawning | | Functions | Registry | Pooled function execution | ### Event Bus Asynchronous pub/sub for inter-component communication. #### Design - Single dispatcher goroutine processes all events - Publishers enqueue actions without waiting for subscriber delivery - Pattern matching supports exact values, `*`, `**`, and segment alternation - Context-based lifecycle ties subscriptions to cancellation #### Event Flow ```mermaid sequenceDiagram participant P as Publisher participant B as EventBus participant S as Subscribers P->>B: Send(ctx, Event) B->>B: Match patterns B->>S: Deliver on subscriber channel S->>S: Execute callback ``` #### Common Topics Every event carries a `System` and a `Kind`. The built-in systems publish: | System | Kind | Purpose | |--------|------|---------| | `registry` | `entry.create`, `entry.update`, `entry.delete`, `entry.accept`, `entry.reject` | Entry mutations | | `registry` | `registry.begin`, `registry.commit`, `registry.discard` | Transaction boundaries | | `process` | `factory.register`, `factory.delete`, `factory.accept`, `factory.reject` | Factory registration for process kinds | | `supervisor` | `service.register`, `service.remove`, `service.update`, `service.start`, `service.stop` | Service lifecycle | ### Registry Versioned storage for entry definitions. #### Features - **Versioned State** - Each mutation creates new version - **History** - In-memory history by default; optional SQLite-backed history for a durable audit trail (history_type: sqlite) - **Event-driven** - Publishes events on mutations #### Entry Lifecycle ```mermaid flowchart LR YAML[YAML Files] --> Parser Parser --> Stages[Pipeline Stages] Stages --> Registry Registry --> Validation Validation --> Active ``` Pipeline stages transform entries: | Stage | Purpose | |-------|---------| | Override | Apply config overrides | | Disable | Remove entries by pattern | | Link | Resolve requirements and dependencies | | Bytecode | Compile Lua to bytecode | | EmbedFS | Collect filesystem entries | ### Relay Message routing between processes across nodes. #### Three-Tier Routing ```mermaid flowchart LR subgraph Router Local[Local Node] --> Peer[Registered Peers] Peer --> Inter[Internode] end Local -.- L[This node] Peer -.- P[Registered peer receiver] Inter -.- I[Other cluster nodes] ``` 1. **Local** - Direct delivery within same node 2. **Peer** - Deliver to a receiver registered for that node ID (an external peer such as a Temporal worker) 3. **Internode** - Fall back to the cluster internode transport, installed by the cluster component after boot #### Mailbox Each node has a mailbox with worker pool: - FNV-1a hashing assigns senders to workers - Preserves per-sender message ordering - Workers process messages concurrently - Back-pressure when queue fills ### AppContext Sealed dictionary for component references. | Property | Behavior | |----------|----------| | Before seal | Single-threaded writes during boot | | After seal | Lock-free reads, panics on write | | Duplicate keys | Panic | | Type safety | Typed getter functions | Components attach services during the Load phase. After boot completes, AppContext is sealed, allowing lock-free reads and preventing further writes. ### Shutdown Graceful shutdown proceeds in reverse dependency order: 1. SIGINT/SIGTERM triggers shutdown 2. Supervisor stops managed services 3. Components with `Stopper` interface receive `Stop()` 4. Infrastructure cleanup Second signal forces immediate exit. ### See Also - [Scheduler](internals/scheduler.md) - Process execution - [Event Bus](internals/events.md) - Pub/sub system - [Registry](internals/registry.md) - State management - [Command Dispatch](internals/dispatch.md) - Yield handling --- # "Registry Internals" ## Registry Internals The registry stores versioned entry state, supports transactions and history, and propagates changes through the event bus. The Go and query fragments on this page document internal data structures and finder syntax; they are not standalone application examples. ### Entry Storage Entries are stored as an ordered slice with a hash map index for O(1) lookups: ```go type Entry struct { ID ID // namespace:name Kind Kind // Entry type Meta attrs.Bag // Author metadata Data payload.Payload // Content Registry EntryMetadata // Registry-owned provenance } type EntryMetadata struct { Owner string // Deployment source that supplied the entry Root bool // Dependency declaration selected by the deployment } ``` Entry IDs use Go's `unique` package for interning—identical IDs share memory. `Registry` is owned by the registry, not the entry author. `Owner` is assigned from the deployment source; `Root` is set from the `dependency_root` write-side field on an `ns.dependency` entry. The ordinary entry APIs return only `ID`, `Kind`, `Meta` and `Data`; provenance is read through the snapshot state API. ### Snapshot `Registry.Snapshot()` returns one atomic view: the version, the entries at that version, and the registry-owned state metadata for that same version. ```go type Snapshot struct { Registry StateMetadata Version Version Entries State } type StateMetadata struct { Resolution *DependencyResolution } ``` Reading version, entries and resolution as one value prevents a caller from pairing entries with a resolution from a different version. The selected module graph is stored once per snapshot rather than repeated on every entry. ### Overlays `OverlayWriter` is an optional registry capability for process-local entries: ```go type OverlayWriter interface { ApplyOverlay(context.Context, string, uint64, ChangeSet) (uint64, error) GetOverlay(string) (State, uint64, error) } ``` Overlay entries are grouped under a logical owner string. They join effective state and pass through the same topology sort and handler transitions as durable entries, so services start and stop for them normally, but they never produce a history version. They are empty after a cold boot and must be reconciled by their owning control service. Writes are optimistically concurrent: `GetOverlay` returns the owner's current generation, and `ApplyOverlay` commits only if that generation is still current, otherwise it returns a retryable `Conflict`. Each successful apply issues a new process-unique generation, and a tombstone is retained for owners that mutated so an ABA sequence cannot be mistaken for an unchanged overlay. The composition rules validated on every apply: - An entry may be created only if no durable entry and no overlay entry holds its ID. - Only the owning identity may update or delete its overlay entries. - Overlay entries may not carry registry-owned metadata, and may not use kinds claimed by registry directives. - A delete may not remove an entry that a surviving entry depends on. - Dependency edges may not cross owner boundaries, and durable entries may not depend on overlay entries. ### Version Chain Each version points to its parent. Path computation uses a graph algorithm to find the shortest route between any two versions: ```mermaid flowchart LR v0[v0] --> v1[v1] --> v2[v2] --> v3[v3] --> vN[vN] ``` ### ChangeSets A changeset is an ordered list of operations transforming one state to another: | Operation | OriginalEntry | Purpose | |-----------|---------------|---------| | Create | nil | Add new entry | | Update | old value | Modify existing | | Delete | deleted value | Remove entry | `OriginalEntry` enables reversal—updates store the previous value, deletes store what was removed. #### Building Deltas `BuildDelta(oldState, newState)` generates minimal operations: 1. Compare states, identify changes 2. Sort deletes in reverse dependency order (dependents first) 3. Sort creates/updates in forward dependency order (dependencies first) #### Squashing Multiple changesets merge by tracking final state per entry: ``` Create + Update = Create (with updated value) Create + Delete = ∅ (cancel out) Update + Delete = Delete Delete + Create = Update ``` ### Transactions ```mermaid sequenceDiagram participant R as Registry participant B as EventBus participant H as Handlers R->>B: registry.begin loop Each Operation R->>B: entry.create/update/delete B->>H: dispatch to listeners H-->>B: accept or reject B-->>R: confirmation end alt All accepted R->>B: registry.commit else Any rejected R->>B: registry.discard R->>R: rollback end ``` By default, the registry waits 30 seconds for listeners to accept or reject each operation. `registry.event_wait_timeout` changes this per-operation timeout. On rejection, the registry rolls back by computing and applying the inverse delta. #### Non-propagating Entries The following kinds skip the event bus by default: - `registry.entry` - Application configs - `ns.requirement` - Namespace requirements - `ns.dependency` - Module dependencies - `ns.definition` - Module metadata (readme, wiki, license, authors) This is the default set; `registry.dispatch_internal_kinds` in the runtime config replaces it. ### Dependency Resolution Entries can declare dependencies on other entries. The resolver extracts dependencies via registered patterns: ```go resolver.RegisterPattern(registry.DependencyPattern{ Path: "meta.server", AllowWildcard: true, }) ``` Dependencies are extracted from entry Meta and Data fields, then used for topological sorting during state transitions. #### Dependency Access Policy External dependency access is a request-scoped context value, not a global flag: | Policy | Effect | |--------|--------| | `DependencyAccessUnspecified` | Callers choose; the caller's own default applies | | `DependencyAccessOnline` | External resolution and artifact download are permitted | | `DependencyAccessVerifiedOffline` | External access is forbidden; resolution uses locked manifests and locally present artifacts | `LoadState()` defaults to verified-offline when the context specifies nothing, so boot replays a stored graph without reaching the network. Restoring a deployment baseline switches the context to online because it must fetch the modules that baseline names. Under verified-offline a manifest provider serving only locked modules replaces the hub provider, and a missing artifact fails as missing evidence rather than triggering a download. ### Version History History backends: | Implementation | Use Case | |----------------|----------| | SQLite | Production persistence | | PostgreSQL | Production persistence, shared across nodes | | Memory | Default when `history_type` is unset; testing | | Nil | No history | SQLite uses WAL mode with tables for versions, changesets (MessagePack encoded), and metadata. PostgreSQL is selected with `registry.history_type: postgres` plus `history_dsn`/`history_schema` (see [Configuration](guides/configuration.md#registry)). History also persists the exact dependency resolution for each version: when an `ns.dependency` change is applied, the resolved module graph is stored content-addressed alongside the changeset. Boot and rollback replay the stored graph instead of re-solving, so a version always reconciles with the versions it was resolved with. The history schema migrates automatically on first boot after an upgrade; a pre-existing version is resolved once on first visit and checkpointed. #### Navigation Path computation finds the shortest route between versions: ```go Path(v0, v3) = [v1, v2, v3] // Apply changesets forward Path(v3, v1) = [v2, v1] // Apply reversed changesets ``` `LoadState()` replays history from a baseline without creating new versions—used during boot. ### Finder Query engine with LRU caching for searching entries: | Operator | Prefix | Example | |----------|--------|---------| | Root-field glob | `.` root field | `.kind=function.*` | | Regex | `~` | `~meta.path=/api/.*` | | Contains | `*` | `*meta.tags=backend` | | Prefix | `^` | `^meta.name=user` | | Suffix | `$` | `$meta.path=Handler` | Cache invalidates on version change. Glob matching applies to the root fields `.kind`, `.name`, `.ns`, and `.id`. Unprefixed `meta.*` criteria use equality matching. ### See Also - [Registry](concepts/registry.md) - High-level concepts - [Events](internals/events.md) - Event bus details --- # "Scheduler" ## Scheduler The scheduler executes processes on workers with local deques, inject queues, a global queue, and work stealing. This is an implementation reference. Its Go structures and diagrams describe the pinned runtime scheduler, not APIs implemented by application code. ### Process Interface The scheduler works with any type implementing the `Process` interface: ```go type Process interface { Init(ctx context.Context, method string, input payload.Payloads) error Step(events []Event, out *StepOutput) error Close() } ``` | Method | Purpose | |--------|---------| | `Init` | Prepare process with entry method name and input arguments | | `Step` | Advance state machine with incoming events, write yields to output | | `Close` | Release resources | The `method` parameter in `Init` specifies which entry point to invoke. A process instance can expose multiple entry points, and the caller selects which one to execute. The scheduler calls `Step()` repeatedly, passing events (yield completions, messages) and collecting yields (commands to dispatch). The process writes its status and any yields to the `StepOutput` buffer. ```go type Event struct { Type EventType // EventYieldComplete or EventMessage Tag uint64 // Correlation tag for yield completions Data any // Result data or message payload Error error // Error if yield failed } ``` ### Structure The scheduler spawns `GOMAXPROCS` workers by default. Each worker has a local deque for cache-friendly LIFO access and a per-worker MPSC inject queue for requeued work that has affinity to that worker, including yield completions and message wakes. A global FIFO queue handles new submissions and affinity-less re-queues. Processes are tracked by PID for message routing. ### Work Finding ```mermaid flowchart TD W[Worker needs work] --> L{Local deque?} L -->|has items| LP[Pop from bottom LIFO] L -->|empty| I{Inject queue?} I -->|has items| IP[Pop + drain up to 16 to local] I -->|empty| G{Global queue?} G -->|has items| GP[Pop + batch transfer up to 16] G -->|empty| S[Scan other workers from rotating start] S --> SH[Steal up to half, capped at 32] ``` Workers check sources in priority order: | Priority | Source | Pattern | |----------|--------|---------| | 1 | Local deque | LIFO pop, lock-free, cache-friendly | | 2 | Inject queue | MPSC pop of affine requeues/events, drain up to 16 to local | | 3 | Global queue | FIFO pop with batch transfer | | 4 | Other workers | Scan from a rotating start index and steal up to half, capped at 32 items per attempt | When popping from the inject or global queue, workers take one item and move up to 16 more to their local deque. ### Chase-Lev Deque Each worker owns a Chase-Lev work-stealing deque: ```go type Deque struct { buffer atomic.Pointer[dequeBuffer] top atomic.Int64 // Thieves steal from here (CAS) bottom atomic.Int64 // Owner pushes/pops here } ``` The owner pushes and pops from the bottom (LIFO) without a mutex; popping the last item uses CAS to coordinate with thieves. Thieves steal from the top (FIFO) using CAS. This gives the owner cache-friendly access to recently-pushed items while distributing older work to stealers. `StealHalfInto` takes up to half the available items in one CAS operation, limited by the destination buffer. Worker steal attempts use a 32-item buffer. ### Adaptive Spinning Before blocking on the condition variable, workers spin adaptively: | Spin Count | Action | |------------|--------| | < 4 | Tight loop | | 4-15 | Yield thread (`runtime.Gosched`) | | >= 16 | Block on condition variable | ### Process States ```mermaid stateDiagram-v2 [*] --> Ready: Submit Ready --> Running: CAS by worker Running --> Complete: done Running --> Blocked: yields commands Running --> Idle: waiting for messages Blocked --> Ready: CompleteYield Idle --> Ready: Send arrives ``` | State | Description | |-------|-------------| | Ready | Queued for execution | | Running | Worker is executing Step() | | Blocked | Waiting for yield completion | | Idle | Waiting for messages | | Complete | Execution finished | A wakeup flag handles races: if a handler calls `CompleteYield` while the worker still owns the process (Running), it sets the flag. The worker checks the flag after dispatching and re-queues if set. ### Event Queue Each process has an MPSC (multi-producer, single-consumer) event queue: - **Producers**: Command handlers (`CompleteYield`), message senders (`Send`) - **Consumer**: Worker drains events in `Step()` A generation counter guards the queue. Every producer binds to the generation it observed; `Reset` bumps it, so a sender left over from a previous execution cannot push into a reused queue. Ordinary event traffic is unbounded. Accounting is opt-in per message: a message that carries `MaxItems` or `MaxBytes` is admitted against a per-topic budget, and the tightest limit seen for a topic wins. A message holds its reservation until the consuming process releases it, and terminals never consume backlog capacity. When a topic's budget is exhausted, the queue appends one synthetic message in the overflowing message's place, carrying `message queue limit exceeded` followed by a terminal payload. Further traffic on that topic is discarded until the queue is reset, so a bounded subscription ends with an error terminal rather than growing without bound. ### Message Routing The scheduler implements `relay.Receiver` to route messages to processes. `Send` delegates to `SendContext` with a background context; `SendContext` checks cancellation before the target lookup and before admission, because admission itself is non-blocking and irreversible once it succeeds. Both look up the target PID in the `byPID` map and push the package onto the process queue under the processor's current generation. Admission is three-way: | Result | Meaning | Package ownership | |--------|---------|-------------------| | Accepted | The queue took the package | Queue, released by the scheduler after processing | | Dropped | A per-topic budget overflowed and the queue retained nothing but its own overflow terminal | Caller, released immediately | | Rejected | The queue is closed or the generation is stale | Caller; `SendContext` returns `ErrProcessClosed` | An accepted or dropped push then wakes the process if it is idle or blocked. It re-queues via injectOrGlobal, which pushes to the last worker's per-worker inject queue when the process has a known worker affinity, and falls back to the global queue otherwise. ### Shutdown On shutdown, the scheduler sends cancel events to all tracked processes and waits for them to complete or timeout. Workers exit once no work remains. ### See Also - [Command Dispatch](internals/dispatch.md) - How yields reach handlers - [Process Model](concepts/process-model.md) - High-level concepts --- # "Command Dispatch" ## Command Dispatch Command dispatch routes process yields to handlers and returns correlated results through process event queues. This is an extension and implementation reference. The custom command and dispatcher fragments assume an existing Go package, boot graph, command API, and service-specific error handling. ### Flow ```mermaid sequenceDiagram participant P as Process participant W as Worker participant R as Registry participant H as Handler P->>W: yield(command, tag) W->>R: getHandler(cmdID) R-->>W: handler W->>H: Handle(cmd, tag, receiver) H-->>H: async work H->>W: CompleteYield(tag, result) W->>P: queue event, wake P->>P: resume with result ``` ### Command Registry The registry stores handlers in a hybrid structure: ```go type Registry struct { handlers [256]Handler // System commands: O(1) index extended map[CommandID]Handler // Extended commands: map lookup frozen atomic.Bool // Lock-free after boot } ``` System commands (0-255) use array indexing. Extended commands use map lookup. After `Freeze()`, all lookups are lock-free. #### Command ID Ranges | Range | Module | Examples | |-------|--------|----------| | 1-9 | process | Send, Spawn, Terminate, Cancel, Monitor, Unmonitor, Link, Unlink, Exec | | 10-29 | clock | Sleep, Ticker, Timer | | 30-39 | socket | Connect, Listen, Accept, Bind, Resolve | | 50-59 | stream | Read, Write, Close, Seek | | 60-69 | http | Request, RequestBatch | | 70-79 | tty | terminal I/O | | 80-89 | websocket | Connect, Send, Receive | | 90-99 | event | Subscribe, Send | | 100-119 | sql | Query, Execute, Prepare, Stmt, Tx ops | | 120-129 | store | Get, Set, Delete, Has | | 130-139 | security | ValidateToken, CreateToken | | 140-149 | function | Call, AsyncStart, AsyncCancel | | 150-159 | exec | ProcessWait | | 160-169, 173-174 | cloudstorage | Upload, Download, List, Presigned URLs, Multipart, OpenReader | | 170-171 | eval | Compile, Run | | 172 | cdc | Subscribe | | 180-189 | workflow | SideEffect, Exec, Version, UpsertAttrs | | 190-199 | contract | Open, Call, AsyncCall, AsyncCancel | | 200-211 | pg (process group) | Join, Leave, GetMembers, GetLocalMembers, WhichGroups, Broadcast, BroadcastLocal, WhichLocalGroups, Monitor, Events, JoinGroups, LeaveGroups | | 256+ | custom | User-defined services | Packages reserve command-ID ownership from `init()` with `MustRegisterCommands()`; ownership collisions panic while packages initialize. During component loading, each service binds its handlers through `Registrar.Register`. The dispatcher is frozen only after those handlers have been installed. ### Defining Commands Commands are data structures with a unique `CommandID`: ```go const MyCommand dispatcher.CommandID = 256 type MyCmd struct { Input string Option int } func (c *MyCmd) CmdID() dispatcher.CommandID { return MyCommand } ``` Reserve the command ID at package initialization: ```go func init() { dispatcher.MustRegisterCommands("myservice", MyCommand) } ``` ### Dispatchers A dispatcher groups related handlers. It implements `RegisterAll` to register handlers and lifecycle methods for setup/teardown: ```go type Handler interface { Handle(ctx context.Context, cmd Command, tag uint64, receiver ResultReceiver) error } type ResultReceiver interface { CompleteYield(tag uint64, data any, err error) } ``` ```go type Dispatcher struct { // service state } func (d *Dispatcher) RegisterAll(register func(id dispatcher.CommandID, h dispatcher.Handler)) { register(myapi.MyCommand, dispatcher.HandlerFunc(d.handleMyCommand)) } func (d *Dispatcher) handleMyCommand(ctx context.Context, cmd Command, tag uint64, receiver ResultReceiver) error { c := cmd.(*myapi.MyCmd) go func() { result := doWork(c) if ctx.Err() == nil { receiver.CompleteYield(tag, result, nil) } }() return nil } ``` Register as a boot component: ```go func MyDispatcher() boot.Component { return boot.New(boot.P{ Name: "dispatcher.myservice", DependsOn: []boot.Name{DispatcherName}, Load: func(ctx context.Context) (context.Context, error) { reg := dispatcher.GetRegistrar(ctx) svc := myservice.NewDispatcher() svc.RegisterAll(reg.Register) return ctx, nil }, }) } ``` ### Yields and Correlation When a process needs async work, it yields a command with a correlation tag: ```go type Yield struct { Cmd Command Tag uint64 // Process-local counter for correlation } ``` The worker extracts yields from `StepOutput` after each step and dispatches them to handlers. Each tag uniquely identifies the request so results can be matched back. ### See Also - [Scheduler](internals/scheduler.md) - Process execution - [Modules](internals/modules.md) - Lua module integration - [Process Model](concepts/process-model.md) - High-level concepts --- # "Event Bus" ## Event Bus The event bus processes queued pub/sub actions on one dispatcher goroutine and delivers matching events to subscriber channels. The Go snippets are implementation and extension fragments. They assume an existing component context, logger, handlers, and application event types. ### Event Structure ```go type Event struct { System string // Component/module (e.g., "registry", "process") Kind string // Event type (e.g., "create", "update", "exit") Path string // Entity identifier Data any // Payload Aux any // In-process dispatcher context; not propagated to processes } ``` ### Bus Architecture ```mermaid flowchart LR subgraph Publishers P1[Component] P2[Component] end subgraph Bus Q[actionQueue] D[dispatcher goroutine] S[subscribers map] end subgraph Subscribers S1[chan Event] S2[chan Event] end P1 & P2 -->|enqueue| Q Q -->|signal| D D -->|match & deliver| S1 & S2 D <-->|manage| S ``` The bus stores state in a simple structure: ```go type Bus struct { subscribers map[SubscriberID]sub subscriberCounter uint64 maxSubscribers int actionQueue []action spareQueue []action actionMu sync.Mutex actionReady chan struct{} // buffered=1 closed atomic.Bool } ``` All mutations go through the dispatcher goroutine, eliminating race conditions without complex locking. ### Actions Four action types flow through the queue: | Action | Behavior | |--------|----------| | Subscribe | Adds subscriber to map, responds on done channel | | Unsubscribe | Removes subscriber, responds on done channel | | Send | Delivers event to matching subscribers | | Stop | Clears subscribers, drains queue, exits loop | Subscribe and Unsubscribe block until the dispatcher confirms. Send is fire-and-forget. The bus accepts at most `DefaultMaxSubscribers` subscriptions (4096 by default); subscriptions beyond the cap fail with `ErrSubscribersCapReached`. `Subscribe` is rejected with `ErrSubscribersCapReached` once the bus holds `DefaultMaxSubscribers` (4096) active subscriptions. `Subscribe` fails immediately when the subscription context is already canceled, and again at the dispatcher if it is canceled before the ownership decision is made — the bus never takes a channel it did not install. `Unsubscribe` is an ownership barrier, not a best-effort hint. It returns only after the dispatcher acknowledges, so the caller can release the channel knowing the bus holds no in-flight send reference. When it arrives after `Stop`, the acknowledgement waits for the dispatcher to finish delivering the batch it already drained. `Stop` is likewise terminal: a second concurrent `Stop` does not return early on the already-closed flag but waits for the dispatcher to drain and exit. ### Queue Swapping The dispatcher uses slice swapping to avoid allocations in steady state: ```go func (b *Bus) processActions() bool { b.actionMu.Lock() actions := b.actionQueue b.actionQueue = b.spareQueue[:0] b.spareQueue = nil b.actionMu.Unlock() for i := range actions { // process action } clear(actions) b.actionMu.Lock() b.spareQueue = actions[:0] b.actionMu.Unlock() return true } ``` Two slices alternate: one for processing, one for new arrivals. The `actionReady` channel is buffered to 1, so signaling never blocks and multiple enqueues coalesce into one wakeup. ### Pattern Matching Subscriptions compile patterns once at subscribe time: ```go type sub struct { subID SubscriberID ctx context.Context system *wildcard.Wildcard kind *wildcard.Wildcard eventCh chan<- Event } ``` The wildcard package supports four pattern types: | Pattern | Matches | |---------|---------| | `registry` | Exact match only | | `*` | Any single segment | | `**` | Zero or more segments | | `(a\|b)` | Alternation within segment | Patterns split on `.` so `registry.*` matches `registry.create` but not `registry.entry.create`. The pattern `registry.**` matches all three of `registry`, `registry.create`, and `registry.entry.create`. ### Event Delivery During Send processing, the dispatcher iterates subscribers: ```go for id, s := range b.subscribers { if s.system != nil && !s.system.Match(a.event.System) { continue } if s.kind != nil && !s.kind.Match(a.event.Kind) { continue } select { case <-a.ctx.Done(): goto cleanup case <-s.ctx.Done(): expiredSubs = append(expiredSubs, id) case s.eventCh <- a.event: } } ``` If a subscriber's context is canceled, it's marked for removal during that delivery pass. The event context can also cancel delivery mid-iteration. ### Lua Process Bridge The events dispatcher bridges Go events to Lua processes. It subscribes once to all events (`"**"`) and routes internally based on process subscriptions: ```go type Dispatcher struct { bus event.Bus node relay.Node subID SubscriberID eventC chan event.Event mu sync.RWMutex subs map[string]*subscription // topic -> subscription } ``` When a Lua process subscribes via `events.subscribe()`, the dispatcher stores the pattern and target PID. Matching events are packaged and sent via relay: ```go func (d *Dispatcher) routeEvent(evt event.Event) { d.mu.RLock() defer d.mu.RUnlock() for _, sub := range d.subs { if !matchPattern(sub.system, evt.System) { continue } if sub.kind != "" && sub.kind != "*" && !matchPattern(sub.kind, evt.Kind) { continue } data := map[string]any{ "system": evt.System, "kind": evt.Kind, "path": evt.Path, } if evt.Data != nil { data["data"] = evt.Data } pkg := relay.NewPackage(pid.PID{}, sub.pid, sub.topic, payload.New(data)) d.node.Send(pkg) } } ``` #### Subscriber Wraps channel subscription with a callback: ```go handler, err := eventbus.NewSubscriber(ctx, bus, "registry", "entry.*", func(evt Event) { // handle }) if err != nil { return err } defer handler.Close() ``` Spawns two goroutines: one reads events and calls the handler, another waits for context cancellation to unsubscribe. #### EventRouter Manages multiple handlers with centralized lifecycle: ```go router, err := eventbus.StartRouter(ctx, bus, WithHandlers(handler1, handler2), WithLogger(log)) if err != nil { return err } defer router.Stop() ``` Each handler implements `Pattern()` and `Handle()`. The router creates a Subscriber for each and closes all on Stop. #### AwaitService Request-response over pub/sub. It keeps a single subscription per `(system, kind)` pair and routes events to waiters by `Path`: ```go svc := eventbus.NewAwaitService(bus) if err := svc.Start(ctx); err != nil { return err } defer svc.Stop() waiter, err := svc.Prepare(ctx, "test", "response.(accept|reject)", "test/path", 5*time.Second) if err != nil { return err } defer waiter.Close() bus.Send(ctx, triggeringEvent) result := waiter.Wait() // returns AwaitResult{Event, Accepted, Error} ``` `Prepare` registers the waiter before the triggering event is sent, avoiding the race where the response arrives before the wait is registered. `Wait` blocks until a matching `Path` event arrives or the timeout (default `DefaultAwaitTimeout`, 30s, when non-positive) expires. `Accepted` is true when the event kind is `accept`, `*.accept`, or `*.accepted`; otherwise the kind is treated as a rejection and any `error` in `Data` surfaces as `Error`. The convenience `Await(ctx, system, kind, path, timeout)` combines Prepare and Wait. The boot infrastructure registers an AwaitService on the context (`event.GetAwaitService`). ### Shutdown 1. `Stop()` atomically sets closed flag and enqueues Stop action 2. Dispatcher clears subscriber map 3. Remaining queued actions are drained: - Subscribe requests get "bus is closed" error - Unsubscribe requests complete immediately - Send events are dropped 4. WaitGroup completes ### See Also - [Registry](internals/registry.md) - Primary event producer - [Command Dispatch](internals/dispatch.md) - Process-to-handler routing --- # "Lua Modules" ## Lua Modules Runtime modules extend the Lua environment with new functionality. Modules can provide deterministic utilities, I/O operations, or async commands that yield to external systems. > The Lua runtime implementation may change in future versions. ### Module Definition Every module uses `luaapi.ModuleDef`: ```go var Module = &luaapi.ModuleDef{ Name: "mymodule", Description: "My custom module", Class: []string{luaapi.ClassDeterministic}, Types: ModuleTypes, // Type definitions for tooling Build: func() (*lua.LTable, []luaapi.YieldType) { mod := lua.CreateTable(0, 2) mod.RawSetString("hello", lua.LGoFunc(helloFunc)) mod.RawSetString("greet", lua.LGoFunc(greetFunc)) mod.Immutable = true return mod, nil }, } ``` The `Build` function returns: - Module table with exported functions - List of yield types for async operations (or nil) Module tables are built once and cached for reuse across all Lua states. ### Module Classification The `Class` field determines where the module can be used: | Class | Description | |-------|-------------| | `ClassDeterministic` | Same input always produces same output | | `ClassNondeterministic` | Output varies (time, random) | | `ClassIO` | External I/O operations | | `ClassNetwork` | Network operations | | `ClassEncoding` | Serialization and encoding | | `ClassTime` | Clock and timer access | | `ClassProcess` | Process control | | `ClassSecurity` | Security context and tokens | | `ClassStorage` | Data persistence | | `ClassWorkflow` | Workflow-safe operations | Workflow processes are compiled with `ClassDeterministic` and `ClassWorkflow` as the allowed classes: a module is available to workflows if it carries at least one of them, otherwise it is restricted to functions and processes. ### Exposing Functions Functions have signature `func(l *lua.LState) int` where the return value is the number of values pushed onto the stack: ```go func greetFunc(l *lua.LState) int { name := l.CheckString(1) // Required argument greeting := l.OptString(2, "Hello") // Optional with default l.Push(lua.LString(greeting + ", " + name + "!")) return 1 } ``` | Method | Description | |--------|-------------| | `l.CheckString(n)` | Required string at position n | | `l.CheckInt(n)` | Required integer | | `l.CheckNumber(n)` | Required number | | `l.CheckTable(n)` | Required table | | `l.OptString(n, def)` | Optional string with default | | `l.OptInt(n, def)` | Optional int with default | ### Tables Tables passed between Go and Lua are mutable by default. Module export tables should be marked immutable: ```go mod := lua.CreateTable(0, 5) mod.RawSetString("func1", lua.LGoFunc(func1)) mod.Immutable = true // Prevent Lua from modifying exports ``` Data tables remain mutable for normal use: ```go result := l.CreateTable(0, 3) result.RawSetString("name", lua.LString("value")) result.RawSetString("count", lua.LNumber(42)) l.Push(result) ``` ### Type System Modules use two separate but complementary typing mechanisms. #### Type Definitions (Tooling) The `Types` field provides type signatures for IDE support and documentation. Types are built with the `typ` package's fluent builders: ```go import ( "github.com/wippyai/go-lua/types/io" "github.com/wippyai/go-lua/types/typ" ) func ModuleTypes() *io.Manifest { m := io.NewManifest("mymodule") objectType := typ.NewInterface("mymodule.Object", []typ.Method{ {Name: "get_value", Type: typ.Func().Param("self", typ.Self). Returns(typ.String, typ.NewOptional(typ.LuaError)).Build()}, {Name: "set_value", Type: typ.Func().Param("self", typ.Self). Param("value", typ.String).Returns(typ.NewOptional(typ.LuaError)).Build()}, }) m.DefineType("Object", objectType) m.SetExport(objectType) return m } ``` **Available type constructs:** | Type | Description | |------|-------------| | `typ.String` | String primitive | | `typ.Number` | Numeric value | | `typ.Integer` | Integer value | | `typ.Boolean` | Boolean value | | `typ.Any` | Any Lua value | | `typ.Self` | Receiver type for methods | | `typ.LuaError` | Error type | | `typ.NewOptional(t)` | Optional value of type t | | `typ.NewInterface(name, methods)` | Object with methods | | `typ.Func()` | Function signature builder | | `typ.NewRecord()` | Struct-like type builder (fields via `.Field`/`.OptField`) | | `typ.NewArray(t)` | Array of element type t | | `typ.NewMap(k, v)` | Map with key/value types | Function builders chain `Param`, `OptParam`, `Variadic`, and `Returns`: ```go // (string, ...any) -> (string, error?) typ.Func(). Param("first", typ.String). Variadic(typ.Any). Returns(typ.String, typ.NewOptional(typ.LuaError)). Build() ``` Records declare fields with `Field` (required) and `OptField` (optional): ```go typ.NewRecord(). Field("key", typ.String). Field("value", typ.Any). OptField("ttl", typ.Number). Build() ``` See the `typ` package in go-lua for the complete type system. #### UserData Bindings (Runtime) `RegisterTypeMethods` creates the actual Go-to-Lua bindings: ```go func init() { value.RegisterTypeMethods(nil, "mymodule.Object", map[string]lua.LGoFunc{ "__tostring": objectToString, // Metamethods }, map[string]lua.LGoFunc{ "get_value": objectGetValue, // Regular methods "set_value": objectSetValue, }, ) } ``` Metatables are immutable and cached globally for thread-safe reuse. | System | Purpose | Defines | |--------|---------|---------| | Type Definitions | IDE, docs, type checking | Signatures | | UserData Bindings | Runtime method calls | Executable functions | ### Async Operations For operations that wait on external systems, return a yield instead of a result. The yield is dispatched to a Go handler and the process resumes when the handler completes. #### Defining Yields Declare yield types in the module's `Build` function: ```go Build: func() (*lua.LTable, []luaapi.YieldType) { mod := lua.CreateTable(0, 1) mod.RawSetString("fetch", lua.LGoFunc(fetchFunc)) mod.Immutable = true yields := []luaapi.YieldType{ {Sample: &FetchYield{}, CmdID: myapi.FetchCommand}, } return mod, yields } ``` #### Creating a Yield Return -1 to signal a yield instead of normal return values: ```go func fetchFunc(l *lua.LState) int { url := l.CheckString(1) yield := AcquireFetchYield() yield.URL = url l.Push(yield) return -1 // Signal yield, not stack count } ``` #### Yield Implementation Yields bridge Lua values and dispatcher commands: ```go type FetchYield struct { *myapi.FetchCmd } func (y *FetchYield) String() string { return "" } func (y *FetchYield) Type() lua.LValueType { return lua.LTUserData } func (y *FetchYield) CmdID() dispatcher.CommandID { return myapi.FetchCommand } func (y *FetchYield) ToCommand() dispatcher.Command { return y.FetchCmd } func (y *FetchYield) Release() { releaseFetchYield(y) } func (y *FetchYield) HandleResult(l *lua.LState, data any, err error) []lua.LValue { if err != nil { return []lua.LValue{lua.LNil, lua.NewLuaError(l, err.Error())} } resp := data.(*myapi.FetchResponse) return []lua.LValue{lua.LString(resp.Body), lua.LNil} } ``` The dispatcher routes the command to a handler. See [Command Dispatch](internals/dispatch.md) for implementing handlers. ### Error Handling Return errors as the second value using structured errors: ```go func myFunc(l *lua.LState) int { result, err := doSomething() if err != nil { lerr := lua.NewLuaError(l, err.Error()). WithKind(lua.Internal). WithRetryable(true) l.Push(lua.LNil) l.Push(lerr) return 2 } l.Push(lua.LString(result)) l.Push(lua.LNil) return 2 } ``` ### Security Check permissions before performing sensitive operations: ```go func myFunc(l *lua.LState) int { ctx := l.Context() if !security.IsAllowed(ctx, "mymodule.action", resource, nil) { l.Push(lua.LNil) l.Push(lua.NewLuaError(l, "permission denied").WithKind(lua.PermissionDenied)) return 2 } // Proceed with operation } ``` ### Testing Basic module tests verify structure and synchronous functions: ```go func TestModule(t *testing.T) { l := lua.NewState() defer l.Close() mod, _ := Module.Build() l.SetGlobal("mymodule", mod) err := l.DoString(` local m = mymodule assert(m.hello() == "Hello, World!") `) if err != nil { t.Fatal(err) } } ``` #### Testing Modules with Yields To test Lua code that uses yielding functions, create a minimal scheduler with the required dispatchers: ```go type testScheduler struct { *actor.Scheduler clock *clock.Dispatcher mu sync.Mutex pending map[string]chan *runtime.Result } func newTestScheduler() *testScheduler { ts := &testScheduler{pending: make(map[string]chan *runtime.Result)} reg := scheduler.NewRegistry() // Register dispatchers for yields your module uses clockSvc := clock.NewDispatcher() clockSvc.RegisterAll(func(id dispatcher.CommandID, h dispatcher.Handler) { reg.Register(id, h) }) ts.clock = clockSvc ts.Scheduler = actor.NewScheduler(reg, actor.WithWorkers(4), actor.WithLifecycle(ts)) return ts } // Stop wraps Scheduler.Stop, which requires a context. func (ts *testScheduler) Stop() { ts.Scheduler.Stop(context.Background()) } // OnStart satisfies process.Lifecycle alongside OnComplete. func (ts *testScheduler) OnStart(context.Context, pid.PID, process.Process) error { return nil } func (ts *testScheduler) OnComplete(_ context.Context, p pid.PID, result *runtime.Result) { ts.mu.Lock() ch, ok := ts.pending[p.UniqID] delete(ts.pending, p.UniqID) ts.mu.Unlock() if ok { ch <- result } } func (ts *testScheduler) Execute(ctx context.Context, p pid.PID, proc process.Process, method string, input payload.Payloads) (*runtime.Result, error) { resultCh := make(chan *runtime.Result, 1) ts.mu.Lock() ts.pending[p.UniqID] = resultCh ts.mu.Unlock() _, err := ts.Scheduler.Submit(ctx, p, proc, method, input) if err != nil { return nil, err } select { case result := <-resultCh: return result, nil case <-ctx.Done(): return nil, ctx.Err() } } ``` Create processes from Lua scripts with the modules you're testing: ```go func bindMyModule(l *lua.LState) error { tbl, _ := mymodule.Module.Build() l.SetGlobal(mymodule.Module.Name, tbl) return nil } func newLuaProcess(script string) *engine.Process { proto, _ := lua.CompileString(script, "test.lua") proc, _ := engine.NewProcess( engine.WithProto(proto), engine.WithModuleBinder(bindMyModule), ) return proc } func TestMyModuleYields(t *testing.T) { sched := newTestScheduler() sched.Start() defer sched.Stop() script := ` local result = mymodule.fetch("http://example.com") return result.status ` ctx, _ := ctxapi.OpenFrameContext(context.Background()) proc := newLuaProcess(script) result, err := sched.Execute(ctx, pid.PID{UniqID: "test"}, proc, "", nil) if err != nil { t.Fatal(err) } // Assert on result } ``` See `runtime/lua/modules/time/integration_test.go` for a complete example. ### See Also - [Command Dispatch](internals/dispatch.md) - Handling yield commands - [Scheduler](internals/scheduler.md) - Process execution --- # "Entry Listeners and Observers" ## Entry Listeners and Observers Entry listeners and observers process registry mutations for matching entry-kind patterns. This is a Go extension reference. The registration and configuration snippets assume an existing boot component, manager, transcoder, and application config type. ### How It Works Boot collects listeners and observers with their kind patterns. When an entry changes: 1. Registry emits event (`entry.create`, `entry.update`, `entry.delete`) 2. Each listener wrapper matches the entry kind against its registered pattern 3. Matching handlers receive the entry 4. Handlers process or reject the entry ### Kind Patterns Handlers subscribe using patterns: | Pattern | Matches | |---------|---------| | `http.service` | Exact match only | | `http.*` | `http.service`, `http.router`, `http.endpoint` | | `function.**` | `function.lua`, `function.lua.bc` | ### Entry Listener Interface Handlers implement `registry.EntryListener`: ```go type EntryListener interface { Add(ctx context.Context, entry Entry) error Update(ctx context.Context, entry Entry) error Delete(ctx context.Context, entry Entry) error } ``` Returning an error from `Add`, `Update`, or `Delete` rejects that operation. ### Listener vs Observer | Type | Purpose | Can Reject | |------|---------|------------| | Listener | Primary handler | Yes | | Observer | Secondary handler (logging, metrics) | No | ```go handlers.RegisterListener("http.*", httpManager) handlers.RegisterObserver("function.*", metricsCollector) ``` Observer errors from `Add`, `Update`, and `Delete` are ignored and do not emit an accept or reject event. A listener or observer that also implements `TransactionListener` participates in transaction barriers, where an error from `Begin`, `Commit`, or `Discard` rejects that transaction phase. ### Registering Handlers Register handlers during boot: ```go func MyService() boot.Component { return boot.New(boot.P{ Name: "myservice", DependsOn: []boot.Name{core.RegistryName}, Load: func(ctx context.Context) (context.Context, error) { handlers := bootpkg.GetHandlerRegistry(ctx) handlers.RegisterListener("myservice.*", manager) return ctx, nil }, }) } ``` ### Decoding Entry Data Use `entry.DecodeEntryConfig` from `system/entry` to unmarshal entry data. `DecodeEntryConfigFromContext` takes the transcoder from the context instead of an argument, and `DecodeEntryConfigRaw` skips placeholder resolution: ```go func (m *Manager) Add(ctx context.Context, ent registry.Entry) error { cfg, err := entry.DecodeEntryConfig[ComponentConfig](ctx, m.dtt, ent) if err != nil { return err } // Process cfg... return nil } ``` The decoder: 1. Resolves `${env:...}` placeholders and `*_env` companion fields against the environment registry 2. Unmarshals `entry.Data` into your config struct 3. Populates `ID` and `Meta` from the entry when the struct leaves them empty 4. Calls `InitDefaults()` if implemented 5. Calls `Validate()` if implemented ### Config Structure Entry configs typically include: ```go type ComponentConfig struct { ID registry.ID `json:"id"` Meta attrs.Bag `json:"meta"` Name string `json:"name"` Timeout int `json:"timeout,omitempty"` } func (c *ComponentConfig) InitDefaults() { if c.Timeout == 0 { c.Timeout = 30 } } func (c *ComponentConfig) Validate() error { if c.Name == "" { return fmt.Errorf("name is required") } return nil } ``` ### Transaction Support For atomic operations across multiple entries, implement `TransactionListener`: ```go type TransactionListener interface { Begin(ctx context.Context) error Commit(ctx context.Context) error Discard(ctx context.Context) error } ``` The registry calls `Begin` before processing a batch, then `Commit` on success or `Discard` on failure. ### See Also - [Registry](internals/registry.md) - Entry storage - [Architecture](internals/architecture.md) - Boot sequence --- # "Contributing" ## Contributing Wippy is developed in the public [wippyai GitHub organization](https://github.com/wippyai). The runtime release line is currently alpha. ### Stability The runtime release line is alpha, and the public repositories do not publish a separate compatibility or migration policy. Pin the runtime and module versions used by an application, and review release notes and documentation before upgrading. ### Issues and Pull Requests Open documentation issues and pull requests in the [documentation repository](https://github.com/wippyai/docs). The public runtime repository currently accepts [pull requests](https://github.com/wippyai/runtime/pulls) but not issues. For a runtime defect report or proposal that does not include a patch, use the published organization contact, [support@wippy.ai](mailto:support@wippy.ai). Contributions must follow the organization [Code of Conduct](https://github.com/wippyai/.github/blob/main/.github/CODE_OF_CONDUCT.md). ### Security The documentation and runtime repositories do not currently publish a dedicated security policy. Report security-sensitive issues privately to [support@wippy.ai](mailto:support@wippy.ai) instead of opening a public issue. ### Support Contact [support@wippy.ai](mailto:support@wippy.ai). --- # "License" ## License This documentation repository and the Wippy runtime are released under the Mozilla Public License 2.0. Individual Wippy modules and components may use different licenses; the `LICENSE` file in each repository is authoritative. ### Mozilla Public License 2.0 Read the exact license in the [documentation repository](https://github.com/wippyai/docs/blob/main/LICENSE) or [runtime repository](https://github.com/wippyai/runtime/blob/main/LICENSE). ### What This Means MPL-2.0 permits commercial use, modification, and distribution. Modified MPL-covered files remain under MPL-2.0, while those files can be combined with separately licensed code in a larger work. When distributing modified MPL-covered files, you must make the source form of those files available under MPL-2.0. Refer to the license text for the complete terms. ### Dependencies Dependencies retain their own licenses. The current public [`wippyai/wasm-runtime`](https://github.com/wippyai/wasm-runtime/blob/main/LICENSE) repository, for example, is licensed under MIT rather than MPL-2.0. --- # "LLM Brief" ## LLM Brief Use this brief as the starting context when generating code for a Wippy project. **Classification: generation reference.** The blocks below are focused contract patterns, not one runnable project. Registry IDs, schemas, policies, and application-specific values such as `user_id`, `config`, and `content` must be defined by the project that uses them. ### What Wippy Is Wippy is a single-binary application runtime built on the actor model. It runs Lua code in isolated processes that communicate through messages rather than shared memory. Its three compute models are functions (stateless and request-scoped), processes (long-lived actors with state), and workflows (durable actors backed by Temporal). Registry-backed behavior can be added or updated without redeploying the runtime. ### Mental Model Everything in Wippy is a **registry entry**. An entry has an ID (`namespace:name`), a kind that determines its behavior, metadata, and data. YAML files are one way to declare entries, but the registry is the runtime source of truth. Entries can also be created, updated, or deleted while the system is running. Kinds determine what an entry does: - `function.lua` — stateless callable function - `process.lua` — long-running actor - `workflow.lua` — durable workflow (Temporal) - `http.service` — HTTP server - `http.router` — route group with middleware - `http.endpoint` — HTTP handler - `db.sql.postgres` / `mysql` / `sqlite` — database connection - `store.memory` / `store.sql` — key-value store - `queue.queue` — message queue - `process.host` — process execution host - `process.service` — supervised process - `contract.definition` / `contract.binding` — typed service interfaces - `registry.entry` — configuration data ### Project Structure ``` myapp/ ├── .wippy.yaml # Runtime configuration ├── wippy.lock # Source directories └── src/ ├── _index.yaml # Entry definitions (namespace: app) ├── api/ │ ├── _index.yaml # namespace: app.api │ └── handler.lua └── workers/ ├── _index.yaml # namespace: app.workers └── task.lua ``` Entry definitions live in `_index.yaml` files: ```yaml version: "1.0" namespace: app.api entries: - name: get_user kind: function.lua source: file://handler.lua method: get_user modules: [sql] - name: get_user.endpoint kind: http.endpoint meta: router: app:api_router method: GET path: /users/{id} func: app.api:get_user ``` ### Writing Functions Functions are stateless: they receive arguments, perform work, and return results. They inherit the caller's context and are canceled when the caller is canceled. ```lua local sql = require("sql") local function get_user(id) local db, err = sql.get("app:main_db") if err then return nil, err end local rows, err = db:query("SELECT * FROM users WHERE id = $1", {id}) if err then return nil, err end if #rows == 0 then return nil, errors.new({kind = errors.NOT_FOUND, message = "user not found"}) end return rows[1] end return get_user ``` For HTTP handlers, use the `http` module: ```lua local http = require("http") local json = require("json") local funcs = require("funcs") local function handler() local req, req_err = http.request() if req_err then return nil, req_err end local res, res_err = http.response() if res_err then return nil, res_err end local id, param_err = req:param("id") if param_err then return nil, param_err end local user, err = funcs.call("app.api:get_user", id) if err then local status_err if errors.is(err, errors.NOT_FOUND) then status_err = res:set_status(404) else status_err = res:set_status(500) end if status_err then return nil, status_err end local write_err = res:write_json({error = err:message()}) if write_err then return nil, write_err end return true end local write_err = res:write_json(user) if write_err then return nil, write_err end return true end return handler ``` ### Writing Processes Processes are actors. Each process has a PID, receives messages through an inbox, and can maintain state across messages. Processes yield while waiting for I/O so other processes can run. ```lua local function worker(initial_config) local inbox = process.inbox() local events = process.events() while true do local r = channel.select { inbox:case_receive(), events:case_receive() } if not r.ok then break end if r.channel == events then local ev = r.value if ev.kind == process.event.CANCEL then break end elseif r.channel == inbox then local msg = r.value local topic = msg:topic() local data, err = msg:payload():data() if err then return nil, err end if topic == "work" then -- Perform the application-specific work here. print(data.item_id) end end end end return worker ``` Spawn processes from other code: ```lua local pid, err = process.spawn("app.workers:task", "app:process_host", config) if err then return nil, err end local ok, send_err = process.send(pid, "work", {item_id = 123}) if send_err then return nil, send_err end return ok ``` ### Writing Workflows Workflows persist execution history so they can resume after crashes or restarts. Workflow code uses normal Lua syntax, while the runtime records function results, sleeps, and random values for deterministic replay. Each `funcs.call()` target below must be registered as an activity on the same Temporal worker through `meta.temporal.activity.worker`. See [Activities](../temporal/activities.md) for the required function metadata. ```lua local funcs = require("funcs") local function compensate(inventory, payment) local _, refund_err = funcs.call("app:refund_payment", payment.id) local _, release_err = funcs.call("app:release_inventory", inventory.id) return refund_err or release_err end local function order_flow(order) local inventory, err = funcs.call("app:reserve_inventory", order.items) if err then return nil, err end local payment, payment_err = funcs.call("app:charge_payment", order.total) if payment_err then local _, release_err = funcs.call("app:release_inventory", inventory.id) return nil, release_err or payment_err end -- Wait for approval signal (can block for days) local msg, open = process.inbox():receive() if not open then local compensation_err = compensate(inventory, payment) return nil, compensation_err or errors.new("workflow inbox closed") end local decision, payload_err = msg:payload():data() if payload_err then local compensation_err = compensate(inventory, payment) return nil, compensation_err or payload_err end if not decision.approved then local compensation_err = compensate(inventory, payment) return nil, compensation_err or errors.new("rejected") end return funcs.call("app:fulfill_order", order.id) end return order_flow ``` #### Calling Functions ```lua local funcs = require("funcs") -- Synchronous local result, err = funcs.call("namespace:function_name", arg1, arg2) if err then return nil, err end -- Asynchronous (returns Future) local future, future_err = funcs.async("namespace:function_name", arg1) if future_err then return nil, future_err end local response_ch = future:response() local _, response_open = response_ch:receive() if not response_open then return nil, errors.new("future response channel closed") end local async_payload, async_err = future:result() if async_err then return nil, async_err end local async_result, decode_err = async_payload:data() if decode_err then return nil, decode_err end -- With context local contextual_exec, contextual_err = funcs.new():with_context({user_id = "123"}) if contextual_err then return nil, contextual_err end local contextual_result, contextual_err = contextual_exec:call("namespace:function_name") if contextual_err then return nil, contextual_err end ``` #### Process Communication ```lua -- Send message (fire-and-forget) local ok, err = process.send(pid, "topic", data) if err then return nil, err end -- Receive messages local inbox = process.inbox() local msg, ok = inbox:receive() if not ok then return nil, errors.new("process inbox closed") end local topic = msg:topic() local data, payload_err = msg:payload():data() if payload_err then return nil, payload_err end -- Monitor another process (receive EXIT on death) local monitored, monitor_err = process.monitor(pid) if monitor_err then return nil, monitor_err end -- Link processes (bidirectional failure notification) local linked_pid, spawn_err = process.spawn_linked("namespace:name", "host") if spawn_err then return nil, spawn_err end ``` #### Channels Go-style channels for coroutine communication: ```lua local ch = channel.new(10) -- buffered ch:send(value) local val, ok = ch:receive() -- Select on multiple channels local r = channel.select { ch1:case_receive(), ch2:case_receive(), timeout:case_receive() } ``` #### Error Handling Functions return `result, error` pairs. Errors are typed objects: ```lua local result, err = some_operation() if err then if errors.is(err, errors.NOT_FOUND) then -- handle not found end return nil, errors.wrap(err, "context message") end ``` Error kinds: `UNKNOWN`, `INVALID`, `NOT_FOUND`, `ALREADY_EXISTS`, `PERMISSION_DENIED`, `TIMEOUT`, `CANCELED`, `UNAVAILABLE`, `INTERNAL`, `CONFLICT`, `RATE_LIMITED`. #### Data Access ```lua -- SQL local sql = require("sql") local db = sql.get("app:main_db") local rows, err = db:query("SELECT * FROM users WHERE active = $1", {true}) db:execute("INSERT INTO users (name) VALUES ($1)", {name}) -- Key-value store local store = require("store") local cache, cache_err = store.get("app:cache") if cache_err then return nil, cache_err end local stored, set_err = cache:set("key", value, 3600) -- TTL in seconds if set_err then cache:release() return nil, set_err end local val, get_err = cache:get("key") cache:release() if get_err then return nil, get_err end -- Queue local queue = require("queue") local published, publish_err = queue.publish("app:tasks", {task = "process", id = 123}) if publish_err then return nil, publish_err end -- Filesystem local fs = require("fs") local vol, volume_err = fs.get("app:storage") if volume_err then return nil, volume_err end local data, read_err = vol:readfile("path/to/file.txt") if read_err then return nil, read_err end local written, write_err = vol:writefile("output.txt", content) if write_err then return nil, write_err end ``` #### HTTP Client ```lua local http_client = require("http_client") local resp, err = http_client.get("https://api.example.com/data", { headers = {Authorization = "Bearer token"}, timeout = "10s" }) if err then return nil, err end local body = resp.body ``` #### Security ```lua local security = require("security") local actor = security.actor() -- who is calling local scope = security.scope() -- what permissions apply if not actor then return nil, errors.new("security actor unavailable") end if not scope then return nil, errors.new("security scope unavailable") end local allowed = security.can("read", "resource:users") -- Token management local ts, store_err = security.token_store("app:tokens") if store_err then return nil, store_err end local token, create_err = ts:create(actor, scope, {expiration = "24h"}) if create_err then ts:close() return nil, create_err end local validated_actor, validated_scope, validate_err = ts:validate(token) ts:close() if validate_err then return nil, validate_err end ``` #### Time ```lua local time = require("time") time.sleep("5s") local now = time.now() local timeout = time.after("30s") -- channel that fires once local ticker = time.ticker("10s") -- ticker:channel() fires every interval ``` #### Registry ```lua local registry = require("registry") local entry, entry_err = registry.get("app.api:get_user") if entry_err then return nil, entry_err end local tests, find_err = registry.find({["meta.type"] = "test"}) if find_err then return nil, find_err end -- Create entries at runtime local snap, snapshot_err = registry.snapshot() if snapshot_err then return nil, snapshot_err end local changes, changes_err = snap:changes() if changes_err then return nil, changes_err end local _, create_err = changes:create({id = "app:new_func", kind = "function.lua", data = {...}}) if create_err then return nil, create_err end local version, apply_err = changes:apply() if apply_err then return nil, apply_err end ``` #### Events ```lua local events = require("events") -- Publish local sent, send_err = events.send("orders", "order.created", "/orders/123", {order_id = "123"}) if send_err then return nil, send_err end -- Subscribe (wildcards supported) local sub, subscribe_err = events.subscribe("orders.*") if subscribe_err then return nil, subscribe_err end local ch = sub:channel() local evt, open = ch:receive() sub:close() if not open then return nil, errors.new("event subscription closed") end ``` ### Module Access Control Each entry receives the restricted base environment and standard libraries, and executable entries also receive the ambient `process` module. Add non-ambient runtime modules to `modules:` and registry-backed libraries to `imports:`. Undeclared non-ambient modules are unavailable. Host Lua facilities such as `os.execute`, `io.open`, `debug.*`, native module loading, and arbitrary `package.path` resolution are not exposed as opt-in runtime modules. The runtime controls availability through its module loader rather than by scanning source code. ```yaml modules: [sql, json, http, time, funcs, store] ``` Workflow entries receive only deterministic modules. The runtime intercepts `time.now()`, `uuid.v4()`, and other non-deterministic calls at the module level, recording results for replay. ### Framework Modules Framework capabilities are distributed as dependencies: - **wippy/llm** — LLM integration (OpenAI, Anthropic, Google). `llm.generate()`, structured output, embeddings, streaming. - **wippy/agent** — Agent framework with tool use, delegation, traits, memory. Agents defined as registry entries. - **wippy/test** — BDD testing. `describe/it` blocks, assertions, mocking. - **wippy/dataflow** — DAG-based workflow orchestration. Function, agent, cycle, parallel nodes. - **wippy/relay** — WebSocket relay with central hub, per-user hubs, plugin routing. - **wippy/views** — Page and component system with template rendering. - **wippy/facade** — Frontend facade and authentication bridge for iframe and Web Fragment pages. ### Conventions - Entry IDs use `namespace:name` format - Names use dots for semantic separation, underscores for words: `get_user.endpoint` - Fallible APIs return `result, error` — always check the error - Processes communicate via message passing, never shared state - Use `channel.select` to multiplex multiple event sources - Let supervision trees handle process failures instead of adding local recovery around every operation - Context (trace IDs, user info, security) propagates automatically through function calls - Workflows must not use non-deterministic operations directly — the runtime handles this for `funcs.call`, `time.sleep`, `uuid.v4`, `time.now` ### Documentation Full documentation is available at [docs.wippy.ai](https://docs.wippy.ai). LLM-friendly endpoints: - Browse structure: `https://wippy.ai/llm/toc` - Search: `https://wippy.ai/llm/search?q=query` - Fetch page: `https://wippy.ai/llm/path/en/` - Batch fetch: `https://wippy.ai/llm/context?paths=path1,path2` ---