# Wippy - Complete Documentation
# What Is Wippy? Concepts and Runtime Overview
## About Wippy
Wippy is an open-source actor-model runtime for applications whose behavior changes while they run. It is designed for automation systems, AI agents, plugin architectures, and other applications that need to evolve without rebuilding or redeploying the runtime.
The foundation is the actor model. Code runs in isolated processes that communicate through messages, with each process managing its own state. Supervised process services can be restarted according to their lifecycle policy.
```lua
local worker, err = process.spawn("app.workers:handler", "app:processes")
if not worker then
return nil, err
end
local ok, monitor_err = process.monitor(worker)
if not ok then
return nil, monitor_err
end
return process.send(worker, "task", {id = 1, data = payload})
```
Runtime definitions live in a central registry. For entry kinds dispatched through the event bus, registered handlers reconcile accepted changes without restarting the entire application. Some internal entry kinds intentionally bypass event dispatch.
```lua
local db, err = registry.get("app.db:postgres")
if not db then
return nil, err
end
local cache, cache_err = registry.get("app.cache:redis")
if not cache then
return nil, cache_err
end
```
For operations that must recover from infrastructure failures, durable workflows persist execution state. This model suits payment flows, multi-step processes, and long-running agent tasks that may need to resume after a restart.
The runtime is distributed as a single binary and configured through project files.
For the project overview and design notes, see [About Wippy](https://wippy.ai/about).
---
# "Installation"
### Install
```bash
curl -fsSL https://hub.wippy.ai/install.sh | bash
```
The install script requires a POSIX shell. On Windows, download the runtime from
[hub.wippy.ai/releases](https://hub.wippy.ai/releases) and place `wippy.exe` on
`PATH`.
### Verify
```bash
wippy version
```
### Initialize Dependency Metadata
```bash
## Create a project directory
mkdir myapp
cd myapp
## Create or update wippy.lock
wippy init
```
`wippy init` writes the dependency lock and its source and module directory settings. It does not scaffold application source files or registry entries. Follow [Hello World](tutorials/hello-world.md) to create a runnable application, then start it with `wippy run`.
The runtime includes HTTP, SQL, storage, and process-hosting capabilities. Add framework modules from the Hub when the application needs them:
```bash
wippy add wippy/test
wippy install
```
### Commands Overview
| Command | Description |
| --------- | ------------- |
| `wippy init` | Create or update `wippy.lock` |
| `wippy run` | Start the runtime |
| `wippy test` | Run the test entrypoint |
| `wippy lint` | Check code for errors |
| `wippy add` | Add a dependency |
| `wippy install` | Install dependencies |
| `wippy update` | Update dependencies |
| `wippy artifacts` | Materialize build-time filesystem artifacts |
| `wippy pack` | Create a snapshot |
| `wippy publish` | Publish to hub |
| `wippy search` | Search for modules |
| `wippy readme` | Fetch a module README from the hub |
| `wippy registry` | Inspect loaded registry entries |
| `wippy auth` | Manage authentication |
| `wippy version` | Print version info |
See [CLI Reference](guides/cli.md) for full documentation.
### Troubleshooting
If the shell cannot find `wippy` after installation, reopen the shell and verify that the installation directory is on `PATH`.
### Next Steps
- [Hello World](../tutorials/hello-world.md) — Create your first application
- [Project Structure](start/structure.md) — Understand the project layout
- [CLI Reference](guides/cli.md) — Review all commands and options
---
# "YAML & Project Structure"
### Directory Layout
```
myapp/
├── .wippy.yaml # Runtime configuration
├── wippy.lock # Source directories and locked modules
├── .wippy/ # Installed modules
└── src/ # Application source
├── _index.yaml # Entry definitions
├── api/
│ ├── _index.yaml
│ └── *.lua
└── workers/
├── _index.yaml
└── *.lua
```
### YAML Definition Files
YAML definitions are loaded into the registry at startup. The registry is the source of truth; YAML files are one way to populate it. Entries can also come from other sources or be created programmatically.
#### Definition File Format
A definition file contains a `namespace` and either an `entries` array or top-level `name` and `kind` fields. The optional `version` marker is conventionally `"1.0"`; the v0.3.32a loader does not require it.
```yaml
version: "1.0"
namespace: app.api
entries:
- name: get_user
kind: function.lua
meta:
comment: Fetches user by ID
source: file://get_user.lua
method: handler
modules:
- sql
- json
- name: get_user.endpoint
kind: http.endpoint
meta:
comment: User API endpoint
method: GET
path: /users/{id}
func: get_user
```
| Field | Required | Description |
|-------|----------|-------------|
| `version` | No | Manifest version marker (conventionally `"1.0"`) |
| `namespace` | Yes | Entry namespace for this file |
| `entries` | Conditional | Array of entry definitions; omit only when using top-level `name` and `kind` |
#### Naming Convention
Use dots (`.`) for semantic separation and underscores (`_`) for words:
```yaml
## Function and its endpoint
- name: get_user # The function
- name: get_user.endpoint # Its HTTP endpoint
## Multiple endpoints for same function
- name: list_orders
- name: list_orders.endpoint.get
- name: list_orders.endpoint.post
## Routers
- name: api.public # Public API router
- name: api.admin # Admin API router
```
Pattern: base_name.variant — dots separate semantic parts, while underscores separate words within a part.
#### Namespaces
Namespaces are dot-separated identifiers:
```
app
app.api
app.api.v2
app.workers
```
Entry full ID combines namespace and name: `app.api:get_user`
#### The Lock File
`wippy.lock` records where Wippy loads definitions from and which module versions are selected:
```yaml
directories:
modules: .wippy
src: ./src
options:
unpack_modules: false
modules:
- name: acme/http
version: v1.2.0
hash: 4ea816fe84ca58a1f0869e5ca6afa93d6ddd72fa09e1162d9e600a7fbf39f0a2
```
| Field | Description |
|-------|-------------|
| `directories.src` | Application source directory, scanned recursively for YAML definition files |
| `directories.modules` | Base directory for vendored modules; packs land under `/vendor/` |
| `options.unpack_modules` | Extract each `.wapp` into a directory beside it instead of loading the pack directly (default `false`) |
| `modules[].name` | Module identifier in `org/module` form |
| `modules[].version` | Selected version |
| `modules[].hash` | Artifact digest the vendored pack must match |
| `modules[].root` | Marks the selected deployment root; at most one module may carry it |
Vendored packs are kept as `.wapp` files. With `unpack_modules: true`, each module is also extracted into a directory, and the verified `.wapp` stays beside it — installation looks for the pack, so a directory whose pack is missing is downloaded again.
A `replacements:` section in `wippy.lock` is deprecated. It still loads, with a warning; declare local module overrides under `workspace.replacements` in a runtime config file instead. See [Dependency Management](guides/dependency-management.md#local-development-with-replacements).
### Entry Definitions
Each item in the `entries` array defines one entry. Kind-specific fields can appear beside `name`, `kind`, and `meta`, as in this example:
```yaml
entries:
- name: hello
kind: function.lua
meta:
comment: Returns hello world
source: file://hello.lua
method: handler
modules:
- http
- json
- name: hello.endpoint
kind: http.endpoint
meta:
comment: Hello endpoint
method: GET
path: /hello
func: hello
```
An explicit `data:` field is also supported. When present, its value is the complete kind-specific payload, so do not mix it with sibling kind-specific fields:
```yaml
entries:
- name: config
kind: registry.entry
data:
environment: production
features:
dark_mode: true
```
#### Metadata
Use `meta` for UI-friendly information:
```yaml
- name: payment_handler
kind: function.lua
meta:
title: Payment Processor
comment: Handles Stripe payments
source: file://payment.lua
```
Use `meta.title` and `meta.comment` for descriptive information that registry consumers and management interfaces can display.
#### Application Entries
Use `registry.entry` kind for application-level configuration:
```yaml
- name: config
kind: registry.entry
meta:
title: Application Settings
type: application
environment: production
features:
dark_mode: true
beta_access: false
```
### Common Entry Kinds
| Kind | Purpose |
|------|---------|
| `registry.entry` | General-purpose data stored without normal event dispatch |
| `function.lua` | Callable Lua function |
| `process.lua` | Long-running process |
| `http.service` | HTTP server |
| `http.router` | Route group |
| `http.endpoint` | HTTP handler |
| `process.host` | Process execution host |
See the [Entry Kinds Guide](guides/entry-kinds.md) for the entry-kind reference.
#### .wippy.yaml
Runtime configuration at project root:
```yaml
version: "1.0"
logger:
encoding: json
logmanager:
min_level: 0
supervisor:
host:
worker_count: 16
```
See the [Configuration Guide](guides/configuration.md) for runtime configuration fields.
#### wippy.lock
Source directories and the selected module graph — see [The Lock File](#the-lock-file) above.
### Referencing Entries
Reference entries by full ID or relative name where the entry kind supports it. HTTP routers and endpoints attach through `meta.server` and `meta.router`, rather than through parent-side child lists:
```yaml
## Router declares itself against a server
- name: api
kind: http.router
meta:
server: app:gateway
prefix: /api
## Endpoint references router by registry ID (cross-namespace works the same way)
- name: get_user.endpoint
kind: http.endpoint
meta:
router: app.api:api
method: GET
path: /users/{id}
func: app.api:get_user
```
### Example Project
```
myapp/
├── .wippy.yaml
├── wippy.lock
└── src/
├── _index.yaml # namespace: app
├── api/
│ ├── _index.yaml # namespace: app.api
│ ├── users.lua
│ └── orders.lua
├── lib/
│ ├── _index.yaml # namespace: app.lib
│ └── database.lua
└── workers/
├── _index.yaml # namespace: app.workers
└── email_sender.lua
```
### See Also
- [Application Architecture](concepts/architecture.md) — Organize an application into slices and layers
- [Entry Kinds Guide](guides/entry-kinds.md) — Review available entry kinds
- [Configuration Guide](guides/configuration.md) — Configure runtime options
- [Custom Entry Kinds](internals/kinds.md) — Implement handlers (advanced)
---
# "CLI Reference"
## CLI Reference
Use the Wippy CLI to initialize projects, run the runtime, manage dependencies, inspect registry entries, and publish modules.
This is a command reference. The examples assume an existing project or module
when the command operates on source, a lock file, registry entries, or publish
metadata; they are not a single end-to-end project.
### Global Flags
Available on all commands:
| Flag | Short | Description |
|------|-------|-------------|
| `--config` | | Config file, repeatable; later files override earlier ones (default: .wippy.yaml). `wippy publish` defines a different command-local option. |
| `--verbose` | `-v` | Enable debug logging |
| `--very-verbose` | | Debug with stack traces |
| `--console` | `-c` | Colorful console logging |
| `--silent` | `-s` | Disable console logging |
| `--event-streams` | `-e` | Stream logs to event bus |
| `--profiler` | `-p` | Enable pprof on localhost:6060 |
| `--memory-limit` | `-m` | Memory limit (e.g., 1G, 512M) |
Memory-limit precedence is `--memory-limit`, then `GOMEMLIMIT`, then the 1 GB default.
The global `--config` option may be passed multiple times to compose config files. Files merge left to right: later files override matching values and keep everything else. Every explicitly named file must exist; without `--config`, the default `.wippy.yaml` is optional. The first file anchors the directory used to resolve relative paths. Configuration applies in order: file composition, then `--profile` selections, then `--set` overrides. See [Configuration](guides/configuration.md#config-composition).
`wippy publish` shadows the global option with a command-local `--config ` option. For that command, the value is the directory containing `wippy.yaml`, not a repeatable runtime configuration file.
### wippy init
Create `wippy.lock`, or update its source and module directory settings if it already exists. This command does not scaffold application source files or registry entries.
```bash
wippy init
wippy init --src-dir ./src --modules-dir .wippy
```
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--src-dir` | `-d` | ./src | Source directory |
| `--modules-dir` | | .wippy | Modules directory |
| `--lock-file` | `-l` | wippy.lock | Lock file path |
### wippy run
Start the runtime or execute a command.
```bash
wippy run # Start runtime
wippy run list # List available commands
wippy run migrate # Run a named custom command
wippy run snapshot.wapp # Run from pack file
wippy run acme/http # Run module from hub
wippy run acme/http@1.2.3 # Run specific version
wippy run --exec app:worker # Start runtime and execute a single process
```
| Flag | Short | Description |
|------|-------|-------------|
| `--override` | `-o` | Override entry values (`namespace:entry:field=value`); `field` may be `kind` to change the entry kind |
| `--set` | | Override a config value (`section.path=value`, repeatable, takes precedence over the config file) |
| `--exec` | `-x` | Execute process and exit (`namespace:entry`) |
| `--host` | | Terminal host ID for `--exec` (auto-detected if only one `terminal.host` exists) |
| `--registry` | | Registry URL for hub modules |
| `--profile` | | Apply a runtime profile from `.wippy.yaml` or packed runtime metadata (repeatable, applied in order) |
Running a hub module (`wippy run org/module`) resolves it once, records it in `wippy.lock`, and vendors the verified packs locally. Subsequent runs of the same reference start from the lock — no network needed. A version selector that no longer matches the lock is rejected with a hint to run `wippy update`.
For a local application, `wippy run` repairs a stale lock before any runtime service starts. It loads the source dependency declarations, and when the lock already satisfies them it re-resolves the graph from local and installed evidence only (verified-offline access, no network). If that offline resolution matches the lock, boot continues unchanged. If it succeeds but differs, it becomes the candidate graph; the hub is asked to resolve only when the offline pass fails or the lock no longer satisfies the source declarations. Packs the candidate graph is missing are downloaded and verified, and only then is `wippy.lock` rewritten. A lock that selects a deployment root is authoritative and is never re-resolved.
`--exec` blocks until the launched process produces its result, then propagates the process exit code as the CLI exit code. Ctrl-C during `--exec` cancels the running process and the runtime still shuts down gracefully; a second signal forces exit.
`--set` writes any runtime configuration value from the command line, merged over `.wippy.yaml` per leaf:
```bash
wippy run --set cluster.enabled=true \
--set cluster.membership.join_addrs=node-2:7946,node-3:7946 \
--set cluster.raft.bootstrap_expect=3
```
Values are coerced by shape: `true` and `false` become booleans, integers and floats become numbers, and other values remain strings. Fields that expect durations parse values such as `5s`.
### wippy test
Run the test entrypoint: the process entry declaring the `test` use case. The runtime boots, executes that entry, and exits. `wippy run` does not auto-run test entrypoints; testing always goes through `wippy test`.
```bash
wippy test # Run tests from the local project
wippy test snapshot.wapp # Run tests from a pack file
wippy test acme/module@1.2.3 # Run tests from a hub module
```
| Flag | Short | Description |
|------|-------|-------------|
| `--override` | `-o` | Override entry values (`namespace:entry:field=value`) |
| `--host` | | Terminal host ID (auto-detected if only one `terminal.host` exists) |
| `--registry` | | Registry URL for hub modules |
| `--set` | | Override a config value (`section.path=value`, repeatable) |
| `--profile` | | Apply a runtime profile (repeatable, applied in order) |
### wippy lint
Check Lua code for type errors and warnings.
```bash
wippy lint
wippy lint --level warning
wippy lint --json
wippy lint --rules
```
Validates source-bearing `function.lua`, `library.lua`, `process.lua`, and
`workflow.lua` entries. Precompiled `.bc` entries do not contain parseable source
and are skipped.
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--lock-file` | `-l` | `wippy.lock` | Lock file path |
| `--level` | | `warning` | Minimum severity: `error`, `warning`, `hint` |
| `--ns` | | | Filter by namespace patterns (e.g. `app`, `lib.*`) |
| `--code` | | | Filter by error codes (e.g. `E0001,E0004`) |
| `--rules` | | `false` | Enable style/quality lint rules |
| `--summary` | | `false` | Group output by error code |
| `--limit` | | `0` | Max diagnostics shown (0 = unlimited) |
| `--json` | | `false` | JSON output |
| `--no-color` | | `false` | Disable colored output |
| `--cache-reset` | | `false` | Clear Lua cache before linting |
| `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable) |
| `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) |
### wippy add
Add a module dependency.
```bash
wippy add acme/http
wippy add acme/http@1.2.3
wippy add acme/http@latest
```
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--lock-file` | `-l` | wippy.lock | Lock file path |
| `--registry` | | | Registry URL |
### wippy install
Install dependencies from lock file.
```bash
wippy install # Install all
wippy install acme/http # Install specific module
wippy install --refresh acme/http # Re-fetch a specific module
```
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--lock-file` | `-l` | wippy.lock | Lock file path |
| `--refresh` | | false | Re-fetch the named modules, or every locked module when no names are supplied, bypassing cache |
| `--force` | | false | Alias for `--refresh` |
| `--repair` | | false | Alias for `--refresh` |
| `--registry` | | | Registry URL |
| `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable) |
| `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) |
### wippy update
Update dependencies and regenerate lock file.
```bash
wippy update # Update all
wippy update acme/http # Update specific module
wippy update acme/http demo/sql # Update multiple
```
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--lock-file` | `-l` | wippy.lock | Lock file path |
| `--src-dir` | `-d` | ./src | Source directory |
| `--modules-dir` | | .wippy | Modules directory |
| `--registry` | | | Registry URL |
| `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable) |
| `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) |
### wippy artifacts
Work with build-time filesystem artifacts.
#### wippy artifacts materialize
Validate and materialize one artifact filesystem out of an existing pack.
```bash
wippy artifacts materialize snapshot.wapp app:package_fs
wippy artifacts materialize snapshot.wapp app:package_fs --root build
```
| Flag | Default | Description |
|------|---------|-------------|
| `--root` | `.wippy` | Materialization root |
The resource is addressed by its full `namespace:name`, must declare `meta.artifact.format`, and that format must be registered in the CLI. The command resolves no module dependencies, does not mutate `wippy.lock`, invokes no package managers, and takes no part in runtime composition. See [Build-time artifacts](guides/artifacts.md#materializing-explicitly).
### wippy pack
Create a snapshot pack (.wapp file).
```bash
wippy pack snapshot.wapp
wippy pack release.wapp --description "Release 1.0"
wippy pack app.wapp --embed app:assets --bytecode "**"
```
| Flag | Short | Description |
|------|-------|-------------|
| `--lock-file` | `-l` | Lock file path |
| `--description` | `-d` | Pack description |
| `--tags` | `-t` | Pack tags (comma-separated) |
| `--meta` | | Custom metadata (key=value) |
| `--embed` | | Embed fs.directory entries (patterns) |
| `--embed-all` | | Embed all fs.directory entries (cannot combine with `--embed`) |
| `--list` | | List fs.directory entries (dry-run) |
| `--exclude-ns` | | Exclude namespaces (patterns) |
| `--exclude` | | Exclude entries (patterns) |
| `--bytecode` | | Compile Lua to bytecode (** for all) |
| `--profile` | | Apply a runtime profile from `.wippy.yaml` before packing (repeatable, applied in order) |
Without `--embed` or `--embed-all`, embed patterns fall back to the `embed:` section of the module manifest `wippy.yaml`. Packing an application also carries embedded resources from its dependency packs, and only the main module's commands are exposed by the resulting pack.
The output file is written atomically: the pack is built into a temporary file in the destination directory, synced, verified, and only then renamed over the target, inheriting the existing file's permissions when one is present. A failed pack leaves the previous file untouched. Naming an output that is also one of the pack's inputs — the same path, or a hard link or symlink resolving to the same file — is refused rather than truncating the input mid-read.
`--meta` cannot write reserved metadata. The key `registry`, and anything under the `wippy.` or `system.` prefixes, is owned by the pack format and rejected.
Resources declaring `meta.artifact.format` are validated while packing, so a malformed artifact fails here rather than in a consumer. See [Build-time artifacts](guides/artifacts.md).
### wippy publish
Publish module to the hub.
```bash
wippy publish
wippy publish --version 1.0.0
wippy publish --dry-run
```
This command reads `wippy.yaml` from the current directory.
| Flag | Description |
|------|-------------|
| `--version` | Version to publish |
| `--dry-run` | Validate without publishing |
| `--label` | Publish as mutable label instead of version |
| `--release-notes` | Release notes |
| `--protected` | Mark version as protected |
| `--embed` | Embed fs.directory entries by id or name |
| `--config` | Path to directory containing wippy.yaml (default: .) |
| `--registry` | Registry URL |
| `--create` | Create the module on the registry if it does not yet exist |
| `--module-visibility` | Visibility for newly created modules (`--create` only): `public` or `private` (default: private) |
| `--module-type` | Module type: `library`, `application`, `agent`, or `plugin` (overrides `type:` in wippy.yaml) |
| `--module-display-name` | Display name for newly created modules (`--create` only) |
The module type is normally declared as `type:` in `wippy.yaml` (see [Publishing](./publishing.md#wippyyaml)); `--module-type` overrides it for a single publish. When neither is set, newly created modules default to `application` with a deprecation warning.
### wippy search
Search for modules in the hub.
Search uses the stored authentication token for the selected registry, when
available. Authenticate with `wippy auth login` to search with your registry
identity; `--registry` selects which registry's credentials are used.
```bash
wippy search http
wippy search "sql driver" --limit 20
wippy search auth --json
```
| Flag | Default | Description |
|------|---------|-------------|
| `--json` | false | Output as JSON |
| `--limit` | 20 | Maximum results |
| `--registry` | | Registry URL |
### wippy auth
Manage registry authentication.
#### wippy auth login
```bash
wippy auth login
wippy auth login --token YOUR_TOKEN
```
| Flag | Description |
|------|-------------|
| `--token` | API token |
| `--registry` | Registry URL |
| `--local` | Store credentials locally |
#### wippy auth logout
```bash
wippy auth logout
```
| Flag | Description |
|------|-------------|
| `--registry` | Registry URL |
| `--local` | Remove local credentials |
#### wippy auth status
```bash
wippy auth status
wippy auth status --json
```
| Flag | Description |
|------|-------------|
| `--json` | Output as JSON |
### wippy readme
Fetch a module README from the hub.
```bash
wippy readme wippy/terminal
wippy readme wippy/terminal@1.2.3
wippy readme --json wippy/terminal@latest
```
| Flag | Description |
|------|-------------|
| `--json` | Output as JSON |
| `--registry` | Registry URL (default: from credentials) |
### wippy registry
Query and inspect registry entries. Both subcommands accept `--profile` and `--set` to control the merged runtime configuration used to load entries.
#### wippy registry list
```bash
wippy registry list
wippy registry list --kind "function.lua.*"
wippy registry list --ns "app.*" --json
wippy registry list --meta "type=api" --meta "enabled=true"
```
| Flag | Short | Description |
|------|-------|-------------|
| `--kind` | `-k` | Filter by kind (glob pattern) |
| `--ns` | `-n` | Filter by namespace (glob pattern) |
| `--name` | | Filter by name (glob pattern) |
| `--meta` | | Filter by metadata (repeatable) |
| `--json` | | Output as JSON |
| `--yaml` | | Output as YAML |
| `--registry-meta` | | Include registry-owned metadata (`owner`, `root`) in JSON or YAML output; requires `--json` or `--yaml` |
| `--lock-file` | `-l` | Lock file path |
Metadata operators for `--meta`:
| Operator | Meaning |
|----------|---------|
| `field=value` | Exact match |
| `field~regex` | Regex match |
| `field*substr` | Contains substring |
| `field^prefix` | Starts with prefix |
| `field$suffix` | Ends with suffix |
#### wippy registry show
```bash
wippy registry show app:http:handler
wippy registry show app:config --yaml
```
| Flag | Short | Description |
|------|-------|-------------|
| `--field` | `-f` | Show specific field |
| `--json` | | Output as JSON |
| `--yaml` | | Output as YAML |
| `--raw` | | Raw output |
| `--lock-file` | `-l` | Lock file path |
### wippy version
Print version information.
```bash
wippy version
wippy version --short
```
### Custom Commands
Any `process.lua` or `process.wasm` entry can be registered as a named command by adding `command` metadata:
```yaml
entries:
- name: migrate_runner
kind: process.lua
meta:
command:
name: migrate
short: Run database migrations
security:
actor:
id: app:migrations
policies:
- app.security:migrations
groups:
- app.security:operators
source: file://runner.lua
method: main
modules:
- io
- registry
- funcs
```
Run it with:
```bash
wippy run migrate
```
List all available commands:
```bash
wippy run list
```
`wippy run list` accepts `--profile` and `--set` so the listing reflects the same merged runtime config `wippy run` would use.
#### Command Metadata Fields
| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Command name used with `wippy run ` |
| `short` | No | Short description shown in `wippy run list` |
| `main` | No | Mark this entry as the default entrypoint. When a pack or hub module is run without a command name, the single `main` entry of that use case is executed; a lone entrypoint is picked even without `main`, and several entrypoints with no `main` is an error |
| `use_case` | No | Entrypoint category, default `run`. The entry declaring `use_case: test` is what `wippy test` executes |
| `security` | No | Security context the command runs under when launched from the CLI |
Any process entry kind works (`process.lua`, `process.wasm`). Command names are not checked for uniqueness; when several loaded entries declare the same name, the first match in registry order runs. Arguments after the command name are passed to the process as string payloads.
#### Command security
A command entry declares the actor and policy scope its CLI launch runs under:
```yaml
entries:
- name: migrate_runner
kind: process.lua
meta:
command:
name: migrate
short: Run database migrations
security:
actor:
id: system.migrations
meta:
role: operator
policies:
- app.security:migrations_policy
groups:
- app.security:operators
source: file://runner.lua
method: main
```
| Field | Description |
|-------|-------------|
| `actor.id` | Actor identity for the launched process |
| `actor.meta` | Actor attributes evaluated by policies |
| `policies` | Registry IDs (`namespace:name`) of individual policies added to the scope |
| `groups` | Registry IDs of policy groups whose policies are added to the scope |
The block lives inside `meta.command` because it applies only to the CLI launch path — the operator started the command on their own deployment, which is the trust anchor. It has no effect on ordinary spawns of the same process entry; those follow the entry's own [`security:` block](guides/entry-kinds.md#process-security).
Declaration is fail-closed and validated before the process starts:
- Unknown fields inside `security` are rejected.
- An empty `security` block (no actor, no policies, no groups) is rejected.
- `security` without a `name` is rejected — a command must be nameable to be launched.
- A policy or group that cannot be resolved refuses the launch; resolution is atomic, so a partial scope is never installed.
When the block omits `actor`, the caller's actor is inherited. When it omits both `policies` and `groups`, the caller's scope is inherited.
#### Development Workflow
```bash
## Initialize dependency lock metadata
wippy init
wippy add wippy/test
wippy add wippy/llm
wippy install
## Check for errors
wippy lint
## Run with debug output
wippy run -c -v
## Override config for local dev
wippy run -o app:db:host=localhost -o app:db:port=5432
```
#### Production Deployment
```bash
## Create release pack with bytecode
wippy pack release.wapp --bytecode "**" --exclude-ns "test.**"
## Run from pack with memory limit
wippy run release.wapp -m 2G
```
#### Debugging
```bash
## Execute single process
wippy run --exec app:worker
## With profiler enabled
wippy run -p -v
## Then: go tool pprof http://localhost:6060/debug/pprof/heap
```
#### Dependency Management
```bash
## Add new dependency
wippy add acme/http@latest
## Force re-download
wippy install --force
## Update specific module
wippy update acme/http
```
#### Publishing
```bash
## Login to hub
wippy auth login
## Validate module
wippy publish --dry-run
## Publish
wippy publish --version 1.0.0 --release-notes "Initial release"
```
### Environment Variables
| Variable | Effect |
|----------|--------|
| `WIPPY_TOKEN` | Registry auth token; overrides stored credentials (a token pushed via `hub.auth.authenticate` ranks higher still) |
| `WIPPY_REGISTRY` | Default registry URL (overridden by `--registry`) |
| `WIPPY_CACHE_DIR` | Cache directory for hub modules run via `wippy run org/module` (default: `~/.wippy/cache`) |
| `GOMEMLIMIT` | Memory-limit fallback when `--memory-limit` is not set |
Values in `.wippy.yaml` may reference OS environment variables with `${env:NAME}`, resolved at file load; a missing variable fails config loading. Bare `${name}` references resolve from the config's `vars:` section instead.
### Configuration File
Create `.wippy.yaml` for persistent settings:
```yaml
logger:
encoding: console
logmanager:
stream_to_events: true
profiler:
enabled: true
address: localhost:6060
override:
app:gateway:addr: ":9090"
app:db:host: "localhost"
```
### See Also
- [Configuration](guides/configuration.md) — Configuration file reference
- [Observability](guides/observability.md) — Monitoring and logging
---
# "Configuration Reference"
## Configuration Reference
Wippy reads runtime configuration from `.wippy.yaml` files.
Use the repeatable `wippy run --set section.path=value` option to override the configuration fields below at launch. To override individual registry *entries* rather than configuration sections, use the `override:` section or `-o`; see [Overriding Entries](guides/entry-kinds.md#overriding-entries).
### Config Composition
`--config` is repeatable; files compose left to right using the same schema:
```bash
wippy run --config .wippy.yaml --config .wippy.local.yaml
```
- Later files override matching values and keep everything else.
- Every explicitly named file must exist. Without `--config`, the default `.wippy.yaml` is optional.
- The first file anchors the directory used to resolve relative paths.
- Filenames carry no reserved meaning; nothing besides the default is auto-discovered.
Configuration applies in this order: composed files, selected `--profile` overlays, and then `--set` overrides. For applications run from packs, packed runtime defaults have lower precedence than all three; see [Publishing Runtime Defaults](guides/publishing.md#publishing-runtime-defaults).
### Profiles
A configuration file may declare named overlays under `profiles:`. Each profile body mirrors the standard configuration sections. Selecting it with `--profile ` applies those values over the merged base configuration:
```yaml
version: "1.0"
vars:
port: 8085
override:
app:db:kind: db.sql.sqlite
disable:
namespaces: ["legacy.**"]
profiles:
pg:
vars:
port: 18085
override:
app:db:kind: db.sql.postgres
disable:
namespaces.add: ["experimental.**"]
```
```bash
wippy run --profile pg
```
- `--profile` is repeatable; profiles compose left to right, after file composition and before `--set`. An unknown name is an error.
- Values merge per leaf (last writer wins). The `profiles:` section itself is stripped from the resolved config.
- The `disable` section supports list operations inside profiles — `namespaces.add`, `namespaces.remove`, `entries.add`, `entries.remove` — so a profile can adjust the base list instead of replacing it.
- `${name}` references interpolate from the merged `vars:` section. OS environment references are not allowed inside profile vars; use `${env:NAME}` in the base config, resolved at file load.
`wippy run`, `test`, and `pack` accept `--profile`; `run list`, `install`, `update`, `lint`, and `registry` accept it as well for workspace profiles (together with `--set`). Applications can ship profiles inside packs — see [Publishing Profiles](guides/publishing.md#publishing-profiles).
### Logger
Controls the zap logger encoder. CLI flags (`-v`, `-c`, `-s`) override the level and output; encoding is the only YAML-configured option.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `encoding` | string | console | Encoder: `console` (humanized) or `json` (structured) |
```yaml
logger:
encoding: json
```
### Log Manager
Controls runtime log routing. Console output is configured via [CLI flags](guides/cli.md) (`-v`, `-c`, `-s`).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `propagate_downstream` | bool | true | Send logs to console/file output |
| `stream_to_events` | bool | false | Publish logs to event bus for programmatic access |
| `min_level` | int | 0 (`-1` with `-v`) | Minimum level: -1=debug, 0=info, 1=warn, 2=error. The CLI writes this key from its flags after the file is read, so a file value is ignored; change it with `--set logmanager.min_level=` |
```yaml
logmanager:
propagate_downstream: true
stream_to_events: false
```
See: [Logger Module](lua/system/logger.md)
### Profiler
Go pprof HTTP server for CPU/memory profiling. Enable with `-p` flag or config.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | false | Start profiler server |
| `address` | string | localhost:6060 | Listen address |
| `read_timeout` | duration | 15s | HTTP read timeout |
| `write_timeout` | duration | 15s | HTTP write timeout |
| `idle_timeout` | duration | 60s | Keep-alive timeout |
```yaml
profiler:
enabled: true
address: "localhost:6060"
```
When enabled with the default address, the profiler is available at `http://localhost:6060/debug/pprof/`.
### Security
Global security behavior. Individual policies are defined as [security.policy entries](guides/entry-kinds.md).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `strict_mode` | bool | true | Deny access when the security context is incomplete |
```yaml
security:
strict_mode: false
```
See: [Security System](system/security.md), [Security Module](lua/security/security.md)
### Registry
Entry storage and version history. The registry holds all configuration entries.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enable_history` | bool | true | Track entry versions |
| `history_type` | string | memory | Storage: `memory`, `sqlite`, `postgres`, `nil` |
| `history_path` | string | .wippy/registry.db | SQLite file path (used when `history_type: sqlite`) |
| `history_dsn` | string | | Postgres DSN (used when `history_type: postgres`) |
| `history_schema` | string | | Postgres schema name (used when `history_type: postgres`) |
| `event_wait_timeout` | duration | 30s | Per-operation wait for listener acknowledgement during a registry apply |
| `dispatch_internal_kinds` | string[] | `[registry.entry, ns.dependency, ns.requirement, ns.definition]` | Entry kinds handled internally instead of dispatched to component listeners |
| `dependency_resolve_timeout` | duration | 0 (none) | Bound on dependency resolution |
| `dependency_download_timeout` | duration | 0 (none) | Bound on each module download and download-URL request |
| `dependency_lock_path` | string | discovered `wippy.lock` | Lock file the dependency handler reads and writes |
| `dependency_vendor_dir` | string | `//vendor` | Directory holding downloaded module packs |
```yaml
registry:
history_type: sqlite
history_path: /var/lib/wippy/registry.db
```
```yaml
registry:
history_type: postgres
history_dsn: ${env:WIPPY_REGISTRY_HISTORY_DSN}
history_schema: wippy_registry
```
See: [Registry Concept](concepts/registry.md), [Registry Module](lua/core/registry.md)
### Artifact
Output root for materialized [build-time artifacts](guides/artifacts.md).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `materialization_root` | string | parent of the dependency vendor directory | Application-owned root under which each artifact format writes its own subtree |
```yaml
artifact:
materialization_root: build/wippy
```
See: [Build-time artifacts](guides/artifacts.md#where-output-lands)
### Workspace
Local module replacements, keyed by `org/module`. Values are directories; relative paths resolve against the first `--config` file's directory, and `null` disables a replacement inherited from an earlier config layer or profile.
```yaml
workspace:
replacements:
acme/http: ../local-http
acme/sql: null
```
Replacements are never written to `wippy.lock`. See [Local Development with Replacements](guides/dependency-management.md#local-development-with-replacements).
### Relay
Message routing between processes across nodes.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `node_name` | string | derived per-instance ID | Identifier for this relay node (default: UUIDv5 of machine-id/hostname + working dir; overridable via `WIPPY_NODE_ID` / `WIPPY_RELAY_NODE_NAME`) |
```yaml
relay:
node_name: worker-1
```
See: [Process Model](concepts/process-model.md)
### Supervisor
Service lifecycle management. Controls the supervisor's internal control mailbox used to dispatch lifecycle events.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `host.buffer_size` | int | 1024 | Internal control mailbox capacity |
| `host.worker_count` | int | 16 | Concurrent dispatcher workers |
```yaml
supervisor:
host:
buffer_size: 2048
worker_count: 32
```
See: [Supervision Guide](guides/supervision.md)
Per-`process.host` workers and queues are configured on the entry itself (`workers`, `queue_size`, `local_queue_size`), not in this global section. See the [Process Host](system/process-host.md) entry kind.
### Lua Runtime
Lua VM caching and expression evaluation.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `cache.enabled` | bool | `type_system.enabled` | Persist compiled bytecode/typecheck cache to disk; follows `type_system.enabled` unless set explicitly |
| `cache.dir` | string | `.wippy/cache/lua` | Cache directory path (relative to the config/working directory) |
| `cache.mode` | string | `readwrite` | Cache mode: `readwrite` (default), `readonly`, `off`; unknown values fall back to `readwrite` |
| `cache.compile.enabled` | bool | true | Persist compiled bytecode (when `cache.enabled`) |
| `cache.typecheck.enabled` | bool | true | Persist typecheck results (when `cache.enabled`) |
| `cache.max_bytes` | int | 1073741824 | On-disk cache size ceiling in bytes |
| `cache.max_entries` | int | 20000 | Maximum cached entries |
| `cache.prune_interval` | int | 256 | Writes between cache prune passes |
| `type_system.enabled` | bool | false | Enable static type checking |
| `type_system.strict` | bool | false | Treat type warnings as errors |
| `invalidation_wait_timeout` | duration | `registry.event_wait_timeout` (30s) | Wait for code invalidation to be acknowledged after an entry changes |
| `eval.max_steps` | int | 10000 | Default scheduler-step budget for an `eval` run; negative values are rejected |
| `eval.cache_size` | int | 256 | Compiled-program cache entries for evaluated source |
| `eval.cache_ttl` | duration | 0 (no expiry) | Lifetime of a cached compiled program |
```yaml
lua:
cache:
enabled: true
dir: .cache/lua
type_system:
enabled: true
```
See: [Lua Overview](lua/overview.md)
### Scheduler
Core partitioning for the WASM runtime. When enabled, `reserved_cores` CPUs are set aside for WASM execution and the rest serve the actor scheduler; an invalid split (for example more reserved cores than available) is logged and ignored.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `wasm_isolation.enabled` | bool | false | Partition cores between WASM and actor work |
| `wasm_isolation.reserved_cores` | int | 1 | Cores reserved for WASM execution |
```yaml
scheduler:
wasm_isolation:
enabled: true
reserved_cores: 2
```
### Finder
Registry search caching. Used internally for entry lookups.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `query_cache_size` | int | 1000 | Cached query results |
| `regex_cache_size` | int | 100 | Compiled regex patterns |
```yaml
finder:
query_cache_size: 2000
```
### OpenTelemetry
Distributed tracing and metrics export via OTLP.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | false | Enable OTEL |
| `endpoint` | string | localhost:4318 | OTLP endpoint |
| `protocol` | string | http/protobuf | Protocol: grpc, http/protobuf |
| `service_name` | string | wippy-runtime | Service identifier |
| `service_version` | string | | Service version tag |
| `insecure` | bool | true | Allow plaintext OTLP connection |
| `sample_rate` | float | 1.0 | Trace sampling (0.0-1.0) |
| `propagators` | string[] | `[tracecontext, baggage]` | Context propagators |
| `traces_enabled` | bool | true | Export traces |
| `metrics_enabled` | bool | false | Export metrics |
| `http.enabled` | bool | true | Trace HTTP requests |
| `http.extract_headers` | bool | true | Extract trace context from inbound headers |
| `http.inject_headers` | bool | true | Inject trace context into the HTTP response |
| `process.enabled` | bool | true | Trace process lifecycle |
| `process.trace_lifecycle` | bool | true | Emit spans for spawn/terminate |
| `interceptor.enabled` | bool | true | Trace function calls |
| `interceptor.order` | int | 100 | Decoded compatibility field; runtime v0.3.32a registers the interceptor at order 100 regardless of this value |
| `queue.enabled` | bool | true | Trace queue publish/consume |
| `temporal.enabled` | bool | false | Trace Temporal workflows |
```yaml
otel:
enabled: true
endpoint: "http://jaeger:4318"
traces_enabled: true
process:
trace_lifecycle: true
```
Standard OTEL environment variables (`OTEL_SDK_DISABLED`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_INSECURE`, `OTEL_SERVICE_NAME`, `OTEL_SERVICE_VERSION`, `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG`, `OTEL_PROPAGATORS`) override the matching fields.
See: [Observability Guide](guides/observability.md)
### Shutdown
Graceful shutdown behavior.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `timeout` | duration | 30s | Max wait for components to stop |
```yaml
shutdown:
timeout: 60s
```
### Metrics
Internal metrics collection buffer.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `buffer.size` | int | 10000 | Metrics buffer capacity |
| `interceptor.enabled` | bool | true | Auto-track function calls |
```yaml
metrics:
buffer:
size: 20000
interceptor:
enabled: true
```
See: [Metrics Module](lua/system/metrics.md), [Observability Guide](guides/observability.md)
### Prometheus
Prometheus metrics endpoint.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | false | Start metrics server |
| `address` | string | | Listen address; must be set explicitly when `enabled: true`, otherwise the metrics server does not start |
| `max_cardinality` | int | 1024 | Distinct label sets retained per metric (LRU); `0` or less uses the default |
```yaml
prometheus:
enabled: true
address: "0.0.0.0:9090"
```
Exposes `/metrics` endpoint for Prometheus scraping, plus `/livez`.
See: [Observability Guide](guides/observability.md)
### Cluster
Multi-node clustering: gossip membership plus a bounded Raft consensus core. See the [Cluster Guide](guides/cluster.md) for the architecture and operational model; this section is the config-key reference.
#### Top-level
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | false | Enable clustering |
| `name` | string | hostname | Node name; must be unique across the cluster |
| `failure_domain` | string | | Zone/rack label; advertised in gossip so voters spread across domains |
| `kv_crdt_tombstone_retention` | duration | 0 | Age after which `store.kv.crdt` delete tombstones are reclaimed; `0` disables age-based GC |
| `kv_crdt_tombstone_gc_alive_peers` | bool | false | Use the current alive membership as the tombstone acknowledgement set |
#### Membership (gossip)
SWIM gossip via memberlist. Used for node discovery, failure detection, and metadata dissemination.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `membership.bind_addr` | string | 0.0.0.0 | Gossip bind address |
| `membership.bind_port` | int | 7946 | Gossip bind port (TCP+UDP) |
| `membership.advertise_addr` | string | | Address peers use to reach this node (NAT/k8s) |
| `membership.join_addrs` | string | | Comma-separated seed `host:port` pairs |
| `membership.secret_key` | string | | Base64-encoded gossip encryption key (inline) |
| `membership.secret_file` | string | | Path to file holding the gossip encryption key |
| `membership.gossip_interval` | duration | 500ms | Gossip dissemination period |
| `membership.push_pull_interval` | duration | 5s | Full state sync period |
| `membership.dead_node_reclaim_time` | duration | 30s | When a dead node's name/address can be reclaimed |
| `membership.probe_interval` | duration | 1s | Failure-detection probe cycle |
| `membership.probe_timeout` | duration | 200ms | Ack wait per probe |
| `membership.tcp_timeout` | duration | 1s | TCP fallback probe timeout |
| `membership.suspicion_mult` | int | 3 | Suspicion timeout multiplier |
A gossip secret is required. Set `membership.secret_key` or `membership.secret_file` (the file wins if both are given); with neither, the cluster component fails to start. The value is base64-encoded.
The four probe keys inherit memberlist's local-network defaults when unset; raise them for high-latency links (e.g. `probe_interval: 2s`, `probe_timeout: 500ms`, `suspicion_mult: 5`).
#### Internode (transport)
TCP mesh carrying the relay and Raft traffic between nodes. Raft rides this mesh over internode request/reply; there is no separate Raft port.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `internode.bind_addr` | string | 0.0.0.0 | Mesh bind address |
| `internode.bind_port` | int | 0 | Mesh port (0 = auto: 7950-7959, then ephemeral) |
| `internode.auto_port` | bool | true | Discover the actual port at boot, pin it, and advertise it in gossip |
| `internode.advertise_addr` | string | | Additional relay endpoint (IP or DNS name) published for upgraded peers — for NAT or load-balanced reachability |
| `internode.advertise_port` | int | 0 | Port for `advertise_addr` (0 = bind port; requires `advertise_addr`) |
| `internode.identity_key` | string | | Base64-encoded ed25519 private key identifying this node (inline) |
| `internode.identity_key_file` | string | | Path to a file holding that key |
| `internode.trusted_peer_keys` | map | | Base64-encoded ed25519 public key per node name, including this node |
| `internode.tls.enabled` | bool | false | Enable mutual TLS on the internode TCP mesh |
| `internode.tls.cert_file` | string | | PEM node certificate; required with TLS |
| `internode.tls.key_file` | string | | Matching private key; required with TLS |
| `internode.tls.ca_file` | string | | PEM CA bundle for client and server verification; required with TLS |
`advertise_addr`/`advertise_port` publish an additive endpoint in node metadata while the bind endpoint stays advertised unchanged, so mixed-version clusters keep connecting during a rolling upgrade.
Internode identity is mandatory whenever clustering is enabled. `identity_key` and `identity_key_file` are mutually exclusive and one of them must be present; the value decodes (standard or raw base64) to either a 32-byte ed25519 seed or a 64-byte ed25519 private key. `trusted_peer_keys` maps each node name to that node's 32-byte ed25519 public key, and must contain an entry for the local `cluster.name` whose value matches the local identity — otherwise startup fails. See the [Cluster Guide](guides/cluster.md#internode-identity).
TLS requires all three credential paths and TLS 1.2 or later. Unknown settings,
invalid values, unreadable or invalid credentials, and credential paths supplied
while TLS is disabled fail startup. The ed25519 identity configuration remains
required. See [Internode TLS](guides/cluster.md#internode-tls) for an example.
#### Raft (consensus)
The bounded Raft core stores durable state under `raft.data_dir` by default (`~/.wippy/store`). A restarted node rejoins quorum from its peers. [`store.kv.raft`](system/store.md#cluster-kv-stores) entries replicate through this core, and gossip coordinates bootstrap using a `bootstrap_expect` model.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `raft.data_dir` | string | `~/.wippy/store` | Directory for fs-durable Raft state and durable CRDT snapshots (under `/_sys/`). Diskless only when no path resolves (no home dir and none set) |
| `raft.enabled` | bool | true | Run a Raft node; `false` makes this a gossip-only client |
| `raft.role` | string | server | `server` runs a Raft node; `client` is gossip-only |
| `raft.eligible` | bool | true | Whether this node may be selected as a voter or standby; false keeps it outside Raft as a client |
| `raft.priority` | int | 100 | Voter selection priority (lower is preferred) |
| `raft.bootstrap_expect` | int | 1 | Initial quorum size: `0`=join existing, `1`=single-node, `N`=wait for N eligible nodes including the local node, then form quorum |
| `raft.max_voters` | int | 5 | Voter ceiling (must be odd); up to `max_standbys` additional eligible nodes become standbys, and the rest remain clients |
| `raft.max_standbys` | int | 4 | Non-voting members kept warm for promotion; nodes beyond voters+standbys are not Raft members |
| `raft.reconcile_debounce` | duration | 2s | Coalesce window after a gossip event before the voter reconciler runs |
| `raft.reconcile_timeout` | duration | 2s | Bound per reconcile pass |
| `raft.heartbeat_timeout` | duration | 3s | Follower idle wait before starting an election |
| `raft.election_timeout` | duration | 3s | Candidate election timeout (clamped to >= heartbeat) |
| `raft.commit_timeout` | duration | 500ms | Idle leader heartbeat cadence |
| `raft.snapshot_threshold` | uint64 | 8192 | Log entries since last snapshot before a new one |
| `raft.snapshot_interval` | duration | 2m | Snapshot check interval |
| `raft.snapshot_retain` | int | 3 | Snapshots retained |
| `raft.trailing_logs` | uint64 | 10240 | Log entries retained after a snapshot |
| `raft.max_append_entries` | int | 16 | Max entries per AppendEntries RPC |
| `raft.leader_probe_interval` | duration | 3s | Global-registry leader-reachability probe cadence |
| `raft.leader_probe_grace` | int | 3 | Consecutive probe failures before leader is declared unreachable |
| `raft.registry_backend` | string | kv | Cluster name-registry implementation: `kv` (shared kv keyspace) or `fsm` (dedicated Raft FSM) |
| `raft.global_dissem_tombstone_retention` | duration | 0 | How long the global-name dissemination cache keeps delete tombstones |
Single-node (development) — clustering on, bootstraps itself immediately:
```yaml
cluster:
enabled: true
name: dev
membership:
secret_key: "d2lwcHktZG9jcy1nb3NzaXAtc2VjcmV0LTMyYnl0ZXM="
internode:
identity_key: "d2lwcHktZG9jcy1kZXYtbm9kZS1leGFtcGxlc2VlZCE="
trusted_peer_keys:
dev: "rNqImcjOzef28dzvma80mSrCW1px5LBAc5TbaYqAgm0="
raft:
bootstrap_expect: 1
```
Three-node voting cluster — each node lists the others as seeds and waits for all three before forming quorum. Every node carries the same `trusted_peer_keys` map and its own private key:
```yaml
cluster:
enabled: true
name: node-1
failure_domain: us-east-1a
membership:
bind_port: 7946
join_addrs: "node-2:7946,node-3:7946"
secret_file: /etc/wippy/cluster.key
internode:
identity_key_file: /etc/wippy/node-1.key
trusted_peer_keys:
node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0="
node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8="
node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE="
raft:
bootstrap_expect: 3
max_voters: 5
```
Gossip-only client — joins the cluster for naming/messaging but never runs Raft. It still needs its own identity and must appear in every node's trusted map:
```yaml
cluster:
enabled: true
name: edge-7
membership:
join_addrs: "node-1:7946,node-2:7946"
secret_file: /etc/wippy/cluster.key
internode:
identity_key_file: /etc/wippy/edge-7.key
trusted_peer_keys:
node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0="
node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8="
node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE="
edge-7: "7lzP4jBAkC3P+0jq4vtMsC45571BlVXk3mSlOD/Z0SA="
raft:
role: client
```
### LSP
Language Server Protocol server for editor integrations.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | false | Enable the LSP service and TCP server; the HTTP transport also requires this |
| `address` | string | :7777 | TCP listen address |
| `http_enabled` | bool | false | Enable the HTTP transport |
| `http_address` | string | :7778 | HTTP listen address |
| `http_path` | string | /lsp | HTTP endpoint path |
| `http_allow_origin` | string | * | CORS allowed origin |
| `max_message_bytes` | int | 8388608 | Max incoming message size |
```yaml
lsp:
enabled: true
address: ":7777"
http_enabled: true
```
See: [LSP Guide](guides/lsp.md)
### Network Service
Overlay network manager (SOCKS5, I2P, Tailscale drivers).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `state_dir` | string | .wippy/net | Driver state storage directory |
| `default_network` | string | | Default network ID applied when entries omit `network` |
```yaml
network_service:
state_dir: /var/lib/wippy/net
default_network: app:tailscale
```
See: [Network Overlays](system/network.md)
### HTTP Dispatcher
Tuning for the shared HTTP client pool used by HTTP-dispatched functions and outbound requests.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `dispatcher.http.timeout` | duration | 0 (none) | Per-request timeout |
| `dispatcher.http.max_idle_conns` | int | 0 (stdlib) | Max idle connections across all hosts |
| `dispatcher.http.max_idle_per_host` | int | 0 (stdlib) | Max idle connections per host |
| `dispatcher.http.idle_conn_timeout` | duration | 0 (stdlib) | Idle connection timeout |
| `dispatcher.http.max_clients` | int | 0 (unbounded) | Max distinct pooled clients |
```yaml
dispatcher:
http:
timeout: 30s
max_idle_per_host: 32
```
### Modules
Module registry client used by `wippy install`/`update`.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `registry_url` | string | https://hub.wippy.ai | Registry endpoint |
```yaml
modules:
registry_url: https://internal-registry.example.com
```
### Extensions
Native Go plugin extensions loaded at boot (Unix only).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | true | Load extensions |
| `paths` | string[] | | Plugin file paths (relative to config dir) |
```yaml
extensions:
enabled: true
paths:
- ./extensions/myplugin.so
```
### Environment Variables
| Variable | Description |
|----------|-------------|
| `GOMEMLIMIT` | Memory limit fallback when `--memory-limit` flag is not set (precedence: `--memory-limit` flag > `GOMEMLIMIT` > 1G default) |
### See Also
- [CLI Reference](guides/cli.md) — Command-line options
- [Cluster Guide](guides/cluster.md) — Clustering architecture and operations
- [Entry Kinds](guides/entry-kinds.md) — Entry types and fields
- [Observability Guide](guides/observability.md) — Logging, metrics, and tracing
---
# "Cluster"
## Cluster
Wippy runs as a single node by default. Enabling clustering connects nodes through gossip membership and a bounded Raft consensus core, supporting cluster-wide process names, distributed locks, and process-group messaging.
Clustering is disabled until `cluster.enabled` is set to `true`.
### Cluster Capabilities
- **Membership** — every node knows the live set of peers through gossip, with fast failure detection.
- **Cluster-wide process names** — register a process under a name that resolves from any node, with a choice of consistency guarantees (see [Naming](#naming-and-name-scopes)).
- **Distributed locks** — `system.lock` provides cluster-wide mutual exclusion with automatic release when the holder dies (see [Distributed locks](#distributed-locks)).
- **Process groups** — publish to every member of a named group across all nodes (see [Process groups](#process-groups)).
- **Replicated key-value stores** — `store.kv.raft` (strong) and `store.kv.crdt` (eventual) replicate KV data across nodes (see [Store](system/store.md#cluster-kv-stores)).
- **A consensus core** — a small, bounded Raft cluster provides the linearizable backbone the naming and lock primitives build on.
### Architecture: Bounded Raft
Wippy limits Raft membership to a fixed-size core so the leader does not replicate every log entry to every node. Other nodes participate through gossip. Each node has one of three roles in the Raft configuration:
| Role | Count (default) | In Raft config | Receives log replication | Votes |
|------|-----------------|----------------|--------------------------|-------|
| **Voter** | up to 5 (`max_voters`, odd) | yes | yes | yes |
| **Standby** | up to 4 (`max_standbys`) | yes | yes | no |
| **Client** | unbounded | no | no | no |
- **Voters** form the quorum. Writes commit once a majority of voters acknowledge them. `max_voters` is normalized to an odd cap (default 5). With at least three eligible nodes, the reconciler also chooses an odd voter count. With two eligible nodes and a cap greater than one, both are voters; `max_voters: 1` keeps a single voter.
- **Standbys** are non-voting members kept fully replicated and warm. When a voter departs, the leader promotes the highest-ranked standby into the open voter slot, so quorum recovers without waiting for a fresh node to catch up.
- **Clients** are nodes beyond `voters + standbys`. They are not in the Raft configuration, so the leader does not send them log entries. They participate in gossip and route writes to a Raft member, keeping Raft replication bounded by the configured core size.
The `max_voters` and `max_standbys` settings cap the consensus core independently of the total cluster size.
#### Voter Selection
The leader runs a reconciler that, on every membership change (debounced by `raft.reconcile_debounce`, default 2s), recomputes which nodes should be voters and applies the minimal set of promote/demote operations. Selection is deterministic — every node derives the same ordering from the same gossip view — and is driven by three gossip-advertised hints:
- `raft.eligible` — a node with `eligible: false` is excluded from both voter and standby selection and remains outside Raft as a client. Keep a node eligible but below the voter cutoff when it should serve as a standby.
- `raft.priority` — lower value is preferred when filling voter slots; ties break by node ID.
- `failure_domain` — voters are spread across distinct domains (zones/racks) first, reducing the risk that one domain failure removes a voter majority.
Operations are applied in a quorum-preserving order: adds and promotions first, then demotions, then removals.
### Membership and Gossip
Membership uses SWIM gossip (HashiCorp memberlist). Each node binds a gossip port (default **7946**) and continuously exchanges small messages with peers to detect failures and disseminate metadata.
A node joins by pointing at one or more existing nodes:
```yaml
cluster:
enabled: true
name: node-2
membership:
join_addrs: "node-1:7946"
secret_file: /etc/wippy/cluster.key
internode:
identity_key_file: /etc/wippy/node-2.key
trusted_peer_keys:
node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0="
node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8="
```
The first node needs no `join_addrs`; it starts as a seed. Joins retry with backoff, and an isolated node periodically attempts to rejoin. This supports nodes that restart with a new IP, as commonly occurs in Kubernetes.
Gossip is always encrypted with a shared key. Supply it inline as `membership.secret_key` or from a file as `membership.secret_file`; a node started with neither fails to bring the cluster component up. The value is base64-encoded and identical on every node.
Membership changes (`NodeJoined`, `NodeLeft`, `NodeUpdated`) are the events that drive Raft bootstrap, voter reconciliation, process-group sync, and automatic cleanup of names owned by a departed node.
### Internode identity
Every node holds an ed25519 key pair, and every node carries the map of public keys it trusts. Both are mandatory when `cluster.enabled: true`.
```yaml
cluster:
internode:
identity_key_file: /etc/wippy/node-1.key
trusted_peer_keys:
node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0="
node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8="
node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE="
```
| Key | Content |
|-----|---------|
| `internode.identity_key` | The node's private key, inline |
| `internode.identity_key_file` | Path to a file holding that key |
| `internode.trusted_peer_keys` | Node name to public key, for every node in the mesh including this one |
Key format: base64, standard or raw (unpadded) encoding. A private key decodes to either a 32-byte ed25519 seed or a full 64-byte ed25519 private key; a trusted peer key decodes to a 32-byte ed25519 public key. There is no key-generation subcommand — mint keys with any ed25519 tool and base64 the raw bytes:
```bash
## 32-byte seed and its public key, base64-encoded
openssl genpkey -algorithm ed25519 -out node-1.pem
openssl pkey -in node-1.pem -outform DER \
| tail -c 32 | base64 > node-1.key
openssl pkey -in node-1.pem -pubout -outform DER \
| tail -c 32 | base64
```
`identity_key` and `identity_key_file` are mutually exclusive, and one of them is required. `trusted_peer_keys` must contain an entry for the local `cluster.name` whose value is this node's own public key; a missing or mismatched self entry aborts startup. This makes the trusted map a single artifact you can distribute unchanged to every node.
The mesh handshake is mutual. Each side proves knowledge of the shared gossip secret with an HMAC over a transcript binding both node IDs and both nonces, and signs that transcript with its identity key; the peer verifies the signature against the public key it has for that node ID and against the key the peer advertises in gossip. Either check failing closes the connection.
Consequences to plan for:
- The mesh does not interoperate with a node that has no identity. Every node in the cluster must be configured with one.
- A peer whose node ID is absent from `trusted_peer_keys` is rejected, as is one whose gossip-advertised public key disagrees with the trusted entry. Adding a node means distributing its public key to the existing nodes.
- A node ID must be present in the live gossip membership before its key resolves, so a peer that has not joined gossip cannot open a mesh connection.
### Internode TLS
Enable mutual TLS for the internode TCP mesh under `cluster.internode.tls`.
This also protects the relay and Raft traffic carried by that mesh. Add the
following settings to your existing cluster configuration:
```yaml
cluster:
internode:
tls:
enabled: true
cert_file: /etc/wippy/node.crt
key_file: /etc/wippy/node.key
ca_file: /etc/wippy/cluster-ca.pem
```
TLS is disabled by default. Enabling it requires all three nonempty file paths:
a PEM certificate and matching private key, and a PEM CA bundle used to verify
both server and client certificates. The minimum protocol version is TLS 1.2.
Configure compatible certificates and trust roots on every connecting peer.
The ed25519 internode identity and trusted peer map described above are still
required; TLS adds transport protection to the existing mesh authentication.
Unknown TLS settings, invalid types, missing credentials, malformed CA bundles,
and credential paths supplied without `enabled: true` fail startup. An invalid
explicit configuration never silently falls back to plaintext.
### Bootstrap
The initial cluster forms through gossip rather than a static peer list. With the Consul/Nomad-style `bootstrap_expect` setting, each starting node waits until the configured number of eligible nodes, including itself, is visible before forming quorum.
| `bootstrap_expect` | Behavior |
|--------------------|----------|
| `0` | Never self-bootstrap; only join a cluster that already exists |
| `1` | Single-node; bootstrap immediately with self as the only voter |
| `N` | Wait until `N` eligible nodes, including the local node, are stably visible in gossip, then all derive the same voter list and form quorum |
For an `N`-node bootstrap, set the same `bootstrap_expect: N` on every initial node. Each advertises a "pre-bootstrap" status in gossip; once exactly `N` such nodes, including itself, are visible for a short stability window, every node independently computes the identical sorted voter set and forms the cluster. The stability window prevents a brief, partial view from triggering bootstrap early.
Nodes that start later see an already-formed cluster and skip bootstrap entirely — the leader's reconciler adds them as voters or standbys.
### Raft Consensus Core
Raft state is **fs-durable by default**: logs and snapshots are persisted under `cluster.raft.data_dir` (default `~/.wippy/store`, in `_sys/raft`), and [`store.kv.raft`](system/store.md#cluster-kv-stores) replicates through the same core. A restarting node still rejoins gossip and catches up from its peers, so the cluster also tolerates losing a node's disk; durability comes from both the live quorum and on-disk state. A node runs diskless only when no data directory resolves (no configured path and no home directory) — see [Recovery](#recovery-and-failure-modes).
Raft does not open its own listening port. It rides the **internode mesh** — the same TCP connections used for relay traffic between nodes — carrying its RPCs as internode request/reply frames over the mesh's reliable per-class channels. The internode port is auto-selected at boot (range 7950-7959, then ephemeral), pinned, and advertised in gossip so peers can reach it. Nodes must be mutually reachable on both the gossip port and their advertised internode TCP ports.
The Raft FSM holds the global name registry: active `name -> PID` bindings plus in-flight strong reservations. That is what the naming primitives below read and write.
### Naming and Name Scopes
A process can be registered and addressed by name instead of its raw PID. Its **scope** selects the consistency and coordination behavior. Four scopes are available, ordered from local to strongest:
| Scope | Backed by | Visibility | Guarantee |
|-------|-----------|------------|-----------|
| **Local** | per-node map | this node only | Instant, node-local; no coordination |
| **Eventual** | gossip CRDT | cluster-wide | Eventually consistent; converges after gossip rounds |
| **Consistent** | Raft | cluster-wide | Linearizable writes; unique singleton across the cluster |
| **Strong** | Raft + all-node ack | cluster-wide | Consistent, plus every live node acknowledges before the name is active |
How to choose:
- **Local** — names meaningful only on one node, such as a per-node helper. Released when the process exits and requires no cluster coordination.
- **Eventual** — cluster-wide service, group, and presence names where a brief stale window is acceptable. The binding set is fully replicated to every node, so it fits a bounded namespace — not one name per high-cardinality entity such as a per-session process (address those directly by PID). When two origins register the same name, conflict resolution picks a winner and the losing process receives a cancel event (`process.event.CANCEL`) carrying the reason `name revoked: `; it keeps running and can re-register. Names release when the owning node leaves.
- **Consistent** — the standard choice for cluster-wide named singletons. First-write-wins: a second registration of the same name to a different PID fails with "already exists" and returns the current owner. Writes need a quorum, so they stall in a minority partition. Reads come from the local Raft replica and may lag a write by a few milliseconds.
- **Strong** — the small set of control-plane singletons where even a momentary stale read is dangerous. On top of the Consistent guarantee, the registration opens a reservation that every live node must acknowledge before the name becomes authoritative; any node already holding a conflicting binding rejects it immediately. If the deadline passes before all nodes ack, the registration expires and reports which nodes were missing.
Names are released automatically: Local on process exit; Consistent and Strong on process exit (via topology monitoring) and on node departure; Eventual on node departure. Resolution for messaging (`process.send`, `process.terminate`, and similar) consults the planes most-authoritative first — Consistent and Strong (Raft), then Eventual (gossip), then Local — so a cluster-wide name shadows a local one with the same string.
The Lua surface for naming lives on `process.registry` (register/lookup/unregister with a scope) — see the [Process](lua/core/process.md) reference.
### Process Groups
Process groups are a cluster-aware publish/subscribe and membership facility modeled on Erlang's `pg`. A process joins a named group; a broadcast fans out over the internode mesh to the group's members across all nodes, delivered best-effort. Groups are eventually consistent and independent of Raft — they use the gossip membership view to choose recipients — so they keep working even while the consensus core is converging.
Typical operations: join/leave a group, broadcast to all members (or local members only), list members, and monitor a group for join/leave events. On a new node joining, groups reconcile their membership through a direct sync handshake, and a background anti-entropy loop repairs any divergence over time.
See [Process Groups](lua/core/pg.md) for the Lua API and the [`pg.scope` entry kind](system/process-groups.md) for configuration.
### Distributed Locks
`system.lock` is cluster-wide mutual exclusion built on a raft-linearizable conditional write in the shared key-value store. Acquiring a lock performs a set-if-absent of the holder PID at `_sys:lock:`; releasing deletes that entry if it is still held by the caller. Because the conditional write goes through Raft (with off-leader writes forwarded to the leader), it is linearizable, so at most one holder can exist cluster-wide.
```lua
local ok, err = system.lock.acquire("orders.migration")
if not ok then
-- err has kind errors.ALREADY_EXISTS when another process holds the lock.
-- Apply the caller's retry and backoff policy for that case if needed.
return nil, err
end
-- critical section: only one holder cluster-wide
local released, release_err = system.lock.release("orders.migration")
if release_err then
return nil, release_err
end
return released
```
Acquire is fail-fast (non-blocking): if the lock is held, it returns immediately, so callers provide their own retry and backoff. The lock is released if the holder process exits or its node leaves. See the [System](lua/system/system.md) reference for exact signatures.
### Configuration
See [Configuration](guides/configuration.md#cluster) for related cluster settings. These minimal shapes include the mandatory internode identity settings.
These are configuration fragments, not a complete deployment manifest. Replace
node names, addresses, failure domains, paths, and every `${env:...}` identity
placeholder with values generated and distributed for your own cluster.
Single node (development):
```yaml
cluster:
enabled: true
name: dev
membership:
secret_key: "d2lwcHktZG9jcy1nb3NzaXAtc2VjcmV0LTMyYnl0ZXM="
internode:
identity_key: "d2lwcHktZG9jcy1kZXYtbm9kZS1leGFtcGxlc2VlZCE="
trusted_peer_keys:
dev: "rNqImcjOzef28dzvma80mSrCW1px5LBAc5TbaYqAgm0="
raft:
bootstrap_expect: 1
```
Three-node voting cluster (`node-2` and `node-3` differ only in `name`, `identity_key_file`, and `join_addrs`):
```yaml
cluster:
enabled: true
name: node-1
failure_domain: us-east-1a
membership:
join_addrs: "node-2:7946,node-3:7946"
secret_file: /etc/wippy/cluster.key
internode:
identity_key_file: /etc/wippy/node-1.key
trusted_peer_keys:
node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0="
node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8="
node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE="
raft:
bootstrap_expect: 3
```
Gossip-only client (joins for naming/messaging, never runs Raft). It still needs an identity, and the voters need its public key in their own maps:
```yaml
cluster:
enabled: true
name: edge-7
membership:
join_addrs: "node-1:7946,node-2:7946"
secret_file: /etc/wippy/cluster.key
internode:
identity_key_file: /etc/wippy/edge-7.key
trusted_peer_keys:
node-1: "okmamN3PKkMpPwPBurknHy2Wi3dwp/rz+uTM2fF9aD0="
node-2: "PWX+oOYrFdtjUxbgmTkXCFI0KEvG++ZM52HOWfDkqP8="
node-3: "QfP0fgllbj4s95VAztTORhy3bv9mst1l0lwuUNvO/hE="
edge-7: "7lzP4jBAkC3P+0jq4vtMsC45571BlVXk3mSlOD/Z0SA="
raft:
role: client
```
### Ports
| Purpose | Port | Protocol | Config key |
|---------|------|----------|------------|
| Gossip (membership) | 7946 | TCP + UDP | `cluster.membership.bind_port` |
| Internode mesh (relay + Raft) | auto | TCP | `cluster.internode.bind_port` |
Raft is multiplexed over the internode mesh rather than using a separate port. The internode port is assigned automatically and advertised through gossip. The gossip port is predictable by default, but peers must also be able to reach each node's advertised internode TCP port.
### Observability
Cluster health is exposed through the standard [Prometheus endpoint](guides/observability.md) and through liveness health checks.
Key metrics to watch:
| Metric | Meaning |
|--------|---------|
| `raft_state` | 0 = follower, 1 = candidate, 2 = leader |
| `raft_term` | Current Raft term; rapid increases signal election churn |
| `raft_voters` / `raft_non_voters` | Live voters and standbys in the configuration |
| `raft_leader_changes_total` | Leader transitions; should be near-flat in a healthy cluster |
| `raft_voter_churn_burst_total` | Bursts of voter add/remove operations; sustained churn indicates instability |
| `gossip_members{state}` | Counts by state (alive/suspect/dead/left) |
| `gossip_convergence_seconds` | Time between gossip events |
Built-in liveness checks (wired to the liveness endpoint):
- **gossip** — healthy while the node's gossip health score stays low, with a boot grace window so a rejoining node is not killed prematurely.
- **raft last-contact** — a voting follower fails if it has not heard from a leader recently; a standby tolerates a much longer gap; leaders always pass.
- **process-group broadcast** — fails when the process-group service has neither sent nor received any broadcast for its activity ceiling, catching a wedged service or a persistent partition.
### Recovery and Failure Modes
Raft state is fs-durable, but the cluster's primary durability still comes from a live quorum. The practical rules:
- Keep a voter majority alive. With 5 voters you tolerate 2 simultaneous voter failures; standbys are promoted to refill open slots. Drop below a majority and writes (new Consistent/Strong registrations and lock acquisitions) stall until quorum returns. Locally replicated state can still answer some reads, but do not treat those reads as proof that a partitioned node has the latest value.
- The leader proactively evicts a voter that is both heartbeat-silent and gossip-dead, so a dead voter does not permanently block quorum while a standby is promoted in.
- To recover a cluster that has lost quorum, restart the failed nodes. They rejoin gossip and the surviving members fold them back in. Spreading voters across `failure_domain`s reduces the chance that a single zone failure causes quorum loss.
### See Also
- [Configuration](guides/configuration.md#cluster) — related cluster settings
- [Process](lua/core/process.md) — registering and resolving processes by name
- [System](lua/system/system.md) — `system.cluster`, `system.raft`, `system.node`, `system.lock`
- [Observability](guides/observability.md) — metrics and health endpoints
- [Process Model](concepts/process-model.md) — actors, PIDs, and messaging
---
# "Linter"
## Linter
Run `wippy lint` to type-check and statically analyze Lua entries.
### Usage
```bash
wippy lint # Check all Lua entries
wippy lint --level hint # Show all diagnostics including hints
wippy lint --json # Output in JSON format
wippy lint --ns app # Check only the app namespace
wippy lint --summary # Group results by error code
```
### What Gets Checked
The linter validates all Lua entry kinds:
- `function.lua` — Functions
- `library.lua` — Libraries
- `process.lua` — Processes
- `workflow.lua` — Workflows
Bytecode entries hold compiled bytecode (fs/path/hash), not source, so they cannot be parsed or type-checked; the linter only checks source-bearing Lua entries (their `.bc` variants are skipped, though they may still appear in the total entry count).
Each entry is parsed, type-checked, and analyzed for correctness issues.
### Severity Levels
Diagnostics have three severity levels:
| Level | Description |
|-------|-------------|
| `error` | Type errors and correctness issues that must be fixed |
| `warning` | Likely bugs or problematic patterns |
| `hint` | Style suggestions and informational notes |
Control which levels appear with `--level`:
```bash
wippy lint --level error # Errors only
wippy lint --level warning # Warnings and errors (default)
wippy lint --level hint # Everything
```
#### Parse Errors
| Code | Description |
|------|-------------|
| `P0001` | Lua syntax error - source cannot be parsed |
#### Type Check Errors (E-series)
Type checker errors (`E0001`+) report issues found by the type system: type mismatches, undefined variables, invalid operations, and similar correctness problems. These are always reported as errors.
```lua
local x: number = "hello" -- E: string not assignable to number
local function add(a: number, b: number): number
return a + b
end
add("one", "two") -- E: string not assignable to number
```
#### Undeclared Requires
A string-literal `require("name")` whose module is neither in the entry's `imports`/`modules` declarations nor an ambient builtin fails with:
```
require("name") is not declared in _index.yaml imports or modules
```
This check always runs (it is not gated behind `--rules`) and reports as an error. Declare the module to satisfy it:
```yaml
imports:
json: wippy.stdlib:json # alias -> registry id
modules:
- funcs # bare module name
```
Dynamic requires (`require(variable)`) are not inspected. The linter and runtime share the ambient module set, which includes modules available without declaration, such as `process` in executable kinds.
#### Lint Rule Warnings (W-series)
Lint rules provide style and quality checks. Enable them with `--rules`:
```bash
wippy lint --rules
```
| Code | Rule | Description |
|------|------|-------------|
| `W0001` | no-empty-blocks | Empty block statements |
| `W0002` | no-global-assign | Assignment to global variables |
| `W0003` | no-self-compare | Comparison of a value with itself |
| `W0004` | no-unused-vars | Unused local variables |
| `W0005` | no-unused-params | Unused function parameters |
| `W0006` | no-unused-imports | Unused import statements |
| `W0007` | no-shadowed-vars | Variable shadowing outer scope |
Without `--rules`, only type checking (P and E codes) is performed.
#### By Namespace
Check specific namespaces using `--ns`:
```bash
wippy lint --ns app # Exact namespace match
wippy lint --ns "app.*" # All under app
wippy lint --ns app --ns lib # Multiple namespaces
```
Dependencies of selected entries are loaded for type checking but their diagnostics are not reported.
#### By Error Code
Filter diagnostics by code:
```bash
wippy lint --code E0001
wippy lint --code E0001 --code E0004
```
#### By Count
Limit the number of diagnostics shown:
```bash
wippy lint --limit 10 # Show first 10 issues
```
#### Table Format (Default)
Each diagnostic is displayed with source context, file location, and the error message. Results are sorted by entry, severity, and line number.
A summary line shows totals:
```
Checked 42 entries: 5 errors, 12 warnings
```
#### Summary Format
Group diagnostics by namespace and error code:
```bash
wippy lint --summary
```
```
By namespace:
app 15 issues (5 errors, 10 warnings)
lib 2 issues (2 warnings)
By error code:
E0001 [error ] 5 occurrences
E0004 [error ] 3 occurrences
Checked 42 entries: 5 errors, 12 warnings
```
#### JSON Format
Machine-readable output for CI/CD processing:
```bash
wippy lint --json
```
```json
{
"diagnostics": [
{
"entry_id": "app:handler",
"code": "E0001",
"severity": "error",
"message": "string not assignable to number",
"line": 10,
"column": 5
}
],
"total_entries": 42,
"error_count": 5,
"warning_count": 12,
"hint_count": 0
}
```
### Caching
The linter caches results between runs. Cache keys include the source hash, method name, dependencies, and type-system configuration.
Clear the cache if results seem stale:
```bash
wippy lint --cache-reset
```
### CI Integration
In table and summary modes, the command exits non-zero when the filtered result contains errors. Warnings and hints do not affect the exit code, even when `--level warning` or `--level hint` displays them.
JSON mode is different: after successfully encoding the result, `wippy lint --json` exits with code 0 even when `error_count` is non-zero. A CI job using JSON output must parse `error_count` itself. To use the command's exit status as the gate, run a non-JSON invocation:
```bash
wippy lint --level error
```
You can produce a report separately without treating its exit status as the lint result:
```bash
wippy lint --json --level error > lint-results.json
```
Example GitHub Actions step:
```yaml
- name: Lint
run: wippy lint --level warning
```
### Flags Reference
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--level` | | warning | Minimum severity level (error, warning, hint) |
| `--json` | | false | Output in JSON format |
| `--ns` | | | Filter by namespace patterns |
| `--code` | | | Filter by error codes |
| `--limit` | | 0 | Max diagnostics to show (0 = unlimited) |
| `--summary` | | false | Group by error code |
| `--no-color` | | false | Disable colored output |
| `--rules` | | false | Enable lint rules (W-series style/quality checks) |
| `--cache-reset` | | false | Clear cache before linting |
| `--profile` | | | Apply a workspace profile from merged runtime configuration; repeat to apply profiles in order |
| `--set` | | | Override a merged configuration value as `section.path=value`; repeat for multiple overrides |
| `--lock-file` | `-l` | wippy.lock | Path to lock file |
| `--profile` | | | Apply a workspace profile from the merged runtime config (repeatable, applied in order) |
| `--set` | | | Override a merged runtime config value (`section.path=value`, repeatable) |
### See Also
- [CLI](guides/cli.md) — Full CLI reference
- [Types](lua/types.md) — Type-system documentation
- [LSP](guides/lsp.md) — Editor integration with live diagnostics
---
# "Language Server"
## Language Server
Wippy includes a Language Server Protocol (LSP) server for Lua editor features. It runs as part of the Wippy runtime and accepts editor connections over TCP or HTTP.
### Features
- Code completion with type-aware suggestions
- Hover information showing types and signatures
- Go to definition
- Find references
- Document and workspace symbols
- Call hierarchy (incoming and outgoing calls)
- Pull diagnostics for type errors in the current editor overlay after successful parsing
- Signature help for function parameters
### Configuration
Enable the LSP server in `.wippy.yaml`:
```yaml
lsp:
enabled: true
address: ":7777"
```
#### Configuration Fields
| Field | Default | Description |
|-------|---------|-------------|
| `enabled` | false | Enable the LSP service and TCP server |
| `address` | :7777 | TCP listen address |
| `http_enabled` | false | Enable the HTTP transport |
| `http_address` | :7778 | HTTP listen address |
| `http_path` | /lsp | HTTP endpoint path |
| `http_allow_origin` | * | CORS allowed origin |
| `max_message_bytes` | 8388608 | Max incoming message size (bytes) |
#### TCP Transport
The TCP server speaks JSON-RPC 2.0 with standard LSP message framing (Content-Length headers). This is the primary transport for editor integrations.
#### HTTP Transport
The HTTP transport accepts POST requests with JSON-RPC payloads. It supports browser-based editors and web tools, answers CORS preflight `OPTIONS` requests, and includes CORS headers for cross-origin access.
```yaml
lsp:
enabled: true
http_enabled: true
http_address: ":7778"
http_path: "/lsp"
http_allow_origin: "*"
```
### Document URI Scheme
The LSP server uses the `wippy://` URI scheme to identify registry entries:
```
wippy://namespace:entry_name
```
Editors map these URIs to entry IDs in the registry. Both `wippy://` scheme and raw `namespace:entry_name` formats are accepted.
### Indexing
The LSP server maintains an index of code entries. Multiple workers update the index in the background.
Key behaviors:
- Entries are indexed in dependency order (dependencies first)
- Changes trigger re-indexing of affected entries
- Unsaved editor changes are stored in an overlay
- Indexing is incremental; only changed entries are reprocessed
### Supported LSP Methods
| Method | Description |
|--------|-------------|
| `initialize` | Capability negotiation |
| `initialized` | Initialization-complete notification |
| `shutdown` | Shut down the protocol session |
| `exit` | Exit notification |
| `textDocument/didOpen` | Track opened documents |
| `textDocument/didChange` | Full document sync |
| `textDocument/didClose` | Release documents |
| `textDocument/hover` | Type info at cursor |
| `textDocument/definition` | Jump to definition |
| `textDocument/references` | Find all references |
| `textDocument/completion` | Code completion |
| `textDocument/signatureHelp` | Function signatures |
| `textDocument/diagnostic` | File diagnostics |
| `textDocument/documentSymbol` | File symbols |
| `workspace/symbol` | Global symbol search |
| `textDocument/prepareCallHierarchy` | Call hierarchy |
| `callHierarchy/incomingCalls` | Find callers |
| `callHierarchy/outgoingCalls` | Find callees |
### Completion
The completion engine resolves types through the code graph. It provides:
- Member completion after `.` and `:` (fields, methods)
- Local variable completion
- Module-level symbol completion
- Trigger characters: `.`, `:`
### Diagnostics
After a document parses successfully, indexing stores type-checking diagnostics such as mismatches and undefined symbols. Diagnostics use the standard error, warning, information, and hint severities.
Full-document change notifications update the overlay used for diagnostics. Clients retrieve the current stored result with `textDocument/diagnostic`; this server does not push `textDocument/publishDiagnostics` notifications. A parse failure aborts re-indexing before new diagnostics are stored, so the pull result does not report that syntax error and can retain the previous successful result.
### See Also
- [Linter](guides/linter.md) — CLI-based code checking
- [Types](lua/types.md) — Type-system documentation
- [Configuration](guides/configuration.md) — Runtime configuration
---
# "Dependency Management"
## Dependency Management
Wippy resolves module dependencies from source declarations and records exact versions in `wippy.lock`. Published modules are downloaded from the Hub into the project's module directory.
The `acme/*` module names, versions, hashes, and local paths below are illustrative.
Substitute modules and verified digests from your own project or the Hub.
#### wippy.lock
The lock file tracks your project's directory layout and pinned dependencies:
```yaml
directories:
modules: .wippy
src: ./src
modules:
- name: acme/http
version: v1.2.0
hash: 4ea816fe84ca58a1f0869e5ca6afa93d6ddd72fa09e1162d9e600a7fbf39f0a2
- name: acme/sql
version: v2.0.1
hash: b3f9c8e12a456d7890abcdef1234567890abcdef1234567890abcdef12345678
```
| Field | Description |
|-------|-------------|
| `directories.modules` | Where downloaded modules are stored (default: `.wippy`) |
| `directories.src` | Where your source code lives (default: `./src`) |
| `modules[].name` | Module identifier in `org/module` format |
| `modules[].version` | Pinned semantic version |
| `modules[].hash` | Artifact digest the downloaded pack must match; a bare hex value is read as `sha256` |
| `modules[].root` | Marks the selected deployment root; at most one module may carry it |
| `options.unpack_modules` | Extract packs into directories instead of loading them as `.wapp` files (default: `false`) |
#### wippy.yaml
Module metadata for publishing. Required only when you publish your own module:
```yaml
organization: acme
module: http
version: 1.2.0
description: HTTP utilities for Wippy
license: MIT
repository: https://github.com/acme/wippy-http
keywords:
- http
- web
```
| Field | Required | Description |
|-------|----------|-------------|
| `organization` | Yes | Lowercase, alphanumeric with hyphens |
| `module` | Yes | Lowercase, alphanumeric with hyphens |
| `version` | No | Semantic version (set at publish time) |
| `description` | No | Module description |
| `license` | No | SPDX license identifier |
| `repository` | No | Source repository URL |
| `homepage` | No | Project homepage |
| `keywords` | No | Discovery keywords |
| `authors` | No | Author list |
### Declaring Dependencies
Add `ns.dependency` entries in your `_index.yaml`:
```yaml
version: "1.0"
namespace: app
entries:
- name: dependency.http
kind: ns.dependency
component: acme/http
version: "^1.0.0"
- name: dependency.sql
kind: ns.dependency
component: acme/sql
version: ">=2.0.0"
```
#### Version Constraints
| Constraint | Example | Matches |
|------------|---------|---------|
| Exact | `1.2.3` | Only 1.2.3 |
| Caret | `^1.2.0` | >=1.2.0, <2.0.0 |
| Tilde | `~1.2.0` | >=1.2.0, <1.3.0 |
| Range | `>=1.0.0` | 1.0.0 and above |
| Wildcard | `*` | Any version (picks highest) |
| Combined | `>=1.0.0 <2.0.0` | Between 1.0.0 and 2.0.0 |
#### Resolution Rules
- Each module resolves against the **intersection of all declared ranges** across the dependency graph. Incompatible ranges (diamond conflicts) fail resolution with an explicit error rather than silently picking one side.
- A full `wippy update` solves every module from its declared ranges; a targeted update and boot-time repair keep a pinned version that still satisfies every live range.
- **Root parameters win over transitive ones**: when your app and a dependency both bind the same requirement, the parameters on your `ns.dependency` take precedence. Version ranges are never overridden; every declaration joins the intersection.
- A component declared by several root `ns.dependency` entries is controlled by one of them — established declarations before new ones, parameter carriers before plain ones, ties on the lowest entry ID — and the others fold into references to it. A duplicate whose parameters disagree with the controlling declaration is rejected with a conflict error; update the existing dependency instead.
Two resolution failures are reported distinctly. A constraint expression that cannot be satisfied by any release ever — the intersection of live ranges is empty — is a conflict, and the error names the module and every requester that contributed a range. A valid range set for which the hub currently publishes no matching version is an availability failure instead: a later release can make it resolvable without any change to the declarations.
The runtime persists each resolved graph in its registry history and replays it at boot instead of re-solving, so a deployed application boots with exactly the versions that were resolved when the dependency change was applied. `wippy.lock` remains the portable snapshot for source projects.
#### Entry provenance
Provenance is registry-owned, not entry metadata. When entries are loaded, the registry stamps each one with the deployment source that supplied it:
| Field | Description |
|-------|-------------|
| `registry.owner` | Module name (`org/module`) that supplied the entry; empty for application source |
| `registry.root` | Set on `ns.dependency` entries supplied by the deployment root, marking them as root declarations |
Entry authors never write these fields; they are assigned during loading and cannot be forged from an `_index.yaml`. Inspect them with `wippy registry list --registry-meta --json`.
#### Starting a New Project
```bash
wippy init
```
Creates a `wippy.lock` with default directories.
#### Adding Dependencies
```bash
wippy add acme/http # Latest version
wippy add acme/http@1.2.3 # Exact version
wippy add acme/http@latest # Latest label
```
This updates the lock file. Then install:
```bash
wippy install
```
#### Resolving from Source
If your source already declares `ns.dependency` entries:
```bash
wippy update
```
This scans your source directory, resolves all dependency constraints, updates the lock file, and installs modules.
#### Updating Dependencies
```bash
wippy update # Re-resolve all dependencies
wippy update acme/http # Update only acme/http
wippy update acme/http acme/sql # Update specific modules
```
When updating specific modules, other modules stay pinned to their current versions. If the update would require changing non-target modules, you are prompted for confirmation.
#### Installing from Lock File
```bash
wippy install # Install all from lock
wippy install --refresh # Re-fetch every module (--force and --repair are aliases)
```
### Module Storage
Downloaded modules are stored under the `.wippy/vendor/` directory:
```
project/
wippy.lock
src/
_index.yaml
.wippy/
vendor/
acme/
http-v1.2.0.wapp
sql-v2.0.1.wapp
```
By default, modules are kept as `.wapp` files. To extract them into directories:
```yaml
## wippy.lock
options:
unpack_modules: true
```
With unpacking enabled:
```
.wippy/
vendor/
acme/
http-v1.2.0.wapp
http/
wippy.yaml
src/
_index.yaml
...
```
Unpacking never discards the pack. The canonical verified `.wapp` stays beside the extracted directory because it is the only content-addressed evidence for the module, and artifact materialization and repair read resources back out of it. The `.wapp` is what installation checks for: a directory whose pack is missing counts as not installed, and the module is downloaded again. Each install extracts the directory afresh from the verified archive, so hand-edits to a vendored directory do not survive.
Modules resolved from a [workspace replacement](#local-development-with-replacements) are never downloaded or vendored; they load from the local path.
### Local Development with Replacements
For local development, map Hub modules to local directories in the `workspace` section of a runtime configuration file. This is typically a private, ignored file composed over `.wippy.yaml`:
```yaml
## .wippy.workspace.yaml
version: "1.0"
workspace:
replacements:
acme/http: ../local-http
acme/sql: ../local-sql
```
```bash
wippy run --config .wippy.yaml --config .wippy.workspace.yaml
```
Keys are `org/module`, values are directories (relative paths resolve against the first `--config` file's directory). Setting a replacement to `null` disables one inherited from an earlier config layer or profile. Replacements can also live inside a [profile](guides/configuration.md#profiles) so they activate only with `--profile workspace`.
The path is required to exist, and to be a directory, only for a module the lock graph actually selects. A replacement declared for a module that nothing depends on is a resolution input, not a boot input: it can point at a directory that is not checked out on this machine without failing validation.
A replacement changes where a module's source comes from, not which release was chosen. It keeps the selected version, while reconciliation snapshots the current local tree and records its digest and size as the replacement identity. Entries loaded from it shadow the vendored ones with the same ID. When a replacement is declared for a module the lock does not pin a version for, resolution asks the hub for a release version, and until stronger evidence selects one it holds a local-only zero version.
Workspace replacements affect the load graph at boot and are never written to `wippy.lock`. Changes to the local source are reconciled directly, without contacting the hub. The module's source `exclude:` globs from `wippy.yaml` apply to replacement directories too, both when loading entries and when hashing content.
The `replacements:` section in `wippy.lock` is deprecated. It still loads with a warning; move those entries to `workspace.replacements` in a configuration file.
### Load Order
At boot, Wippy loads entries from directories in this order:
1. Source directory (`src`)
2. Replacement directories
3. Vendored module directories
Modules with active replacements skip their vendor path.
### Integrity Verification
Every module in the lock file carries an artifact digest. Boot refuses to load a module whose lock entry has none; `wippy install` accepts such an entry and records the digest the hub serves with the download.
At boot, downloads are staged: the pack is written to a temporary file next to its final location, verified against both the digest pinned in `wippy.lock` and the digest the hub served with the download URL (plus the served size), and only then renamed into place. A staged file that fails verification is deleted. `wippy install` renames the download into its vendor path before verifying it, checks it against the served digest and size only, deletes it on failure, and replaces a lock digest that differs from the served one rather than enforcing it.
A digest mismatch is a hard, non-retryable failure. At boot it is `PermissionDenied`, "module integrity verification failed", raised for a fresh download and for an already-vendored pack, which is re-verified against the lock digest before entries are loaded. `wippy install` reports it as `Internal`: "failed to store module" wrapping "verify cached WAPP: digest mismatch" for a pack already in the vendor directory, and "failed to download module" wrapping "verify downloaded WAPP: digest mismatch" for a fresh download. Nothing retries, re-downloads over the mismatch, or falls back to the served content.
The same check guards resolution. When the hub serves a manifest whose digest differs from the one the lock pins, the manifest cache is refreshed once and re-compared; if it still disagrees, resolution fails naming both digests.
Extracted directories carry their own recorded digest, size, and tree digest, and are re-verified against the recorded values, so a modified vendored tree is detected rather than loaded.
Replacement sources are content-addressed per reconciliation attempt. The runtime
snapshots the current local tree, then verifies that same digest and size before
loading it. A concurrent change fails validation instead of mixing two source
generations. A later reconciliation can accept a new local tree; its old recorded
digest is a checkpoint, not an immutable Hub artifact identity.
On restart, immutable historical artifacts are prefetched separately. Historical
local replacements are reconciled against the final dependency declarations
before loading, so a removed replacement does not require its old directory to
remain on disk. A replacement still selected by the final graph must be present
and valid.
### Build-time Artifacts
A module can ship a filesystem resource marked with `meta.artifact.format` that consumers materialize onto disk instead of reading at runtime. Full and targeted `wippy install` and `wippy update`, cold boot, and runtime dependency operations reconcile those outputs as part of the same transaction that changes the module graph; `artifact.materialization_root` sets the output root. See [Build-time artifacts](guides/artifacts.md).
### See Also
- [Build-time artifacts](guides/artifacts.md) - Declaring, materializing and reconciling artifact resources
- [Building Components](guides/components.md) - The author side: `ns.requirement` and supplying values via `parameters`
- [CLI](guides/cli.md) - Command reference
- [Publishing](guides/publishing.md) - Publishing modules to the hub
- [Project Structure](start/structure.md) - Project layout
---
# "Build-time artifacts"
## Build-time artifacts
A module can ship a directory that consumers use **at build time** rather than
at runtime — most usefully, a package that other modules compile against. Wippy
calls these **artifacts**: ordinary WAPP filesystem resources marked with
`meta.artifact.format`.
This is how a shared package reaches a module in a different repository. A path
alias only resolves inside one repo; an artifact travels with the module.
[The Design Layer](../frontend/design-layer.md) explains *what* belongs in such
a package and what does not; this page is the mechanism that ships it.
### Declaring an artifact
The producer declares a normal `fs.directory` and marks it with a format:
```yaml
## src/_index.yaml
entries:
- name: package_fs
kind: fs.directory
meta:
comment: The npm package consumers materialize at build time.
artifact:
format: node-package
directory: ./package
```
Nothing else changes: the resource is embedded into the WAPP like any other
`fs.directory` — list it under `embed:` in `wippy.yaml` or pass `--embed` to
`wippy publish` and `wippy pack`; a directory that is not embedded is neither
packed nor validated. Declared artifacts are **validated during module publish
and application pack**, so a malformed one fails at publish rather than in a
consumer.
### Formats
A format adapter decides how a directory is validated, what identity it has,
and where it lands. Wippy ships one built-in:
| Format | Owns subtree | Validates |
|---|---|---|
| `node-package` | `npm/` | `package.json` |
`node-package` requires a `name` and a semantic `version`, and **rejects
`preinstall`, `install`, `postinstall` and `prepare` lifecycle scripts** — a
materialized package may not execute anything on install. It writes to
`npm/` under the materialization root.
The format must be registered in the binary doing the work. Hosts may register
additional formats; duplicate names and overlapping roots are rejected.
### Materializing
Most of the time you do not run anything. Materialized outputs are reconciled
automatically during:
- full and targeted `wippy install` and `wippy update`
- cold boot
- Hub-backed dynamic install, update and uninstall
Full install, update, cold boot and runtime dependency reconciliation are
*exact*: stale outputs are pruned. A **targeted** install overlays only the
selected modules and preserves outputs belonging to modules it did not select.
Local module replacements go through the same validation and materialization
lifecycle as packed resources, so a replaced module's artifact behaves like a
published one.
#### Materializing explicitly
For a build step that needs the artifact before the runtime is involved, the
CLI exposes it directly:
```bash
wippy artifacts materialize [--root ]
```
`--root` defaults to `.wippy`. The resource must declare `meta.artifact.format`
and that format must be registered in this CLI.
Be clear about what this command deliberately does **not** do: it does not
resolve module dependencies, does not mutate `wippy.lock`, does not invoke
package managers, and does not participate in runtime composition. It validates
one artifact out of one WAPP and writes it to disk.
#### Where output lands
`artifact.materialization_root` configures the application-owned output root.
Its default is the parent of the dependency vendor directory. Each format owns
a non-overlapping subtree beneath it, so `node-package` output is always under
`/npm/`.
Materialization is transactional. Content is validated and staged, managed
roots are swapped atomically under a process lock, a failure rolls back with
the surrounding registry transaction, and an interrupted swap is recovered on
the next run.
### Worked example: a shared frontend package
A producer module whose only job is to publish a package — it serves nothing at
runtime:
```yaml
## platform/ui-kit/src/_index.yaml
version: "1.0"
namespace: kickside.ui_kit
entries:
- name: package_fs
kind: fs.directory
meta:
artifact:
format: node-package
directory: ./package
```
A consumer materializes it into its own tree before installing dependencies:
```bash
wippy artifacts materialize kickside-ui-kit-1.5.0.wapp \
kickside.ui_kit:package_fs --root ./.wippy
```
That writes `./.wippy/npm/@kickside/ui-kit`. The consumer picks it up with an
ordinary workspaces glob, so resolution is plain node resolution from there on:
```json
{
"workspaces": ["./.wippy/npm/@*/*"]
}
```
```bash
npm install
```
Two things worth copying from this shape:
- **The package is its own module, not a directory inside a bigger one.** The
artifact carries its own `package.json` version, and tying it to a module
that changes for unrelated reasons forces a release of one every time the
other moves.
- **The consumer resolves it as a normal dependency.** Once materialized there
is no Wippy-specific import path, which is what lets the same source build
inside the monorepo and outside it.
#### Authoring the producer
For a package artifact there is usually **nothing to build** — the directory is
the deliverable. A CSS vocabulary package is just files plus a manifest:
```text
platform/ui-kit/
├── src/_index.yaml # declares package_fs as the artifact
└── package/ # the directory that becomes the npm package
├── package.json
├── kx-card.css
└── kx-state.css
```
```json
{
"name": "@kickside/ui-kit",
"version": "1.5.0",
"type": "module",
"sideEffects": ["*.css"],
"exports": {
"./kx-card.css": "./kx-card.css",
"./kx-state.css": "./kx-state.css"
},
"files": ["kx-card.css", "kx-state.css", "package.json"]
}
```
`sideEffects` matters for a CSS-only package: without it a bundler is free to
treat an imported stylesheet as dead code and drop it.
**The package version must equal the module version.** `wippy publish`
validates this and refuses a mismatch, so bump both together. This is also the
reason to give a shared package its *own* module rather than nesting it inside
a larger one — otherwise every unrelated change to the host module forces a
release of the package, and vice versa.
#### Publishing
```bash
## validate without publishing
wippy publish --dry-run --version 1.5.0 --embed package_fs
## publish
wippy publish --create --module-type library --module-visibility public --version 1.5.0 --embed package_fs
```
Declared artifacts are validated as part of publish, so a package.json that
fails the format's rules is rejected here rather than in a consumer's build.
#### The dev loop
Publishing on every edit is not a dev loop. Pack the producer locally and point
the consumer's materialize step at that file instead:
```bash
## from the producer module
wippy pack /tmp/ui-kit-dev.wapp --embed package_fs
## consumers materialize from the local pack rather than the published one
UI_KIT_WAPP=/tmp/ui-kit-dev.wapp make ui-kit MOD=workflows
```
Keep that override as the *only* difference between the dev path and CI — an
environment variable that selects the pack file, with everything downstream
identical. A dev loop that materializes differently from CI stops predicting
CI.
#### Wiring it into make and CI
Make the materialize step a **prerequisite of the consumer's build**, not a
thing a person remembers to run:
```make
UI_KIT_WAPP ?=
build:
@case " $(UI_KIT_CONSUMERS) " in *" $(MOD) "*) $(MAKE) ui-kit MOD=$(MOD);; esac
cd $(call fe_dir,$(MOD)) && npm run build
```
CI then needs no artifact-specific step at all: it runs the same `make build`,
`UI_KIT_WAPP` is unset, so the fetch-and-materialize path runs against the
published version pinned in `build-inputs`. A fresh checkout cannot compile
against a stale or missing package, and a contributor who has never heard of
artifacts still gets a correct build.
### What you still have to hand-roll
`wippy artifacts materialize` is deliberately narrow, so a build that consumes
an artifact currently glues four steps together itself. Knowing which four
saves rediscovering them:
**1. Getting the `.wapp`.** The command takes a *pack file path*, not a module
reference, and does not resolve dependencies — so something has to fetch the
producer first. The workable pattern is a tiny Wippy project whose only job is
to pin and download it:
```yaml
## build-inputs/wippy.lock — a project that exists only to fetch
directories:
modules: .wippy
src: ./src
modules:
- name: kickside/ui-kit
version: 1.5.0
hash: be1eafd5…
```
```bash
( cd build-inputs && wippy install )
wapp=$(ls build-inputs/.wippy/vendor/kickside/ui-kit-*.wapp | grep -v sha256 | sort | tail -1)
```
Pinning it here rather than in the application lock keeps a build-time input
out of the runtime dependency graph.
**2. Materializing once per consumer**, into a root the consumer's package
manager can see:
```bash
wippy artifacts materialize "$wapp" kickside.ui_kit:package_fs --root ./ui/.wippy
```
**3. Wiring the consumer's `package.json`.** Materializing writes files; it
does not edit manifests. npm links the package only if the consumer declares
*both* the workspace glob and the dependency:
```json
{
"workspaces": ["./.wippy/npm/@*/*"],
"dependencies": { "@kickside/ui-kit": "*" }
}
```
The version is `*` because the materialized package carries its own. Script
this and make it idempotent — if the wiring is missing, the build fails much
later with a bare `ENOENT` on a stylesheet, which reads as a missing file
rather than as missing wiring.
**4. Running the package manager.** `materialize` does not invoke one, so
`npm install` is yours to call, after step 3.
Together, in a target that takes the consuming module as a parameter:
```make
ui-kit:
@set -e; \
( cd build-inputs && $(WIPPY) install ); \
wapp=$$(ls build-inputs/.wippy/vendor/kickside/ui-kit-*.wapp | grep -v sha256 | sort | tail -1); \
test -n "$$wapp" || { echo "no ui-kit .wapp; is the module published?"; exit 1; }; \
$(WIPPY) artifacts materialize "$$wapp" kickside.ui_kit:package_fs --root $(DIR)/.wippy; \
cd $(DIR) && node ../../scripts/wire-ui-kit.mjs && npm install --no-audit --no-fund
```
Make the whole target a prerequisite of the consumer's build, so a fresh
checkout cannot compile against a stale or absent package.
### Out of scope
Artifacts intentionally do not introduce a second resolver, package registry,
archive format, lock schema, Hub API, or module manifest. Build-only dependency
semantics, redistribution policy and host ABI validation are separate concerns
and are not solved here.
### Related
- [Dependency Management](./dependency-management.md) — resolving modules and
local replacements
- [Publishing](./publishing.md) — what a published module contains
- [The Design Layer](../frontend/design-layer.md) — why a shared frontend
vocabulary ships as a package in the first place
---
# "Building Components"
## Building Components
A **component** is a reusable Wippy module published to the Hub and mounted into a host application. A component can depend on a database, process host, or router without knowing the host's entry IDs. It declares these dependencies through a **requirement interface**, and the host supplies their values.
This guide covers the author side: declaring that interface and understanding how values flow into your entries. For the consumer side (lock files, version constraints, `wippy add`/`update`) see [Dependency Management](guides/dependency-management.md). For how a component is structured internally see [Application Architecture](concepts/architecture.md).
### The Three Entry Kinds
| Kind | Side | Role |
|------|------|------|
| `ns.definition` | component | Module metadata; required to publish. |
| `ns.requirement` | component | A hole the host must fill, and where to inject the value. |
| `ns.dependency` | host | Mounts a component and supplies values for its requirements. |
### ns.definition
Each published module must have exactly one definition. The definition can carry module metadata, a README reference, and wiki page references.
```yaml
- name: definition
kind: ns.definition
module: jobs # optional module metadata
readme: file://README.md # path to the module's documentation
meta:
title: Durable Jobs
description: Leased job queue with retry and dead-lettering.
```
`module`, `readme`, and `wiki` are definition data; all are optional. `meta` is ordinary entry metadata for management UIs. Release notes are supplied at publish time, not here.
### ns.requirement
A requirement is a **named value with a list of injection targets**. The host supplies the value, and the runtime writes it into each target entry at the specified path.
```yaml
- name: target_db
kind: ns.requirement
meta:
description: SQL database backing every table in this module.
default: app:db
targets:
- entry: app.jobs.migrations:schema
path: .meta.target_db
- entry: app.jobs.persist:lifecycle
path: .db
```
#### `default`: Mandatory or Optional
The `default` field decides whether the host *must* supply a value:
- **`default` present with a non-null value** (including an empty string) → the requirement is **optional**. If the host supplies nothing, the default is used.
- **`default` absent** → the requirement is **mandatory**. With nothing supplied, linking fails under strict mode (and warns otherwise).
An explicitly empty default (default: "") is distinct from an
absent or null default. Empty-string means "optional, falls back to nothing";
absent and default: null both mean "the host must provide this."
Use a non-null default for infrastructure that has a sane in-app convention
(app:db, app:processes); omit it for values only the
host can know.
#### `targets`: Injection Locations
Each target is an `{entry, path}` pair:
- **`entry`** — the entry the value is injected into. A bare name (`schema`) resolves within the requirement's own namespace; a fully-qualified id (`app.jobs.migrations:schema`) targets that entry exactly, across namespaces.
- **`path`** — a dot path into the target entry, e.g. `.meta.target_db`, `.host`, `.database.url`. The leading dot is conventional.
A requirement must declare at least one target.
Append instead of set with the `+=` suffix on the path — useful when several requirements contribute to one list (e.g. middleware):
```yaml
targets:
- entry: app.api:router
path: .middleware+= # appends the value to the list at .middleware
```
#### One Requirement, Multiple Targets
Group targets that need the same value under one requirement. For example, `target_db` can supply every migration's `.meta.target_db` and persistence library's `.db`; `process_host` can supply each supervised service's `.host`; and `api_router` can supply each endpoint's `.meta.router`:
```yaml
- name: process_host
kind: ns.requirement
default: app:processes
targets:
- { entry: app.jobs.service:worker.service, path: .host }
- { entry: app.jobs.service:sweeper.service, path: .host }
```
The host supplies one value, and the runtime writes it to every declared target. The requirement entry contains this wiring directly.
### Consuming a Component
The host mounts a component with `ns.dependency` and fills its requirements through `parameters`:
```yaml
version: "1.0"
namespace: app
entries:
- name: dep.jobs
kind: ns.dependency
component: acme/jobs
version: "^1.0.0"
parameters:
- name: target_db
value: app:db
- name: process_host
value: app:processes
- name: api_router
value: app:api
```
Each `parameter.name` matches a requirement; its `value` is what gets injected into that requirement's targets. Requirements with a default may be omitted; mandatory ones must be supplied.
#### Parameter Name Matching
How a parameter name binds to a requirement:
- **Bare name** (`target_db`) matches a requirement of that name belonging to the component being mounted. It does not cross into a different module's requirements.
- **Qualified name** (`acme.jobs:target_db`) matches that requirement id exactly. Use this to disambiguate when wiring transitive dependencies.
If two dependencies supply **different** values for the same requirement, that is a conflict and is reported (identical values are fine).
### When Values Resolve
Injection happens at the **Link stage** of the build pipeline — at publish, during dependency expansion, and at boot — not at runtime. The stage:
1. Collects every `ns.requirement` and every `ns.dependency` with its parameters.
2. For each requirement, resolves a value: a matching parameter wins; otherwise the default; otherwise (no default) it is unresolved.
3. Writes the resolved value into each target entry at its path (set, or append for `+=`).
Under **strict requirements** an unresolved mandatory requirement fails the build; otherwise it logs a warning and proceeds. By the time entries reach the runtime, every filled requirement has already been baked into its targets.
### Verify Integration with a Mount Test
Unit tests do not verify the assembled module's registry relationships. Add a packaging or mount test against the requirement-injected registry to verify that:
- every supervised `service` points at a process entry that exists,
- every spawned or scheduled id resolves to a real entry,
- every `env.variable`'s storage is registered.
This catches unresolved relationships such as a supervisor referencing an unregistered worker or a test fixture using a harness-only storage ID. See [Supervision](guides/supervision.md) and the [Testing](framework/testing.md) framework.
### See Also
- [Application Architecture](concepts/architecture.md) — how a component is structured internally
- [Dependency Management](guides/dependency-management.md) — lock files, versions, the consumer workflow
- [Publishing Modules](guides/publishing.md) — putting a component on the hub
- [Entry Kinds Guide](guides/entry-kinds.md) — `ns.definition`, `ns.requirement`, `ns.dependency` reference
---
# "Entry Kinds Reference"
## Entry Kinds Reference
This page summarizes the available entry kinds and links to their detailed module and system references.
The YAML and Lua blocks are reference fragments, not one application. Registry IDs,
credentials, data objects, and helpers such as `get_users` or `delete_user` are
illustrative; use the linked module pages for complete return and error contracts.
> Entries reference one another using `namespace:name`. The registry uses these references to resolve dependencies and initialization order.
### See Also
- [Registry](concepts/registry.md) — How entries are stored and resolved
- [Configuration](guides/configuration.md) — YAML configuration format
### Lua Runtime
| Kind | Description |
|------|-------------|
| `function.lua` | Lua function entry point |
| `process.lua` | Long-running Lua process |
| `workflow.lua` | Temporal workflow (deterministic) |
| `library.lua` | Shared Lua library |
| `module.lua` | Lua module surface |
| `function.lua.bc` | Precompiled function bytecode |
| `library.lua.bc` | Precompiled library bytecode |
| `process.lua.bc` | Precompiled process bytecode |
| `workflow.lua.bc` | Precompiled workflow bytecode |
```yaml
- name: handler
kind: function.lua
source: file://handler.lua
method: main
modules:
- http
- json
imports:
utils: app.lib:helpers # Import another entry as module
```
Use imports to reference other Lua entries. They become available via require("alias_name") in your code.
### HTTP Services
| Kind | Description |
|------|-------------|
| `http.service` | HTTP server (binds port) |
| `http.router` | Route prefix and middleware |
| `http.endpoint` | HTTP endpoint (method + path) |
| `http.static` | Static file serving |
```yaml
## HTTP server
- name: gateway
kind: http.service
addr: ":8080"
lifecycle:
auto_start: true
## Router with middleware
- name: api
kind: http.router
meta:
server: gateway
prefix: /api
middleware:
- cors
- ratelimit
## Endpoint
- name: users_list
kind: http.endpoint
meta:
router: app:api
method: GET
path: /users
func: list_handler
```
**Lua API:** See [HTTP Module](lua/http/http.md)
```lua
local http = require("http")
local req = http.request()
local resp = http.response()
resp:set_status(200)
resp:write_json({users = get_users()})
```
### Databases
| Kind | Description |
|------|-------------|
| `db.sql.sqlite` | SQLite database |
| `db.sql.postgres` | PostgreSQL database |
| `db.sql.mysql` | MySQL database |
| `db.cdc.postgres` | Postgres Change Data Capture source (see [CDC](system/cdc.md)) |
| `db.cdc.sqlite` | SQLite Change Data Capture source (see [CDC](system/cdc.md)) |
#### SQLite
```yaml
- name: database
kind: db.sql.sqlite
file: "./data/app.db"
lifecycle:
auto_start: true
## In-memory for testing
- name: testdb
kind: db.sql.sqlite
file: ":memory:"
```
#### PostgreSQL
```yaml
- name: database
kind: db.sql.postgres
host: localhost
port: 5432
database: dbname
username: user
password: pass
options:
sslmode: disable
pool:
max_open: 25
max_idle: 5
max_lifetime: "30m"
lifecycle:
auto_start: true
```
#### MySQL
```yaml
- name: database
kind: db.sql.mysql
host: localhost
port: 3306
database: dbname
username: user
password: pass
options:
parseTime: "true"
lifecycle:
auto_start: true
```
See [Database](system/database.md) for `${env:NAME}` secret references, TLS options, and connection pool tuning. When an env-backed value behind a database entry changes, the pool swaps live — active borrows finish against the old connection settings.
**Lua API:** See [SQL Module](lua/storage/sql.md)
```lua
local sql = require("sql")
local db, err = sql.get("app:database")
local rows, err = db:query("SELECT * FROM users WHERE id = ?", {user_id})
db:execute("INSERT INTO logs (msg) VALUES (?)", {message})
```
### Key-Value Stores
| Kind | Description |
|------|-------------|
| `store.memory` | In-memory key-value store |
| `store.sql` | SQL-backed key-value store |
| `store.kv.raft` | Cluster-replicated, strongly-consistent KV (shared Raft) |
| `store.kv.crdt` | Cluster-replicated, eventually-consistent KV (gossip/CRDT) |
```yaml
## Memory store
- name: cache
kind: store.memory
lifecycle:
auto_start: true
## SQL-backed store
- name: persistent_store
kind: store.sql
database: app:database
table_name: kv_store
lifecycle:
auto_start: true
## Cluster-replicated store (requires clustering)
- name: deployments
kind: store.kv.raft
namespace: deploy
```
The `store.kv.*` kinds need [clustering](guides/cluster.md) enabled. See [Store](system/store.md#cluster-kv-stores) for the consistency tradeoffs.
**Lua API:** See [Store Module](lua/storage/store.md)
```lua
local store = require("store")
local s, err = store.get("app:cache")
s:set("user:123", user_data, 3600) -- TTL in seconds
local data = s:get("user:123")
```
### Queues
| Kind | Description |
|------|-------------|
| `queue.driver.memory` | In-memory queue driver |
| `queue.driver.amqp` | AMQP (RabbitMQ) driver |
| `queue.driver.sqs` | AWS SQS driver |
| `queue.queue` | Queue declaration |
| `queue.consumer` | Queue consumer |
```yaml
## Driver
- name: queue_driver
kind: queue.driver.memory
lifecycle:
auto_start: true
## Queue
- name: jobs
kind: queue.queue
driver: queue_driver
## Consumer
- name: job_consumer
kind: queue.consumer
queue: app:jobs
func: job_handler
concurrency: 4
prefetch: 10
lifecycle:
auto_start: true
```
**Lua API:** See [Queue Module](lua/storage/queue.md)
```lua
local queue = require("queue")
-- Publish a message
queue.publish("app:jobs", {task = "process", id = 123})
-- In a consumer handler: the message body is the handler's argument
local function main(data)
-- access delivery metadata via the current message
local msg = queue.message()
local id = msg:id()
local priority = msg:header("priority")
msg:ack()
end
```
The consumer's func is invoked once per message with the message body as its argument. Use queue.message() inside the handler for the delivery's id(), header()/headers(), and ack()/nack().
### Process Management
| Kind | Description |
|------|-------------|
| `process.host` | Process execution host |
| `process.service` | Supervised process (wraps process.lua) |
| `terminal.host` | Terminal/CLI host |
| `pg.scope` | Process-group scope (see [Process Groups](system/process-groups.md)) |
```yaml
## Process host (where processes run)
- name: processes
kind: process.host
host:
workers: 32 # Worker goroutines (default: NumCPU)
queue_size: 1024 # Global queue capacity
local_queue_size: 256 # Per-worker queue
lifecycle:
auto_start: true
## Process definition
- name: worker_process
kind: process.lua
source: file://worker.lua
method: main
## Supervised process service
- name: worker
kind: process.service
process: app:worker_process
host: app:processes
input: ["arg1", "arg2"]
lifecycle:
auto_start: true
restart:
max_attempts: 10
- name: terminal
kind: terminal.host
lifecycle:
auto_start: true
```
Use process.service when you need a process to run as a supervised service with automatic restart. The process field references a process.lua entry.
Updating a live `process.host` entry rescales `host.workers` in place — running processes, PIDs, and queues are preserved. `host.queue_size`, `host.local_queue_size`, and `lifecycle` are fixed at construction: a live update changing them is rejected, as is resizing workers on a host whose workers are affinity-managed.
#### Process security
`process.lua` and `process.lua.bc` entries accept a top-level `security:` block. It is part of the entry, so it applies to every spawn of that process, on both `process.host` and `terminal.host`:
```yaml
- name: worker_process
kind: process.lua
source: file://worker.lua
method: main
security:
actor:
id: system.worker
meta:
tenant: acme
policies:
- app.security:worker_policy
groups:
- app.security:background_jobs
```
| Field | Description |
|-------|-------------|
| `actor.id` | Actor identity the process runs as; replaces the inherited actor |
| `actor.meta` | Actor attributes policies evaluate |
| `policies` | Registry IDs (`namespace:name`) of policies merged into the scope |
| `groups` | Registry IDs of policy groups whose policies are merged into the scope |
Resolution happens as the process starts and is atomic: if any listed policy or group cannot be resolved, the spawn fails and no partial context is installed. Omitting `actor` inherits the spawner's actor; omitting both `policies` and `groups` inherits the spawner's scope. `function.lua`, `function.lua.bc`, `process.lua`, and `process.lua.bc` all accept the block.
A command entry can additionally declare `meta.command.security`, which applies only when the entry is launched as a CLI command — see [Command security](guides/cli.md#command-security). It does not affect ordinary spawns.
See [Security](system/security.md).
### Temporal (Workflows)
| Kind | Description |
|------|-------------|
| `temporal.client` | Temporal client connection |
| `temporal.worker` | Temporal worker |
```yaml
- name: temporal_client
kind: temporal.client
address: "localhost:7233"
namespace: "default"
auth:
type: none # none, api_key, mtls
lifecycle:
auto_start: true
- name: temporal_worker
kind: temporal.worker
client: temporal_client
task_queue: "main-queue"
lifecycle:
auto_start: true
```
### Cloud Storage
| Kind | Description |
|------|-------------|
| `config.aws` | AWS configuration |
| `cloudstorage.s3` | S3 bucket access |
```yaml
- name: aws
kind: config.aws
region: "us-east-1"
access_key_id: ${env:AWS_ACCESS_KEY_ID}
secret_access_key: ${env:AWS_SECRET_ACCESS_KEY}
- name: uploads
kind: cloudstorage.s3
config: app:aws
bucket: "my-uploads"
endpoint: "" # Optional, for S3-compatible services
```
**Lua API:** See [Cloud Storage Module](lua/storage/cloud.md)
```lua
local cloudstorage = require("cloudstorage")
local storage, err = cloudstorage.get("app:uploads")
storage:upload_object("files/doc.pdf", file_content)
local url = storage:presigned_get_url("files/doc.pdf", {expiration = 3600}) -- seconds, default 3600
```
Use endpoint to connect to S3-compatible services like MinIO or DigitalOcean Spaces.
### File Systems
| Kind | Description |
|------|-------------|
| `fs.directory` | Directory access |
| `fs.embed` | Read-only embedded filesystem |
```yaml
- name: data_dir
kind: fs.directory
directory: "./data"
auto_init: true # Create if not exists
mode: "0755" # Permissions
```
**Lua API:** See [Filesystem Module](lua/storage/filesystem.md)
```lua
local fs = require("fs")
local filesystem, err = fs.get("app:data_dir")
local file = filesystem:open("output.txt", "w")
file:write("Hello, World!")
file:close()
```
### Environment
| Kind | Description |
|------|-------------|
| `env.storage.memory` | In-memory env storage |
| `env.storage.file` | File-based env storage |
| `env.storage.os` | OS environment |
| `env.storage.static` | Read-only static key-value storage |
| `env.storage.router` | Env router (multiple storages) |
| `env.variable` | Environment variable |
```yaml
- name: os_env
kind: env.storage.os
- name: file_env
kind: env.storage.file
file_path: ".env"
auto_create: true
- name: defaults
kind: env.storage.static
values:
PUBLIC_API_HOST: "https://api.example.com"
APP_ENV: "production"
- name: app_env
kind: env.storage.router
storages:
- app:os_env
- app:file_env
- app:defaults
```
**Lua API:** See [Env Module](lua/system/env.md)
```lua
local env = require("env")
local api_key = env.get("API_KEY")
env.set("CACHE_TTL", "3600")
```
The router tries storages in order. First match wins for reads; writes go to the first storage in the list.
### Templates
| Kind | Description |
|------|-------------|
| `template.jet` | Individual Jet template |
| `template.set` | Template set configuration |
```yaml
## Template set with engine configuration
- name: templates
kind: template.set
engine:
development_mode: false
extensions:
- ".jet"
- ".html.jet"
## Individual template
- name: email_template
kind: template.jet
source: file://templates/email.jet
set: app:templates
```
**Lua API:** See [Template Module](lua/text/template.md)
```lua
local templates = require("templates")
local set, err = templates.get("app:templates")
local html = set:render("email", {
user = "Alice",
message = "Welcome!"
})
```
### Security
| Kind | Description |
|------|-------------|
| `security.policy` | Security policy with conditions |
| `security.policy.expr` | Expression-based policy |
| `security.token_store` | Token storage |
```yaml
## Condition-based policy
- name: admin_policy
kind: security.policy
policy:
actions: "*"
resources: "*"
effect: allow
conditions:
- field: "actor.meta.role"
operator: eq
value: "admin"
## Expression-based policy
- name: owner_policy
kind: security.policy.expr
policy:
actions: "*"
resources: "*"
effect: allow
expression: 'actor.id == meta.owner_id || actor.meta.role == "admin"'
groups:
- operators
```
Policy groups are formed by the policies themselves: a policy lists the group IDs it belongs to under `groups:`, and a group is the set of policies naming it. There is no separate group entry kind. Group IDs are registry IDs — a bare name resolves in the declaring policy's namespace, so `operators` above becomes `app.security:operators` when declared in namespace `app.security`. Entries reference groups by their full `namespace:name`.
**Lua API:** See [Security Module](lua/security/security.md)
```lua
local security = require("security")
-- Check permission before action
if security.can("delete", "users", {user_id = id}) then
delete_user(id)
end
-- Get current actor
local actor = security.actor()
```
Every policy in scope is evaluated. A deny from any matching policy wins over every allow; with no deny, a matching allow grants access. Order does not matter.
### Contracts (Dependency Injection)
| Kind | Description |
|------|-------------|
| `contract.definition` | Interface with method specifications |
| `contract.binding` | Maps contract methods to function implementations |
```yaml
## Define the contract interface
- name: greeter
kind: contract.definition
methods:
- name: greet
description: Returns a greeting message
- name: greet_with_name
description: Returns a personalized greeting
input_schemas:
- format: "application/schema+json"
definition: {"type": "string"}
output_schemas:
- format: "application/schema+json"
definition: {"type": "string"}
## Implementation functions
- name: greeter_greet
kind: function.lua
source: file://greeter_greet.lua
method: main
- name: greeter_greet_name
kind: function.lua
source: file://greeter_greet_name.lua
method: main
## Bind contract methods to implementations
- name: greeter_impl
kind: contract.binding
contracts:
- contract: app:greeter
default: true
methods:
greet: app:greeter_greet
greet_with_name: app:greeter_greet_name
```
Usage from Lua:
```lua
local contract = require("contract")
-- Open binding by ID
local greeter, err = contract.open("app:greeter_impl")
-- Call methods
local result = greeter:greet()
local personalized = greeter:greet_with_name("Alice")
-- Check if instance implements contract
local is_greeter = contract.is(greeter, "app:greeter")
```
**Lua API:** See [Contract Module](lua/core/contract.md)
Mark one binding as default: true to use it when opening a contract without specifying a binding ID. A contract may have only one default binding.
### Execution
| Kind | Description |
|------|-------------|
| `exec.native` | Native command execution |
| `exec.docker` | Docker container execution |
```yaml
- name: native_exec
kind: exec.native
default_work_dir: "/app"
command_whitelist:
- "ls"
- "cat"
- name: docker_exec
kind: exec.docker
image: "python:3.11-slim"
default_work_dir: "/workspace"
auto_remove: true
memory_limit: 536870912 # 512MB
command_whitelist:
- "python"
```
### WASM Runtime
| Kind | Description |
|------|-------------|
| `function.wat` | WebAssembly function (WAT text format) |
| `function.wasm` | WebAssembly function (binary) |
| `process.wasm` | WebAssembly process |
```yaml
## WAT text is inline source
- name: sum_wat
kind: function.wat
source: file://sum.wat
method: sum
transport: payload # or wasi-http
## Binary WASM is loaded from a filesystem entry and verified by hash
- name: sum
kind: function.wasm
fs: app:modules
path: sum.wasm
hash: sha256:2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
method: sum
transport: payload
```
`function.wasm` and `process.wasm` take `fs`, `path`, and `hash` — there is no `source` field on a binary entry; `source` belongs to `function.wat` only. `hash` is required and must be `sha256:`; the module is rejected if the bytes do not match.
See [WASM Overview](wasm/overview.md).
### Networks
| Kind | Description |
|------|-------------|
| `network` | Base network overlay |
| `network.socks5` | SOCKS5 proxy overlay |
| `network.i2p` | I2P network overlay |
| `network.tailscale` | Tailscale overlay |
Referenced by `http.service` via `network:`, by `funcs`/`process` via the `network` option, and by `http_client` via the `overlay_network` option. See [Network](system/network.md).
### Registry Primitives
| Kind | Description |
|------|-------------|
| `registry.entry` | Plain data entry with no service behind it (app-specific config) |
| `ns.definition` | Namespace definition |
| `ns.requirement` | Namespace requirement declaration |
| `ns.dependency` | Namespace dependency |
The `ns.*` kinds are authored like any other entry: a component declares `ns.definition` and `ns.requirement`, and a host declares `ns.dependency`. See [Building Components](guides/components.md).
### Lifecycle Configuration
Supervisor-managed service entries expose lifecycle configuration. The block below belongs inside a service entry that supports it:
```yaml
lifecycle:
auto_start: true # Start automatically
start_timeout: 10s # Max startup time
stop_timeout: 10s # Max shutdown time
stable_threshold: 5s # Uninterrupted run time before retry accounting resets
requires:
- app:database
restart: # Retry policy
initial_delay: 1s
max_delay: 90s
backoff_factor: 2.0
max_attempts: 0 # 0 = infinite
```
Use depends_on to ensure entries start in the correct order. The supervisor starts a dependent entry only after each of its dependencies has completed its own start.
### Entry Reference Format
Entries are referenced using `namespace:name` format:
```yaml
## Definition
namespace: app.users
entries:
- name: handler
kind: function.lua
## Reference from another entry
func: app.users:handler
```
### Overriding Entries
Any entry's fields — including its `kind` — can be overridden at launch without editing the source YAML, using the `override:` config section or the `-o` CLI flag. Keys use `namespace:entry:path` format:
```yaml
override:
app:gateway:addr: ":9090" # data field (a bare path targets data.*)
app:worker:meta.priority: high # meta field
app:db:kind: db.sql.postgres # the entry's typed kind
app:db:data.kind: custom # a payload field literally named "kind"
```
| Path | Targets |
|------|---------|
| `kind` | The entry's typed kind (must be a non-empty string) |
| `data.` or bare `` | A field in the entry's data payload |
| `meta.` | A field in the entry's metadata |
The same overrides apply from the CLI:
```bash
wippy run -o app:db:kind=db.sql.postgres -o app:gateway:addr=:9090
```
CLI (`-o`) values coerce by shape (`true`/`false` to bool, numbers to numbers, otherwise string); `override:` section values keep their YAML type. To override global [configuration](guides/configuration.md) sections instead of entries, use `--set`.
---
# "Observability"
## Observability
Wippy exposes application and runtime behavior through logging, metrics, distributed tracing, and runtime statistics.
### Overview
Three observability areas are configured at boot:
| Pillar | Backend | Configuration |
|--------|---------|---------------|
| Logging | Zap (JSON structured) | `logger` and `logmanager` |
| Metrics | Prometheus | `prometheus` |
| Tracing | OpenTelemetry | `otel` |
#### Logger Encoding
```yaml
logger:
encoding: json # json or console
```
Level and output are controlled by CLI flags (`-v`, `-c`, `-s`); only `encoding` is read from YAML.
#### Log Manager
The log manager controls log propagation and event streaming:
```yaml
logmanager:
propagate_downstream: true # Propagate to child components
stream_to_events: false # Forward logs to event bus
min_level: 0 # -1=debug, 0=info, 1=warn, 2=error (wippy run sets 0, or -1 with -v)
```
When `stream_to_events` is enabled, log entries become events that processes can subscribe to via the event bus.
The embedded log-manager default is `-1`, but `wippy run` applies its CLI logging choice at startup: info (`0`) by default and debug (`-1`) with `-v` or `--very-verbose`.
#### Automatic Context
Logs emitted from Lua via the [logger module](lua/system/logger.md) automatically include:
- `pid` - Current process PID
- `location` - Entry ID and caller line (e.g., `app.api:handler:45`)
### Prometheus Metrics
```yaml
prometheus:
enabled: true
address: "localhost:9090"
```
Metrics are exposed at `/metrics` on the configured address; the same listener serves `/livez`. `max_cardinality` (default 1024) caps the number of live label sets per exporter; the least recently updated series are evicted beyond it.
#### Scrape Configuration
```yaml
## prometheus.yml
scrape_configs:
- job_name: 'wippy'
static_configs:
- targets: ['localhost:9090']
scrape_interval: 15s
```
For the Lua metrics API, see [Metrics Module](lua/system/metrics.md).
### OpenTelemetry
OpenTelemetry (OTEL) provides distributed tracing and optional metrics export.
#### Basic Configuration
```yaml
otel:
enabled: true
endpoint: "localhost:4318"
protocol: http/protobuf # grpc or http/protobuf
service_name: my-app
service_version: "1.0.0"
insecure: true # Use plaintext for a local collector
sample_rate: 1.0 # 0.0 to 1.0
traces_enabled: true
metrics_enabled: false
propagators:
- tracecontext
- baggage
```
#### Trace Sources
All trace sources are on by default once `otel.enabled` is true; each can be disabled individually:
```yaml
otel:
enabled: true
endpoint: "localhost:4318"
service_name: my-app
# HTTP request tracing
http:
enabled: true
extract_headers: true # Read incoming trace context
inject_headers: true # Write trace context to the HTTP response
# Process lifecycle tracing
process:
enabled: true
trace_lifecycle: true # Trace spawn/exit events
# Queue message tracing
queue:
enabled: true
# Function call tracing
interceptor:
enabled: true
```
When OTEL is enabled, HTTP tracing and propagation, process tracing and lifecycle spans, function interception, queue tracing, and trace export are enabled by default. Temporal tracing and metric export default to disabled. The pinned runtime registers the function interceptor at order 100; although an `interceptor.order` value can be decoded from configuration, it does not change that registration order.
#### Temporal Workflows
Enable tracing for Temporal workflows:
```yaml
otel:
enabled: true
endpoint: "localhost:4318"
service_name: my-app
temporal:
enabled: true
```
When enabled, the Temporal SDK's tracing interceptor is registered for both client and worker operations.
Traced operations include:
- Workflow starts and completions
- Activity executions
- Child workflow calls
- Signal and query handling
#### What Gets Traced
| Component | Span Name | Attributes |
|-----------|-----------|------------|
| HTTP requests | `{METHOD} {route}` | http.method, http.url, http.host, http.route |
| Function calls | Function ID | process.pid, frame.id |
| Process lifecycle | `{source}.started/terminated` | process.pid |
| Queue messages | `{queue}.publish` | messaging.operation, messaging.destination.name |
| Temporal workflows | Workflow/Activity name | workflow.id, run.id |
#### Context Propagation
The configured integrations propagate trace context through:
- **HTTP → Function**: W3C Trace Context headers
- **Function → Function**: Frame context inheritance
- **Process → Process**: Spawn context
- **Queue publish → consume**: Message headers
#### Environment Variables
OTEL can be configured via environment:
| Variable | Description |
|----------|-------------|
| `OTEL_SDK_DISABLED` | Set to `true` to disable OTEL |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Collector endpoint; an `http://` or `https://` scheme is removed before exporter setup |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` or `http/protobuf` |
| `OTEL_EXPORTER_OTLP_INSECURE` | Set to `true` to use a plaintext collector connection |
| `OTEL_SERVICE_NAME` | Service name |
| `OTEL_SERVICE_VERSION` | Service version |
| `OTEL_TRACES_SAMPLER` | `always_on`, `always_off`, `traceidratio`, or `parentbased_traceidratio` |
| `OTEL_TRACES_SAMPLER_ARG` | Sample rate (0.0-1.0) |
| `OTEL_TRACES_SAMPLER` | `always_on`, `always_off`, `traceidratio`, or `parentbased_traceidratio` (ratio from `OTEL_TRACES_SAMPLER_ARG`) |
| `OTEL_EXPORTER_OTLP_INSECURE` | Set to `true` to allow non-TLS connections |
| `OTEL_PROPAGATORS` | Propagator list |
### Runtime Statistics
The `system` module provides internal runtime statistics:
```lua
local system = require("system")
-- Memory statistics
local mem, mem_err = system.memory.stats()
-- mem.alloc, mem.heap_alloc, mem.heap_objects, etc.
-- Goroutine count
local count, count_err = system.runtime.goroutines()
-- Supervisor states
local states, states_err = system.supervisor.states()
```
These functions return `value, error`. They require the `system.read` permission in the current security scope.
### See Also
- [Logger Module](lua/system/logger.md) — Lua logging API
- [Metrics Module](lua/system/metrics.md) — Lua metrics API
- [System Module](lua/system/system.md) — Runtime statistics
---
# "Queue Consumers"
## Queue Consumers
Queue consumers deliver messages from a queue to function handlers through a configurable worker pool.
### Overview
```mermaid
flowchart LR
subgraph Consumer
QD[Queue Driver] --> DC[Delivery Channel prefetch=10]
DC --> WP[Worker Pool concurrency]
WP --> FH[Function Handler]
FH --> AN[Ack/Nack]
end
```
### Configuration
| Option | Default | Max | Description |
|--------|---------|-----|-------------|
| `queue` | Required | - | Queue registry ID |
| `func` | Required | - | Handler function registry ID |
| `concurrency` | 1 | 1000 | Worker count |
| `prefetch` | 10 | 10000 | Message buffer size |
| `auto_ack` | false | - | Driver-level auto-ack (AMQP `Consume` autoAck; ignored by the memory driver) |
| `driver_options` | `{}` | - | Driver-specific consumer options |
### Entry Definition
```yaml
- name: order_consumer
kind: queue.consumer
queue: app:orders
func: app:process_order
concurrency: 5
prefetch: 20
lifecycle:
auto_start: true
requires:
- app:orders
```
### Handler Function
The handler function receives the body after the queue's codec decodes it. Use `queue.message()` to access the current delivery and its metadata:
```lua
-- process_order.lua
local queue = require("queue")
local logger = require("logger")
local function main(order)
local msg, msg_err = queue.message()
if msg_err then
return nil, msg_err
end
logger:info("processing order", {
message_id = msg:id(),
order_id = order.id
})
return {processed = true, order_id = order.id}
end
return {main = main}
```
```yaml
- name: process_order
kind: function.lua
source: file://process_order.lua
method: main
modules:
- queue
- logger
```
### Acknowledgment
Unless the handler explicitly settles the delivery, the consumer uses the function invocation result:
| Handler outcome | Action | Effect |
|-----------------|--------|--------|
| Completes without an invocation error | Ack | Message removed from queue |
| Returns or raises an invocation error | Nack | Redelivery is driver-dependent |
Ordinary return values, including `false`, do not select acknowledgment behavior. Call `msg:ack()` or `msg:nack()` to settle explicitly. Settlement is single-shot: the first settlement wins. With AMQP `auto_ack: true`, the broker acknowledges on delivery, so a later handler failure cannot cause broker redelivery.
The handler can settle the message itself with `queue.message()` and `msg:ack()` / `msg:nack()`; the consumer then skips its own ack/nack.
### Worker Pool
- Workers run as concurrent goroutines.
- Each worker processes one message at a time.
- Workers pull from a shared delivery channel. The next idle worker receives the next message, without guaranteed ordering or rotation across workers.
- The prefetch buffer allows the driver to deliver messages ahead of processing.
#### Example
```
concurrency: 3
prefetch: 10
Flow:
1. Driver delivers up to 10 messages to buffer
2. 3 workers pull from buffer concurrently
3. As workers finish, buffer refills
4. Backpressure when all workers busy and buffer full
```
### Graceful Shutdown
During shutdown, the consumer:
1. Stops accepting new deliveries
2. Cancels worker contexts
3. Waits for in-flight handlers, up to the stop timeout
4. Returns a timeout error if workers do not finish
### Queue Declaration
```yaml
## Queue driver (memory for dev/test)
- name: queue_driver
kind: queue.driver.memory
lifecycle:
auto_start: true
## Queue definition
- name: orders
kind: queue.queue
driver: app:queue_driver
queue_name: orders # Override name (default: entry name)
codec: json/plain # Payload codec (optional; json/plain is the default)
dead_letter: # Accepted configuration; not enforced by built-in drivers
queue: app:dlq
max_attempts: 5
driver_options:
memory:
max_length: 10000 # Memory driver: bounded queue size
```
| Field | Description |
|-------|-------------|
| `queue_name` | Override queue name (default: entry ID name) |
| `codec` | Payload codec name |
| `dead_letter.queue` | Registry ID accepted for a dead-letter queue; not enforced by built-in drivers |
| `dead_letter.max_attempts` | Attempt count accepted in configuration; not enforced by built-in drivers |
| `driver_options` | Driver-specific settings keyed by driver name |
No built-in driver currently counts attempts or routes messages from the `dead_letter` block. The runtime does not translate that block into AMQP queue arguments, and ordinary AMQP consumer failures request requeue. Broker-side dead-lettering must therefore be configured and triggered outside this block. The memory driver does not route to a DLQ.
### Memory Driver
The built-in in-memory driver is intended for development and testing:
- Its kind is `queue.driver.memory`.
- Messages are stored in memory.
- Nack attempts to re-enqueue a cloned message at the end of the queue; that attempt can fail when the bounded queue is full.
- Messages do not persist across restarts.
### See Also
- [Message Queue](lua/storage/queue.md) — Queue module reference
- [Queue Configuration](system/queue.md) — Queue drivers and entry definitions
- [Supervision](guides/supervision.md) — Consumer lifecycle
- [Process Management](lua/core/process.md) — Process spawning and communication
---
# "Supervision"
## Supervision
The supervisor manages service startup, dependency order, restarts, and graceful shutdown. Services with `auto_start: true` start when the application boots.
### Lifecycle Configuration
Services register with the supervisor using a `lifecycle` block. For processes, use `process.service` to wrap a process definition:
```yaml
## Process definition (the code)
- name: worker_process
kind: process.lua
source: file://worker.lua
method: main
## Supervised service (wraps the process with lifecycle management)
- name: worker
kind: process.service
process: app:worker_process
host: app:processes
lifecycle:
auto_start: true
startup: required
start_timeout: 30s
stop_timeout: 10s
stable_threshold: 5s
requires:
- app:database
restart:
initial_delay: 2s
max_delay: 60s
max_attempts: 10
```
`host` must reference a configured process host. The `requires` entry must resolve either to another supervised service or, through registry dependency extraction, to a supervised service that owns the referenced resource.
| Field | Default | Description |
|-------|---------|-------------|
| `auto_start` | `false` | Start automatically when supervisor starts |
| `startup` | `required` | Startup policy for an auto-start root: `required` blocks boot on failure; `optional` may fail and keep retrying without blocking independent branches |
| `start_timeout` | `10s` | Maximum time allowed for startup |
| `stop_timeout` | `10s` | Maximum time for graceful shutdown |
| `stable_threshold` | `5s` | Runtime after which a later failure resets the retry counter |
| `requires` | `[]` | Services that must be running first (legacy alias: `depends_on`) |
| `startup` | `required` | `required` reports a failed or blocked auto-start as a transaction error; `optional` lets the service keep retrying in the background without failing the batch |
### Dependency Resolution
The supervisor resolves dependencies from two sources:
1. **Explicit dependencies** declared in `requires` (or the legacy `depends_on`)
2. **Registry-extracted dependencies** from entry references (e.g., `database: app:db` in your config)
```mermaid
graph LR
A[HTTP Server] --> B[Router]
B --> C[Handler Function]
C --> D[Database]
C --> E[Cache]
```
Dependencies start before their dependents. If Service C depends on A and B, both dependencies must reach the `Running` state before C starts.
You do not need to repeat an infrastructure reference in requires when registry dependency extraction can trace that reference to a supervised service. Use requires for lifecycle dependencies that are not already expressed by entry references.
### Restart Policy
When a service fails, the supervisor retries according to its `restart` block:
```yaml
lifecycle:
restart:
initial_delay: 1s # First retry wait
max_delay: 90s # Accepted backoff cap; see current behavior below
backoff_factor: 2.0 # Accepted multiplier; see current behavior below
jitter: 0.1 # ±10% randomization
max_attempts: 0 # 0 = infinite retries
```
In runtime v0.3.32a, the supervisor constructs a new backoff calculator for each retry and takes only its first interval. Each retry therefore waits `initial_delay` with the configured jitter (0.9s–1.1s for the values above). `backoff_factor` and `max_delay` are accepted configuration fields but do not change this schedule in the pinned runtime.
`max_attempts` counts the initial failed start. A value of `1` permits no retry, and `10` permits at most nine follow-up starts. A value of `0` allows unlimited attempts.
When a service runs longer than `stable_threshold`, its retry counter resets, so later failures start from the initial retry delay.
#### Terminal Errors
These errors stop retry attempts:
- Context cancellation
- Explicit termination request
- Errors marked as non-retryable
### Security Context
Services can run with a specific security identity:
```yaml
## Process definition
- name: admin_worker_process
kind: process.lua
source: file://admin_worker.lua
method: main
## Supervised service with security context
- name: admin_worker
kind: process.service
process: app:admin_worker_process
host: app:processes
lifecycle:
auto_start: true
security:
actor:
id: "service:admin-worker"
meta:
role: admin
groups:
- app:admin_policies
policies:
- app:data_access
```
The security context defines:
| Field | Description |
|-------|-------------|
| `actor.id` | Identity string for this service |
| `actor.meta` | Key-value metadata (role, permissions, etc.) |
| `groups` | Policy groups to apply |
| `policies` | Individual policies to apply |
Code running in the service inherits this security context. The `security` module can use it for permission checks:
```lua
local security = require("security")
if security.can("delete", "users") then
-- allowed
end
```
When no security block is configured, the supervisor adds no service-specific actor or policy scope; any security values already present in the parent context remain inherited. In strict mode (default), a check with an incomplete resulting security context is denied. Configure a complete service security context for services that need authorization.
### Re-registration and Replacement
A registry change can re-register an ID that already has a running controller. If the registration carries the same service instance, nothing is disturbed. If it carries a **different** instance — the manager rebuilt the service because its configuration changed — the supervisor retires the existing controller and adopts the replacement.
Retirement covers more than the one service. A running dependent captured the superseded instance, so it cannot keep running against a service that is being replaced underneath it; the retirement closure is the replaced service plus every running service that depends on it, stopped in dependency order (dependents first). Services already stopped are not stopped a second time — a manager that stops its own instance before re-registering does not see a redundant `Stop`.
The handover is transactional:
1. The plan is computed without touching anything, so a planning failure leaves the running set intact.
2. The stop batch runs. **If any stop fails, the handover is rejected**: the services the batch already stopped are brought back up and the error is reported. A service that could not be brought back is named in that error. The supervisor ends up owning the same running set it had before the commit, never a half-retired one.
3. Only after the batch succeeds are the retired controllers dropped and canceled, freeing the superseded service instances.
4. The replacement is created and started through the same dependency-aware sequencer as any other start, and the dependents that were stopped for the handover come back up against the adopted instance.
A service that was running before the replacement is restarted afterwards even when the new registration sets `auto_start: false` — replacing an active service is an update, not an implicit stop. Restarting a stopped dependent is governed by its own restart policy and does not gate the commit.
### Service States
```mermaid
stateDiagram-v2
[*] --> Unknown
Unknown --> Starting
Starting --> Running
Running --> Stopping
Stopping --> Stopped
Stopping --> Failed : timeout/cancel
Stopped --> [*]
Running --> Failed
Starting --> Failed
Failed --> Starting : retry
Running --> Exited
Starting --> Exited
Exited --> [*]
```
The supervisor transitions services through these states:
| State | Description |
|-------|-------------|
| `Unknown` | Registered but not started |
| `Starting` | Startup in progress |
| `Running` | Operating normally |
| `Stopping` | Graceful shutdown in progress |
| `Stopped` | Stop operation completed; service-reported stop details may still contain an error |
| `Exited` | Terminated by explicit request or a non-retryable/terminal error |
| `Failed` | Error occurred, may retry |
### Startup and Shutdown Order
**Startup:** Dependencies start before dependents. Services at the same dependency level can start in parallel.
**Shutdown:** Dependents stop before dependencies, allowing dependent services to finish first.
```
Startup: database → cache → handler → http_server
Shutdown: http_server → handler → cache → database
```
On SIGINT or SIGTERM the runtime begins a graceful shutdown and the whole sequence runs under a single budget, `shutdown.timeout` in the runtime config (default 30s). That budget is a fresh deadline that does not inherit the interrupted context, so a Ctrl-C does not cut component shutdown short; per-service `stop_timeout` still bounds each individual stop within it. A second signal skips the sequence and exits immediately.
```yaml
## .wippy.yaml
shutdown:
timeout: 60s
```
### See Also
- [Process Model](concepts/process-model.md) — Process lifecycle
- [Configuration](guides/configuration.md) — YAML configuration format
- [Security Module](lua/security/security.md) — Permission checks in Lua
---
# "Publishing Modules"
## Publishing Modules
Publishing packages a module and makes a version or mutable label available through the Wippy Hub.
This is a publishing workflow and reference. The `acme/*` modules, URLs, tokens,
credentials, and example source are illustrative; replace them with resources owned
by your organization.
### Prerequisites
1. Create an account on [hub.wippy.ai](https://hub.wippy.ai)
2. Create an organization or join one
3. Have permission to create modules in that organization — the first `wippy publish` registers the module automatically
### Module Structure
```
mymodule/
├── wippy.yaml # Module manifest
├── src/
│ ├── _index.yaml # Entry definitions
│ └── *.lua # Source files
└── README.md # Documentation (optional)
```
### wippy.yaml
Define module metadata in `wippy.yaml`:
```yaml
organization: acme
module: http-utils
type: library
description: HTTP utilities and helpers
license: MIT
repository: https://github.com/acme/http-utils
homepage: https://acme.dev
keywords:
- http
- utilities
authors:
- Acme Engineering
embed:
- acme.http:assets
exclude:
- test/**
- "*.test.lua"
- acme.http:debug_handler
exclude_meta:
stage:
- experimental
metadata:
support_url: https://acme.dev/support
```
| Field | Required | Description |
|-------|----------|-------------|
| `organization` | Yes | Organization name on the Hub |
| `module` | Yes | Module name |
| `type` | No | Module type: `library`, `application`, `agent`, or `plugin` |
| `description` | No | Short description |
| `license` | No | SPDX identifier (MIT, Apache-2.0) |
| `repository` | No | Source repository URL |
| `homepage` | No | Project homepage |
| `keywords` | No | Search keywords |
| `authors` | No | Author list |
| `version` | No | Semantic version; `--version` overrides it |
| `exclude` | No | Patterns to drop: values containing `:` are entry IDs, everything else is a source-file glob |
| `embed` | No | Default `fs.directory` embed patterns when `--embed` is not passed |
| `exclude_meta` | No | Metadata field to values map; entries whose metadata matches are dropped |
| `metadata` | No | Arbitrary key/value metadata carried with the published module |
| `publish.profiles` | No | Which config profiles to ship in the pack (see [Publishing Profiles](#publishing-profiles)) |
| `publish.runtime` | No | Which runtime config sections to ship as pack defaults; `type: application` only |
`exclude` splits by shape rather than by a separate field. `_old/**`, `test/**` and `*.test.lua` filter source files as they are collected; `acme.http:debug_handler` disables a registry entry after entries are decoded. A `**` segment spans any number of directory segments.
`type` controls how the Hub classifies the module and can be changed in a later publish. The `--module-type` flag overrides it for one publish. When omitted, a newly created module defaults to `application` with a deprecation warning.
### Entry Definitions
Define the module's entries in `_index.yaml`:
```yaml
version: "1.0"
namespace: acme.http
entries:
- name: definition
kind: ns.definition
meta:
title: HTTP Utilities
description: Helpers for HTTP operations
readme: file://README.md
wiki:
GUIDE.md: file://docs/GUIDE.md
examples/auth.md: file://docs/auth.md
- name: client
kind: library.lua
source: file://client.lua
modules:
- http_client
- json
```
The `wiki:` map on `ns.definition` publishes documentation pages alongside the README. Keys are page paths, and values are `file://` references. Contents are inlined during packing and served by the Hub as a module wiki.
### Dependencies
Declare dependencies on other modules:
```yaml
entries:
- name: __dependency.wippy.test
kind: ns.dependency
meta:
description: Testing framework
component: wippy/test
version: ">=0.3.0"
```
Version constraints:
| Constraint | Meaning |
|------------|---------|
| `*` | Any version |
| `1.0.0` | Exact version |
| `>=1.0.0` | Minimum version |
| `^1.0.0` | Compatible (same major) |
### Requirements
Define configuration that consumers must provide:
```yaml
entries:
- name: api_endpoint
kind: ns.requirement
meta:
description: API endpoint URL
targets:
- entry: acme.http:client
path: ".meta.endpoint"
default: "https://api.example.com"
```
Targets specify where the value is injected:
- `entry` — Full entry ID to configure
- `path` — Dot path into the target entry for value injection
`default` accepts any scalar type — `default: 20` flows into a numeric target as a number, not a string. The same applies to `parameters[].value` on `ns.dependency` entries, and both accept `${env:NAME}` references, carried verbatim and resolved when the target entry is decoded.
Consumers can configure the target through an override. The `-o` flag accepts a `namespace:entry:field=value` value:
```bash
wippy run -o acme.http:client:meta.endpoint=https://custom.api.com
```
### Imports
Reference other entries:
```yaml
- name: handler
kind: function.lua
source: file://handler.lua
modules:
- json
imports:
client: acme.http:client # Same namespace
utils: acme.utils:helpers # Different namespace
base_registry: :registry # Built-in
```
In Lua:
```lua
local client = require("client")
local utils = require("utils")
```
### Contracts
Define public interfaces:
```yaml
- name: http_contract
kind: contract.definition
meta:
name: HTTP Client Contract
methods:
- name: get
description: Perform GET request
- name: post
description: Perform POST request
- name: http_contract_binding
kind: contract.binding
contracts:
- contract: acme.http:http_contract
methods:
get: acme.http:get_handler
post: acme.http:post_handler
```
#### 1. Authenticate
```bash
wippy auth login
```
#### 2. Prepare
```bash
wippy init
wippy update
wippy lint
```
#### 3. Validate
```bash
wippy publish --dry-run
```
Publish builds the pack the same way with or without `--dry-run`, so validation covers everything the real publish would produce:
- `organization` and `module` must be lowercase alphanumeric with interior hyphens, `version` must be semver, and `type` must be one of the four module types.
- `publish.runtime` is application-owned: declaring `source`, `sections`, or `vars` under it without `type: application` fails.
- Every resource declaring `meta.artifact.format` is inspected by that format. A malformed artifact fails here rather than in a consumer, and two artifacts whose outputs would land in overlapping directories are rejected.
- The `node-package` format additionally requires `package.json` to carry a semantic `version` that **equals the module version being published**, a valid package `name`, and no `preinstall`, `install`, `postinstall`, or `prepare` lifecycle script.
The last rule is the one that bites during a release: bump `version` in `wippy.yaml` and in the artifact's `package.json` together, or the publish stops.
#### 4. Publish
```bash
wippy publish --version 1.0.0
```
With release notes:
```bash
wippy publish --version 1.0.0 --release-notes "Initial release"
```
#### Publish Flags
| Flag | Description |
|------|-------------|
| `--label ` | Publish as a mutable label (e.g. `latest`, `beta`) instead of an immutable version |
| `--protected` | Mark the published version as protected (cannot be deleted or overwritten) |
| `--registry ` | Override the registry URL for this publish |
| `--config ` | Directory containing `wippy.yaml` (default: current dir) |
| `--create` | Register the module on the hub if it does not exist yet, then publish |
| `--module-visibility ` | Visibility for `--create`: `private` (default) or `public` |
| `--module-type ` | Module type: `library`, `application`, `agent`, or `plugin` (overrides `type:` in wippy.yaml) |
| `--module-display-name ` | Display name for `--create` |
#### Embed Static Files
Modules with `fs.directory` entries (static assets, templates, public files) must use `--embed` to include them in the published package. Without it, an `fs.directory` entry is packed without its directory contents.
```bash
wippy publish --version 1.0.0 --embed app:public_files
wippy publish --version 1.0.0 --embed app:assets,app:templates
```
The manifest list and `--embed` flag accept entry IDs or names matching
`fs.directory` entries. The same CLI flag is available on `wippy pack`; a CLI
selection overrides the manifest list for that invocation.
#### First Publish
On its first publish, a module is registered on the Hub as private by default, and the publish retries once. Use `--create` to register it before publishing and set its properties:
```bash
wippy publish --create --version 0.1.0 \
--module-visibility public \
--module-type library \
--module-display-name "HTTP Utils"
```
`--create` is idempotent — for an already-registered module the create step is a no-op. If your account cannot create modules in the organization, the hub returns a permission error instead of publishing.
#### Publishing to a Local Hub
Point `--registry` at a locally running Hub to publish and install without using the public registry. Plain HTTP is allowed only for local hosts: `localhost`, `127.0.0.1`, and the container aliases `host.docker.internal` (Docker Desktop or OrbStack) and `host.containers.internal` (Podman). Other hosts must use HTTPS.
```bash
wippy auth login --registry http://localhost:8080 --token wpy_xxx
wippy publish --registry http://localhost:8080 --create --version 0.1.0
```
The registry and token can also come from the `WIPPY_REGISTRY` and `WIPPY_TOKEN` environment variables. When unset, the registry defaults to `https://hub.wippy.ai`.
#### Quotas
If the organization's private-module quota is exhausted, publishing fails with a message such as `cannot publish: Private-module quota exhausted (5 of 5)...`. Make the module public or ask an organization administrator to raise the quota. Uploads and downloads retry automatically after transient network errors.
### Publishing Runtime Defaults
Applications with `type: application` can include runtime configuration defaults in their packs through `publish.runtime` in `wippy.yaml`:
```yaml
type: application
publish:
runtime:
source: .wippy.yaml # default: .wippy.yaml
sections: [security, registry, override]
vars: [public_url]
```
| Field | Description |
|-------|-------------|
| `source` | Config file the sections are read from (default: `.wippy.yaml`) |
| `sections` | Runtime config sections copied into pack metadata as defaults |
| `vars` | Explicit allowlist of variables to pack even when unreferenced |
Rules:
- Only variables referenced by the selected sections or published profiles are packed (followed transitively); everything else needs a `vars` entry.
- `${env:...}` references in exported config are rejected — publisher environment never leaks into a pack.
- The machine-local sections `boot`, `extensions`, and `workspace` cannot be exported.
- Only the main application pack provides host runtime defaults; runtime metadata in dependency packs is ignored.
At the destination, configuration precedence runs from application-pack defaults through runtime defaults, local configuration files, selected profiles, and finally CLI overrides.
### Publishing Profiles
Root application profiles are exported into the pack's `runtime.profiles` metadata. Publishing does not select or bake a profile — consumers pick one at run time with `wippy run --profile `:
```yaml
publish:
profiles:
enabled: true
source: config/profiles.yaml # default: .wippy.yaml
include: [production] # omit to publish all non-workspace profiles
```
`include: []` publishes none; an unknown name fails the publish. `workspace` sub-sections are never exported, even inside a published profile. See [Configuration](guides/configuration.md#profiles) for declaring profiles.
#### Add Dependency
```bash
wippy add acme/http-utils
wippy add acme/http-utils@1.0.0
wippy install
```
#### Configure Requirements
Override values at runtime:
```bash
wippy run -o acme.http:client:meta.endpoint=https://my.api.com
```
Or in `.wippy.yaml`:
```yaml
override:
acme.http:client:meta.endpoint: "https://my.api.com"
```
#### Import in Your Code
```yaml
## your src/_index.yaml
entries:
- name: __dependency.acme.http
kind: ns.dependency
component: acme/http-utils
version: ">=1.0.0"
- name: my_handler
kind: function.lua
source: file://handler.lua
imports:
http: acme.http:client
```
### Example Module
**wippy.yaml:**
```yaml
organization: acme
module: cache
type: library
description: In-memory caching with TTL
license: MIT
keywords:
- cache
- memory
```
**src/_index.yaml:**
```yaml
version: "1.0"
namespace: acme.cache
entries:
- name: definition
kind: ns.definition
meta:
title: Cache Module
- name: cache
kind: library.lua
source: file://cache.lua
modules:
- time
```
**src/cache.lua:**
```lua
local time = require("time")
local cache = {}
local store = {}
function cache.set(key, value, ttl)
store[key] = {
value = value,
expires = ttl and (time.now():unix() + ttl) or nil
}
end
function cache.get(key)
local entry = store[key]
if not entry then return nil end
if entry.expires and time.now():unix() > entry.expires then
store[key] = nil
return nil
end
return entry.value
end
return cache
```
Publish:
```bash
wippy init
wippy update
wippy lint
wippy publish --version 1.0.0
```
### See Also
- [CLI Reference](guides/cli.md) — Publishing commands and flags
- [Entry Kinds](guides/entry-kinds.md) — Module and dependency entries
- [Configuration](guides/configuration.md) — Runtime configuration and profiles
---
# "Compute Units"
## Compute Units
Wippy provides three ways to run code: functions, processes, and workflows. They share the same underlying machinery but differ in how long they live, where their state goes, and what happens when things fail.
### Functions
Functions run when called and return a result. Treat each call as stateless:
durable or shared state belongs in a database or store. Function pools can
reuse Lua states, so module globals and closure upvalues are worker-local and
are not a reliable cross-call store.
```lua
local funcs = require("funcs")
local result, err = funcs.call("app.math:add", 2, 3)
if err then
return nil, err
end
```
Functions execute in the caller's context. If the caller is canceled or exits, its running function calls are canceled as well.
Use functions for HTTP handlers, data transformations, and anything that should complete quickly and return a result.
### Processes
Processes are actors. They maintain state across multiple messages, run independently of whoever started them, and communicate through message passing.
```lua
local pid, err = process.spawn("app.workers:handler", "app:processes")
if err then return nil, err end
local ok, send_err = process.send(pid, "job", {task = "process_data"})
if send_err then return nil, send_err end
return ok
```
After being spawned, a process runs independently of the code that created it. Processes can monitor or link to one another and can participate in supervision trees that restart failed children.
The scheduler multiplexes thousands of processes across a worker pool. Each process yields when waiting for I/O, letting others run.
Use processes for background jobs, service daemons, and anything that needs to outlive its creator or maintain state across messages.
### Workflows
Workflows are for durable operations that must recover from interruptions. A
workflow provider such as Temporal records execution history and replays it to
rebuild state after crashes, restarts, or infrastructure changes.
```lua
-- The provider records this workflow so a worker restart can replay it.
local pid, err = process.spawn("app.orders:process", "app:temporal_worker", order_id)
if err then return nil, err end
return pid
```
Durability adds latency because workflow operations are recorded. Use workflows when recovery is more important than the lower latency of functions or processes, such as for multi-step business processes and long-running orchestration.
Wippy records supported workflow operations so they produce the same results during replay. Workflow code uses the same Lua syntax as other compute units.
### How They Compare
| | Functions | Processes | Workflows |
|---|---|---|---|
| **State** | Call-local; do not depend on worker reuse | In memory | Rebuilt from persisted history |
| **Lifetime** | Single call | Until exit or crash | Persists across restarts |
| **Communication** | Return value + messages | Message passing | Activity calls + messages |
| **Failure handling** | Caller handles | Supervision trees | Provider recovery; retries follow policy |
| **Latency** | Lowest | Low | Higher |
### Same Code, Different Behavior
Many modules adapt to their context automatically. For example, `time.sleep()`
yields in both functions and processes so other work can run; in a workflow,
the provider also records the timer so replay does not start a second timer.
---
# "Application Architecture"
## Application Architecture
A Wippy application is a **graph of registry entries** represented by source files. Code lives in entries such as `function.lua` and `process.lua`; `_index.yaml` files declare how functions, routes, services, and libraries connect. Application structure determines how that graph is divided into namespaces so it remains composable, testable, and bootable as it grows.
This page explains one way to organize that graph. For file format, naming, and `_index.yaml` placement, see [YAML & Project Structure](start/structure.md). For entry definitions, see the [Entry Kinds Guide](guides/entry-kinds.md).
### Feature Slices
A useful default is to organize by **feature** rather than file type. A slice owns one capability end to end—its database access, long-running processes, HTTP surface, and shared vocabulary—and lives under one namespace prefix:
```
src/app/jobs/ namespace: app.jobs
src/app/auth/ namespace: app.auth
src/app/billing/ namespace: app.billing
```
Feature slices keep related behavior within one folder, making a capability easier to read, test, change, or remove without tracing it across top-level `handlers/`, `models/`, and `services/` directories.
### Layers within a slice
For larger slices, separate code by **what touches the outside world**. This applies ports-and-adapters (hexagonal) architecture through **sub-namespaces**:
```
src/app/jobs/ namespace: app.jobs ← shared vocabulary
consts.lua config.lua types.lua
persist/ namespace: app.jobs.persist ← database adapters (sql)
service/ namespace: app.jobs.service ← processes, workers
api/ namespace: app.jobs.api ← http.endpoints
```
Keep imports flowing from outer layers toward inner layers:
```
api → service → persist → { consts, config, types }
```
The slice root contains shared vocabulary and does not import its own children. Children may import the root. Avoid direct imports between slices; place shared definitions in a common parent namespace such as `app.core:types`.
Namespaces organize entry IDs but do not create dependencies or injection seams by themselves. Explicit imports, kind-specific references, and ns.requirement targets create those relationships. A consistent direction keeps the resulting graph explicit. See Why this shape.
A small slice can use one `_index.yaml` for its libraries and endpoint. The important property is the **import direction**, not the number of folders.
### Shared Vocabulary
Three files commonly appear at the root of a slice. They contain definitions shared by the slice's layers:
| File | Holds | Capabilities |
|------|-------|--------------|
| `consts.lua` | State machines, enums, queue tiers, registry IDs of processes. The values that mirror your database `CHECK` constraints. | none |
| `config.lua` | Env-tunable knobs with a helper that applies a code default only when `env.get(KEY)` returns `errors.NOT_FOUND` and propagates permission or backend errors. No `env.variable` entry is required for a value to be optional. | `env` |
| `types.lua` | Entity shapes (`type Job = { ... }`) — the rows the persistence layer returns. | none |
`consts` and `types` declare **no host capabilities**; they are pure `library.lua` entries that return a table. Keeping domain vocabulary free of I/O also makes it testable without a database or process host.
Keep this vocabulary **slice-private**. Place constants and types shared across slices in a common parent namespace and import them rather than copying them.
### Capabilities by Layer
Lua entries declare non-ambient modules in `modules:` and registry-backed dependencies in `imports:`. A layered slice can keep those dependencies aligned with responsibility:
- `persist/*` declares `sql`, keeping database access in the persistence layer.
- `service/*` keeps process orchestration and service dependencies in the service layer. The `process` and `channel` globals are ambient and do not need `modules:` declarations.
- `api/*` declares modules such as `http` and imports the functions or libraries it calls.
- The root vocabulary needs no non-ambient modules or infrastructure imports.
This limits module visibility to a known layer. It is not an authorization grant: ABAC policies independently decide whether guarded operations such as `db.get` are allowed at runtime. To review code that can request a database handle, inspect `persist/`, its declared modules, and the policies attached to its execution context.
### Applications and Components
The same shape can support a single application or a published library; the difference is **who supplies its dependencies**.
An **application** is the top-level, deployable graph. It owns the concrete infrastructure — the `http.service`, the `process.host`, the database connection — under a root namespace (conventionally `app`), and wires everything together itself.
A **component** is a publishable module mounted into a host. Because it does not know the host's database or router IDs, it declares an interface of `ns.requirement` entries that the host supplies. Internally, a component can use the same layers, vocabulary, and import direction as an application slice.
These are two points on a spectrum:
- **Single app, internal slices** — slices live under `src/app/`, share the app's infrastructure directly by referencing `app:db`, `app:processes`. No requirement interface is needed because nothing external mounts them.
- **Multi-component composition** — each component is its own publishable module with an `ns.definition` and an `ns.requirement` interface, composed by a host through `ns.dependency`. The host fills each requirement (database, process host, router) once.
Choose based on whether the slice will be **consumed by a host you do not control**. Reusable components need a requirement interface; internal slices can reference application infrastructure directly. The packaging changes with reuse, while the internal layering can remain the same.
See [Building Components](guides/components.md) for the requirement/dependency mechanism, and [Dependency Management](guides/dependency-management.md) for the lock-file side.
### Why Use This Shape :id=why-this-shape
This structure supports composition, capability review, and boot-order analysis:
**Requirement targets are the injection seam.** Distinct namespaces make target IDs legible, but `ns.requirement.targets` performs the injection. A host can supply a database ID to persistence entries and a process-host ID to service entries. Directly referencing `app:db` instead couples the component to that host convention.
**One-way references keep registry transitions resolvable.** The registry extracts declared dependency paths and topologically orders changes so dependencies are created before their dependents and deleted after them. The direction `api → service → persist → root` helps keep that graph acyclic. A parent namespace is only an organizational convention; the shared entries still need explicit references.
**Modules scoped by layer have a clear boundary.** Each Lua chunk can resolve its declared imports and non-ambient modules; undeclared registry modules fail closed at module resolution. Runtime policy checks remain a separate boundary. When persistence entries alone declare `sql`, the code that can request a database handle is easier to identify and audit.
**The layering supports different test scopes.** Vocabulary can be tested without infrastructure. Persistence tests can use a database without starting workers. A whole-module **mount test** then checks the integration seams: every supervised service points to a process, every spawned ID resolves, and every requirement is filled.
### See Also
- [YAML & Project Structure](start/structure.md) — file format, naming, namespaces
- [Building Components](guides/components.md) — `ns.definition`, `ns.requirement`, mounting
- [Dependency Management](guides/dependency-management.md) — lock files, consuming modules
- [Registry](concepts/registry.md) — how entries are stored and resolved
- [Entry Kinds Guide](guides/entry-kinds.md) — every entry kind
- [Process Model](concepts/process-model.md) — services, supervision, hosts
---
# "Registry"
## Registry
The registry is Wippy's versioned store for entry points, services, resources, and other runtime definitions. Most runtime entry kinds are reconciled through event-bus transactions; internal kinds such as `registry.entry` and namespace metadata bypass event dispatch by default.
### Entries
The registry holds **entries**—typed definitions with unique IDs:
```
app.api:get_user → HTTP handler
app.workers:email_sender → Background process
app:database → Database connection
app:templates → Template set
```
Each entry has an `ID` (namespace:name format), a `kind` that determines its handler, arbitrary `meta` fields, and kind-specific `data`.
Alongside that authored content the registry keeps its own provenance for each entry: the `owner`, meaning the deployment source the entry came from, and `root`, marking a dependency declaration the deployment selected. This state is assigned by the registry, not written by the entry author, and it is kept separate from `meta` so the two can never be confused. It is read through the snapshot state API rather than the ordinary entry APIs—see [Registry module](lua/core/registry.md#snapshot-state).
For how the registry functions as an authorization layer, see the [Security Model](concepts/security-model.md).
### Kind Handlers
When a dispatched entry is submitted, its `kind` selects the registered handler. The handler validates and reconciles the corresponding runtime resource: an `http.service` entry manages an HTTP server, a `function.lua` entry manages a function pool, and a `db.sql.postgres` entry manages a connection pool. See the [Entry Kinds Guide](guides/entry-kinds.md) for available kinds and [Custom Entry Kinds](internals/kinds.md) for handler implementation.
### Live Updates
Entries can be added, updated, or removed while the system runs. For dispatched kinds, a registry transaction asks participating handlers to accept or reject each operation before commit. A rejection discards the transaction and applies the inverse transition. Related topology changes produce one new registry version.
Version history supports backward and forward transitions when history is enabled. Memory history is the default and lasts for the process lifetime; SQLite and PostgreSQL backends persist history across restarts.
YAML and JSON definition files are source manifests that the boot loader converts into entries. They are not serialized registry snapshots. See [Registry module](lua/core/registry.md) for programmatic access.
### See Also
- [YAML & Project Structure](start/structure.md) — Definition files
- [Custom Entry Kinds](internals/kinds.md) — Implement kind handlers
- [Process Model](concepts/process-model.md) — Understand process execution
---
# Security Model - Process Isolation, Capability Control, and Data Boundaries
## Security Model
Wippy's security model defines what your code can access, what it cannot, and who enforces those boundaries. It is worth reading before you build, because it works at two layers that most frameworks collapse into one: the runtime isolates each process so dangerous capabilities are simply absent, and an attribute-based policy layer governs which registry capabilities a process is allowed to use. Understanding both changes how you structure an application.
### Trust Model
Wippy's isolation layer gives a process no ambient authority. A fresh Lua or WASM process cannot touch the file system, the network, the host OS, or other processes' memory, because those capabilities are not present in its environment. Capabilities arrive only through the registry: functions, tools, connections, and configuration the process is explicitly granted.
On top of that, access to registry capabilities is governed by attribute-based access control (ABAC). Every guarded operation is checked against the current actor's security scope, a set of policies that allow or deny an action on a resource, optionally conditioned on actor and resource metadata. This is declarative: you define policies in configuration, not in application code.
When a process runs with both an actor and a scope, access is deny-by-default: a request is allowed only if a policy explicitly permits it and none denies it. **Strict mode** governs the incomplete case, when no actor or scope is established. It is **on by default**, so an incomplete context is denied; setting `security.strict_mode: false` in the runtime config opts into the permissive behavior instead. The consequence to plan for is that a process with no declared security context fails every check under the default — give such a process a `security:` block on its entry, or start it through a path that supplies one. Combined with least-privilege policies, this gives you fail-closed authorization on top of deny-by-absence isolation. See the [Security reference](system/security.md) for policy syntax, evaluation rules, and the shape of the `security:` block.
### Process Isolation
Every unit of execution in Wippy runs in an isolated process with its own embedded interpreter (Lua or WASM).
**What a process has:** its own memory space (a baseline overhead of ~13 KB for Lua). A scoped view of the registry. An actor identity and a security scope. A supervised lifecycle with crash recovery and restart limits.
**What a process does not have:** access to the file system (except through registry-controlled filesystem entries). Access to the network (except through granted HTTP client or tool modules). Access to other processes' memory. Access to the Go runtime hosting it. Access to environment variables (except through granted environment entries).
**How isolation is enforced:** each Lua process starts from a minimal standard library. File I/O, OS process access, dynamic code loading, and networking are never loaded, so they are not present in the environment, and the process cannot restore what does not exist. Module loading is restricted: `require` resolves only the modules and registry entries the process is explicitly granted, with no file system search path. WASM processes achieve equivalent isolation through WASI: only the host functions and mounted filesystem entries configured for that entry are reachable.
This is not sandboxing via runtime permissions (like seccomp or AppArmor). It is sandboxing via absence. Dangerous capabilities are never loaded, so they cannot be exploited, bypassed, or escalated.
### Capability Control
The registry is Wippy's capability store, and security policies are its authorization layer.
**Every capability is a registry entry.** Functions, tools, agent definitions, database connections, environment references, configuration values, and scheduled tasks are all registry entries with a declared kind, schema, and metadata. Entries are validated by their kind handler when registered.
**Entry IDs are namespaced.** An ID has the form `namespace:name` with a single colon, and namespaces are hierarchical via dot-separated segments, for example `tenant_acme.tools:read` (namespace `tenant_acme.tools`, name `read`). Policies match actions and resources, and resource patterns can target a namespace prefix, so a single rule can cover an entire namespace.
**Policies decide access.** Each capability access (a registry lookup, a function call, a database handle, a file open) is checked against the actor's scope. A policy declares the actions and resources it covers, an allow or deny effect, and optional conditions on actor and resource metadata. Evaluation happens per access, not once at startup: if any policy denies, access is denied; if at least one allows and none denies, it is allowed; if no policy matches, access is denied. (When the context has no actor or scope at all, that incomplete case is resolved by strict mode rather than by policy evaluation.)
**A context is declared, not inherited from thin air.** Functions inherit the caller's actor and scope. A spawned process inherits them too: its frame is forked from the spawner's, and the `security:` block on its own entry then modifies that inherited context — an `actor` it names replaces the inherited actor, and the policies and policy groups it lists by registry ID are merged into the inherited scope. Resolution is atomic — if any named policy or group is missing, the spawn fails rather than proceeding with a partial scope. A CLI command can additionally declare `meta.command.security`, applied only on the trusted launch path where the operator started the command themselves.
**Tool arguments are schema-shaped.** A tool declares a JSON Schema for its inputs. That schema is given to the model so it generates conforming arguments, and access to the tool is policy-checked before the call runs.
### Data Boundaries
**Database connections are registry entries.** A process does not assemble its own connection string. It requests a connection by registry ID, and that request is policy-checked before a handle is returned. A process whose policies do not grant Tenant B's database entry cannot obtain a handle to it.
**LLM API keys live in the environment system.** Keys for Claude, GPT, and other providers are read from the environment system (for example OS environment variables exposed through an `env.storage.os` entry, referenced by `env.variable` entries whose reads are policy-checked via the `env.get` action). The provider reads them internally; they are not passed in process arguments or returned to calling code.
**File and blob storage follow the same model.** A process reads or writes through filesystem or cloud-storage registry entries, each access policy-checked. WASM processes access files only through filesystem entries explicitly mounted for that entry.
### Agent Security
Agents are LLM-powered processes with tool use. They make decisions at runtime that your code does not directly control, so their boundaries matter. Wippy handles this through the same registry and policy mechanisms as any other process.
**Tool access.** An agent can only invoke tools listed in its definition, and each tool execution runs through `funcs.call`, which is policy-checked. A denied call fails before the tool function runs. An agent designed to read customer data but not delete it either has no delete tool in its definition or is denied that action by policy.
**External and MCP tools.** Wippy can consume external tools and expose its own over the Model Context Protocol. Consumed tools run through the same function-call path and policy checks as native tools. Tools Wippy exposes to external MCP clients are gated by scoped, revocable access tokens that limit which actions a client may perform.
**Structured output.** The LLM module can request schema-constrained (structured) output using the provider's native structured-output support, so an agent's output can be held to a declared shape.
**Observability.** With OpenTelemetry enabled, LLM provider calls and tool invocations are traced, and token usage is recorded through the usage-tracker contract. This gives you an audit trail of what an agent called and what it spent. See [Observability](guides/observability.md).
**Self-modification boundaries.** An agent permitted to create tools in one namespace can be denied write access to its own definition in another. Registry writes are policy-checked actions, so a deny policy on the agent's own namespace prevents it from editing itself or granting itself new access.
### Multi-Tenant Enforcement
For deployments where multiple customers share a single Wippy instance, isolation is enforced by policy evaluation before any operation runs, not by application code checking tenant IDs.
**Tenant isolation is policy-enforced.** Give each tenant an actor and a scope whose policies cover only that tenant's namespaces. With strict mode on, a tenant's process is denied access to resources outside its scope before its code runs. Effective isolation depends on writing those per-tenant policies; the runtime enforces them, but it does not infer tenancy for you.
**Cross-tenant access is explicit.** A capability shared across tenants lives in a shared namespace that each tenant's policies allow. Sharing is opt-in per namespace.
**Concurrency is bounded at the host.** Process hosts bound concurrency through worker pools. Process groups (`pg.scope`) provide isolated, cluster-wide membership and broadcast namespaces and can cap group and member counts. Per-tenant CPU or memory ceilings are not a built-in runtime feature; enforce those at the infrastructure layer.
A dedicated Multi-Tenant Architecture guide is planned.
### Scope and Limitations
Wippy's security model covers process isolation, capability control, and data boundaries. The following are outside the runtime's scope and remain your infrastructure's responsibility.
**Data encryption at rest.** Database, disk, and blob-storage encryption are handled by the underlying infrastructure (PostgreSQL TDE, disk encryption, and similar). Wippy assumes the storage layer handles encryption.
**Network-level isolation.** Process isolation happens at the application layer. Network segmentation between Wippy and its dependencies (database, LLM APIs, external services) is handled by infrastructure: VPCs, security groups, firewalls.
**Identity management.** Authentication (verifying who a user is) is handled by your auth layer. Wippy's security model starts after authentication: it controls what an authenticated user's processes can do, not who the user is. Tokens that carry an actor and scope can be issued and validated through a token store.
**Infrastructure audit logs.** Wippy's tracing covers process-level operations: function calls, tool calls, process activity. Infrastructure-level access (SSH to the server, database admin operations) should be audited by infrastructure tools.
### Common Questions
**Can one tenant's agent access another tenant's data?**
Not when each tenant's resources are scoped by policy. With per-tenant policies and strict mode, the runtime denies access to resources outside the tenant's scope before the agent's code runs.
**Can an agent escalate its own permissions?**
Only if its policies allow writing to its own definition. Registry writes are policy-checked, so a deny policy on the agent's own namespace prevents self-modification. An agent that can create tools in one namespace cannot grant itself access to namespaces its scope does not already cover.
**How do I see what an agent did?**
With OpenTelemetry enabled, LLM and tool calls are traced, and token usage is recorded through the usage-tracker contract. See [Observability](guides/observability.md).
**What happens if an agent behaves unexpectedly?**
It is contained by the sandbox: no file system, no network, no OS, no access to other processes beyond what it was granted. It can only call tools in its definition that policy permits, and those calls are logged.
**Is tenant isolation enforced by my code or by the runtime?**
By the runtime. The policy engine evaluates each access before the operation runs. Your job is to write the per-tenant policies; the runtime enforces them.
**How are external MCP tools secured?**
Tools consumed over MCP run through the same function-call path and policy checks as native tools. Tools Wippy exposes to external MCP clients are gated by scoped, revocable access tokens. Connecting an MCP service does not bypass the security model.
### Security Reference
| Concern | Wippy's approach |
|---------|------------------|
| Process isolation | Separate interpreter per process (Lua or WASM), no shared memory |
| Default access | Unmatched policies deny when both an actor and a scope are set; strict mode, on by default, denies when no actor or scope is established |
| Context declaration | `security:` block on the entry (actor, policies, groups); resolution is atomic and fail-closed |
| Supply chain | Module packs verified by digest at install and at boot; a mismatch refuses the module |
| Node-to-node trust | Mutually authenticated internode mesh; ed25519 identity per node, explicit trusted-peer map |
| Workflow propagation | Actor and scope carried to Temporal as a signed, audience-bound header; verification failure fails the execution |
| Capability control | Registry entries governed by attribute-based security policies (actor, scope, action, resource) |
| Data boundaries | Connections and storage are registry entries; each access is policy-checked by entry ID |
| API key management | Stored in the environment system, read internally by providers, not exposed to process code |
| Agent tool control | Tools limited to the agent's definition; each call checked via `funcs.call` policy |
| External tools (MCP) | Same function-call path and policy checks; exposed tools gated by scoped tokens |
| Agent audit trail | OpenTelemetry tracing (when enabled) plus usage-tracker records |
| Multi-tenant isolation | Per-tenant policies and scopes evaluated by the runtime before each operation |
| Concurrency limits | Bounded by host worker pools; no per-tenant CPU/memory ceilings built in |
| Self-modification | Deny policies on registry-write actions prevent agents from editing their own definitions |
### See Also
- [Security reference](system/security.md) - Policies, scopes, actors, token stores, and the `security:` block
- [Dependency Management](guides/dependency-management.md#integrity-verification) - Module digest verification
- [Cluster](guides/cluster.md#internode-identity) - Internode identity and peer trust
- [Temporal Workflows](temporal/workflows.md#security-context) - Signed context propagation
- [Registry](concepts/registry.md) - The capability store
- [Process Model](concepts/process-model.md) - Process isolation and lifecycle
- [Agents](framework/agents.md) - Agent definitions and tool use
---
# "Process Model"
## Process Model
Wippy executes code in isolated processes: lightweight state machines that communicate through messages rather than shared memory. This actor model gives each process its own state and lifecycle.
This page explains the lifecycle and isolation model. Use the [Process Management reference](../lua/core/process.md) for spawn, messaging, monitoring, registry, and upgrade APIs. See [Process Host and Services](../system/process-host.md) for runtime-managed service fields.
### State Machine Execution
Each process initializes, advances through execution, yields on blocking operations, and closes when complete. The scheduler multiplexes processes across a worker pool and runs other work while a process waits for I/O.
Processes support multiple concurrent yields, allowing code to start several asynchronous operations and wait for any or all of them without spawning additional processes.
```mermaid
flowchart LR
Ready --> Running
Running --> Blocked
Running --> Idle
Blocked --> Running
Idle --> Running
Running --> Complete
```
Processes are not limited to Lua. The runtime also supports WebAssembly modules through the `process.wasm` kind, and its process architecture can support other state-machine implementations.
Processes are lightweight but not free. Each process carries a small baseline cost for its state, inbox, and scheduler bookkeeping, and dynamic allocations grow that footprint during execution.
### Process Hosts
Wippy can run multiple process hosts within one runtime, each with its own capabilities and security boundaries. Privileged system processes can run in a host separate from hosts that execute user sessions.
Some hosts are specialized. The Terminal host, for example, uses one scheduler
worker and supplies terminal I/O context to accepted processes; it does not
enforce a one-process lifetime limit. Separate hosts allow one deployment to
run processes with different trust levels.
### Security Model
Each process executes under an actor identity and security policy. This is typically the user who initiated the call, while system processes use a system actor with different privileges.
Access control applies at multiple levels. Security policy can restrict individual process operations and message delivery between hosts. The policy attached to the current actor determines which operations are permitted.
For the security implications of process isolation, see the [Security Model](concepts/security-model.md).
### Spawning Processes
Create background processes with `process.spawn()`:
```lua
local pid, err = process.spawn("app.workers:handler", "app:processes", arg1, arg2)
if err then return nil, err end
return pid
```
The first argument is the registry entry, the second is the process host, and remaining arguments pass to the process.
Spawn variants control lifecycle relationships:
| Function | Behavior |
|----------|----------|
| `spawn` | Start an independent process |
| `spawn_monitored` | Receive EXIT events when child exits |
| `spawn_linked` | Abnormal exit propagates in either direction; with `trap_links: true`, the peer receives `LINK_DOWN` instead of failing |
### Message Passing
Processes communicate through messages rather than shared memory:
```lua
local ok, err = process.send(target_pid, "topic", payload)
if err then return nil, err end
return ok
```
Messages from the same sender arrive in order. Messages from different senders may interleave. Delivery is fire-and-forget—use request-response patterns when you need confirmation.
Processes can register in a local name registry and be addressed by name instead of PID (e.g., `session_manager`). Names can also be registered cluster-wide for cross-node addressing via `process.registry` using EVENTUAL (gossip-based), CONSISTENT, or STRONG (both Raft-backed) scopes.
### Supervision
Any process can supervise other processes by monitoring them. A supervisor starts monitored children, watches for EXIT events, and decides whether to restart them after failure.
```lua
local worker, spawn_err = process.spawn_monitored("app.workers:handler", "app:processes")
if spawn_err then return nil, spawn_err end
local event, open = process.events():receive()
if not open then return nil, errors.new("process event channel closed") end
if event.kind == process.event.EXIT and event.result.error then
local replacement, restart_err = process.spawn_monitored("app.workers:handler", "app:processes")
if restart_err then return nil, restart_err end
worker = replacement
end
```
At the runtime level, services can start and supervise long-running processes. Define a `process.service` entry to have the runtime manage a process:
```yaml
- name: worker.service
kind: process.service
process: app.workers:handler
host: app:processes
lifecycle:
auto_start: true
restart:
max_attempts: 5
initial_delay: 1s
```
The service starts automatically and integrates with the runtime's lifecycle management. At the pinned runtime, the initial failed start counts toward `max_attempts`, so `5` permits at most four follow-up starts. Each retry waits for `initial_delay` with jitter; the delay does not increase between attempts.
### Process Upgrading
Running processes can upgrade their code without losing identity. Call `process.upgrade()` to swap to a new definition while preserving PID, mailbox, and supervision relationships:
```lua
process.upgrade("app.workers:v2", current_state)
```
The first argument is the new registry entry (or nil to reload the current definition). Additional arguments pass to the new version, letting you carry state across the upgrade. The process resumes execution with the new code immediately.
The runtime caches compiled prototypes to avoid repeated compilation. If an upgrade fails, the process crashes and normal supervision behavior applies; a monitoring parent can restart it or escalate the failure.
### Scheduling
The actor scheduler uses work-stealing across CPU cores. Each worker has a local queue for cache locality, plus a global queue for distributing work. Processes yield on blocking operations so other processes can run on the worker pool.
---
# "Cluster"
## Cluster
A single Wippy node is a complete runtime. A **cluster** connects several nodes so processes can use cluster-wide names, route messages across nodes, and coordinate through locks, groups, and a shared consensus core.
Clustering is opt-in (`cluster.enabled`). This page explains the model your code sees; for topology, configuration, and operations see the [Cluster Guide](guides/cluster.md).
### Cluster Model
Nodes discover one another through **gossip** (SWIM). A node joins through a seed, after which membership and failure information converge without a central coordinator. A bounded **Raft** core provides linearizable consensus through a dynamically reconciled voter set, while other nodes participate through gossip.
The application-facing model has three parts: **names**, **routing**, and **coordination primitives**.
### Naming
A process is normally addressed by its PID. In a cluster, it can also be registered under a **name** and reached by that name from other nodes. The selected **scope** determines the consistency guarantee and coordination cost:
| Scope | Visibility | Guarantee | Use it for |
|-------|------------|-----------|------------|
| **Local** | this node | instant, no coordination | node-local helpers |
| **Eventual** | cluster-wide | converges after gossip; conflicts resolve and notify the loser | service, group, and bounded presence names |
| **Consistent** | cluster-wide | linearizable singleton via Raft | the standard cluster-wide named service |
| **Strong** | cluster-wide | Consistent, plus every live node acknowledges before the name is active | control-plane singletons and locks |
The scopes are ordered as `Local < Eventual < Consistent < Strong` by consistency and coordination cost. Select the least costly scope that meets the required guarantee. Names are registered through [`process.registry`](lua/core/process.md). Local names are removed when the process exits; Consistent and Strong names are also reaped on process exit or node departure. Eventual names are removed explicitly or when their origin node leaves, not automatically when only the owning process exits.
### Routing
Routing connects a registered name to the process that owns it:
- **Reads are local.** Every node resolves a name from its own replica or gossip-disseminated cache — no network round-trip to look up a name. This keeps resolution fast and keeps working during partitions.
- **Resolution has a fixed order.** A name is resolved across the planes most-authoritative first — Consistent and Strong (Raft), then Eventual (gossip), then Local — so a cluster-wide name shadows a local one of the same string.
- **Writes route to the authority.** A Consistent or Strong registration goes through the Raft leader; a node that isn't the leader forwards the write and waits for the result. Once committed, the active binding is disseminated over gossip so every node — including those not in the Raft core — can resolve the name locally afterward.
- **Messaging routes by PID.** When you `process.send` to a name, it resolves to a PID and the relay delivers the message to the owning node. Your code addresses a process the same way whether it lives on this node or another — location is transparent.
Applications register and resolve names without addressing the authority node directly. After resolution, messages route to the node that owns the target PID.
### Primitives
Clustering exposes a small set of coordination building blocks:
- **Membership and identity** — the live set of nodes and this node's identity and role. Use it to discover peers or shard work. See [`system.cluster`](lua/system/system.md) and [`system.node`](lua/system/system.md).
- **Consensus state** — the Raft leader, term, and this node's role, for diagnostics and leader-aware logic. See [`system.raft`](lua/system/system.md).
- **Cluster-wide names** — register and resolve processes by name and scope, the foundation everything else builds on. See [`process.registry`](lua/core/process.md).
- **Distributed locks** — cluster-wide mutual exclusion with at most one holder, released automatically if the holder dies. See [`system.lock`](lua/system/system.md).
- **Process groups** — join named groups and broadcast to every member across all nodes, Erlang-style. See [Process Groups](lua/core/pg.md).
These primitives share membership and routing infrastructure. Consistent and Strong names and distributed locks use the Raft core. Process groups use gossip membership to discover peers, send changes over the relay, and periodically exchange full state for convergence.
### See Also
- [Cluster Guide](guides/cluster.md) — Topology, configuration, and operations
- [Process Management](lua/core/process.md) — Spawning, messaging, and the name registry
- [Process Groups](lua/core/pg.md) — Named groups and broadcast
- [System](lua/system/system.md) — `system.cluster`, `system.node`, `system.raft`, `system.lock`
- [Process Model](concepts/process-model.md) — Processes, PIDs, and messaging
---
# "Functions"
## Functions
Functions are call-and-return entry points. A function inherits its caller's
context and is canceled when the caller is canceled. Pools can reuse Lua states,
so module globals and closure upvalues may survive on one worker but are not
shared consistently across calls. Store durable or shared state outside the
function. Use functions for HTTP handlers, API endpoints, and other operations
that complete within a request lifecycle.
### Calling Functions
Call functions synchronously with `funcs.call()`:
```lua
local funcs = require("funcs")
local result, err = funcs.call("app.api:get_user", user_id)
if err then return nil, err end
return result
```
For non-blocking execution, use `funcs.async()`:
```lua
local future, err = funcs.async("app.process:analyze", data)
if err then
return nil, err
end
local ch = future:response()
local payload, open = ch:receive()
if not open then
return nil, "future response channel closed"
end
local result, err = payload:data()
if err then
return nil, err
end
```
See the [funcs module](lua/core/funcs.md) for function invocation and executor options.
### Context Propagation
Each call creates a frame with its own context scope. Child functions inherit parent context without explicit passing:
```lua
local ctx = require("ctx")
local trace_id = ctx.get("trace_id")
local user_id = ctx.get("user_id")
```
Add context when calling:
```lua
local funcs = require("funcs")
local exec, err = funcs.new():with_context({trace_id = "abc-123"})
if err then return nil, err end
local result, err = exec:call("app.api:process", data)
if err then return nil, err end
return result
```
Security context propagates the same way. Called functions see the caller's actor and can check permissions. See the [security module](lua/security/security.md) for access control APIs.
### Registry Definition
At the registry level, a function entry looks like this:
```yaml
- name: get_user
kind: function.lua
source: file://handlers/user.lua
method: get
pool:
type: lazy
max_size: 16
```
Functions can be invoked by other runtime components—HTTP handlers, queue consumers, scheduled jobs—and are subject to permission checks based on the caller's security context.
### Pools
Functions run on pools that manage execution. The pool type determines scaling behavior.
**Inline** runs in the caller's goroutine without a worker pool. It is used for embedded contexts.
**Static** maintains a fixed number of workers. Requests queue when all workers are busy, which keeps worker concurrency fixed.
```yaml
pool:
type: static
size: 8
buffer: 512
```
**Lazy** starts without workers and creates them on demand. Idle workers are removed after a timeout.
```yaml
pool:
type: lazy
max_size: 32
```
**Adaptive** adjusts the worker count based on measured throughput and current load.
```yaml
pool:
type: adaptive
max_size: 256
```
If you don't specify a pool type, the runtime selects one based on your configuration. Set `workers` for static, `max_size` for lazy, or explicitly set `type` for full control. With neither set, the pool is lazy with a maximum of 16 workers.
### Interceptors
Function calls pass through an interceptor chain. Interceptors can handle cross-cutting concerns separately from the function implementation.
```yaml
- name: my_function
kind: function.lua
source: file://handler.lua
method: main
meta:
options:
retry:
max_attempts: 3
initial_delay: 100
backoff_factor: 2.0
```
Built-in interceptors include retry with exponential backoff. Runtime
integrations written in Go can register additional interceptors for logging,
metrics, tracing, authorization, circuit breaking, or request transformation;
Lua application entries can configure only interceptors installed by the
runtime.
The chain runs before and after each call. Each interceptor can modify the request, short-circuit execution, or wrap the response.
### Contracts
Functions can expose their input/output schemas as contracts. Contracts define method signatures that enable runtime validation and documentation generation.
```lua
local contract = require("contract")
local sender, err = contract.get("app.email:sender")
if err then return nil, err end
local email, err = sender:open("app.email:sender_impl")
if err then return nil, err end
local result, err = email:send({to = "user@example.com", subject = "Hello"})
if err then return nil, err end
return result
```
Contracts allow callers to use an interface while selecting an implementation separately. This supports testing, multi-tenant deployments, and gradual migrations.
### Functions vs Processes
Functions inherit the caller's context and lifecycle. When the caller is canceled, its function calls are canceled as well. This suits execution within HTTP handlers and queue consumers.
Processes run independently with host context. They outlive their creator and communicate through messages. Use processes for background work; use functions for request-scoped operations.
---
# "Workflows"
## Workflows
Workflows persist the state of long-running operations so execution can recover after crashes and restarts. They suit processes such as payments, order fulfillment, and multi-step approvals.
### Why Use Workflows
Functions keep in-flight state in memory, while workflows persist execution state:
| Aspect | Functions | Workflows |
|--------|-----------|-----------|
| State | Call-local | Rebuilt from persisted history |
| Worker crash | In-flight call fails | Replays from recorded history |
| Duration | Seconds to minutes | Hours to months |
| Application failure | Returned to caller | Ends or retries according to provider policy |
### How Workflows Work
Workflow code looks like regular Lua code:
```lua
local funcs = require("funcs")
local time = require("time")
local result, err = funcs.call("app.api:charge_card", payment)
if err then return nil, err end
time.sleep("24h")
local status, err = funcs.call("app.api:check_status", result.id)
if err then return nil, err end
if status == "failed" then
local _, refund_err = funcs.call("app.api:refund", result.id)
if refund_err then return nil, refund_err end
end
return status
```
The workflow engine intercepts calls and records their results. After a crash, it replays execution from the recorded history.
Inside a workflow, each `funcs.call()` target runs as a Temporal activity. A
target `function.*` entry must register with a worker through
`meta.temporal.activity.worker`; unregistered entries are not available to the
workflow. A `process.*` activity target additionally needs
`meta.options.default_host` (or the legacy `meta.default_host`) so it is
registered in the function registry used by the Temporal worker. See
[Activities](../temporal/activities.md) for the function activity example and
activity options.
Workflow authors must still write deterministic code. Wippy limits workflow
modules to those classified as Deterministic or Workflow and supplies
replay-safe implementations for supported operations. funcs.call()
runs a recorded activity, time.sleep() uses a workflow timer,
uuid.v4() records a side effect, and time.now() reads the
workflow's deterministic time reference.
#### Saga Pattern
Compensate on failure:
```lua
local funcs = require("funcs")
local inventory, err = funcs.call("app.inventory:reserve", items)
if err then return nil, err end
local payment, err = funcs.call("app.payments:charge", amount)
if err then
local _, compensation_err = funcs.call("app.inventory:release", inventory.id)
return nil, compensation_err or err
end
local shipping, err = funcs.call("app.shipping:create", order)
if err then
local _, refund_err = funcs.call("app.payments:refund", payment.id)
local _, release_err = funcs.call("app.inventory:release", inventory.id)
return nil, refund_err or release_err or err
end
return {inventory = inventory, payment = payment, shipping = shipping}
```
#### Waiting for Signals
Wait for external events (approval decisions, webhooks, user actions):
```lua
local funcs = require("funcs")
local _, err = funcs.call("app.approvals:submit", request)
if err then return nil, err end
local inbox = process.inbox()
local msg, open = inbox:receive() -- blocks until signal arrives
if not open then return nil, errors.new("workflow inbox closed") end
local decision, payload_err = msg:payload():data()
if payload_err then return nil, payload_err end
if decision.approved then
return funcs.call("app.orders:fulfill", request.order_id)
else
return funcs.call("app.notifications:send_rejection", request)
end
```
### Choosing a Compute Model
| Use Case | Choose |
|----------|--------|
| HTTP request handling | Functions |
| Data transformation | Functions |
| Background jobs | Processes |
| User session state | Processes |
| Real-time messaging | Processes |
| Payment processing | Workflows |
| Order fulfillment | Workflows |
| Multi-day approvals | Workflows |
### Starting Workflows
Workflows use `process.spawn()` with a workflow host:
```lua
-- Spawn workflow on temporal worker
local pid, err = process.spawn("app.workflows:order_processor", "app:temporal_worker", order_data)
if err then return nil, err end
-- Send signals to workflow
local ok, err = process.send(pid, "update", {status = "approved"})
if err then return nil, err end
return ok
```
The caller uses the same spawn API. The host determines whether the entry runs
on a `temporal.worker` or a `process.host`. Persisted history and replay apply
only to the Temporal-hosted path. A workflow entry run through an ordinary
process host has in-memory process semantics and does not gain Temporal
durability.
When a workflow spawns children via process.spawn(), they become child workflows on the same provider, maintaining durability guarantees.
### Failure and Supervision
Processes can run as supervised services using `process.service`:
```yaml
## Process definition
- name: session_handler
kind: process.lua
source: file://session_handler.lua
method: main
## Supervised service wrapping the process
- name: session_manager
kind: process.service
process: app:session_handler
host: app:processes
lifecycle:
auto_start: true
restart:
max_attempts: 10
```
Workflows do not use process supervision trees. The workflow provider manages
persistence and recovery; application-level retries follow the configured
workflow and activity policies.
### Configuration
Workflow definition (spawned dynamically):
```yaml
- name: order_processor
kind: workflow.lua
source: file://order_processor.lua
method: main
meta:
temporal:
workflow:
worker: app:temporal_worker
modules:
- funcs
- time
```
Every function or process invoked through `funcs.call()` also declares the
activity worker. For example:
```yaml
- name: charge_card
kind: function.lua
source: file://charge_card.lua
method: main
meta:
temporal:
activity:
worker: app:temporal_worker
```
Workflow provider:
```yaml
- name: temporal_worker
kind: temporal.worker
client: app:temporal_client
task_queue: "orders"
lifecycle:
auto_start: true
```
See [Temporal](https://temporal.io) for production workflow infrastructure.
### See Also
- [Functions](concepts/functions.md) — Request-scoped calls
- [Process Model](concepts/process-model.md) — Stateful background work
- [Supervision](guides/supervision.md) — Process restart policies
---
# "Lua Runtime"
## Lua Runtime
Lua is Wippy's primary runtime for I/O-bound work and business logic. Code runs in isolated processes that communicate through message passing rather than shared memory.
This page is a conceptual overview. Its code blocks are isolated reference snippets; names such as `inbox`, `events`, and `handle_message` stand for values or callbacks supplied by the surrounding application.
For the design tradeoffs behind Lua and its relationship to WebAssembly, see [Why Wippy Uses Lua](why-lua.md).
### Processes
Lua code runs inside **processes**: isolated execution contexts managed by the scheduler. Each process:
- has its own memory space;
- yields during blocking operations such as I/O and channel access;
- can be monitored and supervised; and
- can run alongside thousands of other processes on one machine.
```lua
local pid, err = process.spawn("app.workers:handler", "app:processes")
if err then
return nil, err
end
local sent, send_err = process.send(pid, "task", {data = "work"})
if send_err then
return nil, send_err
end
```
Executable Lua entries receive `process` as an ambient global. It can also be loaded with `require("process")` without adding it to the entry's `modules` list. See [Process Management](lua/core/process.md) for spawning, linking, and supervision.
### Channels
Channels provide communication between concurrent tasks:
```lua
local sync_ch = channel.new() -- unbuffered
local buffered = channel.new(10)
buffered:send("work") -- completes while buffer space is available
local val, ok = buffered:receive() -- val is "work" and ok is true
```
See [Channels](lua/core/channel.md) for select and patterns.
### Coroutines
Within a process, use lightweight coroutines for concurrent work:
```lua
coroutine.spawn(function()
local data = fetch_data()
ch:send(data)
end)
do_other_work() -- continues immediately
```
The scheduler manages spawned coroutines, so callers do not manually yield or resume them.
### Select
Use `channel.select` to wait for multiple event sources:
```lua
local r = channel.select {
inbox:case_receive(),
events:case_receive(),
timeout:case_receive()
}
if r.channel == timeout then
-- timed out
elseif r.channel == events then
handle_event(r.value)
else
handle_message(r.value)
end
```
### Globals
The following globals are available without `require` and do not need to be listed in `modules:`:
- `channel` - Go-style channels
- `payload` - the entry's input payload
- `process` - process spawning, messaging, monitoring, and lifecycle operations
- `print`, `subscribe`, `unsubscribe` - logging and pub/sub
- `os`, `table`, `math`, `string`, `coroutine`, `errors` - standard libraries
### Modules
Built-in runtime modules that are not ambient are loaded with `require()` and must appear in the entry's `modules:` allowlist. Executable entries receive `process` as an ambient global; `require("process")` is also allowed and does not require a `modules:` declaration.
```lua
local process = require("process")
local json = require("json")
local sql = require("sql")
local http = require("http_client")
```
Available modules depend on entry configuration. See [Entry Definitions](lua/entries.md).
Registry libraries use the same `require("alias")` syntax but are declared separately in the entry's `imports:` map.
### Language and Library Support
Wippy uses Lua 5.3 syntax with a [gradual type system](lua/types.md) inspired by Luau. Types are first-class runtime values that can be used for validation, passed as arguments, and inspected at runtime.
External Lua libraries (LuaRocks, etc.) are not supported. The runtime provides its own module system with built-in extensions for I/O, networking, and system integration.
For custom extensions, see [Modules](internals/modules.md) in the internals documentation.
### Error Handling
Functions commonly return `result, error` pairs:
```lua
local data, err = json.decode(input)
if err then
return nil, errors.wrap(err, "decode failed")
end
```
This snippet assumes `json` is enabled in the entry's `modules` list and `input` contains the string to decode. See [Error Handling](lua/core/errors.md) for patterns.
### What's Next
- [Entry Definitions](lua/entries.md) - Configure entry points
- [Channels](lua/core/channel.md) - Channel patterns
- [Process Management](lua/core/process.md) - Spawning and supervision
- [Functions](lua/core/funcs.md) - Cross-process calls
---
# "Why Wippy Uses Lua"
## Why Wippy Uses Lua
Wippy uses Lua as its primary runtime language because it fits the platform's process-isolation and embedding requirements. This page explains that design choice and its tradeoffs; it is not a general ranking of programming languages.
This is a conceptual design note rather than a runnable tutorial. It describes runtime properties and points to the reference pages that define the concrete APIs.
### Runtime Requirements
Wippy runs user-defined logic in isolated processes. Each process has its own memory and receives only the capabilities exposed by the runtime. Because many processes can run concurrently, the embedded language must support:
- **Low per-process overhead.** Memory use must remain practical as process counts grow.
- **Capability isolation.** The runtime must control the modules, functions, and system operations available to each process.
- **In-process embedding.** Wippy's Go core must be able to create, configure, and stop a language environment for each process.
- **Controlled module loading.** Modules must come from the runtime's allowlist or declared registry imports rather than arbitrary file-system paths.
- **A small language surface.** Application code should remain readable and straightforward to generate, review, and lint.
#### Python
Python offers a large application and data ecosystem, but its interpreter, import model, and package assumptions do not match Wippy's per-process embedding and capability model. Python services can still integrate with Wippy over explicit service boundaries.
#### JavaScript
JavaScript runtimes offer several embedding options. Their module and package ecosystems, however, require a separate integration layer to provide the registry-scoped loading model Wippy uses. Wippy chose Lua's smaller host-controlled runtime surface for application code.
#### Go
Go is used for Wippy's core runtime. Compiled Go code and plugins do not provide the same isolated, per-process embedded environment required for user-defined application logic.
#### WebAssembly
WebAssembly fills a complementary role rather than replacing Lua as the primary authoring language. Its division of responsibilities is described in [Lua and WebAssembly](#lua-and-webassembly).
#### Host-Controlled Embedding
Lua is designed to run inside a host application. Wippy creates an environment for each process, connects it to the scheduler and registry, and controls its globals and module loader. `require` reads only modules already installed in that environment: the always-available base modules and standard libraries, the executable entry's ambient `process` module, built-in runtime modules allowed by `modules:`, and registry libraries declared through `imports:`. It does not search file-system paths or install packages from the network. Different entries can therefore receive different module sets without application-level loading rules.
#### Language Surface
Lua has a compact syntax and a small standard environment. Wippy adds type annotations and linting so code can be checked incrementally without changing the underlying execution model.
#### Cooperative Scheduling
Lua coroutines map to Wippy's cooperative scheduling model. A process can yield during channel or I/O operations while the scheduler runs other work.
### Tradeoffs
Lua does not provide an in-process package ecosystem comparable to pip or npm. Wippy supplies built-in runtime modules through an allowlist and application libraries through registry imports rather than installing packages from the network. Workloads that depend on large external libraries can run as services or as WebAssembly components.
Lua may also be unfamiliar to developers coming from other languages. The syntax is compact, but teams still need conventions, review, and linting for production code.
### Lua and WebAssembly
Wippy provides two complementary runtimes:
- **Lua** is the primary runtime for application logic, tools, and agents.
- **WebAssembly** runs compiled workloads and existing code that can target WASM.
Lua and WASM process entries use Wippy's process model; Lua and WASM functions are exposed through registered function entries. Both integrations are configured through the registry and runtime security policies. Lua code can call registered WASM functions, and WASM processes can call registered Lua functions.
### See Also
- [Lua Runtime Overview](lua/overview.md) - The Lua runtime and its modules
- [Types](lua/types.md) - Type annotations, generics, and unions
- [Linter](guides/linter.md) - Static analysis for Lua
- [WASM Runtime](wasm/overview.md) - Running compiled code in the sandbox
---
# "Type System"
## Type System
> **Experimental.** The type system is still evolving, and some limitations are expected.
Wippy's gradual type system supports incremental annotations and flow-sensitive checking. Types are non-nullable by default.
This page is a language reference, not a complete program. Each code block is an isolated type-checking example, and alternatives within a block are not necessarily meant to be combined. Names such as `get_data`, `get_user`, `call`, and `User` represent application code; lines marked `ERROR` intentionally demonstrate diagnostics. These examples use language syntax and built-in type values, so they do not require runtime modules.
### Primitives
```lua
local n: number = 3.14
local i: integer = 42 -- integer is subtype of number
local s: string = "hello"
local b: boolean = true
local a: any = "anything" -- dynamic member and method access
local u: unknown = { source = "example" } -- must narrow before use
```
#### `any` and `unknown`
```lua
-- any: dynamic member and method access
local a: any = get_data()
a.foo.bar.baz() -- no error, may crash at runtime
local s: string = a -- ERROR: any is not assignable to string
-- unknown: safe unknown, must narrow before use as a concrete type
local u: unknown = get_data()
u.foo -- no error: member access on unknown behaves like any
local n: number = u -- ERROR: unknown not assignable to number, narrow first
if type(u) == "table" then
-- u narrowed to table here
end
```
### Nil Safety
Types are non-nullable by default. Use `?` for optional values:
```lua
local x: number = nil -- ERROR: nil not assignable to number
local y: number? = nil -- OK: number? means "number or nil"
local z: number? = 42 -- OK
```
#### Control Flow Narrowing
The type checker tracks control flow:
```lua
local function process(x: number?): number
if x ~= nil then
return x -- x is number here
end
return 0
end
-- Early return pattern
local user, err = get_user(123)
if err then return nil, err end
-- user narrowed to non-nil here
-- Or default
local val = get_value() or 0 -- val: number
```
### Union Types
```lua
local val: number | string = get_value()
if type(val) == "number" then
print(val + 1) -- val: number
else
print(val:upper()) -- val: string
end
```
#### Literal Types
```lua
type Status = "pending" | "active" | "done"
local s: Status = "pending" -- OK
local s: Status = "invalid" -- ERROR
```
### Function Types
```lua
local function add(a: number, b: number): number
return a + b
end
-- Multiple returns
local function div_mod(a: number, b: number): (number, number)
return math.floor(a / b), a % b
end
-- Error returns (Lua idiom)
local function fetch(url: string): (string?, error?)
-- returns (data, nil) or (nil, error)
end
-- First-class function types
local double: (number) -> number = function(x: number): number
return x * 2
end
```
#### Variadic Functions
```lua
local function sum(...: number): number
local total: number = 0
for _, v in ipairs({...}) do
total = total + v
end
return total
end
```
### Record Types
```lua
type User = {name: string, age: number}
local u: User = {name = "alice", age = 25}
```
#### Optional Fields
```lua
type Config = {
host: string,
port: number,
timeout?: number,
debug?: boolean
}
local cfg: Config = {host = "localhost", port = 8080} -- OK
```
### Generics
```lua
local function identity(x: T): T
return x
end
local n: number = identity(42)
local s: string = identity("hello")
```
#### Constrained Generics
```lua
type HasName = {name: string}
local function greet(obj: T): string
return "Hello, " .. obj.name
end
greet({name = "Alice"}) -- OK
greet({age = 30}) -- ERROR: missing 'name'
```
### Intersection Types
Combine multiple types:
```lua
type Named = {name: string}
type Aged = {age: number}
type Person = Named & Aged
local p: Person = {name = "Alice", age = 30}
```
### Tagged Unions
```lua
type Result =
{ok: true, value: T}
| {ok: false, error: E}
type LoadState =
{status: "loading"}
| {status: "loaded", data: User}
| {status: "error", message: string}
local function render(state: LoadState): string
if state.status == "loading" then
return "Loading..."
elseif state.status == "loaded" then
return "Hello, " .. state.data.name
elseif state.status == "error" then
return "Error: " .. state.message
end
end
```
### The `never` Type
`never` is the bottom type: it has no possible values.
```lua
function fail(msg: string): never
error(msg)
end
```
### Error Handling Pattern
The checker understands the common Lua `value, error` return pattern:
```lua
local value, err = call()
if err then
-- value is nil here
return nil, err
end
-- value is non-nil here, err is nil
print(value)
```
### Non-Nil Assertion
Use `!` to assert an expression is non-nil:
```lua
local user: User? = get_user()
local name = (user!).name -- assert user is non-nil
```
`!` is a type-checker assertion only - it narrows the type to non-nil but emits no runtime check. If the value is actually nil, the following operation fails with the usual error (e.g. indexing nil). Use when you know a value cannot be nil but the type checker cannot prove it.
#### Runtime Validation
Call a type as a function to validate a value. Validation returns the original value with the requested static type; it does not convert or coerce the value:
```lua
local data: any = get_json()
local user = User(data) -- validates and returns User
local name = user.name -- safe field access
```
This works with primitives and custom types:
```lua
local x: any = get_value()
local s = string(x) -- requires an existing string
local n = integer(x) -- requires an existing integer
local b = boolean(x) -- requires an existing boolean
type Point = {x: number, y: number}
local p = Point(data) -- validates record structure
```
For example, `string(42)` raises a validation error; use `tostring(42)` when conversion is intended.
#### Type:is() Method
`Type:is` validates without throwing and returns either `(value, nil)` or `(nil, error)`:
```lua
type Point = {x: number, y: number}
local data: any = get_input()
local p, err = Point:is(data)
if p then
local sum = p.x + p.y -- p is valid Point
else
return nil, err -- validation failed
end
```
The result narrows in conditionals:
```lua
if Point:is(data) then
local p: Point = data -- data narrowed to Point
end
```
#### Unsafe Cast
Use `::` or `as` for unchecked casts:
```lua
local data: any = get_data()
local user = data :: User -- no runtime check
local user = data as User -- same as ::
```
Use sparingly. Unsafe casts bypass validation and can cause runtime errors if the value doesn't match the type.
### Type Reflection
Types are first-class values that provide introspection methods.
#### Kind and Name
```lua
type Num = number
print(Num:kind()) -- "number"
print(Point:kind()) -- "record"
print(Point:name()) -- "Point"
```
#### Record Fields
Iterate over record fields:
```lua
type User = {name: string, age: number}
for name, typ in User:fields() do
print(name, typ:kind())
end
-- name string
-- age number
```
Access individual field types:
```lua
local nameType = User.name -- type of 'name' field
print(nameType:kind()) -- "string"
```
#### Collection Types
```lua
type NumberList = {number}
print(NumberList:elem():kind()) -- "number"
type ScoreMap = {[string]: number}
print(ScoreMap:key():kind()) -- "string"
print(ScoreMap:val():kind()) -- "number"
```
#### Optional Types
```lua
type MaybeNumber = number?
print(MaybeNumber:kind()) -- "optional"
print(MaybeNumber:inner():kind()) -- "number"
```
#### Union Types
```lua
type Status = "pending" | "active" | "done"
for variant in Status:variants() do
print(variant)
end
```
#### Function Types
```lua
type Predicate = (number, string) -> boolean
for param in Predicate:params() do
print(param:kind())
end
print(Predicate:ret():kind()) -- "boolean"
```
`typeof(expression)` is type syntax, not a runtime reflection function. Use it in an alias such as `type Config = typeof(default_config)`; the resulting alias is the runtime type value.
#### Type Comparison
```lua
type Num = number
type Int = integer
print(Num == Num) -- true
print(Int <= Num) -- true (subtype)
print(Int < Num) -- true (strict subtype)
```
#### Types as Table Keys
```lua
type Point = {x: number, y: number}
type Line = {from: Point, to: Point}
local handlers = {}
handlers[Point] = function() return "point handler" end
handlers[Line] = function() return "line handler" end
local h = handlers[Point]
if h then h() end
```
### Type Annotations
Add types to function signatures:
```lua
-- Parameter and return types
local function process(input: string): number
return #input
end
-- Local variable types
local count: number = 0
-- Type aliases
type StringArray = {string}
type StringMap = {[string]: number}
```
### Type Validators
Attach validation constraints to type aliases with annotations, then call the type or use `Type:is()` to enforce them at runtime:
```lua
type NonNegative = number @min(0)
type Percentage = number @min(0) @max(100)
type Email = string @pattern("^.+@.+$")
local x = NonNegative(1)
local percent, err = Percentage:is(50)
local email = Email("test@example.com")
```
An annotation on a local variable is checked statically by the linter. It does not insert an automatic runtime check at assignment; runtime enforcement occurs when a type value validates a value.
#### Built-in Validators
| Validator | Applies to | Example |
|-----------|------------|---------|
| `@min(n)` | number | `type Positive = number @min(1)` |
| `@max(n)` | number | `type Percentage = number @max(100)` |
| `@min_len(n)` | string, array | `type NonEmpty = string @min_len(1)` |
| `@max_len(n)` | string, array | `type ShortName = string @max_len(10)` |
| `@pattern(regex)` | string | `type Email = string @pattern("^.+@.+$")` |
#### Record Field Validators
```lua
type User = {
age: number @min(0) @max(150),
name: string @min_len(1) @max_len(100)
}
```
#### Array Element Validators
```lua
local scores: {number @min(0) @max(100)} = {85, 90}
```
#### Union Member Validators
```lua
local id: number @min(1) | string @min_len(1) = 1
```
### Variance Rules
| Position | Variance | Description |
|----------|----------|-------------|
| Readonly field | Covariant | Can use subtype |
| Mutable field | Quasi-invariant | Normally invariant; fresh literals and refinements may widen to their base type |
| Function parameter | Contravariant | Can use supertype |
| Function return | Covariant | Can use subtype |
### Subtyping
- `integer` is a subtype of `number`
- `never` is a subtype of all types
- All types are subtypes of `any`
- Union subtyping: `A` is subtype of `A | B`
### Gradual Adoption
Types can be added incrementally; untyped code continues to work:
```lua
-- Existing code works unchanged
function old_function(x)
return x + 1
end
-- New code gets types
function new_function(x: number): number
return x + 1
end
```
Useful starting points include:
1. Function signatures at API boundaries
2. HTTP handlers and queue consumers
3. Critical business logic
### Type Checking
Run the type checker with:
```bash
wippy lint
```
The command reports type errors without executing the code.
---
# "Lua Entry Kinds"
## Lua Entry Kinds
Lua entry kinds define how source code is loaded and executed as a function, process, workflow, or library.
This page is a configuration reference. YAML blocks are partial entry definitions intended to be placed under an `entries:` mapping in a Wippy index; they are not complete applications by themselves. Referenced source files, imports, dependencies, process hosts, and security policies must exist in the surrounding project.
### Entry Kinds
| Kind | Description |
|------|-------------|
| `function.lua` | Stateless function, runs on demand |
| `process.lua` | Long-running actor with state |
| `workflow.lua` | Durable workflow (Temporal) |
| `library.lua` | Shared code imported by other entries |
Each kind has a precompiled bytecode counterpart (`function.lua.bc`, `library.lua.bc`, `process.lua.bc`, `workflow.lua.bc`) produced by `wippy pack --bytecode '**'` (or a pattern like `--bytecode 'app:**'`). Authors write `.lua` entries; the bytecode kinds are emitted when packing with that flag.
`module.lua` is reserved for built-in module definitions created by the runtime. It is not an authorable source entry and has no bytecode counterpart.
### Common Fields
All Lua entries share these fields:
| Field | Required | Description |
|-------|----------|-------------|
| `name` | yes | Unique name within namespace |
| `kind` | yes | One of the Lua kinds above |
| `source` | yes | Inline Lua source or a `file://path.lua` reference resolved when the registry is loaded |
| `method` | function/process/workflow | Function to export (libraries don't use it) |
| `modules` | no | Allowed modules for `require()` |
| `imports` | no | Other entries as local modules |
| `meta` | no | Searchable metadata |
`pool` applies only to `function.lua`. `security` applies to `function.lua` and `process.lua`.
### `function.lua`
A `function.lua` entry runs on demand, with each invocation handled independently.
```yaml
- name: handler
kind: function.lua
source: file://handler.lua
method: main
modules:
- http
- json
```
Use functions for HTTP handlers, data transformations, and utilities.
### `process.lua`
A `process.lua` entry is a long-running actor that maintains state and communicates through messages.
```yaml
- name: worker
kind: process.lua
source: file://worker.lua
method: main
modules:
- sql
```
Choose a process for background workers, service daemons, and stateful actors.
To run as a supervised service:
```yaml
- name: worker_service
kind: process.service
process: app:worker
host: app:processes
lifecycle:
auto_start: true
restart:
max_attempts: 10
```
### `workflow.lua`
A `workflow.lua` entry defines a durable workflow whose state is persisted to Temporal.
```yaml
- name: order_processor
kind: workflow.lua
source: file://order_workflow.lua
method: main
modules:
- workflow
- time
```
Use workflows for multi-step business processes and long-running orchestration.
### `library.lua`
A `library.lua` entry provides shared code that other entries can import.
```yaml
- name: helpers
kind: library.lua
source: file://helpers.lua
modules:
- json
- base64
```
Other entries reference it via `imports`:
```yaml
- name: handler
kind: function.lua
source: file://handler.lua
method: main
imports:
helpers: app.lib:helpers
```
In Lua code:
```lua
local helpers = require("helpers")
helpers.format_date(timestamp)
```
### Modules
The `modules` field controls which modules can be loaded with `require()`:
```yaml
modules:
- http
- json
- sql
```
`channel`, `payload`, `print`, `process`, `subscribe`, and `unsubscribe` are loaded as Lua globals — they don't need to appear in `modules:`. `require("process")` is also allowed without a `modules:` declaration.
Only listed built-in modules and aliases declared under `imports` are available. The module allowlist limits access to runtime capabilities, makes dependencies explicit, and restricts workflows to workflow-compatible module classes.
See [Lua Runtime](lua/overview.md) for available modules.
### Imports
Import other entries as local modules:
```yaml
imports:
utils: app.lib:utils # require("utils")
auth: app.auth:helpers # require("auth")
```
The key becomes the module name in Lua code. The value is the entry ID (`namespace:name`).
### Function Pools
Use `pool` to configure how a function entry executes:
```yaml
- name: handler
kind: function.lua
source: file://handler.lua
method: main
pool:
type: adaptive # explicit; omit to use auto-select (lazy)
max_size: 16 # cap for elastic growth
```
| Field | Pools | Description |
|-------|-------|-------------|
| `type` | all | Scheduler implementation (see table below) |
| `workers` | static | Worker thread count (falls back to `size`, then 8) |
| `size` | static | Worker count when `workers` is unset; with `type` omitted, `size` without `max_size` selects an inline pool |
| `buffer` | static | Task queue capacity (default: `workers * 64`) |
| `max_size` | lazy, adaptive | Upper bound for elastic growth (default: 16; 100 when `type` is omitted) |
| Type | Behavior |
|------|----------|
| `inline` | Synchronous execution in the caller's goroutine. No isolation between calls. |
| `lazy` | Zero idle workers, spawn on demand, tear down when idle. |
| `static` | Fixed-size channel-based pool. Predictable under steady load. |
| `adaptive` | Auto-scaling pool — grows under load, shrinks when idle. |
When `type` is omitted, the pool is auto-selected from the other fields: a lazy pool by default, a static pool if `workers` is set, an inline pool if only `size` is set.
### Metadata
Use `meta` to attach searchable routing and discovery fields:
```yaml
- name: api_handler
kind: function.lua
meta:
type: handler
version: "2.0"
tags: [api, users]
source: file://api.lua
method: handle
modules:
- http
- json
- registry
```
Metadata is searchable via the registry:
```lua
local registry = require("registry")
local handlers, err = registry.find({["meta.type"] = "handler"})
if err then
return nil, err
end
```
The query returns all matching registry entries. The Lua code belongs to an executable entry whose `modules` list includes `registry`, such as the `api_handler` entry above.
### See Also
- [Entry Kinds](guides/entry-kinds.md) - Reference for all entry kinds
- [Compute Units](concepts/compute-units.md) - Functions vs processes vs workflows
- [Lua Runtime](lua/overview.md) - Available modules
---
# "Standard Lua Libraries"
## Standard Lua Libraries
These core Lua libraries are available in every executable Lua entry without `require()`.
This is an API reference. Signature blocks list available functions, while the longer blocks are isolated examples or partial patterns rather than complete entries. Names such as `check_health` and `process_request` represent application callbacks.
#### Type and Conversion
```lua
type(value) -- Returns: "nil", "number", "string", "boolean", "table", "function", "thread", "userdata"
tonumber(s [,base]) -- Convert to number, optional base (2-36)
tostring(value) -- Convert to string, calls __tostring metamethod
```
#### Assertions and Errors
```lua
assert(v [,msg]) -- Raises error if v is false/nil, returns v otherwise
error(msg [,level]) -- Raises error at specified stack level (default 1)
pcall(fn, ...) -- Protected call, returns ok, result_or_error
xpcall(fn, errh) -- Protected call with error handler function
```
#### Table Iteration
```lua
pairs(t) -- Iterate all key-value pairs
ipairs(t) -- Iterate array portion (1, 2, 3, ...)
next(t [,index]) -- Get next key-value pair after index
```
#### Metatables
```lua
getmetatable(obj) -- Get metatable (or __metatable field if protected)
setmetatable(t, mt) -- Set metatable, returns t
```
#### Raw Table Access
Bypass metamethods for direct table access:
```lua
rawget(t, k) -- Get t[k] without __index
rawset(t, k, v) -- Set t[k]=v without __newindex
rawequal(a, b) -- Compare without __eq
```
#### Utilities
```lua
select(index, ...) -- Return args from index onwards
select("#", ...) -- Return number of args
unpack(t [,i [,j]]) -- Return t[i] through t[j] as multiple values
print(...) -- Print values (uses structured logging in Wippy)
```
#### Global Variables
```lua
_G -- The global environment table
_VERSION -- Lua version string
```
### Table Manipulation
The `table` library provides in-place array operations, sorting, concatenation, and unpacking:
```lua
table.insert(t, [pos,] value) -- Insert value at pos (default: end)
table.remove(t [,pos]) -- Remove and return element at pos (default: last)
table.concat(t [,sep [,i [,j]]]) -- Concatenate array elements with separator
table.sort(t [,comp]) -- Sort in place, comp(a,b) returns true if a < b
table.unpack(t [,i [,j]]) -- Unpack table elements as multiple values
table.create(narr, nhash) -- Preallocate table with array and hash capacity
table.freeze(t) -- Make table immutable, returns t
table.isfrozen(t) -- true if table is immutable
```
```lua
local items = {"a", "b", "c"}
table.insert(items, "d") -- {"a", "b", "c", "d"}
table.insert(items, 2, "x") -- {"a", "x", "b", "c", "d"}
table.remove(items, 2) -- {"a", "b", "c", "d"}, returns "x"
local csv = table.concat(items, ",") -- "a,b,c,d"
table.sort(items, function(a, b)
return a > b -- Descending order
end)
```
### String Operations
String functions are also available as methods on string values.
#### Pattern Matching
```lua
string.find(s, pattern [,init [,plain]]) -- Find pattern, returns start, end, captures
string.match(s, pattern [,init]) -- Extract matching substring
string.gmatch(s, pattern) -- Iterator over all matches
string.gsub(s, pattern, repl [,n]) -- Replace matches, returns string, count
```
#### Case Conversion
```lua
string.upper(s) -- Convert to uppercase
string.lower(s) -- Convert to lowercase
```
#### Substrings and Characters
```lua
string.sub(s, i [,j]) -- Substring from i to j (negative indexes from end)
string.len(s) -- String length (or use #s)
string.byte(s [,i [,j]]) -- Numeric codes of characters
string.char(...) -- Create string from character codes
string.rep(s, n) -- Repeat string n times
string.reverse(s) -- Reverse string
```
#### Formatting
```lua
string.format(fmt, ...) -- Printf-style formatting
string.pack(fmt, ...) -- Pack values into a binary string
string.unpack(fmt, s [,pos]) -- Unpack binary string, returns values and next position
string.packsize(fmt) -- Size in bytes of a packed format
```
Format specifiers: `%d` (integer), `%f` (float), `%s` (string), `%q` (quoted), `%x` (hex), `%o` (octal), `%e` (scientific), `%%` (literal %)
```lua
local s = "Hello, World!"
-- Pattern matching
local start, stop = string.find(s, "World") -- 8, 12
local word = string.match(s, "%w+") -- "Hello"
-- Substitution
local new = string.gsub(s, "World", "Wippy") -- "Hello, Wippy!"
-- Method syntax
local upper = s:upper() -- "HELLO, WORLD!"
local part = s:sub(1, 5) -- "Hello"
```
#### Patterns
| Pattern | Matches |
|---------|---------|
| `.` | Any character |
| `%a` | Letters |
| `%d` | Digits |
| `%w` | Alphanumeric |
| `%s` | Whitespace |
| `%p` | Punctuation |
| `%c` | Control characters |
| `%x` | Hexadecimal digits |
| `%z` | Zero (null) |
| `[set]` | Character class |
| `[^set]` | Negated class |
| `*` | 0 or more (greedy) |
| `+` | 1 or more (greedy) |
| `-` | 0 or more (lazy) |
| `?` | 0 or 1 |
| `^` | Start of string |
| `$` | End of string |
| `%b()` | Balanced pair |
| `(...)` | Capture group |
Uppercase versions (`%A`, `%D`, etc.) match the complement.
### Math Functions
The `math` library provides numeric constants and common mathematical operations.
#### Constants {id="math-constants"}
```lua
math.pi -- 3.14159...
math.huge -- Largest representable float
math.mininteger -- Minimum integer
math.maxinteger -- Maximum integer
```
#### Basic Operations
```lua
math.abs(x) -- Absolute value
math.min(...) -- Minimum of arguments
math.max(...) -- Maximum of arguments
math.floor(x) -- Round down
math.ceil(x) -- Round up
math.modf(x) -- Integer and fractional parts
math.fmod(x, y) -- Floating-point remainder
```
#### Powers and Roots
```lua
math.sqrt(x) -- Square root
math.pow(x, y) -- x^y (or use x^y operator)
math.exp(x) -- e^x
math.log(x) -- Natural log
math.log10(x) -- Base-10 log
math.frexp(x) -- Mantissa and exponent
math.ldexp(m, e) -- m * 2^e
```
#### Trigonometry
```lua
math.sin(x) math.cos(x) math.tan(x) -- Radians
math.asin(x) math.acos(x) math.atan(x)
math.atan2(y, x) -- Arc tangent of y/x
math.sinh(x) math.cosh(x) math.tanh(x) -- Hyperbolic
math.deg(r) -- Radians to degrees
math.rad(d) -- Degrees to radians
```
#### Random Numbers
```lua
math.random() -- Random float [0,1)
math.random(n) -- Random integer [1,n]
math.random(m, n) -- Random integer [m,n]
math.randomseed(x) -- No effect; the generator is auto-seeded
```
`math.random` is nondeterministic. Do not use it for decisions that must replay identically in a workflow; `math.randomseed` cannot make it deterministic.
#### Type Conversion
```lua
math.tointeger(x) -- Convert to integer or nil
math.type(x) -- "integer", "float", or nil
math.ult(m, n) -- Unsigned less-than comparison
```
### Coroutines
The `coroutine` library provides coroutine creation and control. See [Channels and Coroutines](lua/core/channel.md) for channel-based concurrency patterns.
```lua
coroutine.create(fn) -- Create coroutine from function
coroutine.resume(co, ...) -- Start/continue coroutine
coroutine.yield(...) -- Suspend coroutine, return values to resume
coroutine.status(co) -- "running", "suspended", "normal", "dead"
coroutine.running() -- Current coroutine (nil if main thread)
coroutine.wrap(fn) -- Create coroutine as callable function
```
#### Spawning Concurrent Coroutines
Wippy adds `coroutine.spawn` for scheduler-managed concurrent work:
```lua
coroutine.spawn(fn) -- Spawn function as concurrent coroutine
```
```lua
local time = require("time")
-- Spawn background task
coroutine.spawn(function()
while true do
check_health()
time.sleep("30s")
end
end)
-- Continue main execution immediately
process_request()
```
This partial pattern assumes the entry lists `time` in `modules:` and provides the `check_health` and `process_request` functions. The spawned coroutine runs concurrently in the same Lua process; `process_request()` is reached immediately, and each health check is followed by a 30-second sleep.
### Error Handling
The global `errors` table creates and classifies structured errors. See [Error Handling](lua/core/errors.md) for the complete API.
#### Constants {id="error-constants"}
```lua
errors.UNKNOWN -- Unclassified error
errors.INVALID -- Invalid argument or input
errors.NOT_FOUND -- Resource not found
errors.ALREADY_EXISTS -- Resource already exists
errors.PERMISSION_DENIED -- Permission denied
errors.TIMEOUT -- Operation timed out
errors.CANCELED -- Operation cancelled
errors.UNAVAILABLE -- Service unavailable
errors.INTERNAL -- Internal error
errors.CONFLICT -- Conflict (e.g., concurrent modification)
errors.RATE_LIMITED -- Rate limit exceeded
```
#### Functions {id="error-functions"}
```lua
-- Create error from string
local err = errors.new("something went wrong")
-- Create error with metadata
local err = errors.new({
message = "User not found",
kind = errors.NOT_FOUND,
retryable = false,
details = {user_id = 123}
})
-- Wrap existing error with context
local wrapped = errors.wrap(err, "failed to load profile")
-- Check error kind
if errors.is(err, errors.NOT_FOUND) then
-- handle not found
end
-- Get call stack from error
local stack = errors.call_stack(err)
```
#### Error Methods
```lua
err:message() -- Get error message string
err:kind() -- Get error kind (e.g., "NOT_FOUND")
err:retryable() -- true, false, or nil (unknown)
err:details() -- Get details table or nil
err:stack() -- Get stack trace as string
```
### Restricted Features
The following standard Lua features are unavailable in Wippy processes:
| Feature | Alternative |
|---------|-------------|
| `load`, `loadstring`, `loadfile`, `dofile` | Use [Dynamic Evaluation](lua/dynamic/eval.md) module |
| `collectgarbage` | Automatic GC |
| `rawlen` | Use `#` operator |
| Standard `io.*` file library | Use [File System](lua/storage/filesystem.md) module; the `io` module in Wippy is [Terminal I/O](lua/system/io.md) |
| `os.execute`, `os.exit`, `os.getenv`, `os.remove`, `os.rename`, `os.tmpname` | Use [Command Execution](lua/dynamic/exec.md), [Environment](lua/system/env.md) modules |
| `string.dump` | Not available |
| `debug.*` | Not available |
| `utf8.*` | Not available |
| `package.loadlib` | Native libraries not supported |
### See Also
- [Channels and Coroutines](lua/core/channel.md) - Go-style channels for concurrency
- [Error Handling](lua/core/errors.md) - Creating and handling structured errors
- [OS Time](lua/system/ostime.md) - System time functions
---
# "Errors"
## Errors
The global `errors` table creates and inspects structured errors with categories, details, and retry metadata. It is available without `require`.
This is an API reference. Each code block is an isolated snippet, not a complete entry. Variables such as `err` refer to an error returned or created by surrounding application code; the wrapping example assumes `db` is an application-provided database client.
### Creating Errors
```lua
-- Simple message (kind defaults to UNKNOWN)
local err = errors.new("something went wrong")
-- With kind, retryable, and details
local err = errors.new({
message = "user not found",
kind = errors.NOT_FOUND,
retryable = false,
details = {user_id = 123}
})
```
`errors.new` accepts either a string message or a table with at least a `message` field. The `(kind, message)` form is not supported.
### Wrapping Errors
Wrap an error to add context while preserving its kind, retry metadata, and details:
```lua
local data, err = db:query("SELECT * FROM users")
if err then
return nil, errors.wrap(err, "failed to load users")
end
```
### Error Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `err:kind()` | string | Error category |
| `err:message()` | string | Error message |
| `err:retryable()` | boolean/nil | Whether operation can be retried |
| `err:details()` | table/nil | Structured metadata |
| `err:stack()` | string | Lua stack trace |
| `tostring(err)` | string | Full representation |
### Checking Kind
```lua
if errors.is(err, errors.INVALID) then
-- handle invalid input
end
-- Or compare directly
if err:kind() == errors.NOT_FOUND then
-- handle missing resource
end
```
### Error Kinds
| Constant | Use Case |
|----------|----------|
| `errors.NOT_FOUND` | Resource doesn't exist |
| `errors.ALREADY_EXISTS` | Resource already exists |
| `errors.INVALID` | Bad input or arguments |
| `errors.PERMISSION_DENIED` | Access denied |
| `errors.UNAVAILABLE` | Service temporarily down |
| `errors.INTERNAL` | Internal error |
| `errors.CANCELED` | Operation was canceled |
| `errors.CONFLICT` | Resource state conflict |
| `errors.TIMEOUT` | Operation timed out |
| `errors.RATE_LIMITED` | Too many requests |
| `errors.UNKNOWN` | Unspecified error |
### Call Stack
Use `errors.call_stack` to inspect a structured call stack:
```lua
local stack = errors.call_stack(err)
if stack then
print("Thread:", stack.thread)
for _, frame in ipairs(stack.frames) do
print(frame.source .. ":" .. frame.line, frame.name)
end
end
```
### Retryable Errors
Retryability is error metadata, not a property guaranteed by an error kind. Check the value returned by `err:retryable()` rather than inferring it from `err:kind()`. A result of `nil` means the error does not specify whether retrying is appropriate.
```lua
if err:retryable() then
-- safe to retry
end
```
### Error Details
```lua
local err = errors.new({
message = "validation failed",
kind = errors.INVALID,
details = {
errors = {
{field = "email", message = "invalid format"},
{field = "age", message = "must be positive"}
}
}
})
local details = err:details()
for _, e in ipairs(details.errors) do
print(e.field, e.message)
end
```
---
# "Time & Duration"
## Time & Duration
The `time` module provides time values, durations, time-zone handling, parsing, formatting, sleeps, and timers. Supported workflow time calls are recorded so they can replay deterministically.
This is an API reference. Code blocks are isolated examples or partial scheduling patterns, not a complete entry. Names such as `do_work`, `try_operation`, `make_request`, `send_reminder`, `user_activity`, `check_health`, and `process` represent application callbacks, channels, or data. Where a snippet assigns an error return to `_`, it assumes the shown literal is valid; handle errors when values can come from input or configuration.
### Loading
```lua
local time = require("time")
```
Add `time` to the executable entry's `modules:` list before requiring it. The ambient `channel` and `errors` globals used by scheduling examples need no module declaration.
#### `now`
Returns the current time. In workflows, it returns the recorded workflow time reference so execution can replay deterministically.
```lua
local t = time.now()
print(t:format_rfc3339()) -- "2024-12-29T15:04:05Z"
-- Measure elapsed time
local start = time.now()
do_work()
local elapsed = time.now():sub(start)
print("Took " .. elapsed:milliseconds() .. "ms")
```
The timestamp and elapsed-time output are illustrative; `time.now()` supplies the current or recorded workflow time.
**Returns:** `Time`
#### Create from Components
```lua
-- Create specific date/time in UTC
local t = time.date(2024, time.DECEMBER, 25, 10, 30, 0, 0, time.utc)
print(t:format_rfc3339()) -- "2024-12-25T10:30:00Z"
-- Create in specific timezone
local ny, err = time.load_location("America/New_York")
if err then
return nil, err
end
local meeting = time.date(2024, time.JANUARY, 15, 14, 0, 0, 0, ny)
-- Defaults to local timezone if not specified
local t = time.date(2024, 1, 15, 12, 0, 0, 0)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `year` | number | Year |
| `month` | number | Month (1-12 or `time.JANUARY` etc) |
| `day` | number | Day of month |
| `hour` | number | Hour (0-23) |
| `minute` | number | Minute (0-59) |
| `second` | number | Second (0-59) |
| `nanosecond` | number | Nanosecond (0-999999999) |
| `location` | Location | Timezone (optional, defaults to local) |
**Returns:** `Time`
#### Create from a Unix Timestamp
```lua
-- From seconds since epoch
local t = time.unix(1703862245, 0)
print(t:utc():format_rfc3339()) -- "2023-12-29T15:04:05Z"
-- With nanoseconds
local t = time.unix(1703862245, 500000000) -- +500ms
-- Convert JavaScript timestamp (milliseconds)
local js_timestamp = 1703862245000
local t = time.unix(js_timestamp // 1000, (js_timestamp % 1000) * 1000000)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sec` | number | Unix seconds |
| `nsec` | number | Nanoseconds offset |
**Returns:** `Time`
#### Parse from a String
Parse time strings using Go's reference time format: `Mon Jan 2 15:04:05 MST 2006`.
```lua
-- Parse RFC3339
local t, err = time.parse(time.RFC3339, "2024-12-29T15:04:05Z")
if err then
return nil, err
end
-- Parse custom format
local t, err = time.parse("2006-01-02", "2024-12-29")
local t, err = time.parse("15:04:05", "14:30:00")
local t, err = time.parse("2006-01-02 15:04:05 MST", "2024-12-29 14:30:00 EST")
-- Parse in specific timezone
local ny, _ = time.load_location("America/New_York")
local t, err = time.parse("2006-01-02 15:04", "2024-12-29 14:30", ny)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `layout` | string | Go time format layout |
| `value` | string | String to parse |
| `location` | Location | Default time zone (optional) |
**Returns:** `Time, error`
#### Arithmetic
```lua
local t = time.now()
-- Add duration (accepts number, string, or Duration)
local tomorrow = t:add("24h")
local later = t:add(5 * time.MINUTE)
local d, _ = time.parse_duration("1h30m")
local future = t:add(d)
-- Subtract time to get duration
local diff = tomorrow:sub(t) -- returns Duration
print(diff:hours()) -- 24
-- Add calendar units (handles month boundaries correctly)
local next_month = t:add_date(0, 1, 0) -- add 1 month
local next_year = t:add_date(1, 0, 0) -- add 1 year
local last_week = t:add_date(0, 0, -7) -- subtract 7 days
```
| Method | Parameters | Returns | Description |
|--------|------------|---------|-------------|
| `add(duration)` | number/string/Duration | Time | Add duration |
| `sub(time)` | Time | Duration | Difference between times |
| `add_date(years, months, days)` | numbers | Time | Add calendar units |
#### Comparison
```lua
local t1 = time.date(2024, 1, 1, 0, 0, 0, 0, time.utc)
local t2 = time.date(2024, 1, 2, 0, 0, 0, 0, time.utc)
t1:before(t2) -- true
t2:after(t1) -- true
t1:equal(t1) -- true
```
| Method | Parameters | Returns | Description |
|--------|------------|---------|-------------|
| `before(time)` | Time | boolean | Whether this time is before the other value |
| `after(time)` | Time | boolean | Whether this time is after the other value |
| `equal(time)` | Time | boolean | Whether the two values represent the same time |
#### Formatting
```lua
local t = time.now()
t:format_rfc3339() -- "2024-12-29T15:04:05Z"
t:format(time.DATE_ONLY) -- "2024-12-29"
t:format(time.TIME_ONLY) -- "15:04:05"
t:format("Mon Jan 2, 2006") -- "Sun Dec 29, 2024"
```
| Method | Parameters | Returns | Description |
|--------|------------|---------|-------------|
| `format(layout)` | string | string | Format using Go layout |
| `format_rfc3339()` | - | string | Format as RFC3339 |
#### Unix Timestamps
```lua
local t = time.now()
t:unix() -- seconds since epoch
t:unix_nano() -- nanoseconds since epoch
```
#### Components
```lua
local t = time.now()
-- Get date parts
local year, month, day = t:date()
-- Get time parts
local hour, min, sec = t:clock()
-- Individual accessors
t:year() -- e.g., 2024
t:month() -- 1-12
t:day() -- 1-31
t:hour() -- 0-23
t:minute() -- 0-59
t:second() -- 0-59
t:nanosecond() -- 0-999999999
t:weekday() -- 0=Sunday .. 6=Saturday
t:year_day() -- 1-366
t:is_zero() -- true if zero value
```
#### Time-Zone Conversion
```lua
local t = time.now()
t:utc() -- convert to UTC
t:in_local() -- convert to local timezone
t:in_location(ny) -- convert to specific timezone
t:location() -- get current Location
t:location():string() -- get timezone name
```
| Method | Parameters | Returns | Description |
|--------|------------|---------|-------------|
| `utc()` | - | Time | Convert to UTC |
| `in_local()` | - | Time | Convert to the local time zone |
| `in_location(loc)` | Location | Time | Convert to a specified time zone |
| `location()` | - | Location | Return the current time zone |
#### Rounding
Round or truncate to duration boundaries. **Requires Duration userdata** (not number or string).
```lua
local t = time.now()
local hour_duration, _ = time.parse_duration("1h")
local minute_duration, _ = time.parse_duration("15m")
t:round(hour_duration) -- round to nearest hour
t:truncate(minute_duration) -- truncate to 15-minute boundary
```
| Method | Parameters | Returns | Description |
|--------|------------|---------|-------------|
| `round(duration)` | Duration | Time | Round to nearest multiple |
| `truncate(duration)` | Duration | Time | Truncate to multiple |
#### Create a Duration
```lua
-- Parse from string
local d, err = time.parse_duration("1h30m45s")
local d, err = time.parse_duration("500ms")
local d, err = time.parse_duration("2h30m45s500ms")
-- From number (nanoseconds)
local d, err = time.parse_duration(time.SECOND)
local d, err = time.parse_duration(5 * time.MINUTE)
-- Valid units: ns, us, ms, s, m, h
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `value` | number/string/Duration | Duration to parse |
**Returns:** `Duration, error`
#### Duration Methods
```lua
local d, _ = time.parse_duration("1h30m45s500ms")
d:hours() -- 1.5126...
d:minutes() -- 90.75...
d:seconds() -- 5445.5
d:milliseconds() -- 5445500
d:microseconds() -- 5445500000
d:nanoseconds() -- 5445500000000
```
#### Named Locations
Load a time zone by its IANA name, such as `America/New_York`, `Europe/London`, or `Asia/Tokyo`.
```lua
local ny, err = time.load_location("America/New_York")
if err then
return nil, err
end
local tokyo, _ = time.load_location("Asia/Tokyo")
local london, _ = time.load_location("Europe/London")
-- Convert between timezones
local t = time.now():utc()
print("UTC:", t:format(time.TIME_ONLY))
print("New York:", t:in_location(ny):format(time.TIME_ONLY))
print("Tokyo:", t:in_location(tokyo):format(time.TIME_ONLY))
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | IANA time-zone name |
**Returns:** `Location, error`
#### Fixed-Offset Locations
Create a time zone with a fixed UTC offset.
```lua
-- UTC+5:30 (India Standard Time)
local ist = time.fixed_zone("IST", 5*3600 + 30*60)
-- UTC-8 (Pacific Standard Time)
local pst = time.fixed_zone("PST", -8*3600)
local t = time.date(2024, 1, 15, 12, 0, 0, 0, ist)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Zone name |
| `offset` | number | UTC offset in seconds |
**Returns:** `Location`
#### Built-In Locations
```lua
time.utc -- UTC timezone
time.localtz -- Local system timezone
```
#### `sleep`
Suspend execution for the specified duration. Workflow execution records the sleep for deterministic replay.
```lua
time.sleep("5s")
time.sleep(500 * time.MILLISECOND)
-- Backoff pattern
for attempt = 1, 3 do
local ok = try_operation()
if ok then break end
time.sleep(tostring(attempt) .. "s")
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `duration` | number/string/Duration | Sleep time |
#### `after`
Returns a channel that receives one value after the duration. The channel can be used with `channel.select`.
```lua
-- Simple timeout
local timeout, err = time.after("5s")
if err then return nil, err end
timeout:receive() -- blocks for 5 seconds
-- Timeout with select
local response_ch = make_request()
local timeout_ch, err = time.after("30s")
if err then return nil, err end
local result = channel.select{
response_ch:case_receive(),
timeout_ch:case_receive()
}
if result.channel == timeout_ch then
return nil, errors.new({message = "Request timed out", kind = errors.TIMEOUT})
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `duration` | number/string/Duration | Time to wait |
**Returns:** `Channel, error`
#### `timer`
Creates a one-shot timer that fires after the specified duration and can be stopped or reset.
```lua
local timer, err = time.timer("5s")
if err then
return nil, err
end
-- Wait for timer
timer:response():receive()
send_reminder()
-- Reset on activity
local idle_timer, err = time.timer("5m")
if err then
return nil, err
end
local idle_ch = idle_timer:response()
while true do
local r = channel.select{
user_activity:case_receive(),
idle_ch:case_receive()
}
if r.channel == idle_ch then
logout_user()
break
end
idle_timer:reset("5m")
end
-- Stop timer
timer:stop()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `duration` | number/string/Duration | Time until fire |
**Returns:** `Timer, error`
| Timer Method | Parameters | Returns | Description |
|--------------|------------|---------|-------------|
| `response()` | - | Channel | Get timer channel |
| `channel()` | - | Channel | Alias for response() |
| `stop()` | - | boolean | Cancel timer |
| `reset(duration)` | number/string/Duration | boolean | Reset with new duration |
#### `ticker`
Creates a repeating timer that fires at regular intervals.
```lua
-- Periodic task
local ticker, err = time.ticker("30s")
if err then
return nil, err
end
local ch = ticker:response()
while true do
local tick_time = ch:receive()
check_health()
end
```
The loop above is intended for a long-running process. A separate finite rate-limiting pattern is:
```lua
-- Rate limiting
local ticker, err = time.ticker("100ms")
if err then
return nil, err
end
for _, item in ipairs(items) do
ticker:response():receive()
process(item)
end
ticker:stop()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `duration` | number/string/Duration | Interval between ticks |
**Returns:** `Ticker, error`
| Ticker Method | Parameters | Returns | Description |
|---------------|------------|---------|-------------|
| `response()` | - | Channel | Get ticker channel |
| `channel()` | - | Channel | Alias for response() |
| `stop()` | - | boolean | Stop ticker |
#### Duration Units
Duration constants are expressed in nanoseconds and can be combined with arithmetic.
```lua
time.NANOSECOND -- 1
time.MICROSECOND -- 1,000
time.MILLISECOND -- 1,000,000
time.SECOND -- 1,000,000,000
time.MINUTE -- 60 * SECOND
time.HOUR -- 60 * MINUTE
-- Example usage
time.sleep(5 * time.SECOND)
local timeout, err = time.after(30 * time.SECOND)
if err then return nil, err end
```
#### Format Layouts
```lua
time.RFC3339 -- "2006-01-02T15:04:05Z07:00"
time.RFC3339NANO -- "2006-01-02T15:04:05.999999999Z07:00"
time.RFC822 -- "02 Jan 06 15:04 MST"
time.RFC822Z -- "02 Jan 06 15:04 -0700"
time.RFC850 -- "Monday, 02-Jan-06 15:04:05 MST"
time.RFC1123 -- "Mon, 02 Jan 2006 15:04:05 MST"
time.RFC1123Z -- "Mon, 02 Jan 2006 15:04:05 -0700"
time.DATE_TIME -- "2006-01-02 15:04:05"
time.DATE_ONLY -- "2006-01-02"
time.TIME_ONLY -- "15:04:05"
time.KITCHEN -- "3:04PM"
time.STAMP -- "Jan _2 15:04:05"
time.STAMP_MILLI -- "Jan _2 15:04:05.000"
time.STAMP_MICRO -- "Jan _2 15:04:05.000000"
time.STAMP_NANO -- "Jan _2 15:04:05.000000000"
```
#### Months
```lua
time.JANUARY -- 1
time.FEBRUARY -- 2
time.MARCH -- 3
time.APRIL -- 4
time.MAY -- 5
time.JUNE -- 6
time.JULY -- 7
time.AUGUST -- 8
time.SEPTEMBER -- 9
time.OCTOBER -- 10
time.NOVEMBER -- 11
time.DECEMBER -- 12
```
#### Weekdays
```lua
time.SUNDAY -- 0
time.MONDAY -- 1
time.TUESDAY -- 2
time.WEDNESDAY -- 3
time.THURSDAY -- 4
time.FRIDAY -- 5
time.SATURDAY -- 6
```
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Invalid duration format | `errors.INVALID` | no |
| Parse failed | `errors.INVALID` | no |
| Empty location name | `errors.INVALID` | no |
| Location not found | `errors.NOT_FOUND` | no |
| Duration <= 0 (timer/ticker) | `errors.INVALID` | no |
```lua
local t, err = time.parse(time.RFC3339, "invalid")
if err then
if errors.is(err, errors.INVALID) then
print("Invalid format:", err:message())
end
return nil, err
end
local loc, err = time.load_location("Unknown/Zone")
if err then
if errors.is(err, errors.NOT_FOUND) then
print("Location not found:", err:message())
end
return nil, err
end
```
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Channels and Coroutines"
## Channels and Coroutines
Channels exchange values between concurrent tasks. They can be buffered or unbuffered and can be combined with `channel.select` to coordinate multiple operations.
This is an API reference. The basic blocks are isolated snippets; the timeout, fan-in, and non-blocking sections are partial patterns whose named channels and callbacks come from the surrounding application. The worker-pool block is a complete in-process example.
The `channel` and `coroutine` globals are always available. Channels coordinate coroutines within one Lua process; use process messaging, functions, or queues across process boundaries.
### Creating Channels
An unbuffered channel (size 0) requires a sender and receiver to be ready before a transfer completes. A buffered channel allows sends to complete while buffer space is available.
```lua
-- Unbuffered: synchronizes sender and receiver
local sync_ch = channel.new()
-- Buffered: queue up to 10 messages
local work_queue = channel.new(10)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `size` | integer | Buffer capacity (default: 0 for unbuffered) |
**Returns:** `channel`
### Sending Values
Sending blocks until a receiver is ready on an unbuffered channel or until buffer space is available on a buffered channel.
```lua
-- Send work to a worker pool
local tasks = {"task-a", "task-b"}
local jobs = channel.new(100)
for i, task in ipairs(tasks) do
jobs:send(task) -- Blocks if buffer full
end
jobs:close() -- Signal no more work
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `value` | any | Value to send |
**Returns:** `boolean`
Sending to a closed channel raises an error.
### Receiving Values
Receiving blocks until a value is available or the channel is closed.
```lua
-- Worker consuming from job queue
while true do
local job, ok = jobs:receive()
if not ok then
break -- Channel closed, no more work
end
process(job)
end
```
Here, `jobs` is the application-provided queue and `process` is its task-processing callback.
**Returns:** `any, boolean`
- `value, true` — a value was received
- `nil, false` — the channel is closed and empty
### Closing Channels
Closing a channel causes pending senders to receive an error and pending receivers to receive `nil, false`. Closing an already closed channel is a no-op.
```lua
local results = channel.new(10)
-- Producer fills results
for _, item in ipairs(data) do
results:send(process(item))
end
results:close() -- Signal completion
```
This isolated producer snippet assumes `data` and the `process` callback are provided by the application.
### Selecting from Multiple Channels
`channel.select` waits on multiple channel operations at the same time. It can coordinate event sources, timeouts, and non-blocking checks.
```lua
local result = channel.select(cases)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `cases` | table | Array of select cases |
| `default` | boolean | If true, returns immediately when no case ready |
**Returns:** `table`
- For a channel case: `{channel, value, ok}` — `channel` is the case's channel, `value` is the received/sent value, `ok` is false for a closed-channel receive.
- For the default branch (when no case is ready and `default = true`): `{default = true, ok = true}`.
#### Timeout Pattern
Use `time.after()` to add a timeout to a channel wait.
```lua
local time = require("time")
local result_ch = application_response_channel
local timeout, err = time.after("5s")
if err then
return nil, err
end
local r = channel.select {
result_ch:case_receive(),
timeout:case_receive()
}
if r.channel == timeout then
return nil, errors.new({ kind = errors.TIMEOUT, message = "Operation timed out" })
end
return r.value
```
This partial pattern assumes the entry lists `time` in `modules:` and the application supplies `application_response_channel`. `time.after` returns one channel on success; invalid or non-positive durations return `nil, error`.
#### Fan-in Pattern
Handle values from multiple sources in one loop.
This process-entry pattern uses ambient `process`, while the application supplies the shutdown signal and the two handler functions.
```lua
local events = process.events()
local inbox = process.inbox()
local shutdown = channel.new()
while true do
local r = channel.select {
events:case_receive(),
inbox:case_receive(),
shutdown:case_receive()
}
if r.channel == shutdown then
break
elseif r.channel == events then
handle_event(r.value)
else
handle_message(r.value)
end
end
```
#### Non-Blocking Check
Use a default case to check for available data without blocking.
In this isolated pattern, `ch` and the `process` callback come from the application.
```lua
local r = channel.select {
ch:case_receive(),
default = true
}
if r.default then
-- Nothing available, do something else
elseif not r.ok then
-- The channel is closed
else
process(r.value)
end
```
### Creating Select Cases
Create send and receive cases for `channel.select`:
```lua
-- Send case - completes when channel can accept value
ch:case_send(value)
-- Receive case - completes when value available
ch:case_receive()
```
Values in the cases table that are not send or receive cases are ignored. Make sure the table contains at least one valid case unless it also has a default branch.
### Worker Pool Pattern
```lua
local items = {1, 2, 3, 4}
local num_workers = 2
local function process_item(item)
return item * 2
end
local work = channel.new(#items)
local results = channel.new(#items)
-- Spawn workers
for _ = 1, num_workers do
coroutine.spawn(function()
while true do
local item, ok = work:receive()
if not ok then
return
end
results:send(process_item(item))
end
end)
end
-- Feed work
for _, item in ipairs(items) do
work:send(item)
end
work:close()
-- Collect results
local processed = {}
while #processed < #items do
local result, ok = results:receive()
if not ok then break end
table.insert(processed, result)
end
```
After the loop, `processed` contains `2`, `4`, `6`, and `8`; result order depends on coroutine scheduling. The workers share channels because they are coroutines in the same Lua process.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Send on closed channel | runtime error | no |
| `cases` argument to select is not a table | runtime error | no |
### See Also
- [Process Management](lua/core/process.md) - Process spawning and communication
- [Message Queue](lua/storage/queue.md) - Queue-based messaging
- [Functions](lua/core/funcs.md) - Function invocation
---
# "Process Management"
## Process Management
The `process` global provides process spawning, messaging, monitoring, linking, naming, and lifecycle control.
It is available without `require()` and does not need to be listed in `modules:`.
This is an API reference. Its call-form blocks use placeholders such as `id`, `host`, `destination`, `topic`, and `name` for values supplied by application code; they are not standalone programs. Calls shown with an `err` result return their documented value on success or a failure sentinel plus `error`; the sentinel is normally `nil`, while `process.set_options` returns `false`. Application control flow should handle the error.
### Process Information
Read the current frame ID or process ID:
```lua
local frame_id, err = process.id() -- Registry ID of the current function, process, or workflow definition
if err then return nil, err end
local pid, err = process.pid() -- Process ID
if err then return nil, err end
```
### Sending Messages
Send one or more payload values to a process by PID or registered name:
```lua
local ok, err = process.send(destination, topic, ...)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `destination` | string | PID or registered name |
| `topic` | string | Topic name (cannot start with `@`) |
| `...` | any | Payload values |
**Permission:** `process.send` on target PID
### Spawning Processes
```lua
-- Basic spawn
local pid, err = process.spawn(id, host, ...)
-- With monitoring (receive EXIT events)
local pid, err = process.spawn_monitored(id, host, ...)
-- With linking (receive LINK_DOWN on abnormal exit)
local pid, err = process.spawn_linked(id, host, ...)
-- Both linked and monitored
local pid, err = process.spawn_linked_monitored(id, host, ...)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Process source ID (e.g., `"app.workers:handler"`) |
| `host` | string | Host ID (e.g., `"app:processes"`) |
| `...` | any | Arguments passed to spawned process |
All variants require `process.spawn` on the process ID. The monitored variants also require `process.spawn.monitored`, and the linked variants require `process.spawn.linked`. At runtime v0.3.32a, only the module-level `spawn()` checks `process.host` on the host ID; the specialized module-level variants do not perform that host permission check.
### Process Control
```lua
-- Forcefully terminate a process
local ok, err = process.terminate(destination)
-- Request graceful cancellation with an optional reason
local ok, err = process.cancel(destination, "shutting down")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `destination` | string | PID or registered name |
| `reason` | string | Optional reason delivered to the target |
**Permissions:** `process.terminate`, `process.cancel` on target PID
### Monitoring and Linking
Add or remove monitoring and links for an existing process:
```lua
-- Monitoring: receive EXIT events when target exits
local ok, err = process.monitor(destination)
local ok, err = process.unmonitor(destination)
-- Linking: bidirectional, receive LINK_DOWN on abnormal exit
local ok, err = process.link(destination)
local ok, err = process.unlink(destination)
```
**Permissions:** `process.monitor`, `process.unmonitor`, `process.link`, `process.unlink` on target PID
### Process Options
```lua
local options = process.get_options()
local ok, err = process.set_options({trap_links = true})
```
| Field | Type | Description |
|-------|------|-------------|
| `trap_links` | boolean | Whether LINK_DOWN events are delivered to events channel |
| `upgradable` | boolean | Opt in to OUTDATED events when the process's code is invalidated |
### Inbox and Events
Use the inbox and event channels to receive messages and lifecycle events:
```lua
local inbox = process.inbox() -- Message objects from @inbox topic
local events = process.events() -- Lifecycle events from @events topic
```
#### Event Types
| Constant | Description |
|----------|-------------|
| `process.event.CANCEL` | Cancellation requested |
| `process.event.EXIT` | Monitored process exited |
| `process.event.LINK_DOWN` | Linked process terminated abnormally |
| `process.event.OUTDATED` | The process's code or an imported dependency changed in the registry |
#### Event Fields
| Field | Type | Description |
|-------|------|-------------|
| `kind` | string | Event type constant |
| `from` | string | Source PID (absent for OUTDATED) |
| `result` | table | For EXIT/LINK_DOWN: a {value, error} record; the process return value is at `result.value` and any error at `result.error` |
| `reason` | string | For CANCEL: why the process is being cancelled |
| `sources` | string[] | For OUTDATED: registry IDs that changed or were transitively affected |
`OUTDATED` is delivered only to processes that opt in with `process.set_options({upgradable = true})`. Multiple invalidations are combined into one pending event containing the union of their `sources`. Handle the event by calling [`process.upgrade`](#process-upgrade).
### Topic Subscription
Subscribe to a custom message topic:
```lua
local ch, err = process.listen(topic, options)
if err then return nil, err end
local ok, err = process.unlisten(ch)
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `topic` | string | Topic name (cannot start with `@`) |
| `options.message` | boolean | If true, receive Message objects; if false, raw payloads |
### Message Objects
The inbox and listeners configured with `{message = true}` return message objects:
```lua
local msg = inbox:receive()
msg:topic() -- string: topic name
msg:from() -- string: sender PID (empty string when unknown)
msg:payload() -- Payload: wrapper (call :data() to extract); nil when empty, table of wrappers for several values
msg:payload():data() -- any: actual payload value
```
### Synchronous Call
`process.exec` spawns a process and waits for its result:
```lua
local result, err = process.exec(id, host, ...)
```
**Permissions:** `process.exec` on process id, `process.host` on host id
### Process Upgrade
Upgrade the current process while preserving its PID:
The two snippets below are alternative call forms, not sequential operations.
```lua
-- Upgrade to new version, passing state
process.upgrade(id, ...)
```
```lua
-- Keep same definition, re-run with new state
process.upgrade(nil, preserved_state)
```
`process.upgrade` is a terminal control transfer: it clears the current execution and starts the requested definition with the same PID. Code after the call does not run in the old execution.
### Context Spawner
Create a spawner that supplies custom context to child processes:
```lua
local spawner = process.with_context({request_id = "123"})
```
**Permission:** `process.context` on "context"
#### Spawner with Options
`process.with_options(options)` creates a spawner with spawn-time options, such as a network selector, rather than context values:
```lua
local spawner = process.with_options({network = "app:tor_proxy"})
```
| Option | Type | Description |
|--------|------|-------------|
| `network` | string | Registry ID of a `network.*` entry to use for the child's outbound connections |
| `terminal` | string | Viewport grant that attaches a virtual terminal to the child |
**Permission:** `process.context` on "context"; selecting a network additionally requires `network.select` on that network ID.
#### Terminal Attachment
A `terminal` grant comes from `viewport:grant()` and gives the child a terminal port of its own, so it can use the [TTY](lua/system/tty.md) module exactly as it would on a terminal host:
```lua
local view = assert(tty.viewport({width = 80, height = 24}))
local child = assert(process.with_options({terminal = assert(view:grant())})
:spawn_monitored("app:child", "app:workers"))
```
The grant is one-shot and is consumed at admission: a rejected start leaves it unresolved and reusable, a child that resolves the port consumes it permanently, and a host that does not support terminal attachments rejects the spawn rather than dropping the option. The spawning process keeps reading the child's frames through the viewport it created. See [Terminal](system/terminal.md#composable-terminals).
#### SpawnBuilder Methods
`SpawnBuilder` is immutable; each configuration method returns a new instance:
```lua
spawner:with_context(values) -- Add context values
spawner:with_actor(actor) -- Set security actor
spawner:with_scope(scope) -- Set security scope
spawner:with_name(name) -- Register name at start; if taken, spawn returns the existing PID and queued messages go to it
spawner:with_message(topic, ...) -- Queue message to send after spawn
spawner:with_options(options) -- Merge spawn-time options (e.g. network)
```
**Permission:** `process.security` on "security" for `:with_actor()` and `:with_scope()`
#### Spawner Spawn Methods
```lua
spawner:spawn(id, host, ...)
spawner:spawn_monitored(id, host, ...)
spawner:spawn_linked(id, host, ...)
spawner:spawn_linked_monitored(id, host, ...)
```
All `SpawnBuilder` spawn methods require `process.host` on the host ID in addition to the applicable `process.spawn`, `process.spawn.monitored`, and `process.spawn.linked` permissions.
#### Spawner Exec
```lua
local result, err = spawner:exec(id, host, ...)
```
This method runs the target process synchronously with the builder's context, actor, and scope, then returns its result. A deferred worker can use `with_actor` and `with_scope` to execute with an owner's identity.
**Permissions:** `process.exec` on process id, `process.host` on host id
### Name Registry
Register a process under a name so callers can use the name instead of its PID. Functions that accept a `destination`, including `send`, `terminate`, `cancel`, `monitor`, and `link`, also accept registered names.
```lua
local ok, err = process.registry.register(name) -- self, local scope
local pid, err = process.registry.lookup(name)
local ok, err = process.registry.unregister(name)
```
#### Scope
The optional `scope` argument selects the name's consistency guarantee and defaults to `LOCAL`. See the [Cluster Guide](guides/cluster.md#naming-and-name-scopes) for the complete model.
| Constant | Visibility | Guarantee |
|----------|------------|-----------|
| `process.registry.LOCAL` | this node only | Instant, node-local |
| `process.registry.EVENTUAL` | cluster-wide | Eventually consistent (gossip) |
| `process.registry.CONSISTENT` | cluster-wide | Linearizable singleton (Raft) |
| `process.registry.STRONG` | cluster-wide | Consistent + every live node acknowledges |
On a standalone node, only `LOCAL` is available; cluster scopes require [clustering](guides/cluster.md).
#### register
```lua
local ok, err = process.registry.register(name, pid, scope)
```
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `name` | string | yes | | Name to register |
| `pid` | string | no | self | PID to register; defaults to the calling process |
| `scope` | number | no | `LOCAL` | One of the scope constants above |
Returns `true` on success, or `nil, error` on failure. Conflicts (name already registered to a different PID) return `errors.ALREADY_EXISTS`. Registering the same name to the same PID is idempotent. A `STRONG` registration blocks until every live node acknowledges or the reservation deadline expires; on timeout it returns an error.
Registering on behalf of a different PID additionally requires the `process.registry.foreign` permission on the target PID.
#### lookup
```lua
local pid, err = process.registry.lookup(name)
```
Returns the registered PID string, or `nil, error` with kind `errors.NOT_FOUND` when the name is not registered.
#### unregister
```lua
local ok, err = process.registry.unregister(name, scope)
```
`scope` defaults to `LOCAL` and must match the scope the name was registered under. For `CONSISTENT` and `STRONG`, the owning process is the one allowed to unregister; unregistering a name owned by another PID returns `false`. Names also release automatically when the owning process exits (and, for cluster scopes, when its node leaves), so explicit unregister is for early release.
### Permissions
Permission checks evaluate the caller's security actor against the target resource.
#### Policy Evaluation
Policies can allow or deny an operation based on:
- **Actor**: The security principal making the request
- **Action**: The operation being performed (e.g., `process.send`)
- **Resource**: The target (PID, process id, host id, or name)
- **Attributes**: Additional context including `pid` (caller's process ID)
#### Permission Reference
| Permission | Functions | Resource |
|------------|-----------|----------|
| `process.spawn` | `spawn*()` | process id |
| `process.spawn.monitored` | `spawn_monitored()`, `spawn_linked_monitored()` | process id |
| `process.spawn.linked` | `spawn_linked()`, `spawn_linked_monitored()` | process id |
| `process.host` | module-level `spawn()`, all `SpawnBuilder` spawn methods, `exec()` | host id |
| `process.send` | `send()` | target PID |
| `process.exec` | `exec()` | process id |
| `process.terminate` | `terminate()` | target PID |
| `process.cancel` | `cancel()` | target PID |
| `process.monitor` | `monitor()` | target PID |
| `process.unmonitor` | `unmonitor()` | target PID |
| `process.link` | `link()` | target PID |
| `process.unlink` | `unlink()` | target PID |
| `process.context` | `with_context()`, `with_options()` | "context" |
| `process.security` | `:with_actor()`, `:with_scope()` | "security" |
| `process.registry.register` | `registry.register()` | name |
| `process.registry.unregister` | `registry.unregister()` | name |
| `process.registry.foreign` | `registry.register()` | target PID |
Cluster name scopes are authorized by scope-suffixed variants of these actions (`process.registry.register.eventual`, `.consistent`, `.strong`, and the matching `unregister` actions), so a policy can grant local naming separately from cluster-wide naming.
#### Multiple Permissions
Some operations require multiple permissions:
| Operation | Required Permissions |
|-----------|---------------------|
| `spawn()` | `process.spawn` + `process.host` |
| module-level `spawn_monitored()` | `process.spawn` + `process.spawn.monitored` |
| module-level `spawn_linked()` | `process.spawn` + `process.spawn.linked` |
| module-level `spawn_linked_monitored()` | `process.spawn` + `process.spawn.monitored` + `process.spawn.linked` |
| `SpawnBuilder:spawn()` | `process.spawn` + `process.host` |
| `SpawnBuilder:spawn_monitored()` | `process.spawn` + `process.spawn.monitored` + `process.host` |
| `SpawnBuilder:spawn_linked()` | `process.spawn` + `process.spawn.linked` + `process.host` |
| `SpawnBuilder:spawn_linked_monitored()` | `process.spawn` + `process.spawn.monitored` + `process.spawn.linked` + `process.host` |
| `exec()` | `process.exec` + `process.host` |
| spawn with custom actor/scope | spawn permissions + `process.security` |
### Errors
| Condition | Kind |
|-----------|------|
| No context found | `errors.INTERNAL` |
| Frame context not found | `errors.INTERNAL` |
| Missing required arguments | `errors.INVALID` |
| Reserved topic prefix (`@`) | `errors.INVALID` |
| Destination is neither a PID nor a registered name | `errors.NOT_FOUND` |
| Name not registered | `errors.NOT_FOUND` |
| Permission denied | `errors.PERMISSION_DENIED` |
| Name already registered | `errors.ALREADY_EXISTS` |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Channels](lua/core/channel.md) - In-process coroutine coordination
- [Message Queue](lua/storage/queue.md) - Queue-based messaging
- [Functions](lua/core/funcs.md) - Function invocation
- [Supervision](guides/supervision.md) - Process lifecycle management
- [Cluster](guides/cluster.md) - Name scopes and cluster-wide naming
---
# "Process Groups"
## Process Groups
Process groups organize processes under dynamic names and broadcast messages to group members across the cluster. A process can join multiple groups, and cluster-wide membership is eventually consistent.
This is an API reference. Its snippets assume an existing `pg.scope`, an executable entry running with process context, and policies that authorize the documented operations. The blocks demonstrate individual calls or partial subscription flows rather than a standalone application.
For the scope entry kind and its configuration, see [Process Groups](system/process-groups.md). For the broader clustering model, see the [Cluster Guide](guides/cluster.md).
### Loading
```lua
local pg = require("pg")
```
Add `pg` to the executable entry's `modules:` list before requiring it.
### Opening a Scope
A process group belongs to a **scope**, represented by a `pg.scope` registry entry. Open the scope to obtain an instance for group operations:
```lua
local group, err = pg.open("app:pg")
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Scope entry ID (format: `"namespace:name"`) |
**Returns:** `pg.Instance, error`
**Permission:** `pg.open` on the scope `id`
The instance is released automatically during execution-frame cleanup. Call `release()` to release it earlier. Other operations are methods on the instance and use `:` syntax.
### Joining and Leaving
The calls below are independent forms; choose the single-group or batch join needed by the application and pair it with the corresponding leave operations.
```lua
local ok, err = group:join("workers") -- single group
if err then return nil, err end
```
```lua
local ok, err = group:join({"workers", "all"}) -- batch
if err then return nil, err end
```
```lua
local ok, err = group:leave("workers")
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `group` | string \| string[] | Group name, or a list of names for a batch operation |
**Returns:** `boolean, error`
A process can join the same group more than once and must leave the same number of times to depart fully. For a batch, `leave` is best-effort and returns an error only when the process was not a member of any named group.
**Permissions:** `pg.join` / `pg.leave` on each group name
### Listing Members
```lua
local members, err = group:get_members("workers") -- all nodes
if err then return nil, err end
local local_members, err = group:get_local_members("workers") -- this node only
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `group` | string | Group name |
**Returns:** `string[], error` — an array of PID strings (empty for an unknown group)
**Permissions:** `pg.get_members` / `pg.get_local_members` on the group name
### Listing Groups
```lua
local groups, err = group:which_groups() -- all groups in the cluster
if err then return nil, err end
local local_groups, err = group:which_local_groups() -- groups with a local member
if err then return nil, err end
```
**Returns:** `string[], error` — group names that currently have at least one member
**Permissions:** `pg.which_groups` / `pg.which_local_groups`
### Broadcasting
Broadcast sends a message from the calling process to every group member under `topic`. Members receive it with `process.listen(topic)`.
```lua
local ok, err = group:broadcast("workers", "task", {id = 42}) -- all nodes
if err then return nil, err end
ok, err = group:broadcast_local("workers", "task", {id = 42}) -- this node only
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `group` | string | Target group |
| `topic` | string | Message topic |
| `...` | any | Zero or more payload values |
**Returns:** `boolean, error`
**Permissions:** `pg.broadcast` / `pg.broadcast_local` on the group name
### Monitoring a Group
`monitor` subscribes to join and leave events for one group and returns an atomic snapshot of its current members. No membership change can occur between the snapshot and subscription setup without being observed.
```lua
local sub, members, err = group:monitor("workers")
if err then
return nil, err
end
for _, pid in ipairs(members) do
-- current members at subscription time
end
local ch = sub:channel()
local event, open = ch:receive() -- {kind = "member.joined" | "member.left", path = "workers", data = {...}}
if not open then
return nil, errors.new("Process-group subscription closed")
end
sub:close() -- unsubscribe; sub:close({flush = true}) drains queued events first
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `group` | string | Group to watch |
**Returns:** `pg.Subscription, string[], error` — the subscription and a snapshot of current members
**Permission:** `pg.monitor` on the group name
### Watching All Groups
`events` subscribes to membership changes for every group in the scope and returns a snapshot mapping groups to their members.
```lua
local sub, snapshot, err = group:events()
if err then
return nil, err
end
-- snapshot: { ["workers"] = {pid, ...}, ["all"] = {pid, ...} }
local event, open = sub:channel():receive()
if not open then
return nil, errors.new("Process-group subscription closed")
end
sub:close()
```
**Returns:** `pg.Subscription, table, error`
**Permission:** `pg.events`
#### Event Fields
Events delivered on a subscription channel carry:
| Field | Type | Description |
|-------|------|-------------|
| `system` | string | Always `"pg"` |
| `kind` | string | `"member.joined"` or `"member.left"` |
| `path` | string | The group name |
| `data` | table | `{Group = string, PIDs = string[]}` — the affected members |
Subscription channels are buffered (capacity 64). If a slow consumer fills the buffer, further events are retained in the process mailbox in order and delivered once the consumer drains the channel (the subscription stalls rather than dropping events).
### Releasing
```lua
group:release()
```
`release` frees the instance immediately and is idempotent. After release, every other group operation returns an error. Cleanup also runs automatically at the end of the execution frame.
**Returns:** `boolean`
### Permissions
| Permission | Method | Resource |
|------------|--------|----------|
| `pg.open` | `pg.open()` | scope id |
| `pg.join` | `join()` | group name |
| `pg.leave` | `leave()` | group name |
| `pg.get_members` | `get_members()` | group name |
| `pg.get_local_members` | `get_local_members()` | group name |
| `pg.which_groups` | `which_groups()` | (none) |
| `pg.which_local_groups` | `which_local_groups()` | (none) |
| `pg.broadcast` | `broadcast()` | group name |
| `pg.broadcast_local` | `broadcast_local()` | group name |
| `pg.monitor` | `monitor()` | group name |
| `pg.events` | `events()` | (none) |
### Errors
| Condition | Kind |
|-----------|------|
| Permission denied | `errors.PERMISSION_DENIED` |
| Missing or empty argument | `errors.INVALID` |
| Scope not found | `errors.INTERNAL` |
| Leave a group with no membership | `errors.NOT_FOUND` |
| Instance released | `errors.INVALID` |
| Group/member or action-queue limit reached | `errors.RATE_LIMITED` (retryable) |
| Service stopped, backpressure, or open circuit | `errors.UNAVAILABLE` |
| Broadcast timed out | `errors.TIMEOUT` (retryable) |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Process Groups](system/process-groups.md) - Scope entry kind and configuration
- [Cluster](guides/cluster.md) - Membership, naming, and the clustering model
- [Process Management](lua/core/process.md) - Spawning and messaging individual processes
---
# "Function Invocation"
## Function Invocation
The `funcs` module calls registered functions synchronously or asynchronously. An executor can propagate request context, security identity, and implementation-specific call options. This page is an API reference; target IDs, arguments, and application data represent surrounding code.
### Loading
```lua
local funcs = require("funcs")
```
### `call`
Calls a registered function synchronously and waits for its result.
```lua
local result, err = funcs.call("app.api:get_user", user_id)
if err then
return nil, err
end
print(result.name)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `target` | string | Function ID in format "namespace:name" |
| `...args` | any | Arguments passed to the function |
**Returns:** `result, error`
The target uses the `namespace:name` format.
### `async`
Starts a function call and returns a `Future` immediately. Futures allow other work to continue while the call runs and support multiple concurrent calls.
```lua
-- Start heavy computation without blocking
local future, err = funcs.async("app.process:analyze_data", large_dataset)
if err then
return nil, err
end
-- Do other work while computation runs...
-- Wait for result when ready
local ch = future:response()
local _, open = ch:receive()
if not open then
return nil, errors.new("future response channel closed")
end
local payload, result_err = future:result()
if result_err then
return nil, result_err
end
local result, data_err = payload:data()
if data_err then return nil, data_err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `target` | string | Function ID in format "namespace:name" |
| `...args` | any | Arguments passed to the function |
**Returns:** `Future, error`
### `new`
Creates an `Executor` for calls that need custom context, security identity, or call options.
```lua
local exec = funcs.new()
```
**Returns:** `Executor`
### Executor
An executor stores call context and options. Its configuration methods return new executor instances, allowing a base configuration to be reused.
#### `with_context`
Adds request-scoped values that will be available to the called function, such as trace IDs, session data, or feature flags.
```lua
local ctx = require("ctx")
-- Propagate request context to downstream services
local request_id, ctx_err = ctx.get("request_id")
if ctx_err then return nil, ctx_err end
local exec, err = funcs.new():with_context({
request_id = request_id,
feature_flags = {dark_mode = true}
})
if err then return nil, err end
local user, err = exec:call("app.api:get_user", user_id)
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `values` | table | Key-value pairs to add to context |
**Returns:** `Executor, error`
#### `with_actor`
Sets the security actor used for authorization checks in the called function.
```lua
local security = require("security")
local actor = security.actor() -- Get current user's actor
-- Call admin function with user's credentials
local exec, err = funcs.new():with_actor(actor)
if err then return nil, err end
local result, err = exec:call("app.admin:delete_record", record_id)
if err and err:kind() == errors.PERMISSION_DENIED then
return nil, errors.new({kind = errors.PERMISSION_DENIED, message = "User cannot delete records"})
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `actor` | Actor | Security actor (from security module) |
**Returns:** `Executor, error`
#### `with_scope`
Sets the security scope for called functions. The scope defines the permissions available to the call.
```lua
local security = require("security")
local scope = security.new_scope()
local exec, err = funcs.new():with_scope(scope)
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `scope` | Scope | Security scope (from security module) |
**Returns:** `Executor, error`
#### `with_options`
Sets call options such as the retry policy or the overlay network. Options are merged over any preset options of the target function entry.
```lua
-- Retry transient failures up to 5 times with exponential backoff
local exec = funcs.new():with_options({
retry = { max_attempts = 5, initial_delay = 100 }
})
local result, err = exec:call("app.external:fetch_data", query)
if err then
-- All attempts failed, or the error was not retryable
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `options` | table | Call options |
| Option | Type | Description |
|--------|------|-------------|
| `retry.max_attempts` | int | Maximum attempts including the first (1 disables retry) |
| `retry.initial_delay` | int/duration | Delay before first retry (ms or duration string), default `100` |
| `retry.max_delay` | int/duration | Upper bound for the backoff delay (ms or duration string), default `10s` |
| `retry.backoff_factor` | number | Multiplier applied to the delay after each attempt, default `2.0` |
| `retry.jitter` | number | Random jitter fraction applied to each delay, default `0.1` |
| `retry.retry_kinds` | string[] | Only retry errors of these kinds; by default every kind except `Invalid`, `PermissionDenied` and `Internal` is retried |
| `retry.skip_kinds` | string[] | Never retry errors of these kinds |
| `network` | string | Registry ID of an overlay network to route the call's outbound traffic through; requires the `network.select` permission |
Only retryable errors trigger retries; non-retryable errors surface immediately. Temporal activity options are described in [Activities](temporal/activities.md).
The runtime-defined option is:
| Recognized option | Type | Description |
|-------------------|------|-------------|
| `network` | string | Registry ID of the outbound `network.*` entry |
**Returns:** `Executor, error`
Selecting a network requires `network.select` permission on that network ID.
#### `call` and `async`
The executor versions of `call` and `async` use its configured context and options.
```lua
-- Build reusable executor with context
local exec = funcs.new()
:with_context({trace_id = "abc-123"})
:with_options({retry = {max_attempts = 3}})
-- Make multiple calls with same context
local users, users_err = exec:call("app.api:list_users")
if users_err then return nil, users_err end
local posts, posts_err = exec:call("app.api:list_posts")
if posts_err then return nil, posts_err end
```
### Future Invocation Summary
`async()` returns a future representing an in-progress invocation. The methods below cover the caller-facing steps for receiving, inspecting, or canceling that invocation. See [Future](./future.md) for the Future object reference.
#### `response` and `channel`
Returns the channel used to receive the result.
```lua
local time = require("time")
local future, err = funcs.async("app.api:slow_operation", data)
if err then
return nil, err
end
local ch = future:response() -- or future:channel()
local timeout, err = time.after("5s")
if err then
return nil, err
end
local result = channel.select {
ch:case_receive(),
timeout:case_receive()
}
```
**Returns:** `Channel`
The response channel signals completion. After it becomes ready, call `future:result()` to obtain the cached value or the called function's error.
#### `is_complete`
Checks whether the future has completed without blocking.
```lua
while not future:is_complete() do
-- do other work
local _, sleep_err = time.sleep("100ms")
if sleep_err then return nil, sleep_err end
end
local result, err = future:result()
```
**Returns:** `boolean`
#### `is_canceled`
Returns `true` if the future has been marked canceled by its provider. See the cancellation limitation below.
```lua
if future:is_canceled() then
print("Operation was canceled")
end
```
**Returns:** `boolean`
#### `result`
Returns the cached result when complete or `nil` while the operation is pending.
```lua
local value, err = future:result()
if err then
print("Failed:", err:message())
elseif value then
local data, data_err = value:data()
if data_err then return nil, data_err end
print("Got:", data)
end
```
**Returns:** `Payload|table|nil, error|nil`
#### `error`
Returns the operation error when the future has failed.
```lua
local err, has_error = future:error()
if has_error then
print("Error kind:", err:kind())
end
```
**Returns:** `error|nil, boolean`
This method returns a non-retryable `INTERNAL` wrapper for a failed operation. Use `result()` to preserve the called function's original error metadata.
#### `cancel`
Requests cancellation of the asynchronous operation.
```lua
local canceled, err = future:cancel()
if err then return nil, err end
```
**Returns:** `boolean, error`
In runtime v0.3.32a, function and contract futures share one process-global cancellation callback. When both providers are loaded, cancel() and is_canceled() are not a stable cross-provider contract. Do not use cancellation for application correctness; time out locally and ignore a late result until the runtime separates provider cancellation.
### Parallel Operations
Combine `async` with `channel.select` to run and collect multiple calls concurrently.
```lua
-- Start multiple operations in parallel
local f1, err = funcs.async("app.api:get_user", user_id)
if err then return nil, err end
local f2, err = funcs.async("app.api:get_orders", user_id)
if err then return nil, err end
local f3, err = funcs.async("app.api:get_preferences", user_id)
if err then return nil, err end
-- Wait for all to complete using channels
local user_ch = f1:channel()
local orders_ch = f2:channel()
local prefs_ch = f3:channel()
local pending = {
[user_ch] = {name = "user", future = f1},
[orders_ch] = {name = "orders", future = f2},
[prefs_ch] = {name = "preferences", future = f3}
}
local results = {}
while next(pending) do
local cases = {}
for ch in pairs(pending) do
cases[#cases + 1] = ch:case_receive()
end
local r = channel.select(cases)
local completed = pending[r.channel]
pending[r.channel] = nil
local payload, result_err = completed.future:result()
if result_err then
return nil, result_err
end
local data, data_err = payload:data()
if data_err then
return nil, data_err
end
results[completed.name] = data
end
```
### Permissions
Function operations are subject to security policy evaluation.
| Action | Resource | Description |
|--------|----------|-------------|
| `funcs.call` | Function ID | Call a specific function |
| `funcs.context` | `context` | Use `with_context()` to set custom context |
| `funcs.security` | `security` | Use `with_actor()` or `with_scope()` |
| `network.select` | Network ID | Use `with_options({network = ...})` to select an overlay network |
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Target empty | `errors.INVALID` | no |
| Namespace missing | `errors.INVALID` | no |
| Name missing | `errors.INVALID` | no |
| Permission denied | `errors.PERMISSION_DENIED` | no |
| Async outside a process | `errors.INTERNAL` | no |
| Subscribe failed | `errors.INTERNAL` | no |
| Async start dispatch failed | `errors.INTERNAL` | no |
| Function error | varies | varies |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Futures"
## Futures
Futures represent asynchronous operation results. They are returned by `funcs.async()` and asynchronous contract calls. This page is an API reference; the target IDs and arguments in its patterns are application-defined.
### Loading
Futures are not loaded as a module; asynchronous operations create them:
```lua
local funcs = require("funcs")
local future, err = funcs.async("app.compute:task", data)
if err then
return nil, err
end
```
### Response Channel
Use the response channel to wait for completion, then read the cached result from the future:
```lua
local ch = future:response()
local _, open = ch:receive()
if not open then
return nil, errors.new("future response channel closed")
end
local payload, err = future:result()
if err then
return nil, err
end
local result, data_err = payload:data()
if data_err then return nil, data_err end
```
`channel()` is an alias for `response()`.
The channel value is the operation's payload, payload table, or error. Calling `result()` after the channel becomes ready provides one consistent success/error interface and returns the cached value even after the channel is drained.
### Completion Check
Check whether the future has completed without blocking:
```lua
if future:is_complete() then
local result, err = future:result()
end
```
### Cancellation Check
Check whether the future has been marked canceled by its provider:
```lua
if future:is_canceled() then
print("Operation was canceled")
end
```
### Getting Result
Read the cached result without blocking:
```lua
local val, err = future:result()
```
**Returns:**
- Not complete: `nil, nil`
- Canceled: `nil, error` (kind `CANCELED`)
- Error: `nil, error`
- Success: `Payload, nil` or `table, nil` (multiple payloads)
### Getting Error
Read the error when the future has failed:
```lua
local err, has_error = future:error()
if has_error then
print("Failed:", err:message())
end
```
**Returns:** `error, boolean`
When an operation fails, `error()` returns a non-retryable `INTERNAL` wrapper. Use `result()` when the called function's original error kind and retryability must be preserved.
### Canceling
Request cancellation of the asynchronous operation on a best-effort basis:
```lua
local canceled, err = future:cancel()
```
**Returns:** `boolean, error`
Operation may still complete if already in progress.
### Timeout Pattern
```lua
local time = require("time")
local future, err = funcs.async("app.compute:slow", data)
if err then
return nil, err
end
local timeout, err = time.after("5s")
if err then
return nil, err
end
local r = channel.select {
future:channel():case_receive(),
timeout:case_receive()
}
if r.channel == timeout then
future:cancel()
return nil, errors.new({ kind = errors.TIMEOUT, message = "Operation timed out" })
end
local payload, result_err = future:result()
if result_err then
return nil, result_err
end
local value, data_err = payload:data()
if data_err then return nil, data_err end
return value
```
### First-to-Complete
```lua
local f1, err = funcs.async("app.cache:get", key)
if err then
return nil, err
end
local f2, err = funcs.async("app.db:get", key)
if err then
return nil, err
end
local ch1 = f1:channel()
local ch2 = f2:channel()
local r = channel.select {
ch1:case_receive(),
ch2:case_receive()
}
-- The slower operation may still complete; this caller ignores its result.
local winner
if r.channel == ch1 then
winner = f1
else
winner = f2
end
local payload, result_err = winner:result()
if result_err then
return nil, result_err
end
local value, data_err = payload:data()
if data_err then return nil, data_err end
return value
```
### Errors
| Condition | Kind |
|-----------|------|
| Operation canceled | `CANCELED` |
| Async operation failed | `result()` preserves the operation's kind; `error()` reports `INTERNAL` |
---
# "Streams"
## Streams
Streams provide incremental I/O for HTTP, filesystem, and other modules. The modules that own the underlying data create stream objects. This page is an API reference; the scanner loop uses an application-defined `process(token)` callback.
### Obtaining a Stream
```lua
-- From HTTP request body
local stream, err = req:stream()
if err then return nil, err end
-- From filesystem
local fs = require("fs")
local volume, err = fs.get("app:data")
if err then return nil, err end
local stream, err = volume:open("/file.txt", "r")
if err then return nil, err end
```
### Reading
```lua
local chunk, err = stream:read(size)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `size` | integer | Bytes to read (0 = default 32KB chunk) |
**Returns:** `string, error` — `nil, nil` on EOF
### Writing
```lua
local bytes, err = stream:write(data)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to write |
**Returns:** `integer, error` — bytes written
### Seeking
```lua
local pos, err = stream:seek(whence, offset)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `whence` | string | `"set"`, `"cur"`, or `"end"` |
| `offset` | integer | Offset in bytes |
**Returns:** `integer, error` — new position
### Flushing
```lua
local ok, err = stream:flush()
```
`flush` writes buffered data to the underlying destination.
### Stream Information
```lua
local info, err = stream:stat()
```
| Field | Type | Description |
|-------|------|-------------|
| `size` | integer | Total size (-1 if unknown) |
| `position` | integer | Current position |
| `readable` | boolean | Can read |
| `writable` | boolean | Can write |
| `seekable` | boolean | Can seek |
### Closing
```lua
local ok, err = stream:close()
```
`close` releases the stream's resources and can be called more than once.
### Scanner
Create a scanner that tokenizes stream content:
```lua
local scanner, err = stream:scanner(split)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `split` | string | `"lines"`, `"words"`, `"bytes"`, `"runes"` |
#### Scanner Methods
```lua
local has_more, err = scanner:scan() -- advance to next token
local token = scanner:text() -- current token
local err_msg = scanner:err() -- scanner error if any
```
```lua
while true do
local has_token, err = scanner:scan()
if err then return nil, err end
if not has_token then
local scan_err = scanner:err()
if scan_err then return nil, scan_err end -- raw scanner error string
break -- clean EOF
end
process(scanner:text())
end
```
When `scan()` returns `false`, check `scanner:err()` before treating the result
as EOF. Tokenization and underlying read failures are stored on the scanner and
do not appear in `scan()`'s second return value.
### Errors
| Condition | Kind |
|-----------|------|
| Invalid whence/split type | raised as a Lua error (not returned) |
| Stream closed | `INTERNAL` |
| Not readable/writable | `INTERNAL` |
| Read/write failure | `INTERNAL` |
---
# "Request Context"
## Request Context
The `ctx` module reads request-scoped values propagated through [function calls](lua/core/funcs.md) or [process operations](lua/core/process.md). This page is an API reference; the snippets show individual calls inside an executable Lua entry.
### Loading
```lua
local ctx = require("ctx")
```
#### Get a Value
```lua
local value, err = ctx.get("key")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Context key |
**Returns:** `any, error`
#### Get All Values
```lua
local values, err = ctx.all()
```
**Returns:** `table, error`
`ctx.all()` returns an empty table when an execution context is present but has no request values. A missing execution context returns `nil, errors.INTERNAL`.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty key | `errors.INVALID` | no |
| Key not found | `errors.NOT_FOUND` | no |
| No execution context available | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Event Bus"
## Event Bus
The event bus publishes runtime and application activity for monitoring, logging, metrics, and reactive side effects. This page is an API reference; the snippets assume an executable Lua entry with the listed module and permissions.
The event bus is a best-effort publish/subscribe channel, not a reliable transport. Do not depend on it for business-critical delivery. Use process messaging (`process.send`), channels, or the [message queue](lua/storage/queue.md) when delivery is part of application correctness.
### Loading
```lua
local events = require("events")
```
### Subscribing to Events
Subscribe to one system or a system pattern, with an optional event-kind filter:
```lua
-- Subscribe to all order events
local sub, err = events.subscribe("orders.*")
if err then
return nil, err
end
-- Process events
local ch = sub:channel()
while true do
local evt, ok = ch:receive()
if not ok then break end
print(evt.system, evt.kind, evt.path)
-- Process evt.data when the publisher supplied a payload.
end
```
Pass a second argument to restrict delivery to one kind, for example
`events.subscribe("users", "user.created")`. An omitted kind accepts every
kind from the matching system.
| Parameter | Type | Description |
|-----------|------|-------------|
| `system` | string | System pattern (supports wildcards like "test.*") |
| `kind` | string | Event kind filter (optional) |
**Returns:** `Subscription, error`
### Publishing Events
Publish an event to the event bus:
```lua
-- Send order created event
local ok, err = events.send("orders", "order.created", "/orders/123", {
order_id = "123",
customer_id = "456",
total = 99.99
})
if err then
return nil, err
end
-- Send without data
local heartbeat_sent, heartbeat_err = events.send("system", "heartbeat", "/health")
if heartbeat_err then
return nil, heartbeat_err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `system` | string | System identifier |
| `kind` | string | Event kind/type |
| `path` | string | Event path for routing |
| `data` | any | Event payload (optional) |
**Returns:** `boolean, error`
A successful return confirms that the runtime accepted the send. It does not
confirm that any subscriber received or processed the event.
#### Receive Channel
Use the subscription channel to receive events:
```lua
local json = require("json")
local ch = sub:channel()
local evt, ok = ch:receive()
if ok then
print("System:", evt.system)
print("Kind:", evt.kind)
print("Path:", evt.path)
local encoded, encode_err = json.encode(evt.data)
if encode_err then return nil, encode_err end
print("Data:", encoded)
end
```
Each event contains `system`, `kind`, and `path`. The `data` field is present
only when the publisher supplied a non-nil payload.
#### Close a Subscription
Close the subscription to unsubscribe and close its channel:
```lua
local closed = sub:close() -- true
```
Closing is idempotent. After the channel is closed, `receive()` returns
`nil, false` once buffered events are drained.
### Permissions
| Action | Resource | Description |
|--------|----------|-------------|
| `events.subscribe` | system | Subscribe to events from a system |
| `events.send` | system | Send events to a system |
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty system | `errors.INVALID` | no |
| Empty send kind | `errors.INVALID` | no |
| Empty path | `errors.INVALID` | no |
| Policy denied | `errors.INVALID` | no |
| Missing execution or process context | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Entry Registry"
## Entry Registry
Query and modify registered entries. Access metadata, snapshots, and version history.
### Loading
```lua
local registry = require("registry")
```
### Entry Structure
```lua
{
id = "app.lib:assert", -- string: "namespace:name"
kind = "function.lua", -- string: entry type
meta = {type = "test"}, -- table: searchable metadata
data = {...} -- any: entry payload
}
```
Entries read back from `registry.get`, `registry.find`, `snap:entries()`, `snap:get()`, `snap:namespace()` and `snap:find()` carry only these four author-facing fields.
`dependency_root` is a write-side field accepted by `changes:create()` and `changes:update()`. It is a boolean that marks an `ns.dependency` entry as a deployment root. It is never returned by the entry APIs; registry-owned state is read through [`snap:state()`](lua/core/registry.md#snapshot-state).
### Get Entry
```lua
local entry, err = registry.get("app.lib:assert")
```
**Permission:** `registry.get` on entry ID
### Find Entries
```lua
local entries, err = registry.find({[".kind"] = "function.lua"})
local entries, err = registry.find({[".kind"] = "http.endpoint", [".ns"] = "app.api"})
```
Keys prefixed with `.` match entry fields (`.kind`, `.ns`, `.name`, `.id`) and accept `*` globs. Keys prefixed with `meta.` match entry metadata; a leading `~`, `*`, `^` or `$` on a `meta.` key selects regex, contains, prefix or suffix matching. Keys with neither prefix are ignored.
### Parse ID
```lua
local id = registry.parse_id("app.lib:assert")
-- id.ns = "app.lib", id.name = "assert"
```
### Snapshots
Point-in-time view of the registry:
```lua
local snap, err = registry.snapshot() -- current state
local snap, err = registry.snapshot_at(5) -- at version 5
```
#### Snapshot Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `snap:entries()` | `Entry[], error` | All accessible entries |
| `snap:state()` | `State, error` | Entries with registry-owned metadata, plus the resolved module graph |
| `snap:get(id)` | `Entry, error` | Single entry by ID |
| `snap:find(filter)` | `Entry[]` | Filter entries |
| `snap:namespace(ns)` | `Entry[]` | Entries in namespace |
| `snap:version()` | `Version` | Snapshot version |
| `snap:changes()` | `Changes` | Create changeset |
#### Snapshot State
`snap:state()` returns the entry state together with the module graph selected for the snapshot version. Registry-owned provenance is carried on each entry rather than merged into `meta`, so it cannot be confused with authored metadata.
```lua
local snap, err = registry.snapshot()
local state, err = snap:state()
for _, entry in ipairs(state.entries) do
print(entry.id, entry.registry.owner, entry.registry.root)
end
if state.resolution then
print(state.resolution.digest, state.resolution.input_digest)
for _, module in ipairs(state.resolution.modules) do
print(module.name, module.version)
end
end
```
Each entry in `state.entries` has the four author-facing fields plus:
- `registry.owner` - deployment source that supplied the entry
- `registry.root` - `true` when the entry is a dependency declaration selected by the deployment
`state.resolution` describes the module graph of a `registry.snapshot()` view. It is absent on snapshots that carry no graph of their own, including `registry.snapshot_at()` and overlay snapshots:
| Field | Type | Description |
|-------|------|-------------|
| `digest` | string | Content digest of the complete immutable selection |
| `input_digest` | string | Digest of the declared root set |
| `baseline_digest` | string | Digest of the deployment baseline the graph was solved against; omitted when unbound |
| `roots` | array | Authored dependency declarations used as solver inputs |
| `references` | array | Root-shaped declarations folded into an existing root for the same component; omitted when empty |
| `modules` | array | Selected modules |
`roots` and `references` entries have `id`, `component` and `version`. `modules` entries have `name` and `version`, plus `version_id`, `source`, `digest`, `size_bytes` and `protected` when set.
### Versions
```lua
local version, err = registry.current_version()
local versions, err = registry.versions()
print(version:id()) -- numeric ID
print(version:string()) -- display string
local prev = version:previous() -- previous version or nil
local next = version:next() -- next version or nil
```
### History
```lua
local hist, err = registry.history()
local versions, err = hist:versions()
local version, err = hist:get_version(5)
local snap, err = hist:snapshot_at(version)
```
### Changesets
Build and apply modifications:
```lua
local snap, err = registry.snapshot()
local changes = snap:changes()
changes:create({
id = "test:new_entry",
kind = "test.kind",
meta = {type = "test"},
data = {config = "value"}
})
changes:update({
id = "test:existing",
kind = "test.kind",
meta = {updated = true},
data = {new_value = true}
})
changes:delete("test:old_entry")
local new_version, err = changes:apply()
```
**Permission:** `registry.apply` for `changes:apply()`
#### Deleting Entries
`changes:delete()` accepts an ID string, a table with an `id` string, a table with `ns` and `name` strings, or an array of any of those. Arrays may nest, and duplicate IDs collapse into a single delete operation.
```lua
changes:delete("test:old_entry")
changes:delete({id = "test:old_entry"})
changes:delete({ns = "test", name = "old_entry"})
changes:delete({"test:a", {ns = "test", name = "b"}, {"test:c"}})
```
An empty list, a table that references itself, and a value that is neither a string nor a table are rejected with `errors.INVALID`.
#### Changes Methods
| Method | Description |
|--------|-------------|
| `changes:create(entry)` | Add create operation |
| `changes:update(entry)` | Add update operation |
| `changes:delete(id)` | Add delete operation |
| `changes:ops()` | Get pending operations |
| `changes:apply()` | Apply changes, returns new Version |
### Apply Version
Roll back or forward to a specific version:
```lua
local prev = current_version:previous()
local ok, err = registry.apply_version(prev)
```
**Permission:** `registry.apply_version`
### Build Delta
Compute operations to transition between states:
```lua
local from = {{id = "test:a", kind = "test", meta = {}, data = {}}}
local to = {{id = "test:b", kind = "test", meta = {}, data = {}}}
local ops, err = registry.build_delta(from, to)
for _, op in ipairs(ops) do
print(op.kind, op.entry.id) -- "entry.create", "entry.update", "entry.delete"
end
```
### Overlays
An overlay is a process-local set of registry entries owned by a logical identity. Overlay entries take part in ordinary topology and handler transitions, so services start and stop for them exactly as for durable entries, but they never advance registry history and never appear in a version. They exist only in the running process and are empty after a cold boot, so the owning control service reconciles them on startup.
```lua
local snap, err = registry.overlay("data-sources:crm")
```
**Returns:** `Snapshot, error`
The snapshot exposes the owner's overlay entries through the usual methods and reports the current registry version from `snap:version()`. It also captures the overlay generation at the moment it is opened, which is what makes writes safe.
```lua
local snap, err = registry.overlay("data-sources:crm")
if err then return nil, err end
local changes = snap:changes()
changes:create({
id = "data.crm:connection",
kind = "registry.entry",
meta = {},
data = {endpoint = "https://crm.internal"}
})
local version, err = changes:apply()
```
`changes:apply()` on an overlay snapshot writes the overlay and returns the current registry version. No history version is created, so the returned version is unchanged unless a durable change happened concurrently.
#### Concurrency
Each overlay carries a generation counter that increases on every successful apply. `changes:apply()` succeeds only if the generation still matches the one captured when the snapshot was opened. A concurrent apply to the same overlay fails with `errors.CONFLICT` marked retryable: reopen the overlay and rebuild the changeset.
```lua
local last_err
for _ = 1, 3 do
local snap, err = registry.overlay("data-sources:crm")
if err then return nil, err end
local _, apply_err = snap:changes():delete("data.crm:connection"):apply()
if not apply_err then return true end
if not apply_err:retryable() then return nil, apply_err end
last_err = apply_err
end
return nil, last_err
```
#### Restrictions
- The owner string is required and must not be blank.
- A changeset must be non-empty and must not name the same entry twice.
- `create` fails when the ID already exists in durable state or in any overlay.
- `update` and `delete` only work on entries this owner created; any other ID fails with `errors.NOT_FOUND`.
- Overlay entries cannot set `dependency_root` or any other registry-owned metadata.
- Overlay entries cannot use kinds owned by a registry directive, such as `ns.dependency`.
- A delete that removes an entry a surviving entry depends on is rejected.
- Dependencies cannot cross overlay owner boundaries, and durable entries cannot depend on overlay entries.
The rest surface as `errors.CONFLICT` or `errors.INVALID`, and none are retryable: only the generation mismatch above is.
**Permissions:** `registry.overlay.get` on the owner to open and read, `registry.overlay.apply` on the owner to write, and `registry.overlay..` on each entry ID in the changeset.
### Permissions
| Permission | Resource | Description |
|------------|----------|-------------|
| `registry.get` | entry ID | Read entry (also filters find/entries results) |
| `registry.apply` | - | Apply changeset |
| `registry.apply_version` | - | Apply/rollback version |
| `registry.overlay.get` | owner ID | Open and read an overlay snapshot |
| `registry.overlay.apply` | owner ID | Apply an overlay changeset |
| `registry.overlay.create.` | entry ID | Create an overlay entry of that kind |
| `registry.overlay.update.` | entry ID | Update an overlay entry of that kind |
| `registry.overlay.delete.` | entry ID | Delete an overlay entry of that kind |
### Errors
| Condition | Kind |
|-----------|------|
| Entry not found | `errors.NOT_FOUND` |
| Version not found | `errors.NOT_FOUND` |
| Permission denied | `errors.PERMISSION_DENIED` |
| Invalid parameter | `errors.INVALID` |
| No changes to apply | `errors.INVALID` |
| Overlay changed during apply | `errors.CONFLICT` (retryable) |
| Overlay entry owned elsewhere or conflicts with durable state | `errors.CONFLICT` |
| Registry not available | `errors.INTERNAL` |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Contracts"
## Contracts
The `contract` module opens typed service bindings for remote APIs, workflows, and functions. Contracts support schema validation, asynchronous calls, and call-context propagation. This page is an API reference; IDs and values such as `current_user` represent application-owned entries and surrounding handler state.
### Loading
```lua
local contract = require("contract")
```
### Opening a Binding
Open a binding by its registry ID:
```lua
local greeter, err = contract.open("app.services:greeter")
if err then
return nil, err
end
local result, err = greeter:say_hello("Alice")
if err then
return nil, err
end
```
Bindings can also receive scope values, query parameters, or call options:
```lua
-- With scope table
local svc, err = contract.open("app.services:user", {
tenant_id = "acme",
region = "us-east"
})
-- With query parameters (auto-converted: "true"→bool, numbers→int/float)
local api, err = contract.open("app.services:api?debug=true&timeout=5000")
-- With call options (third argument)
local inst, err = contract.open("app.services:flaky", nil, {
retry = { max_attempts = 5, initial_delay = 100 }
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `binding_id` | string | Binding ID; query parameters are supported |
| `scope` | table | Context values (optional, overrides query params) |
| `options` | table | Call options (optional) — e.g. `retry.max_attempts`, `retry.initial_delay` |
**Returns:** `Instance, error`
### Getting a Contract
Retrieve a contract definition for introspection:
```lua
local c, err = contract.get("app.services:greeter")
if err then
return nil, err
end
print(c:id()) -- "app.services:greeter"
local methods = c:methods()
for _, m in ipairs(methods) do
print(m.name, m.description)
end
local method, err = c:method("say_hello")
if err then
return nil, err
end
```
#### Method Definition
| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Method name |
| `description` | string | Method description |
| `input_schemas` | table[] | Input schema definitions (absent when the method declares none) |
| `output_schemas` | table[] | Output schema definitions (absent when the method declares none) |
### Finding Implementations
List the bindings that implement a contract:
```lua
local bindings, err = contract.find_implementations("app.services:greeter")
if err then
return nil, err
end
for _, binding_id in ipairs(bindings) do
print(binding_id)
end
```
The same lookup is available on a contract object:
```lua
local c, err = contract.get("app.services:greeter")
if err then
return nil, err
end
local bindings, err = c:implementations()
if err then
return nil, err
end
```
### Checking Implementation
Check whether an already opened instance implements a contract:
```lua
if contract.is(instance, "app.services:greeter") then
instance:say_hello("World")
end
```
### Calling Methods
A synchronous method call blocks until it completes:
```lua
local calc, err = contract.open("app.services:calculator")
if err then
return nil, err
end
local sum, err = calc:add(10, 20)
if err then
return nil, err
end
local product, err = calc:multiply(5, 6)
if err then
return nil, err
end
```
### Async Calls
Append `_async` to a method name to start it asynchronously:
```lua
local processor, err = contract.open("app.services:processor")
if err then
return nil, err
end
local future, err = processor:process_async(large_dataset)
if err then
return nil, err
end
-- Do other work...
-- Wait for result
local ch = future:response()
local _, open = ch:receive()
if not open then
return nil, errors.new("future response channel closed")
end
local payload, result_err = future:result()
if result_err then return nil, result_err end
local result, data_err = payload:data()
if data_err then return nil, data_err end
```
See [Futures](lua/core/future.md) for future methods.
### Opening via Contract
Open a binding through a contract object. The calls below are alternatives; check the error returned by `contract.get()` and by the selected `open()` call before using the instance.
```lua
local c, err = contract.get("app.services:user")
if err then
return nil, err
end
-- Default binding
local instance, err = c:open()
-- Specific binding
local instance, err = c:open("app.services:user_impl")
-- With scope
local instance, err = c:open(nil, {user_id = 123})
local instance, err = c:open("app.services:user_impl", {user_id = 123})
```
### Adding Context
Create a wrapper with preconfigured context values:
```lua
local ctx = require("ctx")
local c, err = contract.get("app.services:user")
if err then return nil, err end
local request_id, ctx_err = ctx.get("request_id")
if ctx_err then return nil, ctx_err end
local wrapped, err = c:with_context({
request_id = request_id,
user_id = current_user.id
})
if err then return nil, err end
local instance, err = wrapped:open()
```
### Call Options
Use `with_options` to configure retries and other call behavior:
```lua
local c, err = contract.get("app.services:flaky")
if err then return nil, err end
local configured = c:with_options({
retry = { max_attempts = 5, initial_delay = 100 }
})
local inst, err = configured:open("app.services:flaky_impl")
if err then return nil, err end
local result, err = inst:call()
```
Options apply to every method call on the returned instance. Only retryable errors trigger retries; non-retryable errors return immediately. `with_options` can be chained with `with_context`, `with_actor`, and `with_scope`.
| Option | Type | Description |
|--------|------|-------------|
| `retry.max_attempts` | int | Maximum attempts including the first (1 disables retry) |
| `retry.initial_delay` | int/duration | Delay before first retry (ms or duration string), default `100` |
| `retry.max_delay` | int/duration | Upper bound for the backoff delay (ms or duration string), default `10s` |
| `retry.backoff_factor` | number | Multiplier applied to the delay after each attempt, default `2.0` |
| `retry.jitter` | number | Random jitter fraction applied to each delay, default `0.1` |
| `retry.retry_kinds` | string[] | Only retry errors of these kinds; by default every kind except `Invalid`, `PermissionDenied` and `Internal` is retried |
| `retry.skip_kinds` | string[] | Never retry errors of these kinds |
### Security Context
Set the actor and scope used for authorization:
```lua
local security = require("security")
local c, err = contract.get("app.services:admin")
if err then return nil, err end
local secured, err = c:with_actor(security.actor())
if err then return nil, err end
secured, err = secured:with_scope(security.scope())
if err then return nil, err end
local admin, err = secured:open()
if err then return nil, err end
```
Without explicit `with_actor`/`with_scope`, an opened contract inherits the caller's ambient actor and scope. When set, they propagate to the bound implementation functions — every method call on the instance executes under that identity.
### Permissions
| Permission | Resource | Functions |
|------------|----------|-----------|
| `contract.get` | contract id | `get()` |
| `contract.open` | binding id | `open()`, `Contract:open()` |
| `contract.implementations` | contract id | `find_implementations()`, `Contract:implementations()` |
| `contract.call` | method name | sync and async method calls |
| `contract.context` | "context" | `Contract:with_context()` |
| `contract.security` | "security" | `Contract:with_actor()`, `Contract:with_scope()` |
### Errors
| Condition | Kind |
|-----------|------|
| Invalid binding ID format | `errors.INVALID` |
| Contract not found | `errors.NOT_FOUND` |
| Binding not found | `errors.NOT_FOUND` |
| Method not found | `errors.NOT_FOUND` |
| No default binding | `errors.NOT_FOUND` |
| Permission denied | `errors.PERMISSION_DENIED` |
| Call failed | kind of the implementation's error (preserved); `errors.INTERNAL` for dispatch failures |
---
# "JSON Encoding"
## JSON Encoding
The `json` module encodes Lua values as JSON, decodes JSON strings, and validates data with JSON Schema.
This is an API reference. Short expression examples show successful return values; examples that consume the result capture the optional second `error` return.
### Loading
```lua
local json = require("json")
```
Add `json` to the executable entry's `modules:` list before requiring it.
#### `encode`
Encode a Lua value as a JSON string:
```lua
-- Simple values
json.encode("hello") -- '"hello"'
json.encode(42) -- '42'
json.encode(true) -- 'true'
json.encode(nil) -- 'null'
-- Arrays (sequential numeric keys)
json.encode({1, 2, 3}) -- '[1,2,3]'
json.encode({"a", "b"}) -- '["a","b"]'
-- Objects (string keys)
local user = {name = "Alice", age = 30}
json.encode(user) -- JSON object with name="Alice" and age=30; member order is unspecified
-- Nested structures
local order = {
id = "ord-123",
items = {
{sku = "ABC", qty = 2},
{sku = "XYZ", qty = 1}
},
total = 99.50
}
json.encode(order)
-- Structurally equivalent JSON; object-member order is unspecified
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `value` | any | Lua value to encode |
**Returns:** `string, error`
Encoding follows these rules:
- `nil` becomes `null`
- Empty tables become `[]` (or `{}` if created with string keys)
- Tables with sequential 1-based keys become arrays
- Tables with string keys become objects
- Mixed numeric and string keys cause an error
- Sparse arrays (gaps in indices) cause an error
- Inf/NaN numbers become `null`
- Recursive table references cause an error
- Maximum nesting depth is 128 levels
#### `decode`
Decode a JSON string into a Lua value:
```lua
-- Parse object
local user, err = json.decode('{"name":"Bob","active":true}')
if err then
return nil, err
end
print(user.name) -- "Bob"
print(user.active) -- true
-- Parse array
local items, items_err = json.decode('[10, 20, 30]')
if items_err then return nil, items_err end
print(items[1]) -- 10
print(#items) -- 3
-- Parse nested data
local response, response_err = json.decode([[
{
"status": "ok",
"data": {
"users": [
{"id": 1, "name": "Alice"},
{"id": 2, "name": "Bob"}
]
}
}
]])
if response_err then return nil, response_err end
print(response.data.users[1].name) -- "Alice"
-- Handle errors
local data, err = json.decode("not valid json")
if err then
print(err:kind()) -- "Internal" (errors.INTERNAL)
print(err:message()) -- parse error details
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `str` | string | JSON string to decode |
**Returns:** `any, error`
#### `validate`
Validate a Lua value against a JSON Schema:
```lua
-- Define a schema
local user_schema = {
type = "object",
properties = {
name = {type = "string", minLength = 1},
email = {type = "string", format = "email"},
age = {type = "integer", minimum = 0, maximum = 150}
},
required = {"name", "email"}
}
-- Valid data passes
local valid, err = json.validate(user_schema, {
name = "Alice",
email = "alice@example.com",
age = 30
})
if err then return nil, err end
print(valid) -- true
-- Invalid data fails with details
local valid, err = json.validate(user_schema, {
name = "",
email = "not-an-email"
})
if not valid then
print(err:message()) -- validation error details
end
-- Schema can also be a JSON string
local schema_json = '{"type":"number","minimum":0}'
local valid, schema_err = json.validate(schema_json, 42)
if schema_err then return nil, schema_err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `schema` | table or string | JSON Schema definition |
| `data` | any | Value to validate |
**Returns:** `boolean, error`
Schemas are cached by content hash for performance.
#### `validate_string`
Validate a JSON string against a schema without first returning a decoded value:
```lua
local schema = {
type = "object",
properties = {
action = {type = "string", enum = {"create", "update", "delete"}}
},
required = {"action"}
}
-- Validate raw JSON from request body
local body = '{"action":"create","data":{}}'
local valid, err = json.validate_string(schema, body)
if not valid then
return nil, errors.new("Invalid request: " .. err:message()):kind(errors.INVALID)
end
-- Now safe to decode
local request, decode_err = json.decode(body)
if decode_err then return nil, decode_err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `schema` | table or string | JSON Schema definition |
| `json_str` | string | JSON string to validate |
**Returns:** `boolean, error`
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Recursive table reference | `errors.INTERNAL` | no |
| Sparse array (gaps in indices) | `errors.INTERNAL` | no |
| Mixed key types in table | `errors.INTERNAL` | no |
| Nesting exceeds 128 levels | `errors.INTERNAL` | no |
| Invalid JSON syntax | `errors.INTERNAL` | no |
| Input not a string or empty string (decode) | `errors.INVALID` | no |
| Schema compilation failed | `errors.INVALID` | no |
| Validation failed | `errors.INVALID` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "YAML Encoding"
## YAML Encoding
The `yaml` module serializes Lua tables as YAML and parses YAML documents into Lua values.
This is an API reference. Output-only expressions illustrate successful encoding; examples that consume a value capture the optional second `error` return.
### Loading
```lua
local yaml = require("yaml")
```
Add `yaml` to the executable entry's `modules:` list before requiring it.
#### `encode`
Encode a Lua table as YAML:
```lua
-- Simple key-value
local config = {
name = "myapp",
port = 8080,
debug = true
}
local out, err = yaml.encode(config)
if err then return nil, err end
-- YAML mapping containing name, port, and debug.
-- Arrays become YAML lists
local items = {"apple", "banana", "cherry"}
yaml.encode(items)
-- - apple
-- - banana
-- - cherry
-- Nested structures
local server = {
http = {
address = ":8080",
timeout = "30s"
},
database = {
host = "localhost",
port = 5432
}
}
yaml.encode(server)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | table | Lua table to encode |
| `options` | table? | Optional encoding options |
##### Options
| Field | Type | Description |
|-------|------|-------------|
| `field_order` | string[] | Custom field order; listed fields appear in this order |
| `sort_unordered` | boolean | Sort fields not in `field_order` alphabetically |
```lua
-- Control field order in output
local entry = {
zebra = 1,
alpha = 2,
name = "test",
kind = "demo"
}
-- Fields appear in specified order, remaining sorted alphabetically
local result, encode_err = yaml.encode(entry, {
field_order = {"name", "kind"},
sort_unordered = true
})
if encode_err then return nil, encode_err end
-- name: test
-- kind: demo
-- alpha: 2
-- zebra: 1
-- Just sort all fields alphabetically
yaml.encode(entry, {sort_unordered = true})
-- alpha: 2
-- kind: demo
-- name: test
-- zebra: 1
```
**Returns:** `string, error`
#### `decode`
Parse a YAML string into a Lua value:
```lua
-- Parse configuration
local config, err = yaml.decode([[
server:
host: localhost
port: 8080
features:
- auth
- logging
- metrics
]])
if err then
return nil, err
end
print(config.server.host) -- "localhost"
print(config.server.port) -- 8080
print(config.features[1]) -- "auth"
-- Parse from file content
local content = fs.get("app:config"):readfile("config.yaml")
local settings, err = yaml.decode(content)
if err then
return nil, errors.wrap(err, "invalid config file")
end
-- Handle mixed types
local data, data_err = yaml.decode([[
name: test
count: 42
ratio: 3.14
enabled: true
tags:
- lua
- wippy
]])
if data_err then return nil, data_err end
print(type(data.count)) -- "number"
print(type(data.enabled)) -- "boolean"
print(type(data.tags)) -- "table"
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | YAML string to parse |
**Returns:** `any, error` — the value type depends on the YAML content
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Input not a table (encode) | `errors.INVALID` | no |
| Input not a string (decode) | `errors.INVALID` | no |
| Empty string (decode) | `errors.INVALID` | no |
| Invalid YAML syntax | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Base64 Encoding"
## Base64 Encoding
The `base64` module encodes strings and binary data using standard RFC 4648 Base64 and decodes them back to bytes.
This is an API reference. Output-only expressions show successful values; filesystem and transport examples check the optional second `error` return before consuming data. Names such as `username`, `password`, `encoded_image`, and `user_input` are application-supplied strings.
Base64 is an encoding, not encryption or authentication. Do not use it to conceal secrets or to verify that data has not been modified. Send Basic authentication credentials only over TLS and obtain them from application-owned secret storage rather than literals.
### Loading
```lua
local base64 = require("base64")
```
Add `base64` to the executable entry's `modules:` list before requiring it. Filesystem and JSON examples also require `fs` and `json` respectively.
#### `encode`
Encodes a string, including binary data, as Base64.
```lua
-- Encode text
local encoded, err = base64.encode("Hello, World!")
if err then return nil, err end
print(encoded) -- "SGVsbG8sIFdvcmxkIQ=="
-- Encode binary data (e.g., from file)
local image_data = fs.get("app:data"):readfile("photo.jpg")
local image_b64 = base64.encode(image_data)
-- Encode JSON for transport
local json = require("json")
local payload, json_err = json.encode({user = "alice", action = "login"})
if json_err then return nil, json_err end
local token_part, token_err = base64.encode(payload)
if token_err then return nil, token_err end
-- Encode credentials
local credentials, credentials_err = base64.encode(username .. ":" .. password)
if credentials_err then return nil, credentials_err end
local auth_header = "Basic " .. credentials
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to encode (text or binary) |
**Returns:** `string, error` — an empty input returns an empty string
#### `decode`
Decodes a Base64 string to its original bytes.
```lua
-- Decode text
local decoded, decode_err = base64.decode("SGVsbG8sIFdvcmxkIQ==")
if decode_err then return nil, decode_err end
print(decoded) -- "Hello, World!"
-- Decode with error handling
local data, err = base64.decode(user_input)
if err then
return nil, errors.new("Invalid base64 data"):kind(errors.INVALID)
end
-- Decode binary data
local image_data, err = base64.decode(encoded_image)
if err then
return nil, err
end
fs.get("app:data"):writefile("output.jpg", image_data)
-- Decode a base64-wrapped JSON document
local json = require("json")
local doc = json.decode(base64.decode(encoded_json))
```
The final block demonstrates delimiter handling only. It does not parse or verify a signed token format.
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Base64-encoded string |
**Returns:** `string, error` — an empty input returns an empty string
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Input not a string | `errors.INVALID` | no |
| Invalid base64 characters | `errors.INVALID` | no |
| Corrupted padding | `errors.INVALID` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Compression"
## Compression
The `compress` module encodes and decodes strings with gzip, Brotli, Zstandard, raw DEFLATE, and zlib.
This is an API reference with partial HTTP and storage recipes. Every operation materializes its complete input and output as Lua strings; use the archive or stream APIs when data must remain streaming. The examples assume the entry enables `compress` and any separately required modules such as `json` or `http`.
### Loading
```lua
local compress = require("compress")
```
Add `compress` to the executable entry's `modules:` list before requiring it.
### GZIP
Gzip is defined by RFC 1952.
#### Compress {id="gzip-compress"}
```lua
-- Compress for HTTP response
local body, json_err = json.encode(large_response)
if json_err then return nil, json_err end
local compressed, err = compress.gzip.encode(body)
if err then
return nil, err
end
-- Set Content-Encoding header
local header_err = res:set_header("Content-Encoding", "gzip")
if header_err then return nil, header_err end
local write_err = res:write(compressed)
if write_err then return nil, write_err end
-- Maximum compression for storage
local archived, archive_err = compress.gzip.encode(data, {level = 9})
if archive_err then return nil, archive_err end
-- Fast compression for real-time
local fast, fast_err = compress.gzip.encode(data, {level = 1})
if fast_err then return nil, fast_err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to compress |
| `options` | table? | Optional encoding options |
##### Options {id="gzip-compress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `level` | integer | Compression level 1-9 (default: 6) |
**Returns:** `string, error`
#### Decompress {id="gzip-decompress"}
```lua
-- Decompress HTTP request
local content_encoding, header_err = req:header("Content-Encoding")
if header_err then return nil, header_err end
if content_encoding == "gzip" then
local body, body_err = req:body()
if body_err then return nil, body_err end
local decompressed, err = compress.gzip.decode(body)
if err then
return nil, errors.new("Invalid gzip data"):kind(errors.INVALID)
end
body = decompressed
end
-- Decompress with size limit (prevent zip bombs)
local decompressed, err = compress.gzip.decode(data, {max_size = 10 * 1024 * 1024})
if err then
return nil, errors.new("Decompressed size exceeds 10MB limit"):kind(errors.INVALID)
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | GZIP compressed data |
| `options` | table? | Optional decoding options |
##### Options {id="gzip-decompress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) |
**Returns:** `string, error`
### Brotli
Brotli is defined by RFC 7932 and is commonly used for compressed text content.
#### Compress {id="brotli-compress"}
```lua
-- Best for static assets and text content
local compressed, err = compress.brotli.encode(html_content, {level = 11})
if err then return nil, err end
-- Store `compressed` through the application's cache contract if needed.
-- Moderate compression for API responses
local compressed, err = compress.brotli.encode(json_data, {level = 4})
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to compress |
| `options` | table? | Optional encoding options |
##### Options {id="brotli-compress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `level` | integer | Compression level 0-11 (default: 6) |
**Returns:** `string, error`
#### Decompress {id="brotli-decompress"}
```lua
local decompressed, err = compress.brotli.decode(compressed_data)
if err then
return nil, err
end
-- With size limit
local decompressed, err = compress.brotli.decode(data, {max_size = 50 * 1024 * 1024})
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Brotli compressed data |
| `options` | table? | Optional decoding options |
##### Options {id="brotli-decompress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) |
**Returns:** `string, error`
### Zstandard
Zstandard is a general-purpose compression format defined by RFC 8878.
#### Compress {id="zstd-compress"}
```lua
-- Good balance of speed and ratio
local compressed, err = compress.zstd.encode(binary_data)
if err then return nil, err end
-- Higher compression for archival
local archived, archive_err = compress.zstd.encode(data, {level = 19})
if archive_err then return nil, archive_err end
-- Fast mode for latency-sensitive payloads
local fast, fast_err = compress.zstd.encode(data, {level = 1})
if fast_err then return nil, fast_err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to compress |
| `options` | table? | Optional encoding options |
##### Options {id="zstd-compress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `level` | integer | Compression level 1-22 (default: 3) |
| `dict` | string? | Zstd dictionary bytes from `train_dict` (default: none) |
**Returns:** `string, error`
#### Decompress {id="zstd-decompress"}
```lua
local decompressed, err = compress.zstd.decode(compressed_data)
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Zstandard compressed data |
| `options` | table? | Optional decoding options |
##### Options {id="zstd-decompress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) |
| `dict` | string? | Zstd dictionary bytes (must match the dict used to encode) |
**Returns:** `string, error`
#### Dictionaries {id="zstd-dictionaries"}
Train a dictionary from similar sample payloads, then pass it through the `dict` option to `encode` and `decode`. Decoding requires the same dictionary used for encoding.
```lua
local dict, err = compress.zstd.train_dict(samples, { size = 112640 })
if err then return nil, err end
local packed, pack_err = compress.zstd.encode(data, { dict = dict })
if pack_err then return nil, pack_err end
local original, decode_err = compress.zstd.decode(packed, { dict = dict })
if decode_err then return nil, decode_err end
```
##### train_dict(samples, options?)
| Parameter | Type | Description |
|-----------|------|-------------|
| `samples` | string[] | Training samples (at least one >= 8 bytes) |
| `options` | table? | `size` (integer, target dict bytes, 256-1048576, default 114688), `id` (integer, default 0), `level` (integer, 1-22) |
**Returns:** `string, error` (the dictionary bytes)
##### inspect_dict(dict)
| Parameter | Type | Description |
|-----------|------|-------------|
| `dict` | string | Dictionary bytes |
**Returns:** `table, error` — `{id: integer, content_size: integer}`
### Deflate
Raw DEFLATE is defined by RFC 1951 and is also used inside other formats.
#### Compress {id="deflate-compress"}
```lua
local compressed, err = compress.deflate.encode(data, {level = 6})
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to compress |
| `options` | table? | Optional encoding options |
##### Options {id="deflate-compress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `level` | integer | Compression level 1-9 (default: 6) |
**Returns:** `string, error`
#### Decompress {id="deflate-decompress"}
```lua
local decompressed, err = compress.deflate.decode(compressed)
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | DEFLATE compressed data |
| `options` | table? | Optional decoding options |
##### Options {id="deflate-decompress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) |
**Returns:** `string, error`
### Zlib
Zlib wraps DEFLATE data with a header and checksum as defined by RFC 1950.
#### Compress {id="zlib-compress"}
```lua
local compressed, err = compress.zlib.encode(data, {level = 6})
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to compress |
| `options` | table? | Optional encoding options |
##### Options {id="zlib-compress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `level` | integer | Compression level 1-9 (default: 6) |
**Returns:** `string, error`
#### Decompress {id="zlib-decompress"}
```lua
local decompressed, err = compress.zlib.decode(compressed)
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Zlib compressed data |
| `options` | table? | Optional decoding options |
##### Options {id="zlib-decompress-options"}
| Field | Type | Description |
|-------|------|-------------|
| `max_size` | integer | Max decompressed size in bytes (default: 128MB, max: 1GB) |
**Returns:** `string, error`
### Choosing an Algorithm
| Algorithm | Best For | Speed | Ratio | Level Range |
|-----------|----------|-------|-------|-------------|
| gzip | HTTP, wide compatibility | Medium | Good | 1-9 |
| brotli | Static assets, text | Slow | Best | 0-11 |
| zstd | Binary payloads, fast compression | Fast | Good | 1-22 |
| deflate/zlib | Low-level, specific protocols | Medium | Good | 1-9 |
```lua
-- HTTP response based on Accept-Encoding
local accept, header_err = req:header("Accept-Encoding")
if header_err then return nil, header_err end
accept = accept or ""
local body, json_err = json.encode(response_data)
if json_err then return nil, json_err end
local qualities = {}
for item in accept:gmatch("[^,]+") do
local coding = item:match("^%s*([^;%s]+)")
local has_q = item:match(";%s*[qQ]%s*=") ~= nil
local q_text = item:match(";%s*[qQ]%s*=%s*([^;%s,]+)")
local q
if not has_q then
q = 1
elseif q_text == "0" or q_text == "1" or
(q_text and q_text:match("^0%.%d?%d?%d?$")) or
(q_text and q_text:match("^1%.0?0?0?$")) then
q = tonumber(q_text)
end
if coding and q and q >= 0 and q <= 1 then
coding = coding:lower()
qualities[coding] = math.max(qualities[coding] or 0, q)
end
end
local function quality(coding)
if qualities[coding] ~= nil then return qualities[coding] end
if coding == "identity" then
return qualities["*"] == 0 and 0 or 1
end
return qualities["*"] or 0
end
local selected, selected_q = nil, -1
for _, coding in ipairs({"br", "gzip", "identity"}) do
local q = quality(coding)
if q > selected_q then
selected, selected_q = coding, q
end
end
-- Include every field used by this handler or its surrounding middleware.
local vary_fields = {"Accept-Encoding"}
local vary_err = res:set_header("Vary", table.concat(vary_fields, ", "))
if vary_err then return nil, vary_err end
if selected_q <= 0 then
local status_err = res:set_status(http.STATUS.NOT_ACCEPTABLE)
if status_err then return nil, status_err end
local write_err = res:write("No acceptable content encoding")
if write_err then return nil, write_err end
elseif selected == "br" then
local compressed, compress_err = compress.brotli.encode(body)
if compress_err then return nil, compress_err end
local set_err = res:set_header("Content-Encoding", "br")
if set_err then return nil, set_err end
local write_err = res:write(compressed)
if write_err then return nil, write_err end
elseif selected == "gzip" then
local compressed, compress_err = compress.gzip.encode(body)
if compress_err then return nil, compress_err end
local set_err = res:set_header("Content-Encoding", "gzip")
if set_err then return nil, set_err end
local write_err = res:write(compressed)
if write_err then return nil, write_err end
else
local write_err = res:write(body)
if write_err then return nil, write_err end
end
```
This partial handler parses exact coding tokens and RFC q-values, honors explicit rejections such as `br;q=0`, and emits `Vary: Accept-Encoding`. `set_header` replaces an existing `Vary` value, so add every other field used by surrounding middleware to `vary_fields` before setting it. A full HTTP stack may provide a shared negotiation helper instead.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty input | `errors.INVALID` | no |
| Level out of range | `errors.INVALID` | no |
| Invalid compressed data | `errors.INVALID` | no |
| Decompressed size exceeds limit | `errors.INTERNAL` (gzip, zlib, zstd) / `errors.INVALID` (deflate, brotli) | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Archive"
## Archive
Read and write zip/tar archives with bounded memory. Archives are never loaded into RAM nor extracted to disk — peak memory is independent of archive and entry size, so multi-GB archives run on a low-RAM server.
### Loading
```lua
local archive = require("archive")
```
### Formats
Built-in formats are detected by magic bytes, or forced with `opts.format`:
| Format | Random read | Sequential scan | Write |
|--------|:-----------:|:---------------:|:-----:|
| `zip` | yes | yes (local headers) | yes |
| `tar` | yes | yes | yes |
| `tar.gz` | no | yes | yes |
| `tar.zst` | no | yes | yes |
`archive.formats()` returns the list of registered format names.
```lua
local names = archive.formats() -- {"zip", "tar", "tar.gz", "tar.zst", ...}
```
### Options
All entrypoints accept an optional `opts` table:
| Key | Default | Meaning |
|-----|---------|---------|
| `format` | auto | `"zip"`, `"tar"`, `"tar.gz"`, `"tar.zst"`; auto = sniff magic, else extension |
| `max_entries` | 100000 | Reject archives with more entries (decompression-bomb defense) |
| `max_total_bytes` | 2 GiB | Cap on cumulative uncompressed output during read/extract |
| `max_file_bytes` | 1 GiB | Cap on a single entry's uncompressed size |
| `max_inline_bytes` | 16 MiB | Hard cap for the RAM-materializing `read()` call; above it, use `stream()`/`extract()` |
| `buffer_bytes` | 64 KiB | Streaming copy buffer for read/extract/add |
`max_total_bytes`/`max_file_bytes` are work caps, not RAM caps — streaming an entry never holds more than `buffer_bytes` plus the codec's decompression window. The only RAM-sizing knob is `max_inline_bytes`.
### Reading — Random Access
`archive.open(source, ...)` opens a **seekable** source for full random access (zip central directory is read up front; entries decompress on demand). The source may be an `fs.FS` handle plus a path, an open `fs.File`, raw bytes (bytes hold the whole archive in RAM — small archives only), or any random-access reader handed over by another module.
A reader from another module qualifies when it implements `io.ReaderAt` and reports its `Size`; an optional `Name` is used for extension sniffing when `opts.format` is omitted. [`cloudstorage`](lua/storage/cloud.md) `open_reader` is one, which reads a multi-GB archive directly out of object storage. The archive opens nothing in that case and never closes the reader — its owner does.
```lua
local fs = require("fs")
local archive = require("archive")
-- Open by fs handle + path (the module opens the file and owns its lifecycle)
local r, err = archive.open(fs.get("app:uploads"), "incoming.zip")
-- Or from an already-open seekable fs.File
-- local r = archive.open(fs:get("app:uploads"):open("x.zip"))
-- Or from raw bytes (small archives only)
-- local r = archive.open(zip_bytes, { format = "zip" })
-- Or from a random-access reader owned by another module
-- local reader = cloudstorage.get("app:files"):open_reader("incoming.zip")
-- local r = archive.open(reader)
```
**Returns:** `Reader, error`
**Permission:** `archive.read`
#### entries
Iterate the directory (metadata only — no decompression):
```lua
for e in r:entries() do
-- e: name, size, compressed_size, is_dir, mode, modified, method, crc32, type
print(e.name, e.size, e.is_dir)
end
```
#### stat
Get entry metadata by name (no decompression):
```lua
local info, err = r:stat("docs/readme.md")
```
#### read
Materialize a single entry as a Lua string. Errors (`kind = Invalid`) above `max_inline_bytes` — for anything large, use `stream()` or `extract()`:
```lua
local data, err = r:read("docs/readme.md") -- small entries only
```
#### stream
Return the entry as a `stream.Stream` that decompresses on demand. Composes everywhere a stream does — `:scanner()`, `fs:writefile()`, or handed to another module:
```lua
local es, err = r:stream("big.csv")
while true do
local chunk = es:read(65536)
if not chunk then break end
process(chunk)
end
es:close()
```
#### extract
Stream one entry into a destination filesystem:
```lua
local ok, err = r:extract("docs/readme.md", fs.get("app:out"))
-- optional destination path:
-- r:extract("docs/readme.md", fs.get("app:out"), "readme.md")
```
#### extract_all
Stream every entry into a destination filesystem:
```lua
local count, err = r:extract_all(fs.get("app:out"), {
prefix = "job123/", -- prepend to each destination path
strip = 1, -- drop N leading path components
filter = function(e) return not e.is_dir end,
})
```
Entry names are sanitized on extract — `..` segments, absolute paths, and Windows drive/UNC prefixes are rejected (zip-slip defense).
#### close
Close the reader. Idempotent; also auto-closed at task scope.
```lua
r:close()
```
### Reading — Sequential Scan
`archive.scan(source, opts?)` opens a **forward-only** stream (an HTTP upload body, a multipart file stream). Entries are visited in archive order; each entry's reader is valid only until you advance. No random `read(name)`.
```lua
local up = form.files.upload[1]:stream() -- stream.Stream
local s, err = archive.scan(up, { format = "zip" })
for e, entry in s:walk() do -- entry is a stream.Stream
if not e.is_dir then
fs.get("app:uploads"):writefile("job123/" .. e.name, entry)
end
end
s:close()
```
**Returns:** `Walker, error`
**Permission:** `archive.read`
A walker also supports `extract_all` with the same options as the random-access reader, streaming every entry into a destination filesystem in one call:
```lua
local count, err = s:extract_all(fs.get("app:uploads"), { prefix = "job123/" })
```
`tar`, `tar.gz`, and `tar.zst` stream natively. `zip` is parsed via per-entry local headers; entries written with a streaming data descriptor (size/CRC trailing the data) are read by decompressing to the entry boundary. For robust zip handling of large uploads, land the upload as a file first (a bounded sequential copy) then use `archive.open`:
```lua
local dst = fs.get("app:tmp")
dst:writefile("u.zip", req:stream()) -- streaming copy upload → fs file
local r = archive.open(dst, "u.zip") -- robust random access
-- ... entries / extract_all ...
r:close()
dst:remove("u.zip")
```
### Writing
`archive.create(dest, ...)` builds an archive by streaming entries into a destination — a file in an fs (with a path) or a writable `stream.Stream` (e.g. an HTTP response), so a download `.zip` is generated straight to the wire with bounded memory.
```lua
local w, err = archive.create(fs.get("app:tmp"), "out.zip", { format = "zip" })
-- or stream to a response:
-- local w = archive.create(res:stream(), { format = "zip" })
```
**Returns:** `Writer, error`
**Permission:** `archive.write`
#### add
Add an entry from a string, bytes, reader, or `stream.Stream`:
```lua
w:add("notes.txt", "hello")
w:add("from_upload", some_stream, { method = "deflate", mode = tonumber("644", 8) })
```
#### add_file
Stream an entry from a file in a filesystem:
```lua
w:add_file("data/big.bin", fs.get("app:data"), "big.bin")
```
#### add_dir
Add a directory entry:
```lua
w:add_dir("empty/")
```
#### close
Finalize the archive (writes the central directory for zip). Idempotent; also auto-closed at task scope.
```lua
w:close()
```
`add*` options: `{ method = "store"|"deflate", mode, size }`. Tar formats need the entry size up front, so `add()` from a stream or reader into a `tar*` archive requires `size` (strings and `add_file` supply it). The zip writer streams to non-seekable writers using data descriptors, so writing to a response stream works.
### Errors
| Condition | Kind |
|-----------|------|
| Source is not an fs handle, an fs file, bytes or a random-access reader | `errors.INVALID` |
| Unknown / mismatched format | `errors.INVALID` |
| Corrupt or truncated archive | `errors.INVALID` |
| Limit exceeded (entries / total / file / inline) | `errors.INVALID` |
| Random access on a stream-only format (use `scan`) | `errors.UNAVAILABLE` |
| Entry name not found | `errors.NOT_FOUND` |
| Source not readable / destination not writable | `errors.PERMISSION_DENIED` |
| Read a stale streamed entry after the walk advanced | `errors.INTERNAL` |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Filesystem](lua/storage/filesystem.md) - Source and destination filesystems
- [Stream](lua/core/stream.md) - Stream objects handed to and from archives
- [Compression](lua/data/compress.md) - In-memory gzip/deflate/zstd
- [Cloud Storage](lua/storage/cloud.md) - `open_reader` as a random-access archive source
---
# "Payload Encoding"
## Payload Encoding
Payloads carry typed values between functions, processes, services, and workflows. They can be inspected, extracted, or transcoded between supported formats.
This is an API reference with partial transport recipes. Values such as `p`, `input_data`, and the asynchronous target entry come from the surrounding application.
### Loading
`payload` is a global namespace and does not require `require()`.
```lua
payload.new(...) -- direct access
```
### Format Constants
The following constants identify payload formats:
```lua
payload.format.JSON -- "json/plain"
payload.format.YAML -- "yaml/plain"
payload.format.STRING -- "text/plain"
payload.format.BYTES -- "application/octet-stream"
payload.format.MSGPACK -- "application/msgpack"
payload.format.LUA -- "lua/any"
payload.format.GOLANG -- "golang/any"
payload.format.ERROR -- "golang/error"
```
### Creating Payloads
Create a payload from a Lua value:
```lua
-- From table
local p = payload.new({
user_id = 123,
name = "Alice",
roles = {"admin", "user"}
})
-- From string
local str_p = payload.new("Hello, World!")
-- From number
local num_p = payload.new(42.5)
-- From boolean
local bool_p = payload.new(true)
-- From nil
local nil_p = payload.new(nil)
-- From error
local err_p = payload.new(errors.new("something failed"))
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `value` | any | Lua value (string, number, boolean, table, nil, or error) |
**Returns:** `Payload`
### Getting Format
Read the payload's format identifier:
```lua
local p = payload.new({name = "test"})
local format = p:get_format() -- "lua/any"
local str_p = payload.new("hello")
local format2 = str_p:get_format() -- "lua/any"
local err_p = payload.new(errors.new("failed"))
local format3 = err_p:get_format() -- "golang/error"
```
**Returns:** `string` - one of `payload.format.*` constants
### Extracting Data
Extract the payload's Lua value, transcoding when needed:
```lua
local p = payload.new({
items = {1, 2, 3},
total = 100
})
local data, err = p:data()
if err then
return nil, err
end
print(data.total) -- 100
print(data.items[1]) -- 1
```
**Returns:** `any, error`
### Transcoding Payloads
Transcode a payload to another supported format:
```lua
local p = payload.new({
name = "test",
value = 123
})
-- Convert to JSON
local json_p, err = p:transcode(payload.format.JSON)
if err then
return nil, err
end
print(json_p:get_format()) -- "json/plain"
-- Convert to MessagePack (compact binary)
local msgpack_p, err = p:transcode(payload.format.MSGPACK)
if err then
return nil, err
end
-- Convert to YAML
local yaml_p, yaml_err = p:transcode(payload.format.YAML)
if yaml_err then
return nil, yaml_err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `format` | string | Target format from `payload.format.*` |
**Returns:** `Payload, error`
### Unmarshalling
Decode a payload to a Lua value regardless of its source format:
```lua
local data, err = p:unmarshal()
if err then
return nil, err
end
```
`unmarshal()` behaves like `data()`: both transcode non-Lua payloads to the Lua format and return the resulting Lua value. The only difference is that `unmarshal()` returns an error when the transcoded data is not a valid Lua value, whereas `data()` returns `nil`.
**Returns:** `any, error`
### Async Results
Asynchronous function calls return their values in payloads:
This example assumes `app.process:compute` returns exactly one value. With no result, `future:result()` returns `nil`; with multiple results, it returns a Lua table rather than one `Payload`, so callers must handle those shapes separately.
```lua
local funcs = require("funcs")
local future, err = funcs.async("app.process:compute", input_data)
if err then
return nil, err
end
-- Wait for result
local ch = future:response()
local _, ok = ch:receive()
if not ok then
return nil, errors.new("channel closed")
end
local result_payload, result_err = future:result()
if result_err then
return nil, result_err
end
if result_payload == nil then
return nil, errors.new("compute returned no result")
end
-- Extract data from payload
local result, err = result_payload:data()
if err then
return nil, err
end
print(result.computed_value)
```
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Transcoding failure | `errors.INTERNAL` | no |
| Result not valid Lua value | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Excel Spreadsheets"
## Excel Spreadsheets
Read and write Microsoft Excel files (.xlsx). Create workbooks, manage sheets, read cell values, and generate reports with formatting support.
### Loading
```lua
local excel = require("excel")
```
#### New Workbook
Creates a new empty Excel workbook.
```lua
local wb, err = excel.new()
if err then
return nil, err
end
-- Create sheets and add data
wb:new_sheet("Report")
wb:set_cell_value("Report", "A1", "Title")
wb:close()
```
**Returns:** `Workbook, error`
#### Open Workbook
Opens an Excel workbook from a reader object.
```lua
local fs = require("fs")
local vol, err = fs.get("app:data")
if err then
return nil, err
end
local file, err = vol:open("/reports/sales.xlsx", "r")
if err then
return nil, err
end
local wb, err = excel.open(file)
if err then
file:close()
return nil, err
end
-- Read data from workbook
local rows = wb:get_rows("Sheet1")
for i, row in ipairs(rows) do
print("Row " .. i .. ": " .. table.concat(row, ", "))
end
wb:close()
file:close()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `reader` | File | Must implement io.Reader (e.g., fs.File) |
**Returns:** `Workbook, error`
#### Create Sheet
Creates a new sheet or returns existing sheet index.
```lua
local wb = excel.new()
-- Create sheets
local idx1 = wb:new_sheet("Summary")
local idx2 = wb:new_sheet("Details")
local idx3 = wb:new_sheet("Charts")
-- If sheet exists, returns its index
local existing = wb:new_sheet("Summary") -- returns same as idx1
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Sheet name |
**Returns:** `integer, error`
#### List Sheets
Returns list of all sheet names in workbook.
```lua
local wb = excel.new()
wb:new_sheet("Sales")
wb:new_sheet("Expenses")
wb:new_sheet("Summary")
local sheets = wb:get_sheet_list()
-- sheets = {"Sheet1", "Sales", "Expenses", "Summary"}
for _, name in ipairs(sheets) do
print("Sheet:", name)
end
```
**Returns:** `string[], error`
#### Set Cell Value
Sets value of a single cell.
```lua
local wb = excel.new()
wb:new_sheet("Data")
-- Set different value types
wb:set_cell_value("Data", "A1", "Product Name") -- string
wb:set_cell_value("Data", "B1", "Price") -- string
wb:set_cell_value("Data", "C1", "In Stock") -- string
wb:set_cell_value("Data", "A2", "Widget")
wb:set_cell_value("Data", "B2", 29.99) -- number
wb:set_cell_value("Data", "C2", true) -- boolean
wb:set_cell_value("Data", "A3", "Gadget")
wb:set_cell_value("Data", "B3", 49.99)
wb:set_cell_value("Data", "C3", false)
-- Cell references support columns beyond Z
wb:set_cell_value("Data", "AA1", "Extended Column")
wb:set_cell_value("Data", "AB100", "Far cell")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sheet` | string | Sheet name |
| `cell` | string | Cell reference ("A1", "B2", "AA100") |
| `value` | any | string, integer, number, or boolean |
**Returns:** `error`
#### Get All Rows
Gets all rows from a sheet as 2D array.
```lua
local wb = excel.new()
wb:new_sheet("Report")
wb:set_cell_value("Report", "A1", "Name")
wb:set_cell_value("Report", "B1", "Score")
wb:set_cell_value("Report", "A2", "Alice")
wb:set_cell_value("Report", "B2", 95)
wb:set_cell_value("Report", "A3", "Bob")
wb:set_cell_value("Report", "B3", 87)
local rows, err = wb:get_rows("Report")
if err then
return nil, err
end
-- rows[1] = {"Name", "Score"}
-- rows[2] = {"Alice", "95"}
-- rows[3] = {"Bob", "87"}
for i, row in ipairs(rows) do
if i == 1 then
print("Headers:", row[1], row[2])
else
print("Data:", row[1], "scored", row[2])
end
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sheet` | string | Sheet name |
**Returns:** `string[][], error`
All cell values returned as strings. Booleans as "TRUE" or "FALSE", numbers as string representation.
#### Stream Rows
`wb:rows(sheet)` opens a streaming cursor over one sheet. The sheet is decoded incrementally in constant memory, unlike `get_rows` which materializes the entire sheet:
```lua
local cursor, err = wb:rows("Report")
if err then
return nil, err
end
while true do
local batch, err = cursor:read(500)
if err then
cursor:close()
return nil, err
end
if not batch then
break -- end of sheet
end
for _, row in ipairs(batch) do
process(row)
end
end
cursor:close()
```
| Method | Description |
|--------|-------------|
| `cursor:read(n?)` | Read the next batch of up to `n` rows (default 1, max 10000). Returns `string[][], error`; `nil, nil` at end of sheet |
| `cursor:close()` | Release the cursor (idempotent; cursors also close with the workbook) |
Cell values format identically to `get_rows`. Empty rows come back as empty tables, and trailing empty rows are preserved rather than trimmed. After end-of-sheet or an error, subsequent reads keep returning that same state.
#### Write to File
Writes workbook to a writer object.
```lua
local fs = require("fs")
local wb = excel.new()
-- Build report
wb:new_sheet("Monthly Report")
wb:set_cell_value("Monthly Report", "A1", "Month")
wb:set_cell_value("Monthly Report", "B1", "Revenue")
wb:set_cell_value("Monthly Report", "A2", "January")
wb:set_cell_value("Monthly Report", "B2", 45000)
wb:set_cell_value("Monthly Report", "A3", "February")
wb:set_cell_value("Monthly Report", "B3", 52000)
-- Write to file
local vol, err = fs.get("app:output")
if err then
wb:close()
return nil, err
end
local file, err = vol:open("/reports/monthly.xlsx", "w")
if err then
wb:close()
return nil, err
end
local err = wb:write_to(file)
file:close()
wb:close()
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `writer` | File | Must implement io.Writer (e.g., fs.File) |
**Returns:** `error`
#### Serialize to a String
Renders the workbook into an `xlsx` byte string, without a filesystem or a writer. Use it to hand a workbook to an HTTP response, an object store or a queue message.
```lua
local cloudstorage = require("cloudstorage")
local wb = excel.new()
wb:new_sheet("Report")
wb:set_cell_value("Report", "A1", "Total")
wb:set_cell_value("Report", "B1", 45000)
local data, err = wb:bytes()
wb:close()
if err then
return nil, err
end
local storage = cloudstorage.get("app.infra:files")
storage:upload_object("reports/monthly.xlsx", data, {
content_type = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
})
storage:release()
```
**Returns:** `string, error`
The whole workbook is materialized in memory. `write_to` builds the same in-memory buffer and then copies it to the writer, so it saves the Lua string but does not stream a large workbook.
Calling `bytes()` on a closed workbook returns an `errors.INTERNAL` error.
#### Close Workbook
Closes workbook and releases resources.
```lua
local wb = excel.new()
-- ... work with workbook ...
wb:close()
-- Safe to call multiple times
wb:close()
```
**Returns:** `error`
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| No context | `errors.INTERNAL` | no |
| Invalid workbook | `errors.INVALID` | no |
| Workbook closed | `errors.INTERNAL` (`errors.INVALID` from `rows`) | no |
| Not a writer (`write_to`) | `errors.INTERNAL` | no |
| Not a reader (`open`) | raised as an argument error | no |
| Invalid Excel file | `errors.INTERNAL` | no |
| Non-existent sheet | `errors.INTERNAL` (`errors.INVALID` from `rows`) | no |
| Invalid cell reference | `errors.INTERNAL` | no |
| Write failed | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Filesystem](lua/storage/filesystem.md) - File operations for reading/writing Excel files
---
# "HTTP"
## HTTP
The `http` module reads the current server-side request and builds its response, including headers, route data, body content, streaming output, and Server-Sent Events.
This is an API reference with partial handler recipes. Names such as `id`, `data`, `token`, and application callbacks come from the surrounding handler. Request accessors generally return `value, error`, and response mutations return `error`; examples that consume a result check those errors.
For server configuration, see [HTTP Server](http/server.md).
### Loading
```lua
local http = require("http")
```
Add `http` to the executable entry's `modules:` list before requiring it. Examples using `uuid`, `fs`, or `time` require those modules separately.
### Accessing the Request
Read the current HTTP request context:
```lua
local req, err = http.request({
timeout = 5000, -- 5 second body read timeout
max_body = 10485760 -- 10MB max body
})
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `options.timeout` | integer | Body read timeout in ms (default: 300000 / 5 min) |
| `options.max_body` | integer | Max body size in bytes (default: 120MB) |
**Returns:** `Request, error`
### Accessing the Response
Read the current HTTP response context:
```lua
local res, err = http.response()
if err then return nil, err end
```
**Returns:** `Response, error`
#### `method`
Return the request's HTTP method.
```lua
local method, method_err = req:method()
if method_err then return nil, method_err end
if method == http.METHOD.GET then
return get_resource(id)
elseif method == http.METHOD.POST then
local data, body_err = req:body_json()
if body_err then return nil, body_err end
return create_resource(data)
elseif method == http.METHOD.PUT then
local data, body_err = req:body_json()
if body_err then return nil, body_err end
return update_resource(id, data)
elseif method == http.METHOD.DELETE then
return delete_resource(id)
end
```
#### `path`
Return the request path.
```lua
local path, err = req:path()
if err then return nil, err end
print(path) -- "/api/users/123"
-- Route based on path
if path:match("^/api/") then
return handle_api(req)
end
```
#### `query`
Return one query parameter:
```lua
-- GET /search?q=hello&page=2&limit=10
local query, query_err = req:query("q")
if query_err then return nil, query_err end
-- With defaults
local page_text, page_err = req:query("page")
if page_err then return nil, page_err end
local page = tonumber(page_text) or 1
```
#### `query_params`
Return all query parameters. Multiple values for one key are joined with commas.
```lua
-- GET /search?tags=lua&tags=go&active=true
local params, err = req:query_params()
if err then return nil, err end
-- {tags = "lua,go", active = "true"}
for key, value in pairs(params) do
print(key .. ": " .. value)
end
```
#### `header`
Return one request header by name.
```lua
local uuid = require("uuid")
local auth, auth_err = req:header("Authorization")
if auth_err then return nil, auth_err end
if not auth then
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.UNAUTHORIZED)
if status_err then return nil, status_err end
return res:write_json({error = "Missing authorization"})
end
local correlation_id, correlation_err = req:header("X-Correlation-ID")
if correlation_err then return nil, correlation_err end
if not correlation_id then
correlation_id, correlation_err = uuid.v4()
if correlation_err then return nil, correlation_err end
end
```
Lookup is case-insensitive: `req:header("content-type")` and `req:header("Content-Type")` return the same value. A header sent more than once returns its values joined with `", "`. A header that is not present returns `nil`.
#### headers
Gets every request header.
```lua
local headers, err = req:headers()
for name, value in pairs(headers) do
print(name .. ": " .. value)
end
```
**Returns:** `table, error`
Keys are canonical header names (`Content-Type`, `X-Correlation-ID`), regardless of the casing the client sent. Repeated headers are joined with `", "`, as in `req:header()`.
#### content_type
Return the `Content-Type` header:
```lua
local ct, type_err = req:content_type() -- "application/json; charset=utf-8" or nil
if type_err then return nil, type_err end
```
#### `content_length`
Return the `Content-Length` header value:
```lua
local length, length_err = req:content_length() -- number of bytes
if length_err then return nil, length_err end
```
#### `host`
Return the `Host` header:
```lua
local host, host_err = req:host() -- "example.com:8080"
if host_err then return nil, host_err end
```
#### `param`
Return one route parameter from a path pattern such as `/users/:id`:
```lua
-- Route: /users/:id/posts/:post_id
local id, param_err = req:param("id")
if param_err then return nil, param_err end
local valid = false
if id then
local validate_err
valid, validate_err = uuid.validate(id)
if validate_err then return nil, validate_err end
end
if not valid then
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.BAD_REQUEST)
if status_err then return nil, status_err end
return res:write_json({error = "Invalid ID format"})
end
```
#### `params`
Return all route parameters:
```lua
-- Route: /orgs/:org/repos/:repo/issues/:issue
local p, err = req:params()
if err then return nil, err end
-- {org = "acme", repo = "widget", issue = "123"}
local issue = get_issue(p.org, p.repo, p.issue)
```
#### `body`
Read the full request body as a string:
```lua
local body, err = req:body()
if err then return nil, err end
-- Parse XML manually
local is_xml, type_err = req:is_content_type("application/xml")
if type_err then return nil, type_err end
if is_xml then
local data = parse_xml(body)
end
-- Avoid logging raw request bodies; record only non-sensitive metadata.
logger.debug("Request body read", {length = #body})
```
`body()`, `body_json()`, `stream()`, and `parse_multipart()` consume the same request body. Choose one body-reading path per handler. `body()` and `body_json()` enforce the request object's timeout and size limit; `stream()` is incremental and does not apply those two options.
#### `body_json`
Read and parse the request body as JSON:
```lua
local data, err = req:body_json()
if err then
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.BAD_REQUEST)
if status_err then return nil, status_err end
return res:write_json({error = "Invalid JSON: " .. err:message()})
end
-- Validate required fields
if not data.name or not data.email then
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.BAD_REQUEST)
if status_err then return nil, status_err end
return res:write_json({error = "Missing required fields"})
end
local user = create_user(data)
```
#### `has_body`
Check whether the request has a body.
```lua
local has_body, body_state_err = req:has_body()
if body_state_err then return nil, body_state_err end
if has_body then
local data, body_err = req:body_json()
if body_err then return nil, body_err end
process(data)
else
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.BAD_REQUEST)
if status_err then return nil, status_err end
return res:write_json({error = "Request body required"})
end
```
`has_body()` returns `true` only when the request has a body object and a positive `Content-Length`. A chunked request or another request with unknown length can return `false`; handlers that permit such bodies should attempt their chosen body reader and handle its error instead.
#### `is_content_type`
Check whether the request has the specified content type.
```lua
local is_json, type_check_err = req:is_content_type("application/json")
if type_check_err then return nil, type_check_err end
if not is_json then
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(415)
if status_err then return nil, status_err end
return res:write_json({error = "Content-Type must be application/json"})
end
```
#### `accepts`
Check whether the request accepts the specified content type.
```lua
local accepts_json, json_accept_err = req:accepts("application/json")
if json_accept_err then return nil, json_accept_err end
local accepts_html, html_accept_err = req:accepts("text/html")
if html_accept_err then return nil, html_accept_err end
if accepts_json then
return res:write_json(data)
elseif accepts_html then
local type_err = res:set_content_type("text/html; charset=utf-8")
if type_err then return nil, type_err end
return res:write(render_html(data))
else
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.NOT_ACCEPTABLE)
if status_err then return nil, status_err end
return res:write_json({error = "Cannot produce acceptable response"})
end
```
The pinned `accepts()` helper performs exact comma-separated matches plus `*/*`; it does not process media-type parameters, subtype wildcards, or quality weights, and a missing `Accept` header returns `false`. Use application-owned negotiation when those HTTP semantics matter.
#### `remote_addr`
Return the client's remote network address.
```lua
local addr, addr_err = req:remote_addr() -- "192.168.1.100:54321"
if addr_err then return nil, addr_err end
-- Extract the host from IPv4 and bracketed IPv6 addresses
local ip = addr:match("^%[([^%]]+)%]:%d+$")
or addr:match("^([^:]+):%d+$")
or addr
-- Rate limiting by IP
if rate_limiter:is_limited(ip) then
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.TOO_MANY_REQUESTS)
if status_err then return nil, status_err end
return res:write_json({error = "Too many requests"})
end
```
#### `parse_multipart`
Parse multipart form data, including file uploads. The optional `max_memory` argument sets the number of bytes held in memory before data spills to temporary files; the default is 32 MB.
```lua
local uuid = require("uuid")
local form, err = req:parse_multipart() -- or req:parse_multipart(8 * 1024 * 1024)
if err then
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.BAD_REQUEST)
if status_err then return nil, status_err end
return res:write_json({error = "Invalid form data"})
end
-- Access form values
local title = form.values.title
local description = form.values.description
-- Access uploaded files
if form.files.avatar then
local file = form.files.avatar[1]
local filename, name_err = file:name() -- untrusted client metadata
if name_err then return nil, name_err end
local size, size_err = file:size()
if size_err then return nil, size_err end
local content_type, header_err = file:header("Content-Type") -- "image/jpeg"
if header_err then return nil, header_err end
-- Read file content
local stream = file:stream()
local parts = {}
while true do
local chunk, err = stream:read(65536)
if err or not chunk then break end
parts[#parts + 1] = chunk
end
stream:close()
local content = table.concat(parts)
local stream, stream_err = file:stream()
if stream_err then return nil, stream_err end
local stored_name, id_err = uuid.v7()
if id_err then
stream:close()
return nil, id_err
end
local _, write_err = uploads:writefile(stored_name, stream, "wx")
local _, close_err = stream:close()
if write_err then return nil, write_err end
if close_err then return nil, close_err end
end
-- Handle multiple files
if form.files.documents then
for _, file in ipairs(form.files.documents) do
process_document(file)
end
end
```
Multipart field values are strings when a field occurs once and arrays when it occurs repeatedly. Treat uploaded filenames and `Content-Type` values as untrusted metadata; generate the storage name and inspect the content independently when its type matters.
The exclusive `wx` write prevents overwriting an existing object. A failed write does not prove that the target belongs to this request, so the failure path must not remove it blindly. Applications that need cleanup after partial writes should stage uploads under an ownership-tracked temporary name and promote them only after the write succeeds.
#### `stream`
Read the request body as a stream:
```lua
local stream, stream_err = req:stream()
if stream_err then return nil, stream_err end
-- Process in chunks
local read_err
while true do
local chunk
chunk, read_err = stream:read(65536) -- 64KB chunks
if read_err or not chunk then break end
process_chunk(chunk)
end
local _, close_err = stream:close()
if read_err then return nil, read_err end
if close_err then return nil, close_err end
```
#### `set_status`
Set the response status code.
`set_status()` writes the status and commits the response headers immediately. Call `set_header()`, `set_content_type()`, or `set_transfer()` first; later header changes return `errors.INVALID`.
```lua
local status_err = res:set_status(http.STATUS.CREATED)
if status_err then return nil, status_err end
-- Other common choices: 204 No Content, 400 Bad Request,
-- 401 Unauthorized, 403 Forbidden, 404 Not Found, and 500 Internal Error.
```
#### `set_header`
Set one response header.
```lua
local request_id_err = res:set_header("X-Request-ID", correlation_id)
if request_id_err then return nil, request_id_err end
local cache_err = res:set_header("Cache-Control", "max-age=3600")
if cache_err then return nil, cache_err end
local rate_err = res:set_header("X-RateLimit-Remaining", tostring(remaining))
if rate_err then return nil, rate_err end
-- CORS headers
local origin_err = res:set_header("Access-Control-Allow-Origin", "*")
if origin_err then return nil, origin_err end
local methods_err = res:set_header("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE")
if methods_err then return nil, methods_err end
local headers_err = res:set_header("Access-Control-Allow-Headers", "Content-Type, Authorization")
if headers_err then return nil, headers_err end
```
#### `set_content_type`
Set the response content type.
```lua
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
-- Other examples: "text/html; charset=utf-8" or "application/pdf".
```
#### `write`
Write to the response body:
```lua
local write_err = res:write("Hello, World!")
if write_err then return nil, write_err end
-- Build response incrementally
for _, fragment in ipairs({
"",
"
Title
",
"
Content
",
""
}) do
local fragment_err = res:write(fragment)
if fragment_err then return nil, fragment_err end
end
```
#### `write_json`
Encode a value as JSON and write it to the response:
```lua
-- Success response
local write_err = res:write_json({
data = users,
total = count,
page = page
})
if write_err then return nil, write_err end
-- Error response
local type_err = res:set_content_type(http.CONTENT.JSON)
if type_err then return nil, type_err end
local status_err = res:set_status(http.STATUS.BAD_REQUEST)
if status_err then return nil, status_err end
local error_write_err = res:write_json({
error = "Validation failed",
details = {
{field = "email", message = "Invalid format"},
{field = "age", message = "Must be positive"}
}
})
if error_write_err then return nil, error_write_err end
```
`write()`, `write_json()`, `flush()`, and `write_event()` also commit headers. `write_json()` sets `Content-Type: application/json` only when headers have not already been committed.
#### `flush`
Flush buffered response data to the client:
-- Stream progress updates
for i = 1, 100 do
local write_err = res:write(string.format("Progress: %d%%\n", i))
if write_err then return nil, write_err end
local flush_err = res:flush()
if flush_err then return nil, flush_err end
local _, sleep_err = time.sleep("100ms")
if sleep_err then return nil, sleep_err end
end
#### `set_transfer`
Set the transfer mode for a streaming response:
```lua
-- Chunked transfer
local transfer_err = res:set_transfer(http.TRANSFER.CHUNKED)
if transfer_err then return nil, transfer_err end
for chunk in get_chunks() do
local write_err = res:write(chunk)
if write_err then return nil, write_err end
local flush_err = res:flush()
if flush_err then return nil, flush_err end
end
-- Server-Sent Events
local sse_err = res:set_transfer(http.TRANSFER.SSE)
if sse_err then return nil, sse_err end
```
#### `write_event`
Write a Server-Sent Event:
```lua
-- Real-time updates
local transfer_err = res:set_transfer(http.TRANSFER.SSE)
if transfer_err then return nil, transfer_err end
local connected_err = res:write_event({name = "connected", data = {client_id = client_id}})
if connected_err then return nil, connected_err end
for progress in task:progress() do
local event_err = res:write_event({name = "progress", data = {percent = progress}})
if event_err then return nil, event_err end
end
local complete_err = res:write_event({name = "complete", data = {result = result}})
if complete_err then return nil, complete_err end
-- Chat messages
local message_err = res:write_event({name = "message", data = {
from = "alice",
text = "Hello!",
timestamp = time.now():unix()
}})
if message_err then return nil, message_err end
```
#### HTTP Methods
```lua
http.METHOD.GET
http.METHOD.POST
http.METHOD.PUT
http.METHOD.DELETE
http.METHOD.PATCH
http.METHOD.HEAD
http.METHOD.OPTIONS
```
#### Status Codes
```lua
-- Success (2xx)
http.STATUS.OK -- 200
http.STATUS.CREATED -- 201
http.STATUS.ACCEPTED -- 202
http.STATUS.NO_CONTENT -- 204
http.STATUS.PARTIAL_CONTENT -- 206
-- Redirect (3xx)
http.STATUS.MOVED_PERMANENTLY -- 301
http.STATUS.FOUND -- 302
http.STATUS.SEE_OTHER -- 303
http.STATUS.NOT_MODIFIED -- 304
http.STATUS.TEMPORARY_REDIRECT -- 307
http.STATUS.PERMANENT_REDIRECT -- 308
-- Client Error (4xx)
http.STATUS.BAD_REQUEST -- 400
http.STATUS.UNAUTHORIZED -- 401
http.STATUS.PAYMENT_REQUIRED -- 402
http.STATUS.FORBIDDEN -- 403
http.STATUS.NOT_FOUND -- 404
http.STATUS.METHOD_NOT_ALLOWED -- 405
http.STATUS.NOT_ACCEPTABLE -- 406
http.STATUS.CONFLICT -- 409
http.STATUS.GONE -- 410
http.STATUS.UNPROCESSABLE -- 422
http.STATUS.TOO_MANY_REQUESTS -- 429
-- Server Error (5xx)
http.STATUS.INTERNAL_ERROR -- 500 (alias: INTERNAL_SERVER_ERROR)
http.STATUS.NOT_IMPLEMENTED -- 501
http.STATUS.BAD_GATEWAY -- 502
http.STATUS.SERVICE_UNAVAILABLE -- 503
http.STATUS.GATEWAY_TIMEOUT -- 504
http.STATUS.VERSION_NOT_SUPPORTED -- 505
```
#### Content Types
```lua
http.CONTENT.JSON -- "application/json"
http.CONTENT.FORM -- "application/x-www-form-urlencoded"
http.CONTENT.MULTIPART -- "multipart/form-data"
http.CONTENT.TEXT -- "text/plain"
http.CONTENT.STREAM -- "application/octet-stream"
```
#### Transfer Modes
```lua
http.TRANSFER.CHUNKED -- "chunked"
http.TRANSFER.SSE -- "sse"
```
#### Legacy Error-Type Constants
The module exports these compatibility strings, but current request and response methods do not return them. Runtime failures use the structured `errors.*` kinds described below.
```lua
http.ERROR.PARSE_FAILED -- Form/multipart parse error
http.ERROR.INVALID_STATE -- Invalid response state
http.ERROR.WRITE_FAILED -- Response write error
http.ERROR.STREAM_ERROR -- Body stream error
```
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| No HTTP context | `errors.INTERNAL` | no |
| Body too large | `errors.INVALID` | no |
| Read timeout | `errors.INTERNAL` | no |
| Invalid JSON | `errors.INVALID` | no |
| Not multipart | `errors.INVALID` | no |
| Headers already sent | `errors.INVALID` | no |
| Write failed | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "HTTP Client"
## HTTP Client
The `http_client` module sends HTTP requests with headers, query parameters, forms, file uploads, authentication, TLS options, streaming responses, and concurrent batches.
This is an API reference with partial request recipes. URLs, tokens, credentials, request data, and certificate material come from the surrounding application. Examples check `Response, error` before consuming a response and close streamed bodies explicitly.
### Loading
```lua
local http_client = require("http_client")
```
Add `http_client` to the executable entry's `modules:` list before requiring it. JSON and filesystem recipes also require `json` and `fs`.
### HTTP Methods
Convenience methods use the `method(url, options?)` signature and return `Response, error`.
#### GET
Send a `GET` request.
```lua
local resp, err = http_client.get("https://api.example.com/users")
if err then
return nil, err
end
print(resp.status_code) -- 200
print(resp.body) -- response body
```
#### POST
Send a `POST` request.
```lua
local json = require("json")
local body, body_err = json.encode({name = "Alice", email = "alice@example.com"})
if body_err then return nil, body_err end
local resp, err = http_client.post("https://api.example.com/users", {
headers = {["Content-Type"] = "application/json"},
body = body
})
if err then return nil, err end
```
#### PUT
Send a `PUT` request.
```lua
local body, body_err = json.encode({name = "Alice Smith"})
if body_err then return nil, body_err end
local resp, err = http_client.put("https://api.example.com/users/123", {
headers = {["Content-Type"] = "application/json"},
body = body
})
if err then return nil, err end
```
#### PATCH
Send a `PATCH` request.
```lua
local body, body_err = json.encode({status = "active"})
if body_err then return nil, body_err end
local resp, err = http_client.patch("https://api.example.com/users/123", {
headers = {["Content-Type"] = "application/json"},
body = body
})
if err then return nil, err end
```
#### DELETE
Send a `DELETE` request.
```lua
local resp, err = http_client.delete("https://api.example.com/users/123", {
headers = {["Authorization"] = "Bearer " .. token}
})
if err then return nil, err end
```
#### HEAD
A `HEAD` request returns headers without a response body.
```lua
local resp, err = http_client.head("https://cdn.example.com/file.zip")
if err then return nil, err end
local size = resp.headers["Content-Length"]
```
#### Custom Methods
Send a request using an explicit HTTP method string.
```lua
local resp, err = http_client.request("PROPFIND", "https://dav.example.com/folder", {
headers = {["Depth"] = "1"}
})
if err then return nil, err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `method` | string | HTTP method |
| `url` | string | Request URL |
| `options` | table | Request options (optional) |
### Request Options
| Field | Type | Description |
|-------|------|-------------|
| `headers` | table | Request headers `{["Name"] = "value"}` |
| `body` | string | Request body |
| `query` | table | Query parameters `{key = "value"}` |
| `form` | table | Form data (sets Content-Type automatically) |
| `files` | table | File uploads (array of file definitions) |
| `cookies` | table | Request cookies `{name = "value"}` |
| `auth` | table | Basic auth `{user = "name", pass = "secret"}` |
| `timeout` | number/string | Timeout: number in seconds, or string like `"30s"`, `"1m"` |
| `stream` | boolean | Stream response body instead of buffering |
| `max_response_body` | number | Max response size in bytes (0 = default) |
| `unix_socket` | string | Connect via Unix socket path |
| `tls` | table | Per-request TLS configuration (see [TLS Options](#tls-options)) |
| `overlay_network` | string | Route through a [network overlay](system/network.md) — registry ID of a `network.socks5` / `network.tailscale` / `network.i2p` entry |
Selecting `overlay_network` requires `network.select` permission on that network ID.
#### Query Parameters
```lua
local resp, err = http_client.get("https://api.example.com/search", {
query = {
q = "lua programming",
page = "1",
limit = "20"
}
})
if err then return nil, err end
```
#### Headers and Authentication
```lua
local resp, err = http_client.get("https://api.example.com/data", {
headers = {
["Authorization"] = "Bearer " .. token,
["Accept"] = "application/json"
}
})
if err then return nil, err end
-- Or use basic auth
local resp, err = http_client.get("https://api.example.com/data", {
auth = {user = service_user, pass = service_password}
})
if err then return nil, err end
```
Load authentication values from application-owned secret storage and send them only over TLS.
#### Form Data
```lua
local resp, err = http_client.post("https://api.example.com/login", {
form = {
username = username,
password = password
}
})
if err then return nil, err end
```
#### File Upload
```lua
local resp, err = http_client.post("https://api.example.com/upload", {
form = {title = "My Document"},
files = {
{
name = "attachment", -- form field name
filename = "report.pdf", -- original filename
content = pdf_data, -- file content
content_type = "application/pdf"
}
}
})
if err then return nil, err end
```
| File Field | Type | Required | Description |
|------------|------|----------|-------------|
| `name` | string | yes | Form field name |
| `filename` | string | no | Original filename |
| `content` | string | yes* | File content |
| `reader` | userdata | yes* | Alternative: io.Reader for content |
| `content_type` | string | no | Currently ignored: each uploaded part is always sent with `Content-Type: application/octet-stream` regardless of this field |
\* Either `content` or `reader` is required.
The pinned runtime fully reads a `reader` into memory before dispatch, does not close it, and does not surface a non-EOF read failure separately; it can send the bytes accumulated before that failure. Prefer `content` for already-bounded data, and close caller-owned readers after the request. The `content_type` field is parsed but not forwarded by runtime `v0.3.32a`, so uploaded parts use the transport default.
Reader-backed files are supported only by single-request calls in this release. `request_batch` forwards the `content` field but drops a parsed `reader`, so batch file uploads must provide `content`.
#### Timeout
```lua
-- Number: seconds
local resp, err = http_client.get(url, {timeout = 30})
if err then return nil, err end
-- String alternatives use Go duration format: "30s", "1m30s", or "1h".
```
#### TLS Options
Configure mutual TLS and custom CA certificates for one request.
| Field | Type | Description |
|-------|------|-------------|
| `cert` | string | Client certificate in PEM format |
| `key` | string | Client private key in PEM format |
| `ca` | string | Custom CA certificate in PEM format |
| `server_name` | string | Server name for SNI verification |
| `insecure_skip_verify` | boolean | Skip TLS certificate verification |
For mutual TLS, provide `cert` and `key` together. The `ca` field replaces the system certificate pool with a custom CA.
##### mTLS Authentication
```lua
local fs = require("fs")
local certs, volume_err = fs.get("app:certs")
if volume_err then return nil, volume_err end
local cert_pem, cert_err = certs:readfile("client.crt")
if cert_err then return nil, cert_err end
local key_pem, key_err = certs:readfile("client.key")
if key_err then return nil, key_err end
local resp, err = http_client.get("https://secure.example.com/api", {
tls = {
cert = cert_pem,
key = key_pem,
}
})
if err then return nil, err end
```
##### Custom CA
```lua
local fs = require("fs")
local certs, volume_err = fs.get("app:certs")
if volume_err then return nil, volume_err end
local ca_pem, ca_err = certs:readfile("internal-ca.crt")
if ca_err then return nil, ca_err end
local resp, err = http_client.get("https://internal.example.com/api", {
tls = {
ca = ca_pem,
server_name = "internal.example.com",
}
})
if err then return nil, err end
```
##### Insecure Skip Verify
`insecure_skip_verify` disables TLS verification and requires the `http_client.insecure_tls` security permission.
```lua
local resp, err = http_client.get("https://localhost:8443/api", {
tls = {
insecure_skip_verify = true,
}
})
if err then return nil, err end
```
Use `insecure_skip_verify` only for a controlled diagnostic endpoint. It disables both certificate-chain and hostname verification.
### Response Object
| Field | Type | Description |
|-------|------|-------------|
| `status_code` | number | HTTP status code |
| `body` | string | Response body (if not streaming) |
| `body_size` | number | Body size in bytes (-1 if streaming) |
| `headers` | table | Response headers |
| `cookies` | table | Response cookies |
| `url` | string | Final URL (after redirects) |
| `stream` | Stream | Stream object (if `stream = true`) |
```lua
local resp, err = http_client.get("https://api.example.com/data")
if err then
return nil, err
end
if resp.status_code == 200 then
local data, decode_err = json.decode(resp.body)
if decode_err then return nil, decode_err end
print("Content-Type:", resp.headers["Content-Type"])
end
```
### Streaming Responses
Set `stream = true` to process a response incrementally rather than buffering its full body.
```lua
local resp, err = http_client.get("https://cdn.example.com/large-file.zip", {
stream = true
})
if err then
return nil, err
end
-- Process in chunks
local read_err
while true do
local chunk
chunk, read_err = resp.stream:read(65536)
if read_err or not chunk then break end
-- process chunk
end
local _, close_err = resp.stream:close()
if read_err then return nil, read_err end
if close_err then return nil, close_err end
```
| Stream Method | Returns | Description |
|---------------|---------|-------------|
| `read(n?)` | string, error | Read up to `n` bytes (default: implementation buffer) |
| `close()` | boolean, error | Close the stream |
`resp.stream` is a full [stream](lua/core/stream.md) object — `seek`, `stat`, and `scanner` are also available. The caller owns a streamed response body and should close it on every exit; task cleanup is a fallback, not a substitute for prompt release.
### Batch Requests
`request_batch` executes multiple requests concurrently.
```lua
local requests = {
{"GET", "https://api.example.com/users"},
{"GET", "https://api.example.com/products"},
{"POST", "https://api.example.com/log", {body = "event"}}
}
local responses, batch_errors = http_client.request_batch(requests)
if not responses then
return nil, batch_errors -- whole-batch dispatch or validation failure
end
if batch_errors then
for i = 1, #requests do
local err = batch_errors[i]
if err then
print("Request " .. i .. " failed:", err)
end
end
else
-- All succeeded
for i, resp in ipairs(responses) do
print("Response " .. i .. ":", resp.status_code)
end
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `requests` | table | Array of `{method, url, options?}` |
**Returns:** `responses, errors` — arrays indexed by request position
**Notes:**
- Requests execute concurrently
- Streaming (`stream = true`) is not supported in batch
- Reader-backed file uploads are not supported in batch; use `files[].content`
- Result arrays match request order (1-indexed)
#### Encode
Encode a string for inclusion in a URL.
```lua
local encoded = http_client.encode_uri("hello world")
-- "hello+world"
local url = "https://api.example.com/search?q=" .. http_client.encode_uri(query)
```
#### Decode
Decode a string previously encoded with `http_client.encode_uri`.
```lua
local decoded, err = http_client.decode_uri("hello+world")
if err then return nil, err end
-- "hello world"
```
### Permissions
HTTP requests are evaluated against the active security policy.
#### Security Actions
| Action | Resource | Description |
|--------|----------|-------------|
| `http_client.request` | URL | Allow/deny requests to specific URLs |
| `http_client.unix_socket` | Socket path | Allow/deny Unix socket connections |
| `http_client.private_ip` | IP address | Allow/deny access to private IP ranges |
| `http_client.insecure_tls` | URL | Allow/deny insecure TLS (skip verification) |
| `network.select` | Network entry ID | Allow/deny routing through the `overlay_network` given in the request |
#### Checking Access
```lua
local security = require("security")
if security.can("http_client.request", "https://api.example.com/users") then
local resp, request_err = http_client.get("https://api.example.com/users")
if request_err then return nil, request_err end
end
```
#### SSRF Protection
Non-public IP ranges are blocked by default. Access requires the `http_client.private_ip` permission on the address:
- loopback, private (10.x, 172.16-31.x, 192.168.x), link-local unicast and multicast, and the unspecified address
- carrier-grade NAT `100.64.0.0/10`, `192.0.0.0/24`, multicast `224.0.0.0/4`, reserved `240.0.0.0/4`
- documentation and benchmarking ranges `192.0.2.0/24`, `198.18.0.0/15`, `198.51.100.0/24`, `203.0.113.0/24`, `2001:db8::/32`
- IPv6 multicast `ff00::/8`
```lua
local resp, err = http_client.get("http://192.168.1.1/admin")
-- Error: not allowed: private IP 192.168.1.1
```
The check runs at dial time, not on the URL string, and it covers every address the host resolves to. A hostname that resolves to several addresses is checked address by address: a denied address is skipped and the next one is tried, and the request fails only when every candidate is denied or unreachable. A public hostname that resolves to a private address is therefore blocked exactly like a private IP literal.
#### Redirects
Up to nine redirects are followed; the tenth fails with `stopped after 10 redirects`, a count that includes the original request.
Every hop is authorized on its own. Before following a redirect the client evaluates `http_client.request` against the target URL and applies the private-IP check to it, so a permitted URL cannot be used to reach a denied one by redirection. A hop that fails either check aborts the request.
See [Security Model](system/security.md) for policy configuration.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Security policy denied | `errors.PERMISSION_DENIED` | no |
| Private IP blocked | `errors.PERMISSION_DENIED` | no |
| Unix socket denied | `errors.PERMISSION_DENIED` | no |
| Insecure TLS denied | `errors.PERMISSION_DENIED` | no |
| Invalid batch item, batch streaming, or invalid URI escape | `errors.INVALID` | no |
| No context | `errors.INTERNAL` | no |
| Malformed transport URL or network failure | `errors.INTERNAL` | yes |
| Timeout | `errors.INTERNAL` | yes |
Many unsupported option values are ignored rather than returned as structured errors. Invalid Lua argument types and an empty batch raise Lua argument errors. Validate application-supplied option tables before calling the client.
```lua
local resp, err = http_client.get(url)
if err then
if errors.is(err, errors.PERMISSION_DENIED) then
print("Access denied:", err:message())
elseif err:retryable() then
print("Temporary error:", err:message())
end
return nil, err
end
```
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "WebSocket Client"
## WebSocket Client
The `websocket` module creates bidirectional client connections to WebSocket servers.
This is an API reference with partial connection and subscription recipes. Endpoint URLs, tokens, message handlers, and application data come from the surrounding application. The lifecycle examples close the client on every terminal or checked error path; smaller method snippets assume an enclosing owner performs that cleanup.
### Loading
```lua
local websocket = require("websocket")
```
Add `websocket` to the executable entry's `modules:` list before requiring it. The `channel` global is always available; JSON and timeout recipes also require `json` and `time`.
#### `connect`
Open a WebSocket connection with the default options:
```lua
local client, err = websocket.connect("wss://api.example.com/ws")
if err then
return nil, err
end
```
Pass an options table to configure the connection:
```lua
local client, err = websocket.connect("wss://api.example.com/ws", {
headers = {
["Authorization"] = "Bearer " .. token
},
protocols = {"graphql-ws"},
dial_timeout = "10s",
read_timeout = "30s",
compression = websocket.COMPRESSION.CONTEXT_TAKEOVER
})
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `url` | string | WebSocket URL (ws:// or wss://) |
| `options` | table | Connection options (optional) |
**Returns:** `Client, error`
##### Connection Options
| Option | Type | Description |
|--------|------|-------------|
| `headers` | table | HTTP headers for handshake |
| `protocols` | table | WebSocket subprotocols |
| `dial_timeout` | number/string | Connection timeout (ms or "5s") |
| `read_timeout` | number/string | Read timeout |
| `write_timeout` | number/string | Write timeout |
| `compression` | number/string | Compression mode (see Constants), or `"disabled"`, `"context_takeover"`, `"no_context_takeover"` |
| `compression_threshold` | number | Min size to compress (0-100MB) |
| `read_limit` | number | Max message size (0-128MB) |
| `channel_capacity` | number | Receive channel buffer (1-10000) |
**Timeout format:** Numbers are milliseconds. Strings use Go duration syntax such as `"5s"` or `"1m"`.
Invalid timeout strings and out-of-range or unsupported option values are ignored, leaving the corresponding default in effect.
#### Text Messages
Send a text message.
```lua
local json = require("json")
client:send("Hello, Server!")
-- Send JSON
local payload, encode_err = json.encode({
type = "subscribe",
channel = "orders"
})
if encode_err then return nil, encode_err end
client:send(payload)
```
#### Binary Messages
Send a binary message by specifying `websocket.BINARY`.
```lua
client:send(binary_data, websocket.BINARY)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Message content |
| `type` | number | `websocket.TEXT` (1) or `websocket.BINARY` (2) |
Yields until the message is sent. Returns no values.
#### Ping
Send a ping frame.
```lua
client:ping()
```
Yields until the ping is sent. Returns no values.
### Receiving Messages
`channel()` returns the receive channel, and `receive()` is an alias. The first call yields while the runtime creates the subscription; later calls return the same channel immediately. A subscription failure returns `nil, error`. The channel can be used with `channel.select`.
#### Basic Receive
```lua
local ch, err = client:channel()
if err then
client:close()
return nil, err
end
local msg, ok = ch:receive()
if ok then
print("Type:", msg.type) -- "text" or "binary"
print("Data:", msg.data)
end
local _, close_err = client:close()
if close_err then return nil, close_err end
```
#### Message Loop
```lua
local json = require("json")
local ch, err = client:channel()
if err then
client:close()
return nil, err
end
while true do
local msg, ok = ch:receive()
if not ok then
break -- Connection closed
end
if msg.type == "text" then
local data, decode_err = json.decode(msg.data)
if decode_err then
client:close()
return nil, decode_err
end
handle_message(data)
end
end
local _, close_err = client:close()
if close_err then return nil, close_err end
```
#### With Select
```lua
local json = require("json")
local time = require("time")
local ch, ch_err = client:channel()
if ch_err then
client:close()
return nil, ch_err
end
local timeout, timeout_err = time.after("30s")
if timeout_err then
client:close()
return nil, timeout_err
end
while true do
local r = channel.select {
ch:case_receive(),
timeout:case_receive()
}
if r.channel == timeout then
client:ping() -- Keep-alive
timeout, timeout_err = time.after("30s")
if timeout_err then
client:close()
return nil, timeout_err
end
elseif not r.ok then
break
else
local data, decode_err = json.decode(r.value.data)
if decode_err then
client:close()
return nil, decode_err
end
process(data)
end
end
local _, close_err = client:close()
if close_err then return nil, close_err end
```
#### Message Object
| Field | Type | Description |
|-------|------|-------------|
| `type` | string | `"text"` or `"binary"` |
| `data` | string? | Message content (nil for unknown payload types) |
### Closing Connection
Close the connection with an optional status code and reason:
```lua
local _, close_err = client:close(websocket.CLOSE_CODES.NORMAL, "Session ended")
if close_err then return nil, close_err end
-- Omitting both arguments also uses normal close code 1000.
-- Use INTERNAL_ERROR with an application-owned reason for a failed session.
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `code` | number | Close code (1000-4999), default 1000 |
| `reason` | string | Close reason (optional) |
The call yields until the close command completes. Success returns no values; a close failure returns `nil, error`. Capture two results when checking it, because the error is the second result. Values outside the accepted numeric range are ignored and the default code `1000` is used.
The receive channel is owned by the client; do not close it directly. A remote terminal event closes the channel. Calling `client:close()` unsubscribes the receive channel and stops the client-side producer, so use it promptly rather than relying on process shutdown cleanup.
#### Message Types
```lua
-- Numeric (for send)
websocket.TEXT -- 1
websocket.BINARY -- 2
-- Compatibility string constants
websocket.TYPE_TEXT -- "text"
websocket.TYPE_BINARY -- "binary"
websocket.TYPE_PING -- "ping"
websocket.TYPE_PONG -- "pong"
websocket.TYPE_CLOSE -- "close"
```
Receive-channel message objects use only `"text"` and `"binary"`. Ping and pong frames are handled by the transport, and a terminal event closes the channel instead of producing a `"close"` message object.
#### Compression Modes
```lua
websocket.COMPRESSION.DISABLED -- 0 (no compression)
websocket.COMPRESSION.CONTEXT_TAKEOVER -- 1 (sliding window)
websocket.COMPRESSION.NO_CONTEXT -- 2 (per-message)
```
#### Close Codes
| Constant | Code | Description |
|----------|------|-------------|
| `NORMAL` | 1000 | Normal closure |
| `GOING_AWAY` | 1001 | Server shutting down |
| `PROTOCOL_ERROR` | 1002 | Protocol error |
| `UNSUPPORTED_DATA` | 1003 | Unsupported data type |
| `RESERVED` | 1004 | Reserved |
| `NO_STATUS` | 1005 | No status received |
| `ABNORMAL_CLOSURE` | 1006 | Connection lost |
| `INVALID_PAYLOAD` | 1007 | Invalid frame payload |
| `POLICY_VIOLATION` | 1008 | Policy violation |
| `MESSAGE_TOO_BIG` | 1009 | Message too large |
| `MANDATORY_EXTENSION` | 1010 | Required extension not negotiated |
| `INTERNAL_ERROR` | 1011 | Server error |
| `SERVICE_RESTART` | 1012 | Server restarting |
| `TRY_AGAIN_LATER` | 1013 | Server overloaded |
| `BAD_GATEWAY` | 1014 | Gateway error |
| `TLS_HANDSHAKE` | 1015 | TLS handshake failure |
```lua
local _, close_err = client:close(websocket.CLOSE_CODES.NORMAL, "Done")
if close_err then return nil, close_err end
```
#### Real-Time Chat
```lua
local json = require("json")
local function connect_chat(room_id, token, on_message)
local client, err = websocket.connect("wss://chat.example.com/ws", {
headers = {["Authorization"] = "Bearer " .. token}
})
if err then
return nil, err
end
-- Join room. Runtime v0.3.32a does not expose transport send failures.
local join_payload, encode_err = json.encode({
type = "join",
room = room_id
})
if encode_err then
client:close()
return nil, encode_err
end
client:send(join_payload)
-- Message loop
local ch, channel_err = client:channel()
if channel_err then
client:close()
return nil, channel_err
end
while true do
local msg, ok = ch:receive()
if not ok then break end
local data, decode_err = json.decode(msg.data)
if decode_err then
client:close()
return nil, decode_err
end
on_message(data)
end
local _, close_err = client:close()
if close_err then return nil, close_err end
return true
end
```
#### Price Stream with Keep-Alive
```lua
local json = require("json")
local time = require("time")
local client, err = websocket.connect("wss://stream.example.com/prices")
if err then
return nil, err
end
local subscribe_payload, encode_err = json.encode({
action = "subscribe",
symbols = {"BTC-USD", "ETH-USD"}
})
if encode_err then
client:close()
return nil, encode_err
end
client:send(subscribe_payload)
local ch, channel_err = client:channel()
if channel_err then
client:close()
return nil, channel_err
end
local heartbeat, heartbeat_err = time.after("30s")
if heartbeat_err then
client:close()
return nil, heartbeat_err
end
while true do
local r = channel.select {
ch:case_receive(),
heartbeat:case_receive()
}
if r.channel == heartbeat then
client:ping()
heartbeat, heartbeat_err = time.after("30s")
if heartbeat_err then
client:close()
return nil, heartbeat_err
end
elseif not r.ok then
break -- Connection closed
else
local price, decode_err = json.decode(r.value.data)
if decode_err then
client:close()
return nil, decode_err
end
update_price(price.symbol, price.value)
end
end
local _, close_err = client:close()
if close_err then return nil, close_err end
```
### Permissions
WebSocket connections are evaluated against the active security policy.
#### Security Actions
| Action | Resource | Description |
|--------|----------|-------------|
| `websocket.connect` | - | Allow/deny WebSocket connections |
| `websocket.connect.url` | URL | Allow/deny connections to specific URLs |
See [Security Model](system/security.md) for policy configuration.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Connections disabled | `errors.PERMISSION_DENIED` | no |
| URL not allowed | `errors.PERMISSION_DENIED` | no |
| No context | `errors.INTERNAL` | no |
| Connection failed | `errors.INTERNAL` | yes |
| Invalid connection ID returned by the dispatcher | `errors.INTERNAL` | no |
| Subscription failed | `errors.INTERNAL` | yes |
| Missing process context during subscription | `errors.INTERNAL` | no |
| Close failed | `errors.INTERNAL` | no |
An empty URL, a non-table options value, invalid argument types, and a missing execution context or process PID when requesting the receive channel raise Lua errors. They are not returned as structured errors. Runtime `v0.3.32a` does not expose send or ping transport failures to Lua callers.
```lua
local client, err = websocket.connect(url)
if err then
if errors.is(err, errors.PERMISSION_DENIED) then
print("Access denied:", err:message())
elseif err:retryable() then
print("Temporary error:", err:message())
end
return nil, err
end
```
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "HTML Sanitization"
## HTML Sanitization
The `html` module sanitizes untrusted HTML with policies based on [bluemonday](https://github.com/microcosm-cc/bluemonday).
Sanitization parses an HTML fragment and filters it through an allowlist policy. Elements and attributes that the policy does not allow are removed, and the remaining fragment is normalized during serialization.
This is an API reference. Constructor blocks are self-contained policy snippets; later method blocks are partial configuration snippets that assume `policy` is an already-created policy. Sanitized output is suitable only for an HTML element-content context. It is not safe for JavaScript, CSS, URL, or HTML attribute interpolation; use an encoder for the actual output context.
### Loading
```lua
local html = require("html")
```
Add `html` to the executable entry's `modules:` list before requiring it.
### Preset Policies
The module provides three preset policy constructors:
| Policy | Use Case | Allows |
|--------|----------|--------|
| `new_policy` | Custom sanitization | Nothing (build from scratch) |
| `ugc_policy` | User comments, forums | Common formatting (`p`, `b`, `i`, `a`, lists, etc.) |
| `strict_policy` | Plain text extraction | Nothing (strips all HTML) |
All three constructors return `Policy, nil`; policy construction does not currently fail.
#### Empty Policy
Create an empty policy, then add the elements and attributes it should allow:
```lua
local policy, err = html.sanitize.new_policy()
if err then return nil, err end
policy:allow_elements("p", "strong", "em")
policy:allow_attrs("class"):globally()
local clean = policy:sanitize(user_input)
```
**Returns:** `Policy, error`
#### User Content Policy
Create a policy configured for common user-generated formatting:
```lua
local policy, err = html.sanitize.ugc_policy()
if err then return nil, err end
local safe = policy:sanitize('
Hello world
')
-- '
Hello world
'
local xss = policy:sanitize('
Hello
')
-- '
Hello
'
```
**Returns:** `Policy, error`
#### Strict Policy
Create a strict policy that removes HTML and returns plain text:
```lua
local policy, err = html.sanitize.strict_policy()
if err then return nil, err end
local text = policy:sanitize('
Hello world!
')
-- 'Hello world!'
```
**Returns:** `Policy, error`
#### Allow Elements
Allow specific HTML elements:
```lua
local policy, err = html.sanitize.new_policy()
if err then return nil, err end
policy:allow_elements("p", "strong", "em", "br")
policy:allow_elements("h1", "h2", "h3")
policy:allow_elements("a", "img")
local result = policy:sanitize('
Hello world
')
-- '
Hello world
'
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `...` | string | Element tag names |
**Returns:** `Policy`
#### Allow Attributes
Start an attribute rule, then apply it with `on_elements()` or `globally()`:
```lua
policy:allow_attrs("href"):on_elements("a")
policy:allow_attrs("src", "alt"):on_elements("img")
policy:allow_attrs("class", "id"):globally()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `...` | string | Attribute names |
**Returns:** `AttrBuilder`
#### On Specific Elements
Allow attributes only on specified elements:
```lua
policy:allow_elements("a", "img")
policy:allow_attrs("href", "target"):on_elements("a")
policy:allow_attrs("src", "alt", "width", "height"):on_elements("img")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `...` | string | Element tag names |
**Returns:** `Policy`
#### On All Elements
Allow attributes on every permitted element:
```lua
policy:allow_attrs("class"):globally()
policy:allow_attrs("id"):globally()
```
**Returns:** `Policy`
#### With Pattern Matching
Require attribute values to match a regular expression:
```lua
-- Only allow hex colors in style
local builder, err = policy:allow_attrs("style"):matching("^color:#[0-9a-fA-F]{6}$")
if err then
return nil, err
end
builder:on_elements("span")
policy:sanitize('Red')
-- 'Red'
policy:sanitize('Bad')
-- 'Bad'
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `pattern` | string | Go RE2-compatible regular expression |
**Returns:** `AttrBuilder, error`
#### Standard URLs
Enable the standard URL-handling policy. It requires parseable URLs, permits relative URLs plus `mailto`, `http`, and `https`, and adds `rel="nofollow"` to allowed linking elements:
```lua
policy:allow_elements("a")
policy:allow_attrs("href"):on_elements("a")
policy:allow_standard_urls()
```
**Returns:** `Policy`
#### URL Schemes
Allow specific URL schemes:
```lua
policy:allow_url_schemes("https", "mailto")
policy:sanitize('OK')
-- 'OK'
policy:sanitize('XSS')
-- 'XSS'
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `...` | string | Schemes to allow |
**Returns:** `Policy`
#### Relative URLs
Configure whether relative URLs are allowed:
```lua
policy:allow_relative_urls(true)
policy:sanitize('Link')
-- 'Link'
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `allow` | boolean | Allow relative URLs |
**Returns:** `Policy`
#### Require Parseable URLs
Reject URLs that fail to parse cleanly. With `true`, attribute URLs that the HTML sanitizer cannot parse are stripped instead of passed through.
```lua
policy:require_parseable_urls(true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `require` | boolean | Require URLs to be parseable |
**Returns:** `Policy`
#### Nofollow Links
Add `rel="nofollow"` to links:
```lua
policy:allow_attrs("href", "rel"):on_elements("a")
policy:allow_url_schemes("https")
policy:require_parseable_urls(true)
policy:require_nofollow_on_links(true)
policy:sanitize('Link')
-- 'Link'
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `require` | boolean | Add nofollow |
**Returns:** `Policy`
#### Noreferrer Links
Add `rel="noreferrer"` to links:
```lua
policy:require_noreferrer_on_links(true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `require` | boolean | Add noreferrer |
**Returns:** `Policy`
#### External Links in New Tab
Add `target="_blank"` to fully qualified URLs:
```lua
policy:allow_attrs("href", "target"):on_elements("a")
policy:allow_url_schemes("https")
policy:require_parseable_urls(true)
policy:add_target_blank_to_fully_qualified_links(true)
policy:sanitize('Link')
-- 'Link'
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `add` | boolean | Add target blank |
**Returns:** `Policy`
When opening untrusted links in a new tab, also enable `require_noreferrer_on_links(true)` to suppress referrer leakage and mitigate opener access.
#### Allow Images
Permit `` with `align`, `alt`, `height`, `width`, and `src`. This helper also enables the standard URL policy but does not allow data URI images.
```lua
policy:allow_images()
policy:sanitize('')
-- ''
```
**Returns:** `Policy`
#### Allow Data URI Images
Permit syntactically valid Base64-encoded `gif`, `jpeg`, `png`, `svg+xml`, or `webp` data URI images. The sanitizer validates the media type and Base64 encoding, not the decoded image contents. Data URIs can carry active content, so enable them only for content whose image data you trust:
```lua
policy:allow_elements("img")
policy:allow_attrs("src"):on_elements("img")
policy:allow_data_uri_images()
local input = ''
policy:sanitize(input)
-- The data URI is preserved.
```
**Returns:** `Policy`
#### Allow Lists
Permit `ul`, `ol`, `li`, `dl`, `dt`, and `dd`. The helper also allows validated `type` attributes on `ul`, `ol`, and `li`, plus an integer `value` attribute on `li`.
```lua
policy:allow_lists()
policy:sanitize('
'
```
**Returns:** `Policy`
#### Allow Standard Attributes
Permit common attributes: `id`, `title`, `dir`, `lang`.
```lua
policy:allow_elements("p")
policy:allow_standard_attributes()
policy:sanitize('
Hello
')
-- '
Hello
'
```
**Returns:** `Policy`
### Sanitize
Apply a policy to an HTML string:
```lua
local policy, err = html.sanitize.ugc_policy()
if err then return nil, err end
policy:require_nofollow_on_links(true)
local dirty = '
Hello
'
local clean = policy:sanitize(dirty)
-- '
Hello
'
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `html` | string | HTML to sanitize |
**Returns:** `string`
`sanitize` returns only a string. In runtime `v0.3.32a`, the underlying fragment parser can turn malformed input that it cannot parse into an empty string, and the Lua wrapper cannot distinguish that case from valid input whose content the policy removed. Treat sanitization as output filtering, not input validation; validate required content separately when an empty result matters.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Invalid regex pattern | `errors.INVALID` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "SQL Database"
## SQL Database
The `sql` module runs queries against configured PostgreSQL, MySQL, and SQLite databases. It supports parameterized queries, transactions, prepared statements, and query builders.
This page is an API reference. Its snippets assume a configured database, permission to acquire it, and any tables named by the query. They illustrate individual calls rather than a standalone application. The combined recipe at the end states its additional schema and driver assumptions.
For database configuration, see [Database](system/database.md).
### Loading
```lua
local sql = require("sql")
```
### `sql.get`
Acquire a database connection from the resource registry:
```lua
local db, err = sql.get("app.db:main")
if err then
return nil, err
end
local function finish(value, primary_err)
local _, release_err = db:release()
if primary_err then return nil, primary_err end
if release_err then return nil, release_err end
return value
end
local rows, err = db:query("SELECT * FROM users WHERE active = ?", {1})
if err then
return finish(nil, err)
end
return finish(rows)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Resource ID (e.g., "app.db:main") |
**Returns:** `DB, error`
Database leases are released during execution-frame cleanup. Call `db:release()` explicitly when database work finishes, especially in long-running operations.
Placeholders are passed to the database driver unchanged; the runtime does not rewrite them. SQLite and MySQL use `?`, PostgreSQL uses `$1, $2` — write them in the form your driver expects. The examples below use `?` (SQLite/MySQL). For queries that target more than one engine, build them with the [query builder](#query-builder): `run_with` rewrites placeholders to `$1, $2` when the handle is PostgreSQL, and `to_sql` uses the builder's `placeholder_format`.
#### Database Types
```lua
sql.type.POSTGRES -- "postgres"
sql.type.MYSQL -- "mysql"
sql.type.SQLITE -- "sqlite"
sql.type.UNKNOWN -- "unknown"
```
#### Isolation Levels
```lua
sql.isolation.DEFAULT -- "default"
sql.isolation.READ_UNCOMMITTED -- "read_uncommitted"
sql.isolation.READ_COMMITTED -- "read_committed"
sql.isolation.WRITE_COMMITTED -- "write_committed"
sql.isolation.REPEATABLE_READ -- "repeatable_read"
sql.isolation.SERIALIZABLE -- "serializable"
```
#### NULL Value
```lua
local insert = sql.builder.insert("users")
:columns("name", "email")
:values("alice", sql.NULL)
```
#### `sql.as.int`
Coerce a value to the SQL integer type.
```lua
local value = sql.as.int(42)
```
**Returns:** `userdata`
#### `sql.as.float`
Coerce a value to the SQL float type.
```lua
local value = sql.as.float(19.99)
```
**Returns:** `userdata`
#### `sql.as.text`
Coerce a value to the SQL text type.
```lua
local value = sql.as.text("hello")
```
**Returns:** `userdata`
#### `sql.as.binary`
Coerce a value to the SQL binary type.
```lua
local value = sql.as.binary("binary data")
```
**Returns:** `userdata`
#### `sql.as.null`
Return the SQL `NULL` marker.
```lua
local value = sql.as.null()
```
**Returns:** `userdata`
#### `sql.builder.select`
Create a `SELECT` query builder.
```lua
local query = sql.builder.select("id", "name")
:from("users")
:where({active = 1})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `columns` | ...string | Column names (optional) |
**Returns:** `SelectBuilder`
#### `sql.builder.insert`
Create an `INSERT` query builder.
```lua
local query = sql.builder.insert("users")
:columns("name", "email")
:values("alice", "alice@example.com")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name (optional) |
**Returns:** `InsertBuilder`
#### `sql.builder.update`
Create an `UPDATE` query builder.
```lua
local query = sql.builder.update("users")
:set("status", "active")
:where({id = 123})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name (optional) |
**Returns:** `UpdateBuilder`
#### `sql.builder.delete`
Create a `DELETE` query builder.
```lua
local query = sql.builder.delete("users")
:where({active = 0})
:limit(100)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name (optional) |
**Returns:** `DeleteBuilder`
#### `sql.builder.expr`
Create a raw SQL expression for use in `WHERE` or `HAVING` clauses.
```lua
local expr = sql.builder.expr("score BETWEEN ? AND ?", 80, 90)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL expression with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `Sqlizer`
#### `sql.builder.eq`
Create equality conditions from a table.
```lua
local cond = sql.builder.eq({active = 1, status = "open"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.not_eq`
Create inequality conditions from a table.
```lua
local cond = sql.builder.not_eq({status = "closed"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.lt`
Create less-than conditions from a table.
```lua
local cond = sql.builder.lt({age = 18})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.lte`
Create less-than-or-equal conditions from a table.
```lua
local cond = sql.builder.lte({price = 100})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.gt`
Create greater-than conditions from a table.
```lua
local cond = sql.builder.gt({score = 80})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.gte`
Create greater-than-or-equal conditions from a table.
```lua
local cond = sql.builder.gte({age = 21})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.like`
Create `LIKE` conditions from a table.
```lua
local cond = sql.builder.like({name = "john%"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.not_like`
Create `NOT LIKE` conditions from a table.
```lua
local cond = sql.builder.not_like({email = "%@spam.com"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `Sqlizer`
#### `sql.builder.and_`
Combine multiple conditions with `AND`.
```lua
local cond = sql.builder.and_({
sql.builder.eq({active = 1}),
sql.builder.gt({score = 80})
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `conditions` | table | Array of Sqlizer or table conditions |
**Returns:** `Sqlizer`
#### `sql.builder.or_`
Combine multiple conditions with `OR`.
```lua
local cond = sql.builder.or_({
sql.builder.eq({status = "pending"}),
sql.builder.eq({status = "active"})
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `conditions` | table | Array of Sqlizer or table conditions |
**Returns:** `Sqlizer`
### sqlizer:to_sql
Generates the SQL fragment and bind arguments of a condition.
```lua
local frag, args = sql.builder.eq({active = 1}):to_sql()
```
**Returns:** `string, table`
### builder.question
Use `?` placeholders (default). This format is also available as `sql.builder.default_placeholder`.
```lua
local query = sql.builder.select("*")
:from("users")
:placeholder_format(sql.builder.question)
```
#### `sql.builder.dollar`
Use `$1, $2, ...` placeholders.
```lua
local query = sql.builder.select("*")
:from("users")
:placeholder_format(sql.builder.dollar)
```
#### `sql.builder.at`
Use `@p1, @p2, ...` placeholders (SQL Server style). Pass this format to `placeholder_format` like the formats above.
#### `sql.builder.colon`
Use `:1, :2, ...` placeholders. Pass this format to `placeholder_format` like the formats above.
### Connection Methods
A connection handle returned by `sql.get()` provides query, transaction, statement, and pool operations.
#### `db:type`
Return the database type constant.
```lua
local dbtype, err = db:type()
```
**Returns:** `string, error`
#### `db:query`
Run a `SELECT` query and return its rows.
```lua
local rows, err = db:query("SELECT id, name FROM users WHERE active = ?", {1})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL query with ? placeholders |
| `params` | table | Array of bind parameters (optional) |
**Returns:** `table[], error`
#### `db:execute`
Run an `INSERT`, `UPDATE`, or `DELETE` statement.
```lua
local result, err = db:execute("INSERT INTO users (name) VALUES (?)", {"alice"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL statement with ? placeholders |
| `params` | table | Array of bind parameters (optional) |
**Returns:** `table, error`
Returns table with fields:
- `last_insert_id` - Last inserted ID
- `rows_affected` - Number of rows affected
#### `db:prepare`
Create a prepared statement for repeated execution.
```lua
local stmt, err = db:prepare("SELECT * FROM users WHERE id = ?")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL with ? placeholders |
**Returns:** `Statement, error`
#### `db:begin`
Begin a database transaction.
```lua
local tx, err = db:begin({
isolation = sql.isolation.SERIALIZABLE,
read_only = false
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `options` | table | Transaction options (optional) |
Options table fields:
- `isolation` - Isolation level from sql.isolation.* (default: DEFAULT)
- `read_only` - Read-only transaction flag (default: false)
**Returns:** `Transaction, error`
#### `db:release`
Release the database resource back to the pool.
```lua
local ok, err = db:release()
```
**Returns:** `boolean, error`
The operation is idempotent.
#### `db:stats`
Return connection-pool statistics.
```lua
local stats, err = db:stats()
```
**Returns:** `table, error`
Returns table with fields:
- `max_open_connections` - Max allowed open connections
- `open_connections` - Current open connections
- `in_use` - Connections currently in use
- `idle` - Idle connections in pool
- `wait_count` - Total connection wait count
- `wait_duration` - Total wait duration
- `max_idle_closed` - Connections closed due to max idle
- `max_idle_time_closed` - Connections closed due to idle timeout
- `max_lifetime_closed` - Connections closed due to max lifetime
### Prepared Statements
A prepared statement returned by `db:prepare()` can be queried or executed repeatedly.
#### `stmt:query`
Run the prepared statement as a `SELECT` query.
```lua
local rows, err = stmt:query({123})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `params` | table | Array of bind parameters (optional) |
**Returns:** `table[], error`
#### `stmt:execute`
Run the prepared statement as an `INSERT`, `UPDATE`, or `DELETE` statement.
```lua
local result, err = stmt:execute({"alice"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `params` | table | Array of bind parameters (optional) |
**Returns:** `table, error`
Returns table with fields:
- `last_insert_id` - Last inserted ID
- `rows_affected` - Number of rows affected
#### `stmt:close`
Close the prepared statement.
```lua
local ok, err = stmt:close()
```
**Returns:** `boolean, error`
### Transactions
A transaction returned by `db:begin()` provides query, statement, savepoint, commit, and rollback operations.
An active transaction is rolled back automatically during execution-frame cleanup. Commit or roll it back explicitly as soon as its work is complete.
#### `tx:db_type`
Return the database type constant.
```lua
local dbtype, err = tx:db_type()
```
**Returns:** `string, error`
#### `tx:query`
Run a `SELECT` query within the transaction.
```lua
local rows, err = tx:query("SELECT id, name FROM users WHERE active = ?", {1})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL query with ? placeholders |
| `params` | table | Array of bind parameters (optional) |
**Returns:** `table[], error`
#### `tx:execute`
Run an `INSERT`, `UPDATE`, or `DELETE` statement within the transaction.
```lua
local result, err = tx:execute("INSERT INTO users (name) VALUES (?)", {"alice"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL statement with ? placeholders |
| `params` | table | Array of bind parameters (optional) |
**Returns:** `table, error`
Returns table with fields:
- `last_insert_id` - Last inserted ID
- `rows_affected` - Number of rows affected
#### `tx:prepare`
Create a prepared statement within the transaction.
```lua
local stmt, err = tx:prepare("SELECT * FROM users WHERE id = ?")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL with ? placeholders |
**Returns:** `Statement, error`
#### `tx:commit`
Commit the transaction.
```lua
local ok, err = tx:commit()
```
**Returns:** `boolean, error`
#### `tx:rollback`
Roll back the transaction.
```lua
local ok, err = tx:rollback()
```
**Returns:** `boolean, error`
#### `tx:savepoint`
Create a named savepoint within the transaction.
```lua
local ok, err = tx:savepoint("sp1")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Savepoint name (alphanumeric and underscore only) |
**Returns:** `boolean, error`
#### `tx:rollback_to`
Roll back to a named savepoint.
```lua
local ok, err = tx:rollback_to("sp1")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Savepoint name |
**Returns:** `boolean, error`
#### `tx:release`
Release a savepoint.
```lua
local ok, err = tx:release("sp1")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Savepoint name |
**Returns:** `boolean, error`
### SELECT Builder
Build a `SELECT` query one clause at a time.
#### `select:from`
Set the `FROM` clause.
```lua
local query = sql.builder.select("id", "name"):from("users")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name |
**Returns:** `SelectBuilder`
#### `select:join`
Add a `JOIN` clause.
```lua
local query = sql.builder.select("*")
:from("users")
:join("orders ON orders.user_id = users.id")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `join` | string | JOIN clause with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `SelectBuilder`
#### `select:left_join`
Add a `LEFT JOIN` clause.
```lua
local query = sql.builder.select("*")
:from("users")
:left_join("orders ON orders.user_id = users.id")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `join` | string | JOIN clause with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `SelectBuilder`
#### `select:right_join`
Add a `RIGHT JOIN` clause.
```lua
local query = sql.builder.select("*")
:from("users")
:right_join("orders ON orders.user_id = users.id")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `join` | string | JOIN clause with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `SelectBuilder`
#### `select:inner_join`
Add an `INNER JOIN` clause.
```lua
local query = sql.builder.select("*")
:from("users")
:inner_join("orders ON orders.user_id = users.id")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `join` | string | JOIN clause with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `SelectBuilder`
#### `select:where`
Add a `WHERE` condition.
```lua
local query = sql.builder.select("*")
:from("users")
:where({active = 1})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `condition` | string\|table\|Sqlizer | WHERE condition |
| `args` | ...any | Bind arguments (optional, when using string) |
The method accepts three formats:
- String: `where("status = ?", "active")`
- Table: `where({status = "active"})`
- Sqlizer: `where(sql.builder.gt({score = 80}))`
**Returns:** `SelectBuilder`
#### `select:order_by`
Add an `ORDER BY` clause.
```lua
local query = sql.builder.select("*")
:from("users")
:order_by("name ASC", "created_at DESC")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `columns` | ...string | Column names with optional ASC/DESC |
**Returns:** `SelectBuilder`
#### `select:group_by`
Add a `GROUP BY` clause.
```lua
local query = sql.builder.select("status", "COUNT(*)")
:from("users")
:group_by("status")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `columns` | ...string | Column names |
**Returns:** `SelectBuilder`
#### `select:having`
Add a `HAVING` condition.
```lua
local query = sql.builder.select("status", "COUNT(*) as cnt")
:from("users")
:group_by("status")
:having(sql.builder.gt({cnt = 10}))
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `condition` | string\|table\|Sqlizer | HAVING condition |
| `args` | ...any | Bind arguments (optional, when using string) |
**Returns:** `SelectBuilder`
#### `select:limit`
Set the `LIMIT` value.
```lua
local query = sql.builder.select("*")
:from("users")
:limit(10)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | Limit value |
**Returns:** `SelectBuilder`
#### `select:offset`
Set the `OFFSET` value.
```lua
local query = sql.builder.select("*")
:from("users")
:offset(20)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | Offset value |
**Returns:** `SelectBuilder`
#### `select:columns`
Add columns to the `SELECT` list.
```lua
local query = sql.builder.select():columns("id", "name", "email")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `columns` | ...string | Column names |
**Returns:** `SelectBuilder`
#### `select:distinct`
Add the `DISTINCT` modifier.
```lua
local query = sql.builder.select("status")
:from("users")
:distinct()
```
**Returns:** `SelectBuilder`
#### `select:suffix`
Add an SQL suffix.
```lua
local query = sql.builder.select("*")
:from("users")
:suffix("FOR UPDATE")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL suffix with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `SelectBuilder`
#### `select:placeholder_format`
Set the placeholder format.
```lua
local query = sql.builder.select("*")
:from("users")
:placeholder_format(sql.builder.dollar)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `format` | userdata | Placeholder format (sql.builder.*) |
**Returns:** `SelectBuilder`
#### `select:to_sql`
Generate the SQL string and bind arguments.
```lua
local sql_str, args = query:to_sql()
```
**Returns:** `string, table` on success; `nil, error` for an invalid builder state
#### `select:run_with`
Create an executor for the query.
```lua
local executor, err = query:run_with(db)
if err then
return nil, err
end
local rows, err = executor:query()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `db` | DB\|Transaction | Database or transaction handle |
**Returns:** `QueryExecutor, error`
### INSERT Builder
Build an `INSERT` query one clause at a time.
#### `insert:into`
Set the table name.
```lua
local query = sql.builder.insert():into("users")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name |
**Returns:** `InsertBuilder`
#### `insert:columns`
Set the column names.
```lua
local query = sql.builder.insert("users"):columns("name", "email")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `columns` | ...string | Column names |
**Returns:** `InsertBuilder`
#### `insert:values`
Add row values.
```lua
local query = sql.builder.insert("users")
:columns("name", "email")
:values("alice", "alice@example.com")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `values` | ...any | Row values |
**Returns:** `InsertBuilder`
#### `insert:set_map`
Set columns and values from a table.
```lua
local query = sql.builder.insert("users")
:set_map({name = "alice", email = "alice@example.com"})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs |
**Returns:** `InsertBuilder`
#### `insert:select`
Insert rows from a `SELECT` query.
```lua
local select_query = sql.builder.select("name", "email"):from("temp_users")
local query = sql.builder.insert("users")
:columns("name", "email")
:select(select_query)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | SelectBuilder | SELECT query |
**Returns:** `InsertBuilder`
#### `insert:prefix`
Add an SQL prefix.
```lua
local query = sql.builder.insert("users")
:prefix("/* audit import */")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL prefix with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `InsertBuilder`
#### `insert:suffix`
Add an SQL suffix.
```lua
local query = sql.builder.insert("users")
:columns("name")
:values("alice")
:suffix("RETURNING id")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL suffix with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `InsertBuilder`
#### `insert:options`
Add `INSERT` options.
```lua
local query = sql.builder.insert("users")
:options("DELAYED", "IGNORE")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `options` | ...string | INSERT options |
**Returns:** `InsertBuilder`
#### `insert:placeholder_format`
Set the placeholder format.
```lua
local query = sql.builder.insert("users")
:placeholder_format(sql.builder.dollar)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `format` | userdata | Placeholder format (sql.builder.*) |
**Returns:** `InsertBuilder`
#### `insert:to_sql`
Generate the SQL string and bind arguments.
```lua
local sql_str, args = query:to_sql()
```
**Returns:** `string, table` on success; `nil, error` for an invalid builder state
#### `insert:run_with`
Create an executor for the query.
```lua
local executor, err = query:run_with(db)
if err then
return nil, err
end
local result, err = executor:exec()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `db` | DB\|Transaction | Database or transaction handle |
**Returns:** `QueryExecutor, error`
### UPDATE Builder
Build an `UPDATE` query one clause at a time.
#### `update:table`
Set the table name.
```lua
local query = sql.builder.update():table("users")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name |
**Returns:** `UpdateBuilder`
#### `update:set`
Set a column value.
```lua
local query = sql.builder.update("users")
:set("status", "active")
:set("updated_at", sql.builder.expr("NOW()"))
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `column` | string | Column name |
| `value` | any | Column value |
**Returns:** `UpdateBuilder`
#### `update:set_map`
Set multiple columns from a table.
```lua
local query = sql.builder.update("users")
:set_map({status = "active", login_count = 0})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `map` | table | {column = value} pairs; values are plain values, `sql.NULL`, or `sql.as.*` (use `set` for expressions) |
**Returns:** `UpdateBuilder`
#### `update:where`
Add a `WHERE` condition.
```lua
local query = sql.builder.update("users")
:set("status", "active")
:where({id = 123})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `condition` | string\|table\|Sqlizer | WHERE condition |
| `args` | ...any | Bind arguments (optional, when using string) |
**Returns:** `UpdateBuilder`
#### `update:order_by`
Add an `ORDER BY` clause.
```lua
local query = sql.builder.update("users")
:set("rank", 1)
:order_by("score DESC")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `columns` | ...string | Column names with optional ASC/DESC |
**Returns:** `UpdateBuilder`
#### `update:limit`
Set the `LIMIT` value.
```lua
local query = sql.builder.update("users")
:set("status", "active")
:limit(10)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | Limit value |
**Returns:** `UpdateBuilder`
#### `update:offset`
Set the `OFFSET` value.
```lua
local query = sql.builder.update("users")
:set("status", "active")
:offset(5)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | Offset value |
**Returns:** `UpdateBuilder`
#### `update:suffix`
Add an SQL suffix.
```lua
local query = sql.builder.update("users")
:set("status", "active")
:suffix("RETURNING id")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL suffix with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `UpdateBuilder`
#### `update:from`
Add a `FROM` clause.
```lua
local query = sql.builder.update("users")
:set("status", "active")
:from("other_table")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name |
**Returns:** `UpdateBuilder`
#### `update:from_select`
Update rows from a `SELECT` query.
```lua
local select_query = sql.builder.select("*"):from("temp_users")
local query = sql.builder.update("users")
:set("status", "active")
:from_select(select_query, "t")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | SelectBuilder | SELECT query |
| `alias` | string | Table alias |
**Returns:** `UpdateBuilder`
#### `update:placeholder_format`
Set the placeholder format.
```lua
local query = sql.builder.update("users")
:placeholder_format(sql.builder.dollar)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `format` | userdata | Placeholder format (sql.builder.*) |
**Returns:** `UpdateBuilder`
#### `update:to_sql`
Generate the SQL string and bind arguments.
```lua
local sql_str, args = query:to_sql()
```
**Returns:** `string, table` on success; `nil, error` for an invalid builder state
#### `update:run_with`
Create an executor for the query.
```lua
local executor, err = query:run_with(db)
if err then
return nil, err
end
local result, err = executor:exec()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `db` | DB\|Transaction | Database or transaction handle |
**Returns:** `QueryExecutor, error`
### DELETE Builder
Build a `DELETE` query one clause at a time.
#### `delete:from`
Set the table name.
```lua
local query = sql.builder.delete():from("users")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `table` | string | Table name |
**Returns:** `DeleteBuilder`
#### `delete:where`
Add a `WHERE` condition.
```lua
local query = sql.builder.delete("users")
:where({active = 0})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `condition` | string\|table\|Sqlizer | WHERE condition |
| `args` | ...any | Bind arguments (optional, when using string) |
**Returns:** `DeleteBuilder`
#### `delete:order_by`
Add an `ORDER BY` clause.
```lua
local query = sql.builder.delete("users")
:where({active = 0})
:order_by("created_at ASC")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `columns` | ...string | Column names with optional ASC/DESC |
**Returns:** `DeleteBuilder`
#### `delete:limit`
Set the `LIMIT` value.
```lua
local query = sql.builder.delete("users")
:where({active = 0})
:limit(100)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | Limit value |
**Returns:** `DeleteBuilder`
#### `delete:offset`
Set the `OFFSET` value.
```lua
local query = sql.builder.delete("users")
:where({active = 0})
:offset(10)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | Offset value |
**Returns:** `DeleteBuilder`
#### `delete:suffix`
Add an SQL suffix.
```lua
local query = sql.builder.delete("users")
:where({active = 0})
:suffix("RETURNING id")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `sql` | string | SQL suffix with ? placeholders |
| `args` | ...any | Bind arguments (optional) |
**Returns:** `DeleteBuilder`
#### `delete:placeholder_format`
Set the placeholder format.
```lua
local query = sql.builder.delete("users")
:placeholder_format(sql.builder.dollar)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `format` | userdata | Placeholder format (sql.builder.*) |
**Returns:** `DeleteBuilder`
#### `delete:to_sql`
Generate the SQL string and bind arguments.
```lua
local sql_str, args = query:to_sql()
```
**Returns:** `string, table` on success; `nil, error` for an invalid builder state
#### `delete:run_with`
Create an executor for the query.
```lua
local executor, err = query:run_with(db)
if err then
return nil, err
end
local result, err = executor:exec()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `db` | DB\|Transaction | Database or transaction handle |
**Returns:** `QueryExecutor, error`
### Executing Queries
The query executor runs builder-generated queries.
#### `executor:query`
Run the query and return rows for a `SELECT` statement.
```lua
local rows, err = executor:query()
```
**Returns:** `table[], error`
#### `executor:exec`
Run the query and return the result of an `INSERT`, `UPDATE`, or `DELETE` statement.
```lua
local result, err = executor:exec()
```
**Returns:** `table, error`
Returns table with fields:
- `last_insert_id` - Last inserted ID
- `rows_affected` - Number of rows affected
#### `executor:to_sql`
Return the generated SQL and arguments without executing the query.
```lua
local sql_str, args = executor:to_sql()
```
**Returns:** `string, table`
### Permissions
Database access is subject to security policy evaluation.
| Action | Resource | Description |
|--------|----------|-------------|
| `db.get` | Database ID | Acquire database connection |
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty resource ID | `errors.INVALID` | no |
| Permission denied | `errors.PERMISSION_DENIED` | no |
| Resource not found | `errors.NOT_FOUND` | no |
| Resource not database | `errors.INVALID` | no |
| Invalid parameters | `errors.INVALID` | no |
| SQL syntax error | `errors.UNKNOWN` | nil |
| Statement closed | `errors.INVALID` | no |
| Transaction not active | `errors.INVALID` | no |
| Invalid savepoint name | `errors.INVALID` | no |
| Query execution error | `errors.UNKNOWN` | nil |
See [Error Handling](lua/core/errors.md) for working with errors.
### Combined Partial Recipe
This recipe assumes `app.db:main` is a configured SQLite or MySQL database and already contains `users`, `orders`, and `logs` tables with the referenced columns. It uses `?` placeholders; use `$1`, `$2`, and so on for a PostgreSQL resource. Returned rows depend on the application's data. The surrounding application supplies `report_cleanup_error(err)` so rollback or close failures are observable without replacing the initiating operation error.
```lua
local sql = require("sql")
local db, err = sql.get("app.db:main")
if err then return nil, err end
local function finish(value, primary_err)
local _, release_err = db:release()
if primary_err then return nil, primary_err end
if release_err then return nil, release_err end
return value
end
-- Direct query
local users, err = db:query("SELECT id, name FROM users WHERE active = ?", {1})
if err then
return finish(nil, err)
end
for _, user in ipairs(users) do
print(user.id, user.name)
end
-- Builder pattern
local query = sql.builder.select("u.id", "u.name", "COUNT(o.id) as order_count")
:from("users u")
:left_join("orders o ON o.user_id = u.id")
:where(sql.builder.and_({
sql.builder.eq({["u.active"] = 1}),
sql.builder.gte({["u.score"] = 80})
}))
:group_by("u.id", "u.name")
:having(sql.builder.gt({["COUNT(o.id)"] = 0}))
:order_by("order_count DESC")
:limit(10)
local executor, build_err = query:run_with(db)
if build_err then
return finish(nil, build_err)
end
local results, err = executor:query()
if err then
return finish(nil, err)
end
-- Transaction
local tx, err = db:begin({isolation = sql.isolation.SERIALIZABLE})
if err then
return finish(nil, err)
end
local _, err = tx:execute("INSERT INTO users (name) VALUES (?)", {"alice"})
if err then
local _, rollback_err = tx:rollback()
if rollback_err then report_cleanup_error(rollback_err) end
return finish(nil, err)
end
local _, commit_err = tx:commit()
if commit_err then
return finish(nil, commit_err)
end
-- Prepared statements
local stmt, err = db:prepare("INSERT INTO logs (message, level) VALUES (?, ?)")
if err then
return finish(nil, err)
end
for i = 1, 3 do
local _, err = stmt:execute({"log message " .. i, "info"})
if err then
local _, close_err = stmt:close()
if close_err then report_cleanup_error(close_err) end
return finish(nil, err)
end
end
local _, close_err = stmt:close()
if close_err then
return finish(nil, close_err)
end
return finish({users = users, ranked_users = results})
```
---
# "Key-Value Store"
## Key-Value Store
The `store` module provides key-value storage with optional TTLs. It can hold cached data, sessions, and other temporary state.
This page is an API reference. Its snippets assume a configured store, the permissions listed below, and application-provided values such as `owner` or `new_value`. Snippets after acquisition use an existing live `cache` handle and are not standalone functions.
For store configuration, see [Store](system/store.md).
### Loading
```lua
local store = require("store")
```
### Acquiring a Store
Acquire a store resource by registry ID:
```lua
local cache, err = store.get("app:cache")
if err then
return nil, err
end
local _, set_err = cache:set("user:123", {name = "Alice"}, 3600)
if set_err then
cache:release()
return nil, set_err
end
local user, get_err = cache:get("user:123")
cache:release()
if get_err then return nil, get_err end
return user
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Store resource ID |
**Returns:** `Store, error`
### Storing Values
Store a value with an optional TTL:
```lua
-- Simple set
local _, err = cache:set("user:123:name", "Alice")
if err then return nil, err end
-- Set with TTL (expires in 300 seconds)
local ok, ttl_err = cache:set("session:abc", {user_id = 123, role = "admin"}, 300)
if ttl_err then return nil, ttl_err end
return ok
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Key |
| `value` | any | Value (tables, strings, numbers, booleans) |
| `ttl` | number | TTL in seconds (optional, 0 = no expiry) |
**Returns:** `boolean, error`
### Retrieving Values
Retrieve a value by key:
```lua
local errors = require("errors")
local user, err = cache:get("user:123")
if err then
if err:kind() == errors.NOT_FOUND then
return nil -- key missing or expired
end
return nil, err
end
return user
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Key to retrieve |
**Returns:** `any, error`
The method returns `nil` and an `errors.NOT_FOUND` error when the key does not exist or has expired.
### Checking Existence
Check whether a key exists without retrieving its value:
```lua
if cache:has("lock:" .. resource_id) then
return nil, errors.new({ kind = errors.CONFLICT, message = "Resource is locked" })
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Key to check |
**Returns:** `boolean, error`
### Deleting Keys
Remove a key from the store:
```lua
local deleted, err = cache:delete("session:" .. session_id)
if err then return nil, err end
return deleted
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Key to delete |
**Returns:** `boolean, error`
The method returns `true` when it deletes the key and `false` when the key does not exist.
### Reading Entry Metadata
`entry` returns the value together with its `version` — an opaque string used for optimistic concurrency:
```lua
local e, err = cache:entry("user:123")
if err then return nil, err end
if e then
print(e.key, e.value, e.version)
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Key to read |
**Returns:** `Entry, error` — `{key: string, value: any, version: string}`
### Listing Keys
List entries in deterministic key order with pagination:
```lua
local page, err = cache:list({ prefix = "session:", limit = 100 })
if err then return nil, err end
for _, e in ipairs(page.items) do
print(e.key, e.value)
end
-- next page
if page.has_more then
local next_page, next_err = cache:list({ prefix = "session:", after = page.cursor })
if next_err then return nil, next_err end
page = next_page
end
```
| Option | Type | Description |
|--------|------|-------------|
| `prefix` | string | Only keys with this prefix |
| `after` | string | Continue after this cursor (from a previous page) |
| `limit` | integer | Max items per page |
**Returns:** `Page, error` — `{items: Entry[], cursor: string, has_more: boolean}`
### Conditional Writes
`put` writes a value and returns its new `Entry`. Options enable optimistic concurrency:
```lua
local errors = require("errors")
-- create only if the key does not exist
local e, err = cache:put("lock:job-1", owner, { only_if_absent = true })
if err and err:kind() == errors.ALREADY_EXISTS then
-- someone else holds it
elseif err then
return nil, err
end
-- compare-and-set: write only if the version still matches
local cur, read_err = cache:entry("config")
if read_err then return nil, read_err end
local e2, err2 = cache:put("config", new_value, { if_version = cur.version })
if err2 and err2:kind() == errors.CONFLICT then
-- a concurrent writer changed it; re-read and retry
elseif err2 then
return nil, err2
end
```
| Option | Type | Description |
|--------|------|-------------|
| `ttl` | number | TTL in seconds |
| `only_if_absent` | boolean | Write only if the key does not exist |
| `if_version` | string | Write only if the current version matches |
`only_if_absent` and `if_version` are mutually exclusive.
**Returns:** `Entry, error`
Conditional writes require a store whose info().conditional_put is true (the memory and store.kv.raft stores). On store.kv.crdt and store.sql they return an errors.INVALID error — use store.kv.raft when you need conditional writes.
### Store Capabilities
`info` reports the backend and what it supports, so code can adapt to whichever store is bound:
```lua
local info, err = cache:info()
if err then return nil, err end
-- info.backend -> one of store.backend.* (e.g. "kv.raft")
-- info.consistency -> one of store.consistency.* (e.g. "linearizable")
-- info.durable / info.list / info.versioned / info.conditional_put / info.ttl (booleans)
```
**Returns:** `Info, error` — `{id, backend, consistency, durable, list, versioned, conditional_put, ttl}`
#### Constants
| Constant | Values |
|----------|--------|
| `store.backend` | `MEMORY`, `SQL`, `KV_RAFT`, `KV_CRDT`, `UNKNOWN` |
| `store.consistency` | `LINEARIZABLE`, `EVENTUAL`, `LOCAL`, `UNKNOWN` |
```lua
local info, err = cache:info()
if err then return nil, err end
if info.consistency == store.consistency.LINEARIZABLE then
-- safe to use compare-and-set
end
```
### Store Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `get(key)` | `any, error` | Retrieve value by key |
| `entry(key)` | `Entry, error` | Retrieve value with version metadata |
| `set(key, value, ttl?)` | `boolean, error` | Store value with optional TTL |
| `put(key, value, opts?)` | `Entry, error` | Conditional/versioned write, returns the new entry |
| `list(opts?)` | `Page, error` | Paged listing in key order |
| `has(key)` | `boolean, error` | Check if key exists |
| `delete(key)` | `boolean, error` | Remove key |
| `info()` | `Info, error` | Backend, consistency, and capability flags |
| `release()` | `boolean` | Release store back to pool |
### Permissions
Security policy evaluation applies to store operations.
| Action | Resource | Attributes | Description |
|--------|----------|------------|-------------|
| `store.get` | Store ID | - | Acquire a store resource |
| `store.info` | Store ID | - | Inspect store capabilities |
| `store.key.get` | Store ID | `key` | Read a key value (also `entry`) |
| `store.key.set` | Store ID | `key` | Write a key value (also `put`) |
| `store.key.delete` | Store ID | `key` | Delete a key |
| `store.key.has` | Store ID | `key` | Check key existence |
| `store.key.list` | Store ID | `prefix` | List entries |
Permission denials from `store.get`, `get`, `set`, `delete`, and `has` raise a Lua error. The `info`, `entry`, `list`, and `put` methods instead return an `errors.PERMISSION_DENIED` error. Grant the required actions before calling code that cannot tolerate a raised denial.
### Errors
`store.get()` and all methods on the store handle (`get`, `entry`, `set`, `put`, `list`, `has`, `delete`, `info`) return structured errors (use `err:kind()`), except that a permission denial in `store.get`, `get`, `set`, `has` and `delete` raises a Lua error instead.
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty resource ID | `errors.INVALID` | no |
| Resource not found | `errors.INTERNAL` | no |
| Store released | `errors.INVALID` | no |
| Permission denied (`entry`, `put`, `list`, `info`) | `errors.PERMISSION_DENIED` | no |
| `only_if_absent` and key exists | `errors.ALREADY_EXISTS` | no |
| `if_version` mismatch | `errors.CONFLICT` | yes |
| Conditional write on a store without support | `errors.INVALID` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Filesystem"
## Filesystem
The `fs` module reads, writes, and manages files within configured filesystem volumes.
This page is an API reference. Its snippets assume a configured volume and permission to acquire it. Each block is an isolated operation or partial recipe; application values and callbacks such as `config`, `message`, `process`, and `report_cleanup_error` must already exist. `report_cleanup_error(err)` records a close failure without replacing an operation error that already occurred.
For filesystem configuration, see [Filesystem](system/filesystem.md).
### Loading
```lua
local fs = require("fs")
```
### Acquiring a Volume
Acquire a filesystem volume by registry ID:
```lua
local vol, err = fs.get("app:storage")
if err then
return nil, err
end
local content, read_err = vol:readfile("/config.json")
if read_err then return nil, read_err end
return content
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Volume registry ID |
**Returns:** `FS, error`
Volumes do not require explicit release. The system manages them, and a volume becomes unavailable when its filesystem is detached from the registry.
### Reading Files
Read an entire file:
```lua
local json = require("json")
local vol, get_err = fs.get("app:config")
if get_err then return nil, get_err end
local data, err = vol:readfile("/settings.json")
if err then
return nil, err
end
local config, decode_err = json.decode(data)
if decode_err then return nil, decode_err end
return config
```
Use `open()` to stream a large file:
```lua
local errors = require("errors")
local file, err = vol:open("/data/large.csv", "r")
if err then
return nil, err
end
while true do
local chunk, err = file:read(65536)
if err then
if err:kind() == errors.NOT_FOUND then
break -- EOF
end
local _, close_err = file:close()
if close_err then report_cleanup_error(close_err) end
return nil, err
end
process(chunk)
end
local _, close_err = file:close()
if close_err then return nil, close_err end
```
### Writing Files
Write a string or reader-backed stream to a file:
```lua
local json = require("json")
local vol, get_err = fs.get("app:data")
if get_err then return nil, get_err end
-- Overwrite (default)
local encoded, encode_err = json.encode(config)
if encode_err then return nil, encode_err end
local _, write_err = vol:writefile("/config.json", encoded)
if write_err then return nil, write_err end
-- Append
local _, append_err = vol:writefile("/logs/app.log", message .. "\n", "a")
if append_err then return nil, append_err end
-- Exclusive write (fails if exists)
local ok, err = vol:writefile("/lock.pid", tostring(pid), "wx")
if err then return nil, err end
-- Copy from an open file or another reader-backed value
local source, err = vol:open("/incoming/report.csv", "r")
if err then
return nil, err
end
local copied, err = vol:writefile("/archive/report.csv", source)
local _, close_err = source:close()
if err then
if close_err then report_cleanup_error(close_err) end
return nil, err
end
if close_err then return nil, close_err end
return copied
```
| Mode | Description |
|------|-------------|
| `"w"` | Overwrite (default) |
| `"a"` | Append |
| `"wx"` | Exclusive write (fails if file exists) |
Use a file handle for streaming writes:
```lua
local file, open_err = vol:open("/output/report.txt", "w")
if open_err then return nil, open_err end
local _, header_err = file:write("Header\n")
if header_err then
local _, close_err = file:close()
if close_err then report_cleanup_error(close_err) end
return nil, header_err
end
local _, data_err = file:write("Data: " .. value .. "\n")
if data_err then
local _, close_err = file:close()
if close_err then report_cleanup_error(close_err) end
return nil, data_err
end
local _, sync_err = file:sync()
if sync_err then
local _, close_err = file:close()
if close_err then report_cleanup_error(close_err) end
return nil, sync_err
end
local _, close_err = file:close()
if close_err then return nil, close_err end
```
### Checking Paths
```lua
local vol, get_err = fs.get("app:data")
if get_err then return nil, get_err end
-- Check existence
local exists, exists_err = vol:exists("/cache/results.json")
if exists_err then return nil, exists_err end
if exists then
return vol:readfile("/cache/results.json")
end
-- Check if directory
local is_dir, isdir_err = vol:isdir(path)
if isdir_err then return nil, isdir_err end
if is_dir then
process_directory(path)
end
-- Get file info
local info, stat_err = vol:stat("/documents/report.pdf")
if stat_err then return nil, stat_err end
print(info.size, info.modified, info.type)
```
**Stat fields:** `name`, `size`, `mode`, `modified`, `is_dir`, `type`
### Directory Operations
```lua
local vol, get_err = fs.get("app:data")
if get_err then return nil, get_err end
-- Create directory
local _, mkdir_err = vol:mkdir("/uploads/" .. user_id)
if mkdir_err then return nil, mkdir_err end
-- List directory contents
local iter, state = vol:readdir("/documents")
if not iter then return nil, state end
for entry in iter, state do
print(entry.name, entry.type)
end
-- Remove file or empty directory
local removed, remove_err = vol:remove("/temp/file.txt")
if remove_err then return nil, remove_err end
return removed
```
Entry fields: `name`, `type` ("file" or "directory")
`mkdir` creates one directory and does not create missing parents. `remove` accepts files and empty directories only.
### File Handle Methods
When using `vol:open()` for streaming:
| Method | Description |
|--------|-------------|
| `read(size?)` | Read bytes (default: 4096) |
| `write(data)` | Write string data |
| `seek(whence, offset)` | Set position ("set", "cur", "end") |
| `stat()` | Get file info (same fields as `vol:stat`) |
| `sync()` | Flush to storage |
| `close()` | Release file handle |
| `scanner(split?)` | Create line/word scanner |
Call `close()` after finishing with a file handle.
### Scanner
Use a scanner for line-by-line processing:
```lua
local file, err = vol:open("/data/users.csv", "r")
if err then
return nil, err
end
local scanner, err = file:scanner("lines")
if err then
local _, close_err = file:close()
if close_err then report_cleanup_error(close_err) end
return nil, err
end
scanner:scan() -- skip header
while scanner:scan() do
local line = scanner:text()
process(line)
end
local scan_err = scanner:err()
if scan_err then
local _, close_err = file:close()
if close_err then report_cleanup_error(close_err) end
return nil, scan_err
end
local _, close_err = file:close()
if close_err then return nil, close_err end
```
Split modes: `"lines"` (default), `"words"`, `"bytes"`, `"runes"`
`scanner:scan()` returns only a boolean. When it returns `false`, call `scanner:err()` to distinguish clean EOF from a tokenization or underlying read failure. `scanner:err()` returns a structured `INTERNAL` error or `nil`; unlike a stream scanner, a file scanner has no separate scan-dispatch error return.
### Constants
```lua
fs.type.FILE -- "file"
fs.type.DIR -- "directory"
fs.seek.SET -- from start
fs.seek.CUR -- from current
fs.seek.END -- from end
```
### FS Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `readfile(path)` / `read_file(path)` | `string, error` | Read entire file |
| `writefile(path, data, mode?)` / `write_file(path, data, mode?)` | `boolean, error` | Write a string or reader-backed value |
| `exists(path)` | `boolean, error` | Check if path exists |
| `stat(path)` | `table, error` | Get file info |
| `isdir(path)` | `boolean, error` | Check if directory |
| `mkdir(path)` | `boolean, error` | Create directory |
| `remove(path)` | `boolean, error` | Remove file/empty dir |
| `readdir(path)` | `iterator, state` | List directory (use in generic `for` loop) |
| `open(path, mode)` | `File, error` | Open file handle |
| `chdir(path)` | `boolean, error` | Change working dir |
| `pwd()` | `string, error` | Get working dir |
### Permissions
Security policy evaluation applies when a volume is acquired.
| Action | Resource | Description |
|--------|----------|-------------|
| `fs.get` | Volume ID | Acquire filesystem volume |
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty path | `errors.INVALID` | unspecified |
| Path contains a null byte | `errors.INVALID` | no |
| Invalid mode | `errors.INVALID` | unspecified |
| `scanner()` called on a closed file | `errors.INVALID` | unspecified |
| Read, write, seek, stat, or sync called on a closed file | `errors.INTERNAL` | no |
| `close()` called on an already closed file | succeeds | not applicable |
| File-handle read reached EOF | `errors.NOT_FOUND` | unspecified |
| Path not found | `errors.NOT_FOUND` | preserved from the underlying error when available |
| Path already exists | `errors.ALREADY_EXISTS` | unspecified |
| Permission denied | `errors.PERMISSION_DENIED` | no |
| File scanner tokenization or read failed | `errors.INTERNAL` | preserved from the underlying error when available |
`unspecified` means `err:retryable()` returns `nil`; it is not equivalent to `false`.
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Cloud Storage"
## Cloud Storage
Access S3-compatible object storage. Upload, download, list, and manage objects, presign download, upload and multipart-part URLs, and read objects with random access.
For storage configuration, see [Cloud Storage](system/cloudstorage.md).
### Loading
```lua
local cloudstorage = require("cloudstorage")
```
### Acquiring Storage
Acquire a cloud storage resource by registry ID:
```lua
local storage, err = cloudstorage.get("app.infra:files")
if err then
return nil, err
end
local uploaded, upload_err = storage:upload_object("data/file.txt", "content")
storage:release()
if upload_err then return nil, upload_err end
return uploaded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Storage resource ID |
**Returns:** `Storage, error`
### Uploading Objects
Upload content from a string or file:
```lua
local json = require("json")
local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end
-- Upload string content
local body, encode_err = json.encode({
date = "2024-01-15",
total = 1234
})
if encode_err then
storage:release()
return nil, encode_err
end
local ok, err = storage:upload_object("reports/daily.json", body)
if err then
storage:release()
return nil, err
end
-- Upload from file
local fs = require("fs")
local vol, fs_err = fs.get("app:data")
if fs_err then
storage:release()
return nil, fs_err
end
local file, open_err = vol:open("/large-file.bin", "r")
if open_err then
storage:release()
return nil, open_err
end
local uploaded, file_upload_err = storage:upload_object("backups/large-file.bin", file)
local _, close_err = file:close()
storage:release()
if file_upload_err then
if close_err then report_cleanup_error(close_err) end
return nil, file_upload_err
end
if close_err then return nil, close_err end
return uploaded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key/path |
| `content` | string or Reader | Content as string or file reader |
| `options` | table | Optional metadata and conditional write options |
**Returns:** `boolean, error`
#### Upload Options
Attach metadata or guard the write with an options table:
```lua
local uploaded, err = storage:upload_object("reports/daily.json", body, {
content_type = "application/json",
cache_control = "max-age=3600",
metadata = { owner = "team-a", run_id = "1234" }, -- stored as x-amz-meta-*
only_if_absent = true -- fail if the key already exists
})
if err then return nil, err end
return uploaded
```
| Option | Type | Description |
|--------|------|-------------|
| `content_type` | string | MIME type |
| `cache_control` | string | Cache-Control header |
| `content_disposition` | string | Content-Disposition header |
| `content_encoding` | string | Content-Encoding header |
| `metadata` | table | User metadata (string keys/values), stored as `x-amz-meta-*` |
| `headers` | table | Additional request headers (string keys/values) |
| `if_match` | string | Write only if the current object ETag matches |
| `if_none_match` | string | Write only if no object matches the ETag (`"*"` means any) |
| `only_if_absent` | boolean | Write only if the key does not exist (alias for `if_none_match = "*"`) |
A conditional write that fails its precondition returns a `precondition_failed` error.
### Downloading Objects
Download an object to a file writer:
```lua
local fs = require("fs")
local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end
local vol, fs_err = fs.get("app:temp")
if fs_err then
storage:release()
return nil, fs_err
end
local file, open_err = vol:open("/downloaded.json", "w")
if open_err then
storage:release()
return nil, open_err
end
local ok, err = storage:download_object("reports/daily.json", file)
local _, close_err = file:close()
if err then
if close_err then report_cleanup_error(close_err) end
storage:release()
return nil, err
end
if close_err then
storage:release()
return nil, close_err
end
-- Download partial content (first 1KB)
local partial, partial_open_err = vol:open("/partial.bin", "w")
if partial_open_err then
storage:release()
return nil, partial_open_err
end
local partial_ok, partial_err = storage:download_object("backups/large-file.bin", partial, {
range = "bytes=0-1023"
})
local _, partial_close_err = partial:close()
storage:release()
if partial_err then
if partial_close_err then report_cleanup_error(partial_close_err) end
return nil, partial_err
end
if partial_close_err then return nil, partial_close_err end
return partial_ok
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key to download |
| `writer` | Writer | Destination file writer |
| `options.range` | string | Byte range (e.g., "bytes=0-1023") |
| `options.if_match` | string | Download only if the object ETag matches |
| `options.if_none_match` | string | Download only if the ETag does not match |
**Returns:** `boolean, error`
A failed precondition (`if_match`/`if_none_match`) returns a `precondition_failed` error.
### Listing Objects
List objects with optional prefix filtering:
```lua
local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end
local result, err = storage:list_objects({
prefix = "reports/2024/",
max_keys = 100
})
if err then
storage:release()
return nil, err
end
for _, obj in ipairs(result.objects) do
print(obj.key, obj.size, obj.etag)
end
-- Paginate through large results
local token = nil
repeat
local page, page_err = storage:list_objects({
prefix = "logs/",
max_keys = 1000,
continuation_token = token
})
if page_err then
storage:release()
return nil, page_err
end
for _, obj in ipairs(page.objects) do
process(obj)
end
token = page.next_continuation_token
if not page.is_truncated then break end
until false
storage:release()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `options.prefix` | string | Filter by key prefix |
| `options.max_keys` | integer | Maximum objects to return |
| `options.continuation_token` | string | Pagination token |
| `options.include_owner` | boolean | Include each object's `owner` (`id`, `display_name`) |
| `options.include_versions` | boolean | List object versions; each item includes `version_id` |
**Returns:** `table, error`
Result contains `objects`, `is_truncated`, `next_continuation_token`. Each object has `key`, `size`, `etag`, `storage_class`, and optional `last_modified`, `version_id`, and `owner`.
In list results content_type is always empty — S3 list operations do not return it. Use head_object to read an object's content type and metadata.
### Object Metadata
Fetch a single object's metadata without downloading its body:
```lua
local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end
local meta, err = storage:head_object("reports/daily.json")
if err then
storage:release()
return nil, err
end
print(meta.size, meta.etag, meta.content_type)
for k, v in pairs(meta.metadata) do
print("meta", k, v)
end
storage:release()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key |
**Returns:** `table, error`
Result fields:
| Field | Type | Description |
|-------|------|-------------|
| `size` | integer | Object size in bytes |
| `etag` | string | Entity tag |
| `content_type` | string | MIME type |
| `cache_control` | string | Cache-Control header |
| `content_disposition` | string | Content-Disposition header |
| `content_encoding` | string | Content-Encoding header |
| `storage_class` | string | Storage class |
| `version_id` | string | Version ID (present when versioning is enabled) |
| `last_modified` | integer | Last modified time (Unix seconds) |
| `metadata` | table | User metadata (`x-amz-meta-*`) |
| `headers` | table | Raw response headers (lowercased keys) |
A missing object returns a `not_found` error.
### Deleting Objects
Remove multiple objects:
```lua
local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end
local deleted, err = storage:delete_objects({
"temp/file1.txt",
"temp/file2.txt",
"temp/file3.txt"
})
storage:release()
if err then return nil, err end
return deleted
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `keys` | string[] | Array of object keys to delete |
**Returns:** `boolean, error`
Every key is attempted. Deleting a key that does not exist is not an error. When the provider reports per-key failures, the call returns a single error naming each failed key and its provider error code.
### Download URLs
Create a temporary URL that permits downloading an object without storage credentials. A client can use the URL until it expires.
```lua
local storage, err = cloudstorage.get("app.infra:files")
if err then
return nil, err
end
local url, err = storage:presigned_get_url("reports/quarterly.pdf", {
expiration = 3600
})
storage:release()
if err then
return nil, err
end
-- Return URL to client for direct download
return {download_url = url}
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key |
| `options.expiration` | integer | Seconds until URL expires (default: 3600) |
**Returns:** `string, error`
### Upload URLs
Create a temporary URL that permits uploading an object without storage credentials. A client can upload directly to storage until the URL expires.
```lua
local storage, err = cloudstorage.get("app.infra:files")
if err then
return nil, err
end
local url, err = storage:presigned_put_url("uploads/user-123/avatar.jpg", {
expiration = 600,
content_type = "image/jpeg",
content_length = 1024 * 1024
})
storage:release()
if err then
return nil, err
end
-- Return URL to client for direct upload
return {upload_url = url}
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key |
| `options.expiration` | integer | Seconds until URL expires (default: 3600) |
| `options.content_type` | string | Required content type for upload |
| `options.content_length` | integer | Expected upload size in bytes |
**Returns:** `string, error`
### Multipart Uploads
A single presigned PUT caps an object at 5 GiB. A presigned multipart upload splits a larger object into parts that a client uploads directly, then assembles them server-side. Multipart is a provider capability: S3 implements it, and providers without it return `errors.UNAVAILABLE`.
```lua
local storage = cloudstorage.get("app.infra:files")
local mp, err = storage:create_multipart_upload("backups/huge.zip", {
content_type = "application/zip",
metadata = { source = "uploader" },
})
if err then return nil, err end
local urls, err = storage:presigned_part_urls("backups/huge.zip", mp.upload_id, {
count = 3,
expiration = 900,
})
if err then
storage:abort_multipart_upload("backups/huge.zip", mp.upload_id)
return nil, err
end
-- The client PUTs each url and returns the ETag from the response headers.
local done, err = storage:complete_multipart_upload("backups/huge.zip", mp.upload_id, {
{ part_number = 1, etag = etag1 },
{ part_number = 2, etag = etag2 },
{ part_number = 3, etag = etag3 },
})
storage:release()
```
#### create_multipart_upload
Start a multipart upload for a key.
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key of the final object |
| `options` | table | `content_type`, `cache_control`, `content_disposition`, `content_encoding`, `metadata`, `headers` - same semantics as `upload_object` |
**Returns:** `table, error` - the table carries `upload_id`, which identifies the upload for every later part, complete and abort call.
Conditional writes (`if_match`, `if_none_match`, `only_if_absent`) are not part of the multipart protocol and are not accepted here.
#### presigned_part_urls
Generate presigned PUT URLs for parts of an in-progress upload. Each URL is uploaded to with a plain HTTP PUT; the uploader must keep the `ETag` response header of each part for `complete_multipart_upload`.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `key` | string | required | Object key |
| `upload_id` | string | required | From `create_multipart_upload` |
| `options.parts` | int[] | - | Explicit part numbers (1-10000, no duplicates) |
| `options.count` | int | - | Presign parts `1..count` |
| `options.headers` | table | - | Headers required on each part request; they are signed and must also be sent by the uploader |
| `options.expiration` | int | 3600 | Seconds until the URLs expire |
Exactly one of `parts` or `count` is required, and a single call presigns at most 1000 URLs - presign in pages for very large objects.
**Returns:** `table, error` - an array of `{ part_number, url }`.
Every part except the last must be at least 5 MiB; the provider enforces this at completion time.
#### complete_multipart_upload
Assemble the final object from its uploaded parts. Parts may be reported in any order and are sorted by part number before completion.
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key |
| `upload_id` | string | From `create_multipart_upload` |
| `parts` | table | Array of `{ part_number = int, etag = string }` |
**Returns:** `table, error` - `etag`, plus `version_id` and `location` when the provider reports them. An unknown upload ID returns `errors.NOT_FOUND`.
#### abort_multipart_upload
Discard an in-progress upload and free its stored parts.
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Object key |
| `upload_id` | string | From `create_multipart_upload` |
**Returns:** `boolean, error`
An upload that is never completed keeps its parts stored, and billed, until it is aborted. Abort on every failure path, and configure a bucket lifecycle rule as a backstop - see [Cloud Storage](system/cloudstorage.md#multipart-uploads).
### Ranged Readers
`open_reader` opens random access over an object using ranged GETs - no local staging and no full download. Its main consumer is [`archive.open`](lua/data/archive.md), which reads multi-GB archives straight out of object storage with bounded memory.
```lua
local archive = require("archive")
local storage = cloudstorage.get("app.infra:files")
local reader, err = storage:open_reader("uploads/huge.zip", {
block_size = 8 * 1024 * 1024,
cache_blocks = 4,
})
if err then return nil, err end
local r = assert(archive.open(reader))
for e in r:entries() do
print(e.name, e.size)
end
r:close()
reader:close()
storage:release()
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `key` | string | required | Object key |
| `options.block_size` | int | 8388608 | Ranged-GET unit in bytes (64 KiB to 128 MiB) |
| `options.cache_blocks` | int | 4 | Resident LRU blocks (1 to 64) |
`block_size * cache_blocks` may not exceed 256 MiB. A missing object returns `errors.NOT_FOUND`.
**Returns:** `Reader, error`
The object's ETag is pinned when the reader opens and sent as `If-Match` on every ranged read, so an object overwritten mid-read fails the read with the provider's precondition error instead of serving a mix of two object generations; `archive` surfaces it as `errors.INTERNAL`. A provider that cannot supply an ETag returns `errors.UNAVAILABLE`; the reader never serves an unpinned object.
Cache-miss reads perform blocking network IO in the calling task and serialize concurrent readers, so sequential per-entry access - the archive pattern - is the intended shape.
#### Reader Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `size()` | `integer` | Object size in bytes, from the open-time stat |
| `key()` | `string` | Object key the reader reads from |
| `close()` | `boolean, error` | Release the block cache; idempotent |
The reader is closed automatically at task scope if it is not closed explicitly.
### Storage Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `upload_object(key, content, opts?)` | `boolean, error` | Upload string or file content |
| `download_object(key, writer, opts?)` | `boolean, error` | Download to file writer |
| `head_object(key)` | `table, error` | Fetch object metadata |
| `list_objects(opts?)` | `table, error` | List objects with prefix filter |
| `delete_objects(keys)` | `boolean, error` | Delete multiple objects |
| `presigned_get_url(key, opts?)` | `string, error` | Generate temporary download URL |
| `presigned_put_url(key, opts?)` | `string, error` | Generate temporary upload URL |
| `create_multipart_upload(key, opts?)` | `table, error` | Start a presigned multipart upload |
| `presigned_part_urls(key, upload_id, opts)` | `table, error` | Presign PUT URLs for upload parts |
| `complete_multipart_upload(key, upload_id, parts)` | `table, error` | Assemble the object from uploaded parts |
| `abort_multipart_upload(key, upload_id)` | `boolean, error` | Discard an in-progress multipart upload |
| `open_reader(key, opts?)` | `Reader, error` | Open a ranged random-access reader |
| `release()` | `boolean` | Release storage resource |
### Permissions
Security policy evaluation applies to cloud storage operations.
| Action | Resource | Description |
|--------|----------|-------------|
| `cloudstorage.get` | Storage ID | Acquire a storage resource |
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty resource ID | `errors.INVALID` | no |
| Resource not found | `errors.NOT_FOUND` | no |
| Not a cloud storage resource | `errors.INVALID` | no |
| Storage released | `errors.INVALID` | no |
| Empty key | `errors.INVALID` | no |
| Content nil | `errors.INVALID` | no |
| Writer not valid | `errors.INVALID` | no |
| Object not found | `errors.NOT_FOUND` | no |
| Unknown upload ID | `errors.NOT_FOUND` | no |
| Conditional precondition failed | `errors.CONFLICT` | no |
| Object overwritten during a ranged read (surfaced by `archive`) | `errors.INTERNAL` | no |
| Provider does not support multipart uploads | `errors.UNAVAILABLE` | no |
| Provider supplies no ETag for `open_reader` | `errors.UNAVAILABLE` | no |
| Permission denied | raised as a Lua error, not returned | - |
| Provider operation failed | `errors.UNKNOWN` | unset |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Message Queue"
## Message Queue
The `queue` module publishes messages and processes deliveries from configured distributed queues, including RabbitMQ and other AMQP-compatible brokers.
This page is an API reference. Publishing snippets assume the queue entries and permissions already exist. The consumer section is a partial recipe for a handler invoked by `queue.consumer`; it is not a standalone queue deployment.
For queue configuration, see [Queue](system/queue.md).
### Loading
```lua
local queue = require("queue")
```
### Publishing Messages
Publish a message to a queue by ID:
```lua
local ok, err = queue.publish("app:tasks", {
action = "send_email",
user_id = 456,
template = "welcome"
})
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `queue_id` | string | Queue identifier (format: "namespace:name") |
| `data` | any | Message data (tables, strings, numbers, booleans) |
| `headers` | table | Optional message headers |
**Returns:** `boolean, error`
#### Message Headers
Headers carry routing, priority, and tracing metadata. Keys must be strings, and publisher values may be strings, integers, numbers, or booleans:
```lua
local ok, err = queue.publish("app:notifications", {
type = "order_shipped",
order_id = order.id
}, {
priority = 5,
correlation_id = request_id
})
if err then return nil, err end
```
Consumers receive every header value as a string. The `x_original_queue`, `x_dead_letter_reason`, `x_dead_letter_time`, and `attempts` keys are reserved for delivery and dead-letter bookkeeping and must not be set by publishers.
### Accessing Delivery Context
Access the current delivery from within a queue consumer:
```lua
local msg, err = queue.message()
if err then
return nil, err
end
local msg_id, id_err = msg:id()
if id_err then return nil, id_err end
local priority, header_err = msg:header("priority")
if header_err then return nil, header_err end
local all_headers, headers_err = msg:headers()
if headers_err then return nil, headers_err end
```
**Returns:** `Message, error`
This function is available only while a queue consumer is processing a message.
### Message Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `id()` | `string, error` | Unique message identifier |
| `header(key)` | `string, error` | Single header value as a string (nil if missing) |
| `headers()` | `table, error` | All message headers |
| `ack()` | `boolean, error` | Acknowledge processing (single-shot) |
| `nack()` | `boolean, error` | Signal failure for redelivery or dead-letter (single-shot) |
The runtime auto-acks on handler success and auto-nacks on handler error. Call `ack`/`nack` only to settle early. Settlement is single-shot, and a `Message` is invalid after its consumer handler returns.
### Queue Info
```lua
local stats, err = queue.info("app:tasks")
if err then return nil, err end
-- stats may contain: message_count, consumer_count, ready (driver-dependent)
```
**Returns:** `table, error`
### Consumer Pattern
A `queue.consumer` entry binds a queue to the handler referenced by `func`. The handler receives the message payload directly:
```yaml
entries:
- kind: queue.consumer
name: email_worker
queue: app:emails
func: app:email_handler
```
This fragment assumes `app:emails` and the `app:email_handler` function entry already exist. The function source below assumes the application supplies `deliver_email(payload)` and grants any permissions it needs.
```lua
local queue = require("queue")
local logger = require("logger")
local function main(payload)
local msg, msg_err = queue.message()
if msg_err then return nil, msg_err end
local message_id, id_err = msg:id()
if id_err then return nil, id_err end
logger:info("Processing", {
message_id = message_id,
to = payload.to
})
local ok, send_err = deliver_email(payload)
if send_err then return nil, send_err end
return ok
end
return {main = main}
```
Returning an invocation error causes the consumer to nack the unsettled delivery. Redelivery then follows the selected driver's behavior; the built-in dead-letter configuration is not enforced in this release.
### Permissions
Security policy evaluation applies to queue operations.
| Action | Resource | Description |
|--------|----------|-------------|
| `queue.publish` | - | General permission to publish messages |
| `queue.publish.queue` | Queue ID | Publish to specific queue |
The runtime checks the general permission first and the queue-specific permission second.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Queue ID empty | `errors.INVALID` | no |
| Message argument missing or an empty table | `errors.INVALID` | no |
| No delivery context | `errors.INVALID` | no |
| Message released or already settled | `errors.INVALID` | no |
| Publish not allowed | `errors.INVALID` | no |
| Publish failed | `errors.INTERNAL` | no |
| Queue or driver not found for `info` | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Queue Configuration](system/queue.md) - Queue drivers and entry definitions
- [Queue Consumers Guide](guides/queue-consumers.md) - Consumer patterns and worker pools
- [Process Management](lua/core/process.md) - Process spawning and communication
- [Channels](lua/core/channel.md) - Inter-process communication patterns
- [Functions](lua/core/funcs.md) - Async function invocation
---
# "CDC"
## CDC
Subscribe to Change Data Capture streams from [`db.cdc.postgres`](system/cdc.md) and [`db.cdc.sqlite`](system/cdc.md) sources. List configured sources, open a stream, and receive row-level change events over a channel. The API is driver-neutral: both kinds return the same source info and the same change events, and differ only in the [capabilities](system/cdc.md#capabilities) they publish.
### Loading
```lua
local cdc = require("cdc")
```
### `list_sources`
List the configured CDC sources the caller is allowed to see:
```lua
local sources, err = cdc.list_sources()
if err then return nil, err end
for _, s in ipairs(sources) do
print(s.id, s.kind, s.state, s.capabilities.before_images)
end
```
Sources the caller lacks `cdc.source` on are omitted rather than reported as an error.
**Returns:** `table, error`
### `source`
Retrieve one source by its registry entry ID or replication slot name:
```lua
local info, err = cdc.source("app:pg_cdc")
if err then return nil, err end
if info == nil then
-- no such source
end
```
**Returns:** `table, error` (source info, or `nil` if not found)
### `stream`
Open a change stream on a source. The returned `cdc.Stream` exposes a channel that delivers change events:
```lua
local stream, err = cdc.stream("app:pg_cdc", {
tables = { "public.users", "public.orders" },
ops = { "insert", "update" },
buffer = 128,
})
if err then return nil, err end
-- The caller owns stream until close(), release(), or task cleanup.
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | string | required | Source name (entry ID) |
| `opts.tables` | []string | - | Filter to these tables (omit for all captured tables) |
| `opts.ops` | []string | - | Filter to these operations: `insert`, `update`, `delete`, `truncate` |
| `opts.buffer` | int | 64 | Backlog item capacity (1-65536) |
| `opts.max_bytes` | int | 1048576 | Backlog byte budget for this subscriber (1 MiB) |
| `opts.snapshot` | bool | entry default | Request the snapshot/live handoff for this stream |
| `opts.after` | string | - | Opaque resume cursor from a previous event's `cursor` |
Unknown option keys are rejected with `errors.INVALID`. Table names are matched case-insensitively against both the qualified relation and the bare table name. Snapshot rows are filtered by `tables` only; `ops` applies to live changes.
A stream receives a snapshot when either `opts.snapshot` is true or the source entry's `snapshot` field is set; snapshot rows arrive first with `op = "snapshot"`, then the stream continues into live changes with no gap. `opts.after` is reserved for drivers that resume from a cursor — every driver shipped today returns `errors.INVALID` ("cdc operation is not supported by this source") for it, including `db.cdc.postgres` when it reports `capture_resume`.
Filters narrow delivery only. Access to a source is granted by the `cdc.subscribe` permission, never by a filter.
**Returns:** `Stream, error`
The Lua delivery channel has a separate fixed capacity of 64. The `buffer` option controls the PostgreSQL source subscription, not that channel.
#### `channel`
Return the channel that receives change events. The first call subscribes to the source and yields; subsequent calls return the same channel. The first call can return a subscription error. Channel `:receive()` returns `value, true` for a change or `nil, false` when the stream ends:
```lua
local stream, stream_err = cdc.stream("app:pg_cdc")
if stream_err then return nil, stream_err end
local ch, subscribe_err = stream:channel()
if subscribe_err then
stream:close()
return nil, subscribe_err
end
while true do
local change, ok = ch:receive()
if not ok then break end
if change.op == "snapshot" then
seed_row(change.table, change.after)
elseif change.op == "insert" then
handle_new_user(change.table, change.after)
elseif change.op == "update" then
handle_update(change.table, change.before, change.after)
elseif change.op == "delete" then
handle_delete(change.table, change.before)
end
end
local _, close_err = stream:close()
if close_err then return nil, close_err end
```
The stream is lazy: construct it, then call `channel()` before generating the writes it should observe. This is live observation, not replay of changes made before the subscription.
When a source terminates a stream with a failure, the channel delivers an error value before it closes. `receive` is an alias for `channel`.
#### `close`
Stop the subscription and release the stream. The method is idempotent, and the runtime also closes the stream at the end of the task scope. `release` is an alias for `close`.
```lua
local _, err = stream:close()
if err then return nil, err end
```
### Change Event
Each message received on the channel is a change table:
| Field | Description |
|-------|-------------|
| `op` | Operation: `insert`, `update`, `delete`, `snapshot` or `truncate` |
| `schema` | Table schema |
| `table` | Table name |
| `relation` | Qualified relation name |
| `before` | Row state before the change (`update`, `delete`). A full row image is guaranteed only when the source has the `before_images` capability; `db.cdc.postgres` fills it from whatever old tuple the WAL carries, which the table's `REPLICA IDENTITY` controls |
| `after` | Row state after the change (`insert`, `update`, `snapshot`; absent for `delete`) |
| `source` | Source entry ID |
| `source_id` | Source entry ID, as a registry ID |
| `generation` | Source generation that produced the event |
| `cursor` | Opaque per-event position within the source |
| `transaction` | Transaction identifier, when the driver reports one |
| `lsn` | Log sequence number of the change (`db.cdc.postgres`) |
| `commit_lsn` | LSN of the committing transaction (when applicable) |
| `xid` | Transaction ID (when applicable) |
| `unchanged` | Columns whose value was not transmitted (unchanged TOAST values) |
| `error` | Driver-reported error description carried on the event |
`before` and `after` are row maps keyed by column name.
### Source Info
`cdc.source` and each entry of `cdc.list_sources` return the same record:
| Field | Description |
|-------|-------------|
| `id` | Entry ID |
| `kind` | `db.cdc.postgres` or `db.cdc.sqlite` |
| `name` | Source name (the entry ID) |
| `state` | `unknown`, `starting`, `running`, `faulted` or `stopped` |
| `generation` | Current source generation |
| `epoch` | Same value as `generation` |
| `engine` | Engine name, when the driver reports one |
| `db_resource` | Observed SQL resource entry ID (`db.cdc.sqlite`) |
| `slot` | Replication slot name (`db.cdc.postgres`) |
| `publication` | Postgres publication, when configured |
| `tables` | Captured tables, when configured |
| `streaming` | `db.cdc.sqlite`: whether the source is running; `db.cdc.postgres`: the entry's `streaming` protocol setting |
| `failover` | Failover slot mode (`db.cdc.postgres`) |
| `temporary` | Temporary slot (`db.cdc.postgres`) |
| `snapshot` | Entry-level snapshot default |
| `faulted` | Whether the source is in the `faulted` state |
| `error` | Last source error, when one is recorded |
| `admission` | `active`, `snapshots`, `reserved_bytes`, `rejected` |
| `capabilities` | `snapshot`, `capture_resume`, `replayable`, `captures_external_writes`, `before_images`, `coalesced` |
Branch on `capabilities` rather than on `kind`:
```lua
local info = cdc.source("app:changes")
if not info.capabilities.before_images then
-- before is not a guaranteed full row image; keep your own last-known state
end
```
See [CDC sources](system/cdc.md#source-info) for field semantics.
### Permissions
| Action | Resource | Description |
|--------|----------|-------------|
| `cdc.source` | Source entry ID | `cdc.source`; also filters `cdc.list_sources` |
| `cdc.subscribe` | Source entry ID | `cdc.stream`, checked again when the subscription is established |
A denied action returns `errors.PERMISSION_DENIED`.
### Errors
| Condition | Kind |
|-----------|------|
| No context | `errors.INTERNAL` |
| Source name required | `errors.INVALID` |
| Invalid or unknown stream option | `errors.INVALID` |
| `after` on a source without `capture_resume` | `errors.INVALID` |
| Source not registered | `errors.NOT_FOUND` |
| Source not started or replacing | `errors.UNAVAILABLE` |
| Subscription capacity exhausted | `errors.UNAVAILABLE` |
| Permission denied | `errors.PERMISSION_DENIED` |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Change Data Capture](system/cdc.md) - Source configuration and capabilities
- [Channel](lua/core/channel.md) - Channel semantics
- [Database](system/database.md) - SQL database services
---
# "System"
## System
The `system` module reports runtime, memory, process, host, supervisor, and cluster state. It also exposes selected runtime controls.
This is an API reference. Most snippets show one isolated operation; controls such as shutdown, runtime tuning, and distributed locks require explicit policy authorization and application-specific failure handling.
### Loading
```lua
local system = require("system")
```
### Shutdown
Request system shutdown with an exit code. Calling this function from any process or actor terminates the entire system:
```lua
local ok, err = system.exit(0)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `code` | integer | Exit code (0 = success), defaults to 0 |
**Returns:** `boolean, error`
### Listing Modules
List the loaded Lua modules and their metadata:
```lua
local mods, err = system.modules()
```
**Returns:** `table[], error`
Each module table contains:
| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Module name |
| `description` | string | Module description |
| `class` | string[] | Module classification tags |
### Deployment Sources
The `system.source` sub-table reads the normalized deployment baseline: the entry set produced by the sources the application was assembled from, before any registry history is applied.
```lua
local loaded, err = system.source.load()
```
**Returns:** `table, error`
| Field | Type | Description |
|-------|------|-------------|
| `owners` | string[] | Source owners authoritative over the baseline entries |
| `entries` | table[] | Baseline entries with `id`, `kind`, `meta`, `data` |
`owners` is sorted with the application owner first, then the remaining owners alphabetically. The application owner is the string `"application"`.
```lua
local loaded, err = system.source.load()
if err then return nil, err end
for _, owner in ipairs(loaded.owners) do
print(owner)
end
for _, entry in ipairs(loaded.entries) do
print(entry.id, entry.kind)
end
```
The load is taken from one stable source generation, so entries and owners always describe the same baseline. Filesystem paths behind each source are runtime-private and are not exposed; a failed load reports a generic internal error rather than leaking the backing path.
**Permission:** `system.read` on `sources`
### Memory Statistics
Read detailed memory statistics:
```lua
local stats, err = system.memory.stats()
```
**Returns:** `table, error`
The statistics table contains:
| Field | Type | Description |
|-------|------|-------------|
| `alloc` | number | Bytes allocated and in use |
| `total_alloc` | number | Cumulative bytes allocated |
| `sys` | number | Bytes obtained from system |
| `heap_alloc` | number | Bytes allocated on heap |
| `heap_sys` | number | Bytes obtained for heap from system |
| `heap_idle` | number | Bytes in idle spans |
| `heap_in_use` | number | Bytes in non-idle spans |
| `heap_released` | number | Bytes released to OS |
| `heap_objects` | number | Number of allocated heap objects |
| `stack_in_use` | number | Bytes used by stack allocator |
| `stack_sys` | number | Bytes obtained for stack from system |
| `mspan_in_use` | number | Bytes of mspan structures in use |
| `mspan_sys` | number | Bytes obtained for mspan from system |
| `num_gc` | number | Number of completed GC cycles |
| `next_gc` | number | Target heap size for next GC |
### Current Allocation
Read the number of bytes currently allocated:
```lua
local bytes, err = system.memory.allocated()
```
**Returns:** `number, error`
### Heap Objects
Read the number of allocated heap objects:
```lua
local count, err = system.memory.heap_objects()
```
**Returns:** `number, error`
### Memory Limit
Set the memory limit and return its previous value:
```lua
local prev, err = system.memory.set_limit(1024 * 1024 * 100)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `limit` | integer | Memory limit in bytes, -1 for unlimited |
**Returns:** `number, error`
Read the current memory limit:
```lua
local limit, err = system.memory.get_limit()
```
**Returns:** `number, error`
### Force GC
Run garbage collection immediately:
```lua
local ok, err = system.gc.collect()
```
**Returns:** `boolean, error`
### GC Target Percentage
Set the garbage-collection target percentage and return its previous value. A value of 100 triggers collection when the heap doubles:
```lua
local prev, err = system.gc.set_percent(200)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `percent` | integer | GC target percentage |
**Returns:** `number, error`
Read the current garbage-collection target percentage:
```lua
local percent, err = system.gc.get_percent()
```
**Returns:** `number, error`
### Goroutine Count
Read the number of active goroutines:
```lua
local count, err = system.runtime.goroutines()
```
**Returns:** `number, error`
### GOMAXPROCS
Read or set the `GOMAXPROCS` value:
```lua
-- Get current value
local current, err = system.runtime.max_procs()
-- Set new value
local prev, err = system.runtime.max_procs(4)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | If provided, sets GOMAXPROCS (must be > 0) |
**Returns:** `number, error`
### CPU Count
Read the number of logical CPUs:
```lua
local cpus, err = system.runtime.cpu_count()
```
**Returns:** `number, error`
### Process ID
Read the current operating-system process ID:
```lua
local pid, err = system.process.pid()
```
**Returns:** `number, error`
### Hostname
Read the system hostname:
```lua
local hostname, err = system.process.hostname()
```
**Returns:** `string, error`
### Working Directory
Read the runtime's current working directory:
```lua
local dir, err = system.process.cwd()
```
**Returns:** `string, error`
### Process Hosts
List process hosts with worker and queue statistics:
```lua
local hosts, err = system.hosts.list()
```
**Returns:** `table[], error`
Each host table contains:
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Host registry ID |
| `workers` | number | Worker pool size |
| `processes` | number | Active processes on this host |
| `executed` | number | Total steps executed |
| `stolen` | number | Steps stolen from other hosts |
| `queue_depth` | number | Pending items in the host queue |
List processes running on a specific host:
```lua
local procs, err = system.hosts.processes("app:host")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `host_id` | string | Host registry ID |
**Returns:** `table[], error`
Each process table contains:
| Field | Type | Description |
|-------|------|-------------|
| `pid` | string | Process ID |
| `host` | string | Host ID |
| `source` | string | Source entry ID |
| `state` | string | Process state |
| `steps` | number | Steps executed |
| `started_at` | number | Start timestamp (nanoseconds) |
| `parent` | string | Parent PID (omitted if none) |
| `actor_id` | string | Actor ID (omitted if none) |
| `stats` | table | Process-specific stats (optional) |
### Service State
Read the state of a supervised service:
```lua
local state, err = system.supervisor.state("namespace:service")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `service_id` | string | Service ID (e.g., "namespace:service") |
**Returns:** `table, error`
The state table contains:
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Service ID |
| `status` | string | Current status |
| `desired` | string | Desired status |
| `retry_count` | number | Number of retries |
| `last_update` | number | Last update timestamp (nanoseconds) |
| `started_at` | number | Start timestamp (nanoseconds) |
| `details` | string | Optional details (formatted) |
### All Service States
List the states of all supervised services:
```lua
local states, err = system.supervisor.states()
```
**Returns:** `table[], error`
Each state table has the same format as `system.supervisor.state()`.
### Cluster Primitives
The `system.node`, `system.cluster`, `system.raft`, and `system.lock` sub-tables expose the clustering layer. They are most useful when [clustering is enabled](guides/cluster.md); on a standalone node they degrade predictably — `system.raft.*` reports "raft not available", `system.cluster` reports just the local node, and `system.lock` requires the Raft-backed KV store that clustering provides.
Read calls report this node's local view of committed state and do not block on the network.
#### Node Identity
`system.node` reports the current node's identity in the cluster.
```lua
local id, err = system.node.id() -- this node's ID
local addr, err = system.node.addr() -- advertised network address
local role, err = system.node.role() -- "leader" | "voter" | "standby" | "non-member"
```
| Function | Returns | Notes |
|----------|---------|-------|
| `system.node.id()` | `string, error` | Node ID from the relay context |
| `system.node.addr()` | `string, error` | Advertised address (e.g. `10.0.0.1:7946`); errors if membership is unavailable |
| `system.node.role()` | `string, error` | Raft role of this node; returns `"non-member"` (no error) when Raft is not running |
**Permission:** `system.read` on `node`.
#### Cluster Membership
`system.cluster` reports cluster membership and the current leader.
```lua
local members, err = system.cluster.members() -- array of node tables
local leader, err = system.cluster.leader() -- leader node ID, or "" if unknown
local n, err = system.cluster.size() -- count of visible members
```
`system.cluster.members()` returns an array of node tables. The local node is included once and sorts first.
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Node ID |
| `is_local` | boolean | True for the calling node |
| `addr` | string | Advertised address (omitted when unknown) |
| `meta` | table | String-to-string gossip metadata (omitted when none) |
| Function | Returns | Notes |
|----------|---------|-------|
| `system.cluster.members()` | `table[], error` | Errors if no membership information is reachable |
| `system.cluster.leader()` | `string, error` | Current Raft leader's ID; `""` (no error) when the leader is unknown or Raft is absent |
| `system.cluster.size()` | `number, error` | Count of visible members; `0` when no membership info is available |
**Permission:** `system.read` on `cluster`.
#### Raft State
`system.raft` reads this node's local view of the Raft consensus core. Every function returns `nil, error` ("raft not available") when Raft is not running on this node.
```lua
local leader, err = system.raft.is_leader() -- boolean
local member, err = system.raft.is_member() -- boolean: voter or standby
local role, err = system.raft.role() -- same values as system.node.role()
local term, err = system.raft.term() -- current Raft term
local idx, err = system.raft.commit_index() -- highest committed log index
local stats, err = system.raft.stats() -- raw stats map (string -> string)
```
| Function | Returns | Notes |
|----------|---------|-------|
| `system.raft.is_leader()` | `boolean, error` | True iff this node is the current leader |
| `system.raft.is_member()` | `boolean, error` | True iff this node is a voter or standby in the committed configuration |
| `system.raft.role()` | `string, error` | `"leader"` / `"voter"` / `"standby"` / `"non-member"` |
| `system.raft.term()` | `number, error` | Current term; `0` if unavailable from stats |
| `system.raft.commit_index()` | `number, error` | Highest committed log index on this node |
| `system.raft.stats()` | `table, error` | Full raw stats map; keys and values are strings |
**Permission:** `system.read` on `raft`, except `system.raft.stats()` which requires `system.read` on `raft_stats`.
#### Distributed Locks
`system.lock` provides cluster-wide mutual exclusion. A lock is a globally unique name owned by the calling process. It is built on the Raft-replicated system KV store, so at most one holder can exist across the cluster, and the lock auto-releases when the holder process exits or its node leaves — there is no stuck lock to clean up.
```lua
local ok, err = system.lock.acquire("orders.migration")
if not ok then
-- err has kind errors.ALREADY_EXISTS when another process holds the lock.
-- Apply the caller's retry and backoff policy for that case if needed.
return nil, err
end
-- critical section: only one holder cluster-wide
local released, release_err = system.lock.release("orders.migration")
if release_err then
return nil, release_err
end
return released
```
Acquisition is fail-fast: when a lock is already held, the call returns `false` immediately instead of blocking. Callers provide any required retry and backoff. Only the current holder can release a lock; a release attempt by another process is a no-op.
| Function | Returns | Outcomes |
|----------|---------|----------|
| `system.lock.acquire(name)` | `boolean, error` | `true, nil` acquired; `false, error` already held (kind `errors.ALREADY_EXISTS`); `nil, error` on failure |
| `system.lock.release(name)` | `boolean, error` | `true, nil` released; `false, nil` not held or held by another process; `nil, error` on failure |
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Cluster-wide lock name |
**Permission:** `system.lock` on the lock `name` (so policy can restrict which names a caller may lock).
### Permissions
Security policy evaluation applies to system operations.
| Action | Resource | Description |
|--------|----------|-------------|
| `system.read` | `memory` | Read memory statistics |
| `system.read` | `memory_limit` | Read memory limit |
| `system.control` | `memory_limit` | Set memory limit |
| `system.read` | `gc_percent` | Read GC percentage |
| `system.gc` | `gc` | Force garbage collection |
| `system.gc` | `gc_percent` | Set GC percentage |
| `system.read` | `goroutines` | Read goroutine count |
| `system.read` | `gomaxprocs` | Read GOMAXPROCS |
| `system.control` | `gomaxprocs` | Set GOMAXPROCS |
| `system.read` | `cpu` | Read CPU count |
| `system.read` | `pid` | Read process ID |
| `system.read` | `hostname` | Read hostname |
| `system.read` | `cwd` | Read working directory |
| `system.read` | `hosts` | List hosts / host processes |
| `system.read` | `modules` | List loaded modules |
| `system.read` | `sources` | Load the deployment source baseline |
| `system.read` | `supervisor` | Read supervisor state |
| `system.read` | `node` | Read this node's identity |
| `system.read` | `cluster` | Read cluster membership and leader |
| `system.read` | `raft` | Read Raft state |
| `system.read` | `raft_stats` | Read the raw Raft stats map |
| `system.lock` | `` | Acquire or release a distributed lock |
| `system.exit` | - | Trigger system shutdown |
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Permission denied (`system.source.load`, `system.lock.*`) | `errors.PERMISSION_DENIED` | no |
| Permission denied (all other calls) | `errors.INVALID` | no |
| Invalid argument | `errors.INVALID` | no |
| Missing required argument | `errors.INVALID` | no |
| Code manager unavailable | `errors.INTERNAL` | no |
| Service info unavailable | `errors.INTERNAL` | no |
| OS error (hostname, cwd) | `errors.INTERNAL` | no |
| Raft not running on this node | `errors.INTERNAL` | no |
| Membership unavailable | `errors.INTERNAL` | no |
| Lock already held | `errors.ALREADY_EXISTS` | no |
| Lock service unavailable (no Raft on this node) | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Environment Variables"
## Environment Variables
The `env` module reads and updates environment variables exposed by the runtime.
This is an API reference. Its snippets are isolated operations and assume the named variables and security policies already exist.
Variables must be defined in the [Environment System](system/env.md) before they can be accessed. The system controls which storage backends (OS, file, memory) provide values and whether variables are read-only.
### Loading
```lua
local env = require("env")
```
### `get`
Retrieve an environment variable.
```lua
-- Get database connection string
local db_url = env.get("DATABASE_URL")
if not db_url then
return nil, errors.new({ kind = errors.INVALID, message = "DATABASE_URL not configured" })
end
local port, port_err = get_or("PORT", "8080")
if port_err then return nil, port_err end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Variable name |
**Returns:** `string, error`
The function returns `nil, error` when the variable does not exist.
### `set`
Set an environment variable.
```lua
-- Set runtime configuration
local updated, set_err = env.set("APP_MODE", "production")
if set_err then return nil, set_err end
return updated
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | Variable name |
| `value` | string | Value to set |
**Returns:** `boolean, error`
### `get_all`
Retrieve all environment variables accessible to the caller.
```lua
local logger = require("logger")
local vars, vars_err = env.get_all()
if vars_err then return nil, vars_err end
-- Log names only. Values such as connection URLs may contain credentials even
-- when their keys do not include words like SECRET or KEY.
local accessible_keys = {}
for key in pairs(vars) do table.insert(accessible_keys, key) end
logger:debug("accessible environment variables", {keys = accessible_keys})
-- Check required variables
local required = {"DATABASE_URL", "REDIS_URL", "API_KEY"}
for _, key in ipairs(required) do
if not vars[key] then
return nil, errors.new({ kind = errors.INVALID, message = "Missing required env var: " .. key })
end
end
```
**Returns:** `table, error`
### Permissions
Security policy evaluation applies to environment access.
#### Security Actions
| Action | Resource | Description |
|--------|----------|-------------|
| `env.get` | Variable name | Read environment variable |
| `env.set` | Variable name | Write environment variable |
`get_all` has no dedicated security action: it returns only the variables for which the `env.get` action is permitted, filtering each variable name through `env.get`.
#### Checking Access
```lua
local security = require("security")
if security.can("env.get", "DATABASE_URL") then
local url = env.get("DATABASE_URL")
end
```
See [Security Model](system/security.md) for policy configuration.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty key | `errors.INVALID` | no |
| Variable not found | `errors.NOT_FOUND` | no |
| Permission denied | `errors.PERMISSION_DENIED` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Environment System](system/env.md) - Configure storage backends and variable definitions
---
# "Logging"
## Logging
The `logger` module writes structured messages at debug, info, warn, and error levels.
This is an API reference. Each snippet is an isolated logging operation and assumes an execution context with the desired logger configuration.
Log calls return no values. When the execution context provides them, each call also adds the process `pid` and the source `location` derived from the current frame.
### Loading
```lua
local logger = require("logger")
```
#### `logger:debug`
Write a debug-level log message.
```lua
logger:debug("message", {key = "value"})
```
#### `logger:info`
Write an info-level log message.
```lua
logger:info("message", {key = "value"})
```
#### `logger:warn`
Write a warning-level log message.
```lua
logger:warn("message", {key = "value"})
```
#### `logger:error`
Write an error-level log message.
```lua
logger:error("message", {key = "value"})
```
All four log-level methods accept the same parameters:
| Parameter | Type | Description |
|-----------|------|-------------|
| `message` | string | Log message |
| `fields` | table? | Contextual key-value pairs |
Only string keys become field names. Strings, numbers, integers, booleans, errors, and structured Lua values are converted to log fields; non-string keys are ignored.
For `logger:error`, a field named `error` is emitted as an error field and removed from the supplied table before the remaining fields are processed. Do not reuse that table if the `error` entry must remain intact.
#### `logger:with`
Create a child logger that adds the same fields to every message.
```lua
local function request_logger(request_id)
return logger:with({request_id = request_id})
end
request_logger("req-123"):info("message")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `fields` | table | Fields to attach to all logs |
**Returns:** `Logger`
The original logger is unchanged. Child loggers can be chained with additional `with` and `named` calls.
#### `logger:named`
Create a child logger with a name.
```lua
local named = logger:named("auth")
named:info("message")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Logger name |
**Returns:** `Logger`
An empty name raises a Lua argument error. It is not returned as a structured `errors.INVALID` value.
`logger:named("")` raises a Lua argument error (`name cannot be empty`) instead of returning an error value. Logging methods return nothing.
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Terminal I/O"
## Terminal I/O
The `io` module reads from standard input and writes to standard output and standard error in terminal applications.
This is an API reference. Its snippets are isolated calls; a terminal process should propagate returned structured Lua errors when the result affects control flow.
This module is available only to processes running on a Terminal Host, not to regular functions.
### Loading
```lua
local io = require("io")
```
### Writing to Stdout
Write values to standard output without a trailing newline:
```lua
local ok, err = io.write("text", "more")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `...` | any | Variable number of values to write (coerced to string) |
**Returns:** `boolean, error`
### Print with Newline
Write values to standard output, separated by tabs and followed by a newline:
```lua
io.print("value1", "value2", 123)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `...` | any | Variable number of values to print |
**Returns:** `boolean, error`
After terminal context lookup succeeds, output write errors are ignored and the function returns `true`. A missing terminal context returns `nil, "no terminal context"`.
### Writing to Stderr
Write values to standard error, separated by tabs and followed by a newline:
```lua
io.eprint("Error:", message)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `...` | any | Variable number of values to print |
**Returns:** `boolean, error`
After terminal context lookup succeeds, output write errors are ignored and the function returns `true`. A missing terminal context returns `nil, "no terminal context"`.
### Reading Bytes
Read up to `n` bytes from standard input:
```lua
local data, err = io.read(1024)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `n` | integer | Number of bytes to read (default: 1024, values <= 0 become 1024) |
**Returns:** `string, error`. A successful read may return fewer than `n` bytes or an empty string.
### Reading a Line
Read one line from standard input:
```lua
local line, err = io.readline()
```
**Returns:** `string, error`. The trailing `\n` and `\r` are removed. EOF after partial input returns that partial line; EOF without input returns `nil` and a structured error.
### Raw Mode
Enable or disable raw terminal mode, which disables line buffering and echo:
```lua
local ok, err = io.raw(true) -- enable
local ok, err = io.raw(false) -- disable
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `enable` | boolean | `true` to enable, `false` to disable (default: `true`) |
**Returns:** `boolean, error`
Raw mode is reference-counted: each `io.raw(true)` call must be matched by an `io.raw(false)` call. The terminal returns to normal mode automatically when the process exits.
### Flushing Output
Flush the standard-output buffer:
```lua
local ok, err = io.flush()
```
**Returns:** `boolean, error`. The call is a successful no-op when standard output does not implement `Sync()`.
### Command Line Arguments
Retrieve the command-line arguments:
```lua
local args = io.args()
```
**Returns:** `string[]`
`io.args()` never fails. It returns an empty table when no terminal context is available.
### Errors
This module returns structured Lua errors. A missing terminal context uses `errors.UNAVAILABLE`; direct write/flush and invalid yield-response failures use `errors.INTERNAL`. Dispatcher-backed read, readline, and raw-mode failures preserve underlying error metadata when available. `io.args()` has no error return.
---
# "Metrics & Telemetry"
## Metrics & Telemetry
The `metrics` module records application counters, gauges, and histogram observations.
This is an API reference. The snippets show one observation at a time and propagate collector errors.
Every function returns `true, nil` after passing the observation to the active collector. If the execution context has no collector, it returns `nil` and a non-retryable `errors.INTERNAL` error.
Labels are optional. Only entries with both a string key and a string value are recorded; other entries are silently ignored. A non-table labels argument is treated as if no labels were supplied.
Metric names are forwarded without local validation.
### Loading
```lua
local metrics = require("metrics")
```
#### `metrics.counter_inc`
Increment a counter by one.
```lua
local recorded, err = metrics.counter_inc("requests_total", {method = "POST"})
if err then return nil, err end
return recorded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Metric name |
| `labels` | table? | Label key-value pairs |
**Returns:** `boolean, error`
#### `metrics.counter_add`
Add a value to a counter.
```lua
local recorded, err = metrics.counter_add("bytes_total", 1024, {direction = "out"})
if err then return nil, err end
return recorded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Metric name |
| `value` | number | Value to add |
| `labels` | table? | Label key-value pairs |
**Returns:** `boolean, error`
The runtime forwards the value unchanged and does not require it to be positive.
#### `metrics.gauge_set`
Set a gauge to the current value.
```lua
local recorded, err = metrics.gauge_set("queue_depth", 42, {queue = "emails"})
if err then return nil, err end
return recorded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Metric name |
| `value` | number | Current value |
| `labels` | table? | Label key-value pairs |
**Returns:** `boolean, error`
#### `metrics.gauge_inc`
Increment a gauge by one.
```lua
local recorded, err = metrics.gauge_inc("connections", {pool = "db"})
if err then return nil, err end
return recorded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Metric name |
| `labels` | table? | Label key-value pairs |
**Returns:** `boolean, error`
#### `metrics.gauge_dec`
Decrement a gauge by one.
```lua
local recorded, err = metrics.gauge_dec("connections", {pool = "db"})
if err then return nil, err end
return recorded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Metric name |
| `labels` | table? | Label key-value pairs |
**Returns:** `boolean, error`
#### `metrics.histogram`
Record a histogram observation.
```lua
local recorded, err = metrics.histogram("duration_seconds", 0.123, {method = "GET"})
if err then return nil, err end
return recorded
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Metric name |
| `value` | number | Observed value |
| `labels` | table? | Label key-value pairs |
**Returns:** `boolean, error`
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Collector not available | `errors.INTERNAL` | no |
Invalid name or value types raise Lua argument errors instead of returning structured errors.
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "OS Time"
## OS Time
The global `os` table provides timestamps, date formatting, elapsed-time measurement, and time-difference calculations. In a workflow, current-time reads use the workflow's time reference; outside a workflow they use the system clock.
This is an API reference. Timestamp literals and formatted outputs are illustrative; current values depend on the runtime or workflow clock and timezone.
### Loading
The `os` table is global and does not require loading with `require`.
```lua
os.time()
os.date()
os.clock()
os.difftime()
```
### Getting Timestamps
Read a Unix timestamp in seconds since January 1, 1970 UTC:
```lua
-- Current timestamp
local now = os.time() -- 1718462445
-- Specific date/time
local t = os.time({
year = 2024,
month = 12,
day = 25,
hour = 10,
min = 30,
sec = 0
})
```
**Signature:** `os.time([spec]) -> number`
**Parameters:**
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `year` | number | current year | Four-digit year (e.g., 2024) |
| `month` | number | current month | Month 1-12 |
| `day` | number | current day | Day of month 1-31 |
| `hour` | number | 0 | Hour 0-23 |
| `min` | number | 0 | Minute 0-59 |
| `sec` | number | 0 | Second 0-59 |
With no arguments, `os.time()` returns the current Unix timestamp.
When called with a table, missing fields use the defaults shown above. The `year`, `month`, and `day` fields use the current date when omitted.
```lua
-- Just date (time defaults to midnight)
os.time({year = 2024, month = 6, day = 15})
-- Partial (fills in current year/month)
os.time({day = 1}) -- first of current month
```
### Formatting Dates
Format a timestamp as a string or return its date fields in a table:
local now = os.time()
-- Default format
os.date() -- "Sat Jun 15 14:30:45 2024"
-- Custom format
os.date("%Y-%m-%d", now) -- "2024-06-15"
os.date("%H:%M:%S", now) -- "14:30:45"
os.date("%Y-%m-%dT%H:%M:%S", now) -- "2024-06-15T14:30:45"
-- UTC time (prefix format with !)
os.date("!%Y-%m-%d %H:%M:%S", now) -- UTC instead of local
-- Date table
local t = os.date("*t", now)
**Signature:** `os.date([format], [timestamp]) -> string | table`
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `format` | string | `"%c"` | Format string, `"*t"` for table |
| `timestamp` | number | current time | Unix timestamp to format |
#### Format Specifiers
| Code | Output | Example |
|------|--------|---------|
| `%Y` | 4-digit year | 2024 |
| `%y` | 2-digit year | 24 |
| `%m` | Month (01-12) | 06 |
| `%d` | Day (01-31) | 15 |
| `%H` | Hour 24h (00-23) | 14 |
| `%I` | Hour 12h (01-12) | 02 |
| `%M` | Minute (00-59) | 30 |
| `%S` | Second (00-59) | 45 |
| `%p` | AM/PM | PM |
| `%A` | Weekday name | Saturday |
| `%a` | Weekday short | Sat |
| `%B` | Month name | June |
| `%b` | Month short | Jun |
| `%w` | Weekday (0-6, Sunday=0) | 6 |
| `%j` | Day of year (001-366) | 167 |
| `%U` | ISO 8601 week number (01-53, week starts Monday) | 24 |
| `%W` | ISO 8601 week number (01-53, week starts Monday) | 24 |
| `%z` | Timezone offset | -0700 |
| `%Z` | Timezone name | PDT |
| `%c` | Full date/time | Sat Jun 15 14:30:45 2024 |
| `%x` | Date only | 06/15/24 |
| `%X` | Time only | 14:30:45 |
| `%%` | Literal % | % |
#### Date Table
When the format is `"*t"`, `os.date()` returns a table:
```lua
local t = os.date("*t")
```
| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `year` | number | Four-digit year | 2024 |
| `month` | number | Month (1-12) | 6 |
| `day` | number | Day of month (1-31) | 15 |
| `hour` | number | Hour (0-23) | 14 |
| `min` | number | Minute (0-59) | 30 |
| `sec` | number | Second (0-59) | 45 |
| `wday` | number | Weekday (1-7, Sunday=1) | 7 |
| `yday` | number | Day of year (1-366) | 167 |
| `isdst` | boolean | `true` when the zone's UTC offset is nonzero in this release; not a reliable DST indicator | false |
Use `"!*t"` for UTC date table.
### Measuring Elapsed Time
Read the seconds between the current runtime time reference and the OS-time module's initialization time:
```lua
local start = os.clock()
-- do work
for i = 1, 1000000 do end
local elapsed = os.clock() - start
print(string.format("Took %.3f seconds", elapsed))
```
**Signature:** `os.clock() -> number`
Unlike standard Lua's CPU-time definition, this implementation is based on elapsed time. In workflows, it uses the workflow time reference.
### Time Difference
Calculate the difference between two timestamps in seconds:
```lua
local t1 = os.time({year = 2024, month = 1, day = 1})
local t2 = os.time({year = 2024, month = 12, day = 31})
local diff = os.difftime(t2, t1) -- t2 - t1
local days = diff / 86400
print(days) -- 365
```
**Signature:** `os.difftime(t2, t1) -> number`
| Parameter | Type | Description |
|-----------|------|-------------|
| `t2` | number | Later timestamp |
| `t1` | number | Earlier timestamp |
The result is `t2 - t1` in seconds and is negative when `t1 > t2`.
### Platform Constant
The `os.platform` constant identifies the runtime:
```lua
os.platform -- "wippy"
```
---
# "TTY"
## TTY
Terminal input events, styled output, presentation surfaces, and local virtual viewports.
Every function resolves the terminal port attached to the calling process frame. A process on a Terminal Host owns the physical terminal; a process.lua on a regular process.host owns a virtual terminal when it is spawned with a viewport grant. Without either attachment the module returns "no terminal context".
### Loading
```lua
local tty = require("tty")
```
### Model
A **Surface** is one process's exclusive presentation lease on its terminal port. It publishes complete row snapshots; the backend owns diffing and terminal recovery. Only one surface may be open on a port at a time.
A **Canvas** is an in-process styled-cell composition buffer. It clips at cell boundaries and never emits terminal control commands of its own.
A **Viewport** is a local, structured terminal boundary that lets one process host another process's surface without sharing byte streams. The shell decides where viewport content appears and translates input into the child's coordinates; the child sees an ordinary terminal port and does not know whether it is full-screen, tiled, tabbed, or hidden.
Viewports are local to one runtime node. Grants and handles are opaque local capabilities, not serializable network references.
### Input Loop
Start input delivery, subscribe to events, and process them in a loop:
```lua
local tty = require("tty")
local io = require("io")
local function handler()
local events = tty.events()
tty.start()
while true do
local ev, open = events:receive()
if not open then break end
if ev.type == "key" then
if ev.key == "q" or (ev.ctrl and ev.key == "c") then
break
end
local _, print_err = io.print("Key: " .. ev.key)
if print_err then loop_err = print_err; break end
elseif ev.type == "resize" then
local _, print_err = io.print("Size: " .. ev.width .. "x" .. ev.height)
if print_err then loop_err = print_err; break end
end
end
local _, stop_err = tty.stop()
if loop_err then return nil, loop_err end
if stop_err then return nil, stop_err end
return started
end
```
Call `events()` before `start()` so a consumer is ready when the first events arrive. On a virtual port, `start()` opens viewer-to-producer event delivery and `stop()` closes it: a `Viewport:send()` outside that interval fails instead of silently dropping input. Resize delivery is independent of input state.
#### `tty.start()`
Start input delivery for the current port. A physical terminal switches to raw mode.
```lua
local ok, err = tty.start()
```
**Returns:** `boolean, error`
#### `tty.stop()`
Stop input delivery and restore the terminal to normal mode.
```lua
local ok, err = tty.stop()
```
**Returns:** `boolean, error`
#### `tty.events()`
Subscribe to the port's terminal events and return a channel. Events are delivered as tables with a `type` field. Subscribe once and reuse the channel.
```lua
local events, err = tty.events()
```
**Returns:** `EventChannel, error`
`EventChannel` has `receive()` and `case_receive()`, so it composes with `channel.select`.
#### tty.screen_size()
Read the current terminal dimensions.
```lua
local width, height, err = tty.screen_size()
```
**Returns:** `number, number, error`
#### `tty.mouse(enable)`
Enable or disable mouse event tracking.
```lua
local ok, err = tty.mouse(true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `enable` | boolean | `true` to enable, `false` to disable |
**Returns:** `boolean, error`
### Surface
A surface is the port's presentation lease. Acquire one, publish complete frames, and close it when done.
#### tty.surface(options?)
```lua
local surface, err = tty.surface({
alternate_screen = true,
hide_cursor = true,
synchronized_output = true,
})
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `alternate_screen` | boolean | false | Present on the terminal's alternate screen buffer |
| `hide_cursor` | boolean | false | Hide the terminal cursor while the surface is open |
| `synchronized_output` | boolean | false | Wrap each frame in synchronized-output markers |
**Returns:** `Surface, error`
Opening a second surface on a port that already has one fails. A virtual port keeps the options as surface metadata; a physical port translates them into terminal modes and restores them on close.
#### surface:present(rows, options?)
Publish a complete array of row strings. Row `1` is the top line.
```lua
local stats, err = surface:present(rows, {
cursor = {x = 12, y = 3, visible = true},
images = {
{placement_id = "logo", image = logo, x = 2, y = 2, cols = 20, rows = 8, alt = "Logo"},
},
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `rows` | string[] | Complete frame, at most 16384 rows |
| `options.cursor` | table | `{x, y, visible}` in one-based surface coordinates |
| `options.images` | table[] | Complete retained-image placement set for the frame |
Omitting `cursor` preserves the last explicit cursor state. All three cursor fields are required when `cursor` is present.
**Returns:** `stats, error` — an immutable record with `rows`, `changed_rows`, and `bytes_written`. A physical frame identical to the previous one writes nothing.
#### surface:invalidate()
Forget backend presentation state without erasing the logical frame. The next `present` commits even when its rows are unchanged. Use it after an outer terminal resize or when another owner may have disturbed physical state.
**Returns:** `boolean`
#### surface:close()
Release the lease. Idempotent: later calls return the first close result. A physical backend restores terminal modes.
**Returns:** `boolean, error`
#### surface:capabilities()
Return `{images = "native" | "kitty" | "pending" | "none"}`. Start terminal
input before probing. A physical backend may briefly return `pending` while it
queries the terminal; virtual surfaces retain images without probing.
**Returns:** `table, error`
#### surface:clipboard(text)
Write an OSC 52 clipboard request on a physical surface. Text must be valid
UTF-8 and at most 65,536 bytes. Success means the terminal output accepted the
request; terminal policy may still ignore it. Virtual surfaces return an
unsupported error, and the API provides no clipboard read or acknowledgement.
**Returns:** `boolean, error`
### Retained Images
Import a PNG into bounded runtime storage, then place its handle in a complete
surface frame:
```lua
local image = assert(tty.image(png_bytes))
local info = image:info() -- id, format, width, height, bytes
assert(surface:present(rows, {images = {{
placement_id = "preview",
image = image,
x = 1, y = 1, cols = 40, rows = 12,
src = {x = 0, y = 0, width = info.width, height = info.height},
z = 1,
alt = "Preview",
}}}))
```
`tty.image()` validates PNG bytes asynchronously. `image:read()` explicitly
exports the encoded bytes and `image:close()` releases the reference. Source
pixel coordinates are zero-based; destination cell coordinates are one-based.
Omitting `images` from a later `present` clears prior placements. Unsupported
physical terminals display the placement's `alt` text, while virtual surfaces
keep the image resource for viewers.
### Canvas
A canvas is a bounded styled-cell buffer used to compose a frame before presenting it.
#### tty.canvas(width, height)
```lua
local canvas = tty.canvas(width, height)
```
Width is capped at 16384 columns, height at 16384 rows, and the area at 262,144 cells. Out-of-range arguments raise an argument error.
**Returns:** `Canvas`
Drawing accepts styled text, not terminal commands. SGR colors and OSC 8 links are preserved; erase, cursor-motion, and other control-only output is not emitted. Each placement is clipped independently at cell boundaries with grapheme-width awareness, so a clipped escape sequence cannot leak into neighboring content.
#### canvas:clear(fill?)
Clear every cell. An optional styled `fill` string is repeated across each row.
```lua
canvas:clear()
canvas:clear(tty.style():background("#1a1a1a"):render(" "))
```
**Returns:** `boolean`
#### canvas:put(x, y, text, width?)
Place one styled row at one-based `x`, `y` and clip it to `width` cells (default: the canvas width). Coordinates may be negative or past the edge; the placement is clipped rather than rejected. A newline ends the row, so use `put_rows` for multi-row content.
```lua
canvas:put(3, 1, tty.style():bold():render("Title"), 40)
```
**Returns:** `boolean`
#### canvas:put_rows(x, y, rows, width?)
Place an array of styled rows starting at `x`, `y`, one row per line downward. Every entry is validated before anything is drawn.
```lua
canvas:put_rows(2, 2, child_rows, inner_width)
```
**Returns:** `boolean`
#### canvas:rows()
Render the complete row array, ready for `surface:present`.
**Returns:** `string[]`
### Viewport
A viewport is a virtual terminal port. The creating process is its first viewer; the process admitted with its grant is its producer.
#### tty.viewport(options?)
```lua
local view, err = tty.viewport({
width = 80,
height = 24,
page = {foreground = "#e0def4", background = "#191724"},
})
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `width` | number | 80 | Columns, 1 to 65535 |
| `height` | number | 24 | Rows, 1 to 65535 |
| `page` | table | none | Opaque `#RRGGBB` foreground and background defaults |
The area is capped at 262,144 cells.
**Returns:** `Viewport, error`
#### tty.attach(handle)
Add another local viewer to an existing viewport. A handle grants viewing, never presentation ownership, and is not valid on another node.
```lua
local view, err = tty.attach(handle)
```
**Returns:** `Viewport, error`
#### viewport:grant()
Return the one-shot producer capability. Pass it as the `terminal` spawn option:
```lua
local grant = assert(view:grant())
local child = assert(process.with_options({terminal = grant})
:spawn_monitored("app:child", "app:workers"))
```
Admission consumes the grant transactionally: a rejected start restores an unresolved grant, while a process that has resolved the port consumes it permanently. A host that does not support terminal attachments rejects the spawn instead of dropping the option. See [Processes](lua/core/process.md#spawner-with-options).
**Returns:** `string, error`
#### viewport:handle()
Return the local viewer handle for `tty.attach`.
**Returns:** `string`
#### viewport:snapshot(after_revision?)
Read the current dimensions, rows, cursor, and revision. With `after_revision`, return `nil` when the revision is unchanged.
```lua
local frame = view:snapshot(revision)
if frame then
revision = frame.revision
canvas:put_rows(2, 2, frame.rows, inner_width)
end
```
**Returns:** `snapshot` or `nil`
| Field | Type | Description |
|-------|------|-------------|
| `revision` | number | Monotonic revision of this frame |
| `width` | number | Viewport columns |
| `height` | number | Viewport rows |
| `rows` | string[] | Rows last published by the producer |
| `cursor` | table | `{x, y, visible}` in one-based coordinates, absent until the producer publishes explicit cursor state |
| `images` | table[] | Retained image placement metadata |
| `layers` | table[] | Ordered presentation layers |
| `images_omitted` | boolean | Image resources exist but are not retained by this plain snapshot |
A page resolves terminal-default cells and omitted rows to explicit colors. The
creator can change it with `viewport:set_page(page)`; passing `nil` restores the
producer's original rows. Page changes advance the revision without requiring
the producer to repaint.
#### viewport:updates()
Return a channel of coalesced revision watermarks. `receive()` yields the revision number; `case_receive()` composes with `channel.select`.
```lua
local updates = assert(view:updates())
```
Updates are bounded hints, not an event log. A slow viewer receives only the newest watermark and must call `snapshot()` for state. Presentation and resize never block on a slow viewer.
**Returns:** `ViewportUpdateChannel, error`
#### viewport:send(event)
Forward a validated event record to the producer. The producer must have called `tty.start()`; otherwise the call fails rather than dropping the event.
```lua
assert(view:send(event))
assert(view:send({type = "close"}))
```
**Returns:** `boolean, error`
#### viewport:resize(width, height)
Update the viewport geometry. When the size changes, viewers get a new revision and the producer receives a `resize` event.
**Returns:** `boolean, error`
#### viewport:close()
Detach this viewer only. Closing the last viewer does not kill a live producer, and closing the producer's port does not destroy state while viewers remain.
**Returns:** `boolean, error`
#### viewport:mount(recipient_pid, rights)
Issue a process-bound reference for a local or remote viewer. Rights are
independent and default to false:
```lua
local observation = assert(view:mount(agent_pid, {observe = true}))
local control = assert(view:mount(agent_pid, {input = true, resize = true}))
-- In the exact recipient process, on this node or an authenticated mesh peer:
local observer = assert(tty.attach(observation))
local controller = assert(tty.attach(control))
```
A mount is bound to the recipient's full PID and can be redeemed only once.
Mounted viewers cannot create producer grants or delegate further mounts.
Remote mounts use a renewable lease; reconnecting requires a fresh mount and
cannot replay terminal input. Use `viewport:revoke(reference)` to revoke an
issued mount. Closing the owner viewport or ending the owner process revokes
its mounts.
#### viewport:capture()
Atomically pin a viewport revision and its retained image resources:
```lua
local capture = assert(view:capture())
local snapshot = capture:snapshot()
local image = assert(capture:image(snapshot.images[1].image_id))
assert(capture:close())
```
A plain `snapshot()` does not retain image bytes. A capture does until closed;
image handles already acquired from it remain independently owned.
### Event Types
Events are tables with a `type` field that determines which other fields are present. Coordinates are one-based. The same records are accepted by `viewport:send()`.
#### Key Event
```lua
{
type = "key",
key = "a", -- printable character or key name
key_type = "runes", -- "runes" for printable, or special key name
action = "press", -- "press" or "release"
alt = false,
ctrl = false,
shift = false
}
```
#### Mouse Event
Requires `tty.mouse(true)`.
```lua
{
type = "mouse",
action = "press", -- "press", "release", "motion", "wheel"
button = "left", -- button name
x = 10,
y = 5,
alt = false,
ctrl = false,
shift = false
}
```
#### Resize Event
```lua
{type = "resize", width = 120, height = 40}
```
#### Start Event
Emitted once after `tty.start()` with initial dimensions.
```lua
{type = "start", width = 120, height = 40}
```
#### Focus Event
Reports keyboard ownership.
```lua
{type = "focus", focused = true}
```
#### Visibility Event
Reports whether repainting is useful. It does not prescribe application lifecycle or background computation.
```lua
{type = "visibility", visible = true}
```
#### Paste Event
```lua
{type = "paste", text = "pasted content"}
```
#### Close Event
Asks the producer to shut down. A shell sends it through `viewport:send` to request a graceful child exit.
```lua
{type = "close"}
```
### Key Bindings
Create reusable key bindings that match against key events:
```lua
local quit = tty.bind({
keys = {"q", "ctrl+c"},
help = {key = "q/ctrl+c", desc = "quit"}
})
-- In event loop
if quit:matches(ev) then
break
end
```
#### `tty.bind(config)`
| Field | Type | Description |
|-------|------|-------------|
| `keys` | string[] | Required. Key patterns to match (e.g. `"a"`, `"ctrl+c"`, `"enter"`) |
| `help` | table | Optional. `{key = "...", desc = "..."}` for help text |
**Returns:** `KeyBinding`
The type schema requires `keys`. At runtime, an omitted or empty `keys` table creates a binding that never matches.
#### KeyBinding Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `matches(event)` | boolean | Test if a key event matches this binding |
| `set_enabled(bool)` | self | Enable or disable the binding |
| `is_enabled()` | boolean | Check if the binding is enabled |
| `help()` | table | Returns `{key, desc}` help info |
### Styles
Create styled terminal output. Style values are immutable, so each style method returns a new value.
```lua
local tty = require("tty")
local io = require("io")
local title = tty.style()
:bold()
:foreground("#FF0000")
:padding(0, 1)
local box = tty.style()
:border(tty.borders.ROUNDED)
:border_foreground("#00FF00")
:width(40)
:padding(1, 2)
local _, print_err = io.print(box:render(title:render("Hello"), "World"))
if print_err then return nil, print_err end
```
#### `tty.style()`
Create an empty style.
**Returns:** `Style`
#### Style Methods
All methods return a new `Style` and can be chained.
##### Text Decoration
| Method | Parameter | Description |
|--------|-----------|-------------|
| `foreground(color)` | string | Text color (hex `"#FF0000"`, ANSI `"9"`, or name) |
| `background(color)` | string | Background color |
| `bold(enable?)` | boolean | Bold text (default: true) |
| `italic(enable?)` | boolean | Italic text |
| `underline(enable?)` | boolean | Underline text |
| `strikethrough(enable?)` | boolean | Strikethrough text |
| `faint(enable?)` | boolean | Dimmed text |
| `blink(enable?)` | boolean | Blinking text |
| `reverse(enable?)` | boolean | Swap foreground/background |
##### Layout
| Method | Parameter | Description |
|--------|-----------|-------------|
| `width(n)` | number | Fixed width |
| `height(n)` | number | Fixed height |
| `max_width(n)` | number | Maximum width |
| `max_height(n)` | number | Maximum height |
| `padding(...)` | numbers | Padding (CSS-style: top, right, bottom, left) |
| `margin(...)` | numbers | Margin (CSS-style) |
| `align(pos)` | number | Horizontal alignment |
| `align_vertical(pos)` | number | Vertical alignment |
| `inline(enable?)` | boolean | Inline rendering mode |
##### Borders
| Method | Parameter | Description |
|--------|-----------|-------------|
| `border(name, ...)` | string, booleans | Border style, optional per-side toggles |
| `border_foreground(...)` | strings | Border color(s) |
| `border_background(...)` | strings | Border background color(s) |
##### Other
| Method | Description |
|--------|-------------|
| `render(...)` | Render strings with this style applied |
| `copy()` | Create a copy of this style |
#### Border Constants
```lua
tty.borders.NORMAL
tty.borders.ROUNDED
tty.borders.THICK
tty.borders.DOUBLE
tty.borders.HIDDEN
```
#### Alignment Constants
```lua
tty.align.LEFT -- 0
tty.align.CENTER -- 0.5
tty.align.RIGHT -- 1
```
### Text Utilities
The `tty.text` subtable provides layout and measurement functions for styled text.
#### Measurement
```lua
local w = tty.text.width("hello") -- printable width (ANSI-aware)
local h = tty.text.height("a\nb\nc") -- line count
local w, h = tty.text.size("hello\nworld") -- both
```
#### Clipping
```lua
-- Truncate to a printable width, with an optional tail
local head = tty.text.truncate(line, 40)
local head = tty.text.truncate(line, 40, "…")
-- Take the printable cell range [left, right)
local middle = tty.text.cut(line, 10, 30)
```
Both preserve ANSI state and grapheme boundaries, so styled text can be clipped and spliced without breaking escape sequences. `truncate` returns an empty string for a width of zero or less; `cut` returns an empty string when `right` is not greater than `left`.
#### Joining
```lua
-- Join side by side, aligned at top
local row = tty.text.join_horizontal(tty.text.position.TOP, left, right)
-- Stack vertically, centered
local col = tty.text.join_vertical(tty.text.position.CENTER, top, bottom)
```
#### Max Dimensions
```lua
local w = tty.text.max_width({"short", "a longer string"}) -- widest
local h = tty.text.max_height({"one\ntwo", "single"}) -- tallest
```
#### Placement
Place a string within a box with the given dimensions:
```lua
-- Center in a 80x24 box
local out = tty.text.place(80, 24, tty.text.position.CENTER, tty.text.position.CENTER, content)
-- Horizontal only
local out = tty.text.place_horizontal(80, tty.text.position.RIGHT, content)
-- Vertical only
local out = tty.text.place_vertical(24, tty.text.position.BOTTOM, content)
```
#### Position Constants
```lua
tty.text.position.TOP -- 0
tty.text.position.LEFT -- 0
tty.text.position.CENTER -- 0.5
tty.text.position.BOTTOM -- 1
tty.text.position.RIGHT -- 1
```
### Permissions
Access to a physical terminal comes from the process frame. Attaching a
producer with `process.with_options({terminal = grant})` requires
`process.context` on the spawning side. Delegated viewports additionally check:
| Action | Resource | Description |
|--------|----------|-------------|
| `tty.mount` | Owner viewport handle | Issue a process-bound mount |
| `tty.observe` | Owner viewport handle | Read snapshots, updates, and captures |
| `tty.input` | Owner viewport handle | Forward input events |
| `tty.resize` | Owner viewport handle | Resize the viewport |
### See Also
- [Terminal UI](tutorials/tty.md) — build a shell that hosts a child in a viewport
- [Terminal I/O](lua/system/io.md) — stdin/stdout/stderr operations
- [Terminal Host](system/terminal.md) — Terminal host configuration
- [Command Execution](lua/dynamic/exec.md) — PTY processes and terminal sessions
- [Processes](lua/core/process.md) — spawn options, monitoring, lifecycle events
---
# "Hub"
## Hub
The `hub` module reads Wippy Hub modules, versions, dependencies, files, artifacts, and READMEs. It also manages the runtime's Hub credential override and can remove unpinned artifacts from the local cache.
This is an API reference. Catalog coordinates are illustrative; artifact, authentication, and cache operations require matching network access, credentials, lock state, and security policies.
### Loading
```lua
local hub = require("hub")
```
### Per-call Options
Network-backed catalog and artifact calls accept an optional options table with these common keys:
| Key | Type | Description |
|-----|------|-------------|
| `registry` | string | Registry URL override |
| `token` | string | API token override |
| `timeout` | duration/number | Request timeout (e.g. `"3m"` or seconds) |
Pagination-aware calls also accept `page` and `page_size`.
Authentication calls take a registry URL directly. Cache calls and package-handle methods use their own options described below.
### Modules
```lua
local result, err = hub.modules.list({
org = "wippy",
visibility = "public",
type = "library",
sort_order = "downloads_desc",
page = 1,
page_size = 20,
})
-- result = { items, total, page, page_size }
```
| Function | Description |
|----------|-------------|
| `hub.modules.list(opts?)` | List modules with filters |
| `hub.modules.search(query, opts?)` | Search by query string |
| `hub.modules.get(module, opts?)` | Fetch module by `org/name` or module id |
| `hub.modules.readme(module, opts?)` | Fetch README; returns `{content, filename, version}` |
#### List/Search Options
| Option | Values |
|--------|--------|
| `organization_id` / `org` | string |
| `visibility` | `public`, `private`, `internal` |
| `type` | `library`, `application`, `agent`, `plugin` |
| `sort_order` | `name_asc`, `name_desc`, `created_desc`, `updated_desc`, `downloads_desc` |
| `keywords` (search) | array of strings |
| `license` (search) | string |
| `include_deprecated` (search) | boolean |
#### README
```lua
local readme, err = hub.modules.readme("wippy/terminal", {
version = "1.2.3"
})
if err then return nil, err end
print(readme.content)
```
The `version` option accepts either a version string or a table like `{id, version, label}`.
### Versions
```lua
local versions, err = hub.versions.list("wippy/terminal", {
include_yanked = false,
page_size = 50,
})
local v, err = hub.versions.get("wippy/terminal", "1.0.0")
```
| Function | Description |
|----------|-------------|
| `hub.versions.list(module, opts?)` | List versions for a module |
| `hub.versions.get(module, version, opts?)` | Fetch a specific version |
| `hub.versions.inspect(module, version, opts?)` | Inspect a version's artifact (downloads and reads the bundle) |
| `hub.versions.open(module, version, opts?)` | Open a version's artifact as a package handle |
#### Package Handle
`hub.versions.open` downloads an artifact and returns a handle with the fields `version`, `digest`, and `packed`:
```lua
local pkg, err = hub.versions.open("wippy/terminal", "1.2.3")
if err then return nil, err end
local entries, entries_err = pkg:entries({
kind = "function.lua", -- string or string[], omit for all kinds
include_data = false, -- default true
})
-- each entry: { id = "ns:name", kind = "...", meta = {...}, data = }
local _, close_err = pkg:close()
if entries_err then return nil, entries_err end
if close_err then return nil, close_err end
return entries
```
| Method | Description |
|--------|-------------|
| `pkg:metadata()` | Pack metadata map |
| `pkg:entries(opts?)` | Registry entries in the artifact; `opts.kind` filters, `opts.include_data` (default true) controls the `data` field |
| `pkg:resources()` | Embedded resources list |
| `pkg:fs(resource)` | Filesystem handle for an embedded resource |
| `pkg:close()` | Release the handle |
Entry `data` is returned without resolving `${env:...}` references.
### Local Artifact Cache
```lua
local entries, err = hub.cache.list()
local removed, err = hub.cache.remove("wippy/terminal", "1.2.3", {
force = false,
})
local candidates, err = hub.cache.prune({
dry_run = true,
})
```
| Function | Description |
|----------|-------------|
| `hub.cache.list()` | List cached artifacts as `{module, version, size, pinned}` records |
| `hub.cache.remove(module, version, opts?)` | Remove one cached artifact; `opts.force = true` permits removal when the lock file pins it |
| `hub.cache.prune(opts?)` | Remove artifacts not referenced by the lock file; `opts.dry_run = true` only reports candidates |
`hub.cache.remove` and `hub.cache.prune` delete files from the lock-resolved vendor directory unless their dry-run or pin protections apply.
### Dependencies
```lua
local deps, err = hub.dependencies.get("wippy/terminal", "1.0.0")
local users, err = hub.dependents.get("wippy/terminal")
```
| Function | Description |
|----------|-------------|
| `hub.dependencies.get(module, version?, opts?)` | Dependencies for a module version |
| `hub.dependents.get(module, opts?)` | Modules that depend on this one |
### Files
```lua
local files, err = hub.files.list("wippy/terminal", "1.0.0")
```
| Function | Description |
|----------|-------------|
| `hub.files.list(module, version, opts?)` | List files for a version (`version` required); returns `{items, total, page, page_size}` |
### Cache
```lua
local cached, err = hub.cache.list()
-- each entry: { module, version, size, pinned }
local ok, err = hub.cache.remove("wippy/terminal", "1.2.3", { force = true })
local pruned, err = hub.cache.prune({ dry_run = true })
```
| Function | Description |
|----------|-------------|
| `hub.cache.list(opts?)` | List cached artifacts under the vendor directory; `pinned` is `true` when the lock file references the artifact |
| `hub.cache.remove(module, version, opts?)` | Remove one cached artifact; refuses a lock-pinned artifact (kind `errors.CONFLICT`) unless `opts.force` is `true`; returns `true` |
| `hub.cache.prune(opts?)` | Remove every cached artifact the lock file does not reference; with `opts.dry_run = true` nothing is deleted; returns the pruned (or would-be-pruned) entries |
**Permissions:** `hub.cache.list`, `hub.cache.remove` (resource: module name), `hub.cache.prune`
### Authentication
Install a registry token as a runtime override. Hub consumers use it on subsequent calls without requiring a restart:
```lua
local status, err = hub.auth.authenticate("wpy_xxx") -- default registry
local status, err = hub.auth.authenticate("wpy_xxx", "https://hub.example.com")
local status, err = hub.auth.status()
local ok, err = hub.auth.logout()
```
The token strings above are placeholders. Load real credentials from a secret-backed environment entry or another protected source; do not commit them in Lua or registry YAML.
| Function | Description |
|----------|-------------|
| `hub.auth.authenticate(token, registry?)` | Validate the token against the registry and, on success, install it as the runtime override |
| `hub.auth.status(registry?)` | Live-validate the current credential |
| `hub.auth.logout(registry?)` | Clear the runtime token override |
`status` contains `authenticated`, `registry`, and `orgs`. Identity fields (`username`, `user_id`, `scope`, `expires_at`, `expired`) are present only when authenticated. A token that fails validation is not stored; `authenticate` returns `authenticated = false`. The runtime override takes precedence over `WIPPY_TOKEN` and stored credentials.
### Permissions
Each top-level `hub.*` operation checks the matching action name, such as `hub.modules.list`, `hub.versions.open`, `hub.dependencies.get`, `hub.files.list`, `hub.auth.status`, or `hub.cache.prune`. Actions that address a module use the supplied module reference as the security resource; authentication actions use the registry URL. Package-handle methods do not perform another permission check after the authorized `hub.versions.open` call.
### See Also
- [CLI Reference](guides/cli.md) — `wippy readme`, `wippy search`, `wippy publish`
- [Publishing Guide](guides/publishing.md)
---
# "Text Processing"
## Text Processing
The `text` module provides regular expressions, text comparison and patching, and document splitting. This page is an API reference. Its short blocks are isolated calls; longer splitting blocks are partial recipes whose documents, configured filesystem resources, and downstream processing belong to the surrounding application.
### Loading
```lua
local text = require("text")
```
#### `text.regexp.compile`
Compile an RE2-compatible regular expression.
```lua
local re, err = text.regexp.compile("[0-9]+")
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `pattern` | string | RE2 compatible regex pattern |
**Returns:** `Regexp, error`
#### `re:match_string`
Match a string against the compiled expression.
```lua
local ok = re:match_string("abc123")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to match |
**Returns:** `boolean`
#### `re:find_string`
Find the first matching substring.
```lua
local match = re:find_string("abc123def")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to search |
**Returns:** `string | nil`
At this runtime pin, an empty-string match is also represented as `nil`; use a pattern that consumes at least one character when an empty match must be distinguished from no match.
#### `re:find_all_string`
Find all matching substrings.
```lua
local matches = re:find_all_string("a1b2c3")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to search |
**Returns:** `string[]`
#### `re:find_string_submatch`
Find the first match and its capture groups.
```lua
local match = re:find_string_submatch("user@example.com")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to search |
**Returns:** `string[] | nil` (full match + capture groups)
#### `re:find_all_string_submatch`
Find all matches and their capture groups.
```lua
local matches = re:find_all_string_submatch("a=1 b=2")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to search |
**Returns:** `string[][]`
#### `re:find_string_index`
Find the 1-based bounds of the first match.
```lua
local pos = re:find_string_index("abc123")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to search |
**Returns:** `table | nil` ({start, end}, 1-based)
#### `re:find_all_string_index`
Find the bounds of all matches.
```lua
local positions = re:find_all_string_index("a1b2c3")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to search |
**Returns:** `table[] | nil` (nil when there are no matches)
#### `re:replace_all_string`
Replace every matching substring.
```lua
local result = re:replace_all_string("a1b2", "X")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | Input string |
| `repl` | string | Replacement string |
**Returns:** `string`
#### `re:split`
Split a string at matches of the compiled expression.
```lua
local parts = re:split("a,b,c", -1)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `s` | string | String to split |
| `n` | integer | Max parts, -1 for all |
**Returns:** `string[]`
#### `re:num_subexp`
Return the number of capturing subexpressions.
```lua
local count = re:num_subexp()
```
**Returns:** `number`
#### `re:subexp_names`
Return the names of capturing subexpressions.
```lua
local names = re:subexp_names()
```
**Returns:** `string[]`
#### `re:string`
Return the compiled pattern string.
```lua
local pattern = re:string()
```
**Returns:** `string`
### Text Diffing
Compare text versions and generate patches with [go-diff](https://github.com/sergi/go-diff), an implementation of Google's diff-match-patch algorithm.
#### `text.diff.new`
Create a text differ with default or custom options.
```lua
local diff, err = text.diff.new()
local diff, err = text.diff.new(options)
```
**Returns:** `Differ, error`
##### Options {id="diff-options"}
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `diff_timeout` | number | 1.0 | Timeout in seconds |
| `diff_edit_cost` | integer | 4 | Cost of an empty edit |
| `match_threshold` | number | 0.5 | Match tolerance 0-1 |
| `match_distance` | integer | 1000 | Distance to search for match |
| `patch_delete_threshold` | number | 0.5 | Delete threshold |
| `patch_margin` | integer | 4 | Context margin |
#### `diff:compare`
Compare two strings and return operations that transform `text1` into `text2`.
```lua
local diff, diff_err = text.diff.new()
if diff_err then
return nil, diff_err
end
local diffs, err = diff:compare("hello world", "hello there")
if err then
return nil, err
end
-- diffs contains:
-- {operation = "equal", text = "hello "}
-- {operation = "delete", text = "world"}
-- {operation = "insert", text = "there"}
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `text1` | string | Original text |
| `text2` | string | Modified text |
**Returns:** `table, error` (array of {operation, text})
Operations: `"equal"`, `"delete"`, `"insert"`
#### `diff:summarize`
Count the unchanged, inserted, and deleted UTF-8 bytes. For non-ASCII text, these totals are not Unicode code-point or grapheme counts.
```lua
-- `diffs` is the checked result from diff:compare.
local summary = diff:summarize(diffs)
-- summary.equals = 6 (bytes unchanged)
-- summary.deletions = 5 (bytes removed)
-- summary.insertions = 5 (bytes added)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `diffs` | table | Diff array from compare |
**Returns:** `table` ({insertions, deletions, equals})
#### `diff:pretty_text`
Format a diff with ANSI colors for terminal output.
```lua
local formatted, err = diff:pretty_text(diffs)
if err then
return nil, err
end
print(formatted)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `diffs` | table | Diff array from compare |
**Returns:** `string, error`
#### `diff:pretty_html`
Format a diff as HTML with `` and `` elements.
```lua
local html, err = diff:pretty_html(diffs)
if err then
return nil, err
end
-- `html` is an HTML fragment with equal, deleted, and inserted spans.
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `diffs` | table | Diff array from compare |
**Returns:** `string, error`
#### `diff:patch_make`
Create patches that transform one string into another. The patches can be serialized and applied later.
```lua
local text1 = "The quick brown fox jumps over the lazy dog"
local text2 = "The quick red fox jumps over the lazy cat"
local patches, err = diff:patch_make(text1, text2)
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `text1` | string | Original text |
| `text2` | string | Modified text |
**Returns:** `table, error`
#### `diff:patch_apply`
Apply patches to a string and return the result and whether every patch succeeded.
```lua
local result, success = diff:patch_apply(patches, text1)
-- result = "The quick red fox jumps over the lazy cat"
-- success = true
```
Check `success` before treating `result` as the requested transformation. Pass patch tables produced by `patch_make`; at this runtime pin, malformed serialized patch text inside a hand-built table can be skipped rather than reported separately.
| Parameter | Type | Description |
|-----------|------|-------------|
| `patches` | table | Patches from patch_make |
| `text` | string | Text to apply patches to |
**Returns:** `string, boolean`
### Text Splitting
Split documents into chunks while preserving semantic boundaries. The splitters are based on the [langchaingo](https://github.com/tmc/langchaingo) implementation.
#### `text.splitter.recursive`
The recursive splitter tries double newlines, single newlines, spaces, and then individual characters. It moves to the next separator when a chunk exceeds the size limit.
```lua
local splitter, err = text.splitter.recursive({
chunk_size = 1000,
chunk_overlap = 100
})
if err then
return nil, err
end
local long_text = "This is a long text that needs splitting..."
local chunks, split_err = splitter:split_text(long_text)
if split_err then
return nil, split_err
end
```
**Returns:** `Splitter, error`
##### Options {id="recursive-splitter-options"}
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `chunk_size` | integer | 4000 | Max characters per chunk |
| `chunk_overlap` | integer | 200 | Characters repeated between adjacent chunks |
| `keep_separator` | boolean | false | Keep separators in output |
| `separators` | string[] | nil | Custom separator list |
#### `text.splitter.markdown`
The Markdown splitter can keep headings with their content, preserve code blocks, and group table rows.
```lua
local splitter, err = text.splitter.markdown({
chunk_size = 2000,
code_blocks = true,
heading_hierarchy = true
})
if err then
return nil, err
end
local readme = fs.get("app:docs"):readfile("README.md")
local chunks, err = splitter:split_text(readme)
```
This partial recipe requires the entry to enable both `text` and `fs`, a configured `app:docs` filesystem resource, and a readable `README.md` within that resource.
**Returns:** `Splitter, error`
##### Options {id="markdown-splitter-options"}
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `chunk_size` | integer | 4000 | Max characters per chunk |
| `chunk_overlap` | integer | 200 | Characters repeated between adjacent chunks |
| `code_blocks` | boolean | false | Keep code blocks together |
| `reference_links` | boolean | false | Preserve reference links |
| `heading_hierarchy` | boolean | false | Respect heading levels |
| `join_table_rows` | boolean | false | Keep table rows together |
| `separators` | string[] | nil | Custom separator list |
#### `splitter:split_text`
Split a single document into an array of chunks.
```lua
local chunks, err = splitter:split_text(document)
if err then
return nil, err
end
for i, chunk in ipairs(chunks) do
-- Process each chunk (e.g., create embedding, send to LLM)
process(chunk)
end
```
Here, `splitter` is a successfully created splitter, while `document` and `process` are supplied by the application.
| Parameter | Type | Description |
|-----------|------|-------------|
| `text` | string | Text to split |
**Returns:** `string[], error`
#### `splitter:split_batch`
Split multiple documents while preserving their metadata. One input document can produce several chunks, each with the source document's metadata.
```lua
-- Input: pages from a PDF with page numbers
local pages = {
{content = "First page content...", metadata = {page = 1}},
{content = "Second page content...", metadata = {page = 2}}
}
local chunks, err = splitter:split_batch(pages)
if err then
return nil, err
end
-- Output: each chunk knows which page it came from
for _, chunk in ipairs(chunks) do
print("Page " .. chunk.metadata.page .. ": " .. chunk.content:sub(1, 50))
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `pages` | table | Array of {content, metadata} |
**Returns:** `table, error` (array of {content, metadata})
`split_batch` silently skips an item when the item is not a table, its `content` field is missing, empty, or not a string, or splitting that item fails. It still returns the remaining chunks with a `nil` error. Validate every input item before the call and verify any cardinality requirements in application code; do not treat a successful call as proof that every input was represented.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Invalid pattern syntax | `errors.INVALID` | no |
| Internal error | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Template Engine"
## Template Engine
The `templates` module renders [Jet](https://github.com/CloudyKit/jet) templates from configured sets. Templates can use inheritance and includes. This page is an API reference with isolated rendering examples, not a standalone template deployment. The registry IDs and template sources must already be configured, and the executable entry must enable `templates` and have `template.get` permission for the requested set.
For template set configuration, see [Template Engine](system/template.md).
### Loading
```lua
local templates = require("templates")
```
### `templates.get`
Acquire a template set by registry ID:
```lua
local set, err = templates.get("app.views:emails")
if err then
return nil, err
end
-- Use the set...
return set:release()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Template set registry ID |
**Returns:** `Set, error`
### `set:render`
Render a template by name with data:
```lua
local set, get_err = templates.get("app.views:emails")
if get_err then
return nil, get_err
end
local html, err = set:render("welcome", {
user = {name = "Alice", email = "alice@example.com"},
activation_url = "https://example.invalid/activate"
})
set:release()
if err then
return nil, err
end
return html
```
The caller owns every acquired set until `release()` is called. Release it after the final render, including checked error paths; repeated releases are safe. Rendering does not make application-provided values safe for every output context. Keep secrets and one-time URLs out of logs, and apply the escaping or sanitization required where the rendered string is consumed.
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Template name within the set |
| `data` | table | Variables to pass to template (optional) |
**Returns:** `string, error`
### Set Method Summary
The set handle provides these methods:
| Method | Returns | Description |
|--------|---------|-------------|
| `render(name, data?)` | `string, error` | Render template with data |
| `release()` | `boolean` | Release set back to pool |
### Jet Syntax Reference
Jet uses `{{ }}` for expressions and control structures and `{* *}` for comments.
#### Variables
```html
{{ user.name }}
{{ user.email }}
{{ items[0].price }}
```
#### Conditionals
```html
{{ if order.shipped }}
Shipped!
{{ else if order.processing }}
Processing...
{{ else }}
Received.
{{ end }}
```
#### Loops
```html
{{ range items }}
{{ end }}
```
#### Includes
```html
{{ include "partials/header" }}
Content
{{ include "partials/footer" }}
```
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Empty ID | `errors.INVALID` | no |
| Empty template name | `errors.INVALID` | no |
| Permission denied | `errors.PERMISSION_DENIED` | no |
| Template set missing, unavailable, or wrong resource type | `errors.INTERNAL` | no |
| Template not found | `errors.NOT_FOUND` | no |
| Render error | `errors.INTERNAL` | no |
| Render attempted after the set was released | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Tree-sitter Parsing"
## Tree-sitter Parsing
The `treesitter` module parses source code into concrete syntax trees with [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) through the [go-tree-sitter](https://github.com/tree-sitter/go-tree-sitter) bindings.
This page is an API reference with partial parsing recipes. Source strings and query patterns are application input, and node-level snippets assume a live tree from an earlier checked parse. Parsers, trees, queries, and cursors are owned resources: close every successfully created handle when its last dependent operation is complete.
The resulting syntax trees:
- Represent the full structure of source code
- Update incrementally as code changes
- Are robust to syntax errors (partial parsing)
- Support pattern-based queries using S-expressions
### Loading
```lua
local treesitter = require("treesitter")
```
The `treesitter` module is optional and is present only in builds that include the `treesitter` build tag. Official Wippy binaries include it. Source builds can use `make build-wippy` or `go build -tags treesitter`; without the tag, `require("treesitter")` is unavailable.
### Supported Languages
| Language | Aliases | Root Node |
|----------|---------|-----------|
| Go | `go`, `golang` | `source_file` |
| JavaScript | `js`, `javascript` | `program` |
| TypeScript | `ts`, `typescript` | `program` |
| TSX | `tsx` | `program` |
| Python | `python`, `py` | `module` |
| Lua | `lua` | `chunk` |
| PHP | `php` | `program` |
| C# | `csharp`, `cs`, `c#` | `compilation_unit` |
| HTML | `html`, `html5` | `document` |
| Markdown | `markdown`, `md` | `document` |
| SQL | `sql` | - |
```lua
local langs = treesitter.supported_languages()
-- {go = true, javascript = true, python = true, ...}
```
#### Parse Code
```lua
local code = [[
func hello() {
return "Hello!"
}
]]
local tree, err = treesitter.parse("go", code)
if err then
return nil, err
end
local root, root_err = tree:root_node()
if root_err then
tree:close()
return nil, root_err
end
print(root:kind()) -- "source_file"
print(root:child_count()) -- number of top-level declarations
tree:close()
```
#### Query Syntax Tree
```lua
local code = [[
func hello() {}
func world() {}
]]
local tree, parse_err = treesitter.parse("go", code)
if parse_err then
return nil, parse_err
end
local root, root_err = tree:root_node()
if root_err then
tree:close()
return nil, root_err
end
-- Find all function names
local query, query_err = treesitter.query("go", [[
(function_declaration name: (identifier) @func_name)
]])
if query_err then
tree:close()
return nil, query_err
end
local captures, captures_err = query:captures(root, code)
if captures_err then
query:close()
tree:close()
return nil, captures_err
end
for _, capture in ipairs(captures) do
print(capture.name, capture.text)
end
-- "func_name" "hello"
-- "func_name" "world"
query:close()
tree:close()
```
#### Simple Parse
Parse source code with a temporary internal parser.
```lua
local tree, err = treesitter.parse("go", code)
if err then
return nil, err
end
-- Use the tree, then call tree:close().
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `language` | string | Language name or alias |
| `code` | string | Source code |
**Returns:** `Tree, error`
#### Reusable Parser
Create a reusable parser for repeated parsing or incremental updates.
```lua
local parser, parser_err = treesitter.parser()
if parser_err then
return nil, parser_err
end
local _, language_err = parser:set_language("go")
if language_err then
parser:close()
return nil, language_err
end
local tree1, first_err = parser:parse("package main")
if first_err then
parser:close()
return nil, first_err
end
-- Parse another source with the reusable parser. For an incremental update,
-- edit the old tree first as shown in the complete recipe below.
local tree2, second_err = parser:parse("package main\nfunc foo() {}")
if second_err then
tree1:close()
parser:close()
return nil, second_err
end
tree2:close()
tree1:close()
parser:close()
```
**Returns:** `Parser, error`
#### Parser Methods
| Method | Description |
|--------|-------------|
| `set_language(lang)` | Set parser language, returns `boolean, error` |
| `get_language()` | Get current language name |
| `parse(code, old_tree?)` | Parse code, optionally with old tree for incremental parsing |
| `set_timeout(duration)` | Set parse timeout (string like `"1s"` or nanoseconds) |
| `set_ranges(ranges)` | Set byte ranges to parse |
| `reset()` | Reset parser state |
| `close()` | Release parser resources |
Trees created by a reusable parser and the parser itself are independently owned; close each successful handle. Nodes borrow their tree's storage and must not be used after that tree is closed. Cursors borrow a tree as well, so close them before the tree. Queries own separate native resources and also require `close()`; close a query after its captures or matches are no longer needed. Explicit cleanup is deterministic, while the process resource store is only a fallback for handles left open at process teardown.
#### Get Root Node
```lua
local tree, err = treesitter.parse("go", "package main")
if err then
return nil, err
end
local root, root_err = tree:root_node()
if root_err then
tree:close()
return nil, root_err
end
print(root:kind()) -- "source_file"
local source_text, text_err = root:text()
if text_err then
tree:close()
return nil, text_err
end
print(source_text) -- "package main"
tree:close()
```
#### Tree Methods
| Method | Description |
|--------|-------------|
| `root_node()` | Get root node of tree |
| `root_node_with_offset(bytes, point)` | Get root with offset applied |
| `language()` | Get tree's language object |
| `copy()` | Create deep copy of tree |
| `walk()` | Create cursor for traversal |
| `edit(edit_table)` | Apply incremental edit |
| `changed_ranges(other_tree)` | Get ranges that changed |
| `included_ranges()` | Get ranges included during parsing |
| `dot_graph()` | Get DOT graph representation |
| `close()` | Release tree resources |
#### Incremental Editing
Apply an edit before reparsing changed source code:
```lua
local code = "func main() { x := 1 }"
local tree, parse_err = treesitter.parse("go", code)
if parse_err then
return nil, parse_err
end
-- Mark edit: changed "1" to "100" at byte 19
local _, edit_err = tree:edit({
start_byte = 19,
old_end_byte = 20,
new_end_byte = 22,
start_row = 0,
start_column = 19,
old_end_row = 0,
old_end_column = 20,
new_end_row = 0,
new_end_column = 22
})
if edit_err then
tree:close()
return nil, edit_err
end
-- Re-parse with edited tree (faster than full parse)
local parser, parser_err = treesitter.parser()
if parser_err then
tree:close()
return nil, parser_err
end
local _, language_err = parser:set_language("go")
if language_err then
parser:close()
tree:close()
return nil, language_err
end
local new_tree, new_tree_err = parser:parse("func main() { x := 100 }", tree)
if new_tree_err then
parser:close()
tree:close()
return nil, new_tree_err
end
new_tree:close()
parser:close()
tree:close()
```
### Nodes
Nodes represent elements in the syntax tree. In the isolated snippets below, `root`, `node`, and `func_decl` are application-selected nodes borrowed from a still-open tree.
#### Node Types
```lua
local node = root:child(0)
-- Type information
print(node:kind()) -- "package_clause"
print(node:type()) -- same as kind()
print(node:is_named()) -- true for significant nodes
print(node:grammar_name()) -- grammar rule name
```
#### Navigation
```lua
-- Children
local child = node:child(0) -- by index (0-based)
local named = node:named_child(0) -- named children only
local count = node:child_count()
local named_count = node:named_child_count()
-- Siblings
local next = node:next_sibling()
local prev = node:prev_sibling()
local next_named = node:next_named_sibling()
local prev_named = node:prev_named_sibling()
-- Parent
local parent = node:parent()
-- By field name
local name_node = func_decl:child_by_field_name("name")
local field = node:field_name_for_child(0)
```
#### Position Information
```lua
-- Byte offsets
local start = node:start_byte()
local end_ = node:end_byte()
-- Row/column positions (0-based)
local start_pt = node:start_point() -- {row = 0, column = 0}
local end_pt = node:end_point() -- {row = 0, column = 12}
-- Source text
local source_text, err = node:text()
if err then
return nil, err
end
```
#### Error Detection
```lua
if root:has_error() then
-- Tree contains syntax errors
end
if node:is_error() then
-- This specific node is an error
end
if node:is_missing() then
-- Parser inserted this to recover from error
end
if node:is_extra() then
-- Node is an "extra" (e.g. a comment) not required by the grammar
end
```
Other node methods: `descendant_count()` and `named_descendant_for_point_range(start_pt, end_pt)`.
#### S-Expression
```lua
local sexp = node:to_sexp()
-- "(source_file (package_clause (package_identifier)))"
```
### Queries
Tree-sitter queries match syntax-tree patterns written as S-expressions.
#### Create Query
```lua
local query, err = treesitter.query("go", [[
(function_declaration
name: (identifier) @func_name
parameters: (parameter_list) @params
)
]])
if err then
return nil, err
end
-- The owner calls query:close() after the final query operation.
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `language` | string | Language name |
| `pattern` | string | Query pattern in S-expression syntax |
**Returns:** `Query, error`
#### Execute Query
```lua
-- Get all captures (flattened)
local captures, captures_err = query:captures(root, source_code)
if captures_err then
query:close()
tree:close()
return nil, captures_err
end
for _, capture in ipairs(captures) do
print(capture.name) -- "func_name"
print(capture.text) -- actual text
print(capture.index) -- capture index
-- capture.node is the Node object
end
-- Get matches (grouped by pattern)
local matches, matches_err = query:matches(root, source_code)
if matches_err then
query:close()
tree:close()
return nil, matches_err
end
for _, match in ipairs(matches) do
print(match.id, match.pattern)
for _, capture in ipairs(match.captures) do
local captured_text, text_err = capture.node:text()
if text_err then
query:close()
tree:close()
return nil, text_err
end
print(capture.name, captured_text)
end
end
query:close()
tree:close()
```
Passing userdata of the wrong type instead of a Tree-sitter `Node` returns `nil, error`; passing a primitive or table raises a Lua argument error before that check. `root`, `source_code`, and `query` here must come from a still-open tree and a successfully created query. The snippet uses the owning `tree` handle to close both resources before it returns.
#### Query Control
```lua
-- Limit query scope
query:set_byte_range(0, 1000)
query:set_point_range({row = 0, column = 0}, {row = 10, column = 0})
-- Limit matches
query:set_match_limit(100)
if query:did_exceed_match_limit() then
-- More matches exist
end
-- Timeout (string duration or nanoseconds)
query:set_timeout("500ms")
query:set_timeout(1000000000) -- 1 second in nanoseconds
-- Disable patterns/captures
query:disable_pattern(0)
query:disable_capture("func_name")
```
#### Query Inspection
```lua
local pattern_count = query:pattern_count()
local capture_count = query:capture_count()
local name = query:capture_name_for_id(0)
local id = query:capture_index_for_name("func_name")
```
Further inspection methods: `string_count()`, `start_byte_for_pattern(i)`, `end_byte_for_pattern(i)`, `get_match_limit()`, `get_timeout()`, `is_pattern_rooted(i)`, `is_pattern_non_local(i)`, `is_pattern_guaranteed(i)`, `capture_quantifier(pattern, capture)`, `set_max_start_depth(n)`, `get_property_predicates(i)`, `get_property_settings(i)`, `get_text_predicates(i)`.
### Tree Cursor
A tree cursor traverses a tree without creating a node object at every step.
#### Basic Traversal
```lua
local cursor, err = tree:walk()
if err then
return nil, err
end
-- Start at root
print(cursor:current_node():kind()) -- "source_file"
print(cursor:current_depth()) -- 0
-- Navigate
if cursor:goto_first_child() then
print(cursor:current_node():kind())
print(cursor:current_depth()) -- 1
end
if cursor:goto_next_sibling() then
-- moved to next sibling
end
cursor:goto_parent() -- back to parent
cursor:close()
```
#### Cursor Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `current_node()` | `Node` | Node at cursor position |
| `current_depth()` | `integer` | Depth (0 = root) |
| `current_field_name()` | `string?` | Field name if any |
| `current_field_id()` | `integer` | Field ID (0 if none) |
| `current_descendant_index()` | `integer` | Descendant index of current node |
| `goto_parent()` | `boolean` | Move to parent |
| `goto_first_child()` | `boolean` | Move to first child |
| `goto_last_child()` | `boolean` | Move to last child |
| `goto_next_sibling()` | `boolean` | Move to next sibling |
| `goto_previous_sibling()` | `boolean` | Move to previous sibling |
| `goto_descendant(index)` | - | Move to descendant by index |
| `goto_first_child_for_byte(n)` | `integer?` | Move to child containing byte |
| `goto_first_child_for_point(pt)` | `integer?` | Move to child containing point |
| `reset(node)` | - | Reset cursor to node |
| `reset_to(cursor)` | - | Reset cursor to another cursor's position |
| `copy()` | `Cursor` | Create copy of cursor |
| `close()` | - | Release resources |
### Language Metadata
```lua
local lang, err = treesitter.language("go")
if err then
return nil, err
end
print(lang:version()) -- ABI version
print(lang:node_kind_count()) -- number of node types
print(lang:field_count()) -- number of fields
print(lang:parse_state_count()) -- number of parse states
-- Node kind lookup
local kind = lang:node_kind_for_id(1)
local id = lang:id_for_node_kind("identifier", true)
local is_named = lang:node_kind_is_named(1)
-- Field lookup
local field_name = lang:field_name_for_id(1)
local field_id = lang:field_id_for_name("name")
```
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Language not supported | `errors.INVALID` | no |
| Language has no binding | `errors.INVALID` | no |
| Invalid query pattern | `errors.INVALID` | no |
| Invalid positions | `errors.INVALID` | no |
| Parse failed | `errors.INTERNAL` | no |
| No execution context | `errors.INTERNAL` | no |
Closing an already closed parser, tree, query, or cursor is safe. Calling any other method on a closed handle raises a Lua argument error.
See [Error Handling](lua/core/errors.md) for working with errors.
### Query Syntax Reference
Tree-sitter queries use S-expression patterns:
```
; Match a node type
(identifier)
; Match with field names
(function_declaration name: (identifier))
; Capture with @name
(function_declaration name: (identifier) @func_name)
; Multiple patterns
[
(function_declaration)
(method_declaration)
] @declaration
; Wildcards
(_) ; any node
(identifier)+ ; one or more
(identifier)* ; zero or more
(identifier)? ; optional
; Predicates
((identifier) @var
(#match? @var "^_")) ; regex match
```
See [Tree-sitter Query Syntax](https://tree-sitter.github.io/tree-sitter/using-parsers#query-syntax) for complete documentation.
---
# "Security & Access Control"
## Security & Access Control
The `security` module exposes authentication actors, authorization scopes, policies, and token stores. This page is an API reference with partial authorization recipes. Registry IDs, actors, request metadata, token values, application objects such as `user` and `doc`, and callbacks such as `show_admin_features` come from the surrounding application; the examples are not a complete authentication deployment.
Wippy runs in strict security mode by default. The executable entry must enable `security`, have an actor and scope, and authorize the exact operations it calls. In particular, construction and scope changes need `security.actor.create` or `security.scope.create`; registry lookup needs `security.policy.get` or `security.policy_group.get`; token work needs `security.token_store.get` plus the operation-specific token permission. `new_actor`, `new_scope`, `scope:with`, `scope:without`, and permission-denied `token_store` acquisition raise a Lua error instead of returning a structured `error`. Grant these prerequisites in the entry's security context rather than trying to recover after a denial. See [Security Model](system/security.md) for configuration.
### Loading
```lua
local security = require("security")
```
### `actor`
Return the current security actor from the execution context.
```lua
local actor = security.actor()
if actor then
local id = actor:id()
local meta = actor:meta()
-- Use only the fields required for authorization or application logic.
local role = meta.role
end
```
Actor metadata can contain identifiers or personal data. Do not log the complete metadata table or copy secrets into it.
**Returns:** `Actor|nil`
### `scope`
Return the current security scope from the execution context.
```lua
local scope = security.scope()
if scope then
local policies = scope:policies()
for _, policy in ipairs(policies) do
print("Active policy:", policy:id())
end
end
```
**Returns:** `Scope|nil`
### `can`
Check whether the current context allows an action on a resource.
```lua
-- Check read permission
if not security.can("read", "user:" .. user_id) then
return nil, errors.new("Cannot read user data"):kind(errors.PERMISSION_DENIED)
end
-- Check write permission
if not security.can("write", "order:" .. order_id) then
return nil, errors.new("Cannot modify order"):kind(errors.PERMISSION_DENIED)
end
-- Check with metadata
local allowed = security.can("delete", "document:" .. doc_id, {
owner_id = doc.owner_id,
department = doc.department
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `action` | string | Action to check |
| `resource` | string | Resource identifier |
| `meta` | table | Additional metadata (optional) |
**Returns:** `boolean`
### `new_actor`
Create an actor with an ID and metadata.
```lua
-- Create user actor
local actor = security.new_actor("user:" .. user.id, {
role = user.role,
department = user.department,
email = user.email
})
-- Create service actor
local service_actor = security.new_actor("service:payment-processor", {
type = "service",
version = "1.0.0"
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Unique actor identifier |
| `meta` | table | Metadata key-value pairs |
**Returns:** `Actor`
### `new_scope`
Create a custom scope.
```lua
-- Empty scope
local scope = security.new_scope()
-- Scope with policies
local read_policy, read_err = security.policy("app:read-only")
if read_err then
return nil, read_err
end
local scope = security.new_scope({read_policy})
-- Build scope incrementally
local scope = security.new_scope()
local policy1, policy1_err = security.policy("app:read")
if policy1_err then
return nil, policy1_err
end
local policy2, policy2_err = security.policy("app:write")
if policy2_err then
return nil, policy2_err
end
scope = scope:with(policy1):with(policy2)
```
Each alternative above is an isolated construction pattern. `new_scope` and `scope:with` can raise on missing context or permission denial; they do not return `nil, error` for those checks.
**Returns:** `Scope`
### `policy`
Retrieve a policy from the registry.
```lua
local policy, err = security.policy("app:admin-access")
if err then
return nil, err
end
-- Evaluate policy
local result = policy:evaluate(actor, "delete", "user:123")
if result == "allow" then
-- permitted
elseif result == "deny" then
-- forbidden
else
-- undefined, check other policies
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Policy ID "namespace:name" |
**Returns:** `Policy, error`
### `named_scope`
Retrieve a predefined policy group.
```lua
-- Get admin scope
local admin_scope, err = security.named_scope("app:admin")
if err then
return nil, err
end
-- Use for elevated operations
local result = admin_scope:evaluate(actor, "delete", "user:123")
```
Loading a scope does not elevate the current execution context. It produces a value for explicit evaluation or for an API that accepts a scope; the caller still needs permission to perform the protected operation.
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Policy group ID |
**Returns:** `Scope, error`
### `token_store`
Acquire a token store for managing authentication tokens.
```lua
local store, err = security.token_store("app:tokens")
if err then
return nil, err
end
-- Use store...
return store:close()
```
The caller owns an acquired token store until `close()` is called. Close it after the final operation on every checked success or error path; repeated closes are safe. A permission denial during acquisition raises a Lua error, while lookup and resource failures return `nil, error`.
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Token store ID "namespace:name" |
**Returns:** `TokenStore, error`
### `Actor` Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `actor:id()` | string | Actor identifier |
| `actor:meta()` | table | Actor metadata |
#### `with` / `without`
Add or remove policies from scope.
```lua
local scope = security.new_scope()
-- Add policy
local write_policy, err = security.policy("app:write")
if err then
return nil, err
end
scope = scope:with(write_policy)
-- Remove policy
scope = scope:without("app:read-only")
```
`with` and `without` return new immutable scope values and raise when `security.scope.create` is not allowed for the `with` or `without` resource.
#### `evaluate`
Evaluate all policies in scope.
```lua
local result = scope:evaluate(actor, "read", "document:123")
-- "allow", "deny", or "undefined"
if result ~= "allow" then
return nil, errors.new("Access denied"):kind(errors.PERMISSION_DENIED)
end
```
#### `contains`
Check whether the scope contains a policy.
```lua
if scope:contains("app:admin") then
show_admin_features()
end
```
#### `policies`
Return all policies in the scope.
```lua
local policies = scope:policies()
for _, policy in ipairs(policies) do
print(policy:id())
end
```
**Returns:** `Policy[]`
### `Policy` Methods
| Method | Returns | Description |
|--------|---------|-------------|
| `policy:id()` | string | Policy identifier |
| `policy:evaluate(actor, action, resource, meta?)` | string | `"allow"`, `"deny"`, or `"undefined"` |
#### `create`
Create an authentication token.
```lua
local actor = security.new_actor("user:123", {role = "user"})
local scope, scope_err = security.named_scope("app:default")
if scope_err then
return nil, scope_err
end
local store, store_err = security.token_store("app:tokens")
if store_err then
return nil, store_err
end
local token, err = store:create(actor, scope, {
expiration = "24h", -- or milliseconds
meta = {
login_ip = request_ip,
user_agent = user_agent
}
})
store:close()
if err then
return nil, err
end
return token
```
`request_ip` and `user_agent` are application-provided request values. Store only metadata needed for security decisions, apply retention limits, and never log or persist the returned bearer token outside the intended credential store.
| Parameter | Type | Description |
|-----------|------|-------------|
| `actor` | Actor | Actor for the token |
| `scope` | Scope | Permissions scope |
| `options.expiration` | string/number | Duration string or ms |
| `options.meta` | table | Token metadata |
**Returns:** `string, error`
#### `validate`
Validate a token and return its actor and scope.
```lua
local actor, scope, err = store:validate(token)
store:close()
if err then
return nil, errors.new("Invalid token"):kind(errors.PERMISSION_DENIED)
end
```
Here and below, `store` is a live owned handle and `token` is an untrusted bearer credential supplied by the caller. Do not log the token, including on validation or revocation errors.
**Returns:** `Actor, Scope, error`
#### `revoke`
Invalidate a token.
```lua
local ok, err = store:revoke(token)
store:close()
if err then
return nil, err
end
```
**Returns:** `boolean, error`
#### `close`
Release the token store resource.
```lua
store:close()
```
**Returns:** `boolean`
### Permissions
Security policy evaluation applies to security operations.
#### Security Actions
| Action | Resource | Description |
|--------|----------|-------------|
| `security.policy.get` | Policy ID | Access policy definitions |
| `security.policy_group.get` | Group ID | Access named scopes |
| `security.scope.create` | `custom`, `with`, `without` | Create custom scopes (`new_scope`) and add/remove policies (`scope:with`, `scope:without`) |
| `security.actor.create` | Actor ID | Create actors |
| `security.token_store.get` | Store ID | Access token stores |
| `security.token.validate` | Store ID | Validate tokens |
| `security.token.create` | Store ID | Create tokens |
| `security.token.revoke` | Store ID | Revoke tokens |
See [Security Model](system/security.md) for policy configuration.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| No context | `errors.INTERNAL` | no |
| Empty token store ID | `errors.INVALID` | no |
| Permission denied (`policy`, `named_scope`, token `create`/`validate`/`revoke`) | `errors.INVALID` | no |
| Permission denied (`new_scope`, `new_actor`, `token_store`, `scope:with`/`without`) | raised as a Lua error | no |
| Policy not found | `errors.INTERNAL` | no |
| Token store not found | `errors.INTERNAL` | no |
| Token store closed | `errors.INTERNAL` | no |
| Invalid expiration format | `errors.INVALID` | no |
| Token validation failed | `errors.INTERNAL` | no |
```lua
local store, err = security.token_store("app:tokens")
if err then
if errors.is(err, errors.INVALID) then
print("Invalid request:", err:message())
end
return nil, err
end
store:close()
```
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Security Model](../../system/security.md) - Actors, policies, scopes configuration
- [HTTP Middleware](http/middleware.md) - Endpoint and resource firewall
---
# "Encryption & Signing"
## Encryption & Signing
The `crypto` module generates random values, computes HMACs, encrypts and decrypts data, encodes and verifies JWTs, and derives keys. In deterministic workflows, random generation and encryption (which creates a random nonce) run as recorded side effects; replay returns the recorded bytes. Other operations, including HMAC, decryption, JWT processing, PBKDF2, and comparison, run directly.
This page is an API reference. Each code block is an isolated call, not a complete key-management or authentication system. Names such as `data`, `key`, `aad`, `payload`, and `token` are application-provided values. Load keys and passwords through the application's secret-management boundary; do not hard-code, log, or return them in diagnostics. Before consuming any `value, error` result shown here, propagate or handle the error.
### Loading
```lua
local crypto = require("crypto")
```
#### Random Bytes
```lua
local bytes, err = crypto.random.bytes(32)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `length` | integer | Number of bytes (1 to 1,048,576) |
**Returns:** `string, error`
#### Random String
```lua
local str, err = crypto.random.string(32)
local str, err = crypto.random.string(32, "0123456789abcdef")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `length` | integer | Output length in bytes (1 to 1,048,576) |
| `charset` | string? | ASCII byte alphabet to use (default: alphanumeric) |
**Returns:** `string, error`
The implementation selects bytes from the supplied alphabet. A non-ASCII alphabet can be split into invalid UTF-8, and modulo selection is exactly uniform only when the alphabet's byte length divides 256. For uniformly random secret material, use `crypto.random.bytes` and encode the result for the required transport format.
#### Random UUID
```lua
local id, err = crypto.random.uuid()
```
**Returns:** `string, error`
#### HMAC-SHA256
```lua
local hex, err = crypto.hmac.sha256(key, data)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | HMAC key |
| `data` | string | Data to authenticate |
**Returns:** `string, error`
#### HMAC-SHA512
```lua
local hex, err = crypto.hmac.sha512(key, data)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | string | HMAC key |
| `data` | string | Data to authenticate |
**Returns:** `string, error`
#### AES-GCM {id="encrypt-aes-gcm"}
```lua
local encrypted, err = crypto.encrypt.aes(data, key)
local encrypted, err = crypto.encrypt.aes(data, key, aad)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Plaintext to encrypt |
| `key` | string | 16, 24, or 32 bytes (AES-128/192/256) |
| `aad` | string? | Additional authenticated data |
**Returns:** `string, error` (nonce prepended)
Both encryption functions generate a nonce and prepend it to the ciphertext. Do not remove or reuse it, and use the same AAD during decryption. Ciphertext is not a secret-free log value: it can expose length and correlation information.
#### ChaCha20-Poly1305 {id="encrypt-chacha20"}
```lua
local encrypted, err = crypto.encrypt.chacha20(data, key)
local encrypted, err = crypto.encrypt.chacha20(data, key, aad)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Plaintext to encrypt |
| `key` | string | Must be 32 bytes |
| `aad` | string? | Additional authenticated data |
**Returns:** `string, error` (nonce prepended)
#### AES-GCM {id="decrypt-aes-gcm"}
```lua
local plaintext, err = crypto.decrypt.aes(encrypted, key)
local plaintext, err = crypto.decrypt.aes(encrypted, key, aad)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Encrypted data from encrypt.aes |
| `key` | string | Same key used for encryption |
| `aad` | string? | Must match AAD used in encryption |
**Returns:** `string, error`
#### ChaCha20-Poly1305 {id="decrypt-chacha20"}
```lua
local plaintext, err = crypto.decrypt.chacha20(encrypted, key)
local plaintext, err = crypto.decrypt.chacha20(encrypted, key, aad)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Encrypted data from encrypt.chacha20 |
| `key` | string | Same key used for encryption |
| `aad` | string? | Must match AAD used in encryption |
**Returns:** `string, error`
#### Encode
```lua
local token, err = crypto.jwt.encode(payload, secret)
local token, err = crypto.jwt.encode(payload, secret, "HS256")
local token, err = crypto.jwt.encode(payload, private_key_pem, "RS256")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `payload` | table | JWT claims (`_header` for custom header) |
| `key` | string | Secret (HMAC) or PEM private key (RSA) |
| `alg` | string? | HS256, HS384, HS512, RS256 (default: HS256) |
**Returns:** `string, error`
Pass only one of the documented algorithm names. At this runtime pin, an unsupported value passed to `encode` falls back to HS256 instead of returning an error. Validate any configurable algorithm before this call, and do not copy untrusted fields into `_header`; in particular, do not let input override reserved JWT headers such as `alg`.
#### Verify
```lua
local claims, err = crypto.jwt.verify(token, secret)
local claims, err = crypto.jwt.verify(token, secret, "HS256", false)
local claims, err = crypto.jwt.verify(token, public_key_pem, "RS256")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `token` | string | JWT token to verify |
| `key` | string | Secret (HMAC) or PEM public key (RSA) |
| `alg` | string? | Expected algorithm (default: HS256) |
| `require_exp` | boolean? | Require an `exp` claim to be present (default: true); an `exp` that is present is always validated |
**Returns:** `table, error`
Whenever present, `exp` and `nbf` are validated against the JWT library's current wall clock, not the workflow time reference. Setting `require_exp = false` permits a missing `exp` claim; it does not disable validation of a claim that is present. Do not use either time-dependent result for replay-sensitive workflow control; perform the check in an activity or validate time against an explicitly replay-safe value.
Always pass the algorithm expected by the issuer; verification restricts the token to that exact method. Treat returned claims as authenticated data, not automatically authorized application input, and still validate issuer, audience, subject, and application-specific constraints.
#### PBKDF2
```lua
local key, err = crypto.pbkdf2(password, salt, iterations, key_length)
local key, err = crypto.pbkdf2(password, salt, iterations, key_length, "sha512")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `password` | string | Password/passphrase |
| `salt` | string | Salt value |
| `iterations` | integer | Iteration count (max 10,000,000) |
| `key_length` | integer | Desired key length in bytes |
| `hash` | string? | sha256 or sha512 (default: sha256) |
**Returns:** `string, error`
The derived key is raw bytes. Use a fresh random salt for each stored password verifier and store the salt and work-factor parameters alongside the verifier; the salt need not be secret. Do not use a fixed example salt for production password storage.
#### Constant-Time Compare
```lua
local equal = crypto.constant_time_compare(a, b)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `a` | string | First string |
| `b` | string | Second string |
**Returns:** `boolean`
The result is `false` when lengths differ. The underlying constant-time comparison guarantee applies to equal-length inputs, so compare fixed-length digests or other same-length secrets.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Invalid length | `errors.INVALID` | no |
| Empty key | `errors.INVALID` | no |
| Invalid key size | `errors.INVALID` | no |
| Decryption failed | `errors.INTERNAL` | no |
| Token expired | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Hash Functions"
## Hash Functions
The `hash` module computes cryptographic hashes, HMAC values, PBKDF2-derived keys, and non-cryptographic FNV-1 hashes. This page is an API reference of isolated calls. Literal inputs illustrate successful use; when data, secrets, passwords, or salts come from the application, capture and handle the documented second `error` return before consuming the result.
A hash is not encryption and does not conceal low-entropy input. Do not log passwords, HMAC keys, derived keys, or raw secret-dependent digests. Use HMAC-SHA256 or HMAC-SHA512 for new message-authentication designs and PBKDF2 with a unique random salt for password verifiers.
### Loading
```lua
local hash = require("hash")
```
#### MD5
MD5 is not collision-resistant. Use it only for compatibility with protocols that require MD5, not for security decisions.
```lua
local hex = hash.md5("data")
local raw = hash.md5("data", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to hash |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### SHA-1
SHA-1 is not collision-resistant. Use it only for compatibility with protocols that require SHA-1, not for security decisions.
```lua
local hex = hash.sha1("data")
local raw = hash.sha1("data", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to hash |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### SHA-256
```lua
local hex = hash.sha256("data")
local raw = hash.sha256("data", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to hash |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### SHA-512
```lua
local hex = hash.sha512("data")
local raw = hash.sha512("data", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to hash |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### HMAC-MD5
Use HMAC-MD5 only for compatibility with a protocol that requires it; prefer HMAC-SHA256 or HMAC-SHA512 for new designs.
```lua
local hex = hash.hmac_md5("message", "secret")
local raw = hash.hmac_md5("message", "secret", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Message to authenticate |
| `secret` | string | Secret key |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### HMAC-SHA1
Use HMAC-SHA1 only for compatibility with a protocol that requires it; prefer HMAC-SHA256 or HMAC-SHA512 for new designs.
```lua
local hex = hash.hmac_sha1("message", "secret")
local raw = hash.hmac_sha1("message", "secret", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Message to authenticate |
| `secret` | string | Secret key |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### HMAC-SHA256
```lua
local hex = hash.hmac_sha256("message", "secret")
local raw = hash.hmac_sha256("message", "secret", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Message to authenticate |
| `secret` | string | Secret key |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### HMAC-SHA512
```lua
local hex = hash.hmac_sha512("message", "secret")
local raw = hash.hmac_sha512("message", "secret", true)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Message to authenticate |
| `secret` | string | Secret key |
| `raw` | boolean? | Return raw bytes instead of hex |
**Returns:** `string, error`
#### FNV-1 32-bit
Compute a hash for uses such as hash tables and partitioning.
```lua
local n = hash.fnv32("data")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to hash |
**Returns:** `number, error`
#### FNV-1 64-bit
Compute a wider hash for uses such as hash tables and partitioning, reducing collision probability.
```lua
local n = hash.fnv64("data")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | string | Data to hash |
**Returns:** `number, error`
#### PBKDF2
```lua
local key, err = hash.pbkdf2(password, salt, iterations, key_length)
local key, err = hash.pbkdf2(password, salt, iterations, key_length, "sha512")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `password` | string | Password/passphrase (non-empty) |
| `salt` | string | Salt value (non-empty) |
| `iterations` | integer | Iteration count (1 to 10,000,000) |
| `key_length` | integer | Desired key length in bytes |
| `hash` | string? | `sha256` or `sha512` (default: `sha256`) |
**Returns:** `string, error` (raw key bytes)
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Input not a string | `errors.INVALID` | no |
| Secret not a string (HMAC) | `errors.INVALID` | no |
| Empty password/salt, non-positive or excessive iterations, unsupported hash (PBKDF2) | `errors.INVALID` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "UUID Generation"
## UUID Generation
The `uuid` module generates, validates, inspects, parses, and formats UUIDs. In deterministic workflows, v1, v4, and v7 generation runs as a recorded side effect and returns the recorded value during replay. Namespace-based v3 and v5 generation is deterministic and runs directly.
This page is an API reference of isolated calls. Values such as `namespace`, `name`, `input`, and `id` come from the surrounding application. Capture and handle the second `error` return before consuming generated, parsed, inspected, or formatted results. UUIDs are identifiers, not bearer credentials; do not use any UUID version as an authentication token or secret.
### Loading
```lua
local uuid = require("uuid")
```
#### Version 1
Time-based UUID with timestamp and node ID.
Version 1 exposes its creation time and node identifier. Avoid it where those details are sensitive; prefer v4 when only an opaque identifier is needed.
```lua
local id, err = uuid.v1()
```
**Returns:** `string, error`
#### Version 4
Random UUID.
```lua
local id, err = uuid.v4()
```
**Returns:** `string, error`
#### Version 7
A time-ordered UUID that encodes its creation time for chronological indexing. Do not rely on it as a strictly monotonic sequence, especially for values generated within the same timestamp interval.
```lua
local id, err = uuid.v7()
```
**Returns:** `string, error`
#### Version 3
Deterministic UUID from namespace and name using MD5.
```lua
local id, err = uuid.v3(namespace, name)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `namespace` | string | Valid UUID string |
| `name` | string | Value to hash |
**Returns:** `string, error`
#### Version 5
Deterministic UUID from namespace and name using SHA-1.
```lua
local NS_URL = "6ba7b811-9dad-11d1-80b4-00c04fd430c8"
local id, err = uuid.v5(NS_URL, "https://example.com/resource")
if err then
return nil, err
end
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `namespace` | string | Valid UUID string |
| `name` | string | Value to hash |
**Returns:** `string, error`
#### `validate`
```lua
local valid = uuid.validate(input)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `input` | any | Value to check |
**Returns:** `boolean, nil`. Non-string and malformed inputs return `false`; validation does not raise a structured error.
#### `version`
```lua
local ver, err = uuid.version(id)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `uuid` | string | Valid UUID string |
**Returns:** `integer, error`
#### `variant`
```lua
local var, err = uuid.variant(id)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `uuid` | string | Valid UUID string |
**Returns:** `string, error` (RFC4122, Reserved, Microsoft, Future, NCS, or Invalid)
#### `parse`
```lua
local info, err = uuid.parse(id)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `uuid` | string | Valid UUID string |
**Returns:** `table, error`
Returned table fields:
- `version` (integer): UUID version (1, 3, 4, 5, or 7)
- `variant` (string): RFC4122, Reserved, Microsoft, Future, NCS, or Invalid
- `timestamp` (integer): Unix timestamp (v1 and v7 only)
- `node` (string): 6 raw node ID bytes (v1 only)
#### `format`
```lua
local formatted, err = uuid.format(id, "standard")
local formatted, err = uuid.format(id, "simple")
local formatted, err = uuid.format(id, "urn")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `uuid` | string | Valid UUID string |
| `format` | string? | standard (default), simple, or urn |
**Returns:** `string, error`
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Invalid input type | `errors.INVALID` | no |
| Invalid UUID format | `errors.INVALID` | no |
| Unsupported format type | `errors.INVALID` | no |
| Generation failed | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "Dynamic Evaluation"
## Dynamic Evaluation
Execute code dynamically at runtime with sandboxed environments and controlled module access.
### Two Systems
Wippy provides two evaluation systems:
| System | Purpose | Use Case |
|--------|---------|----------|
| `expr` | Expression evaluation | Config, templates, simple calculations |
| `eval_runner` | Full Lua execution | Plugins, user scripts, dynamic code |
### expr Module
Lightweight expression evaluation using the expr-lang syntax.
```lua
local expr = require("expr")
local result, err = expr.eval("x + y * 2", {x = 10, y = 5})
-- result = 20
```
#### Compiling Expressions
Compile once, run many times:
```lua
local program, err = expr.compile("price * quantity")
local total1 = program:run({price = 10, quantity = 5})
local total2 = program:run({price = 20, quantity = 3})
```
#### Supported Syntax
```lua
-- Arithmetic
expr.eval("1 + 2 * 3") -- 7
expr.eval("10 / 2 - 1") -- 4
expr.eval("10 % 3") -- 1
-- Comparison
expr.eval("x > 5", {x = 10}) -- true
expr.eval("x == y", {x = 1, y = 1}) -- true
-- Boolean
expr.eval("a && b", {a = true, b = false}) -- false
expr.eval("a || b", {a = true, b = false}) -- true
expr.eval("!a", {a = false}) -- true
-- Ternary
expr.eval("x > 0 ? 'positive' : 'negative'", {x = 5})
-- Functions
expr.eval("max(1, 5, 3)") -- 5
expr.eval("min(1, 5, 3)") -- 1
expr.eval("len([1, 2, 3])") -- 3
-- Arrays
expr.eval("[1, 2, 3][0]") -- 1
-- String concatenation
expr.eval("'hello' + ' ' + 'world'")
```
### eval_runner Module
Full Lua execution with security controls.
```lua
local runner = require("eval_runner")
local result, err = runner.run({
source = [[
local function double(x)
return x * 2
end
return { double = double }
]],
method = "double",
args = {21}
})
-- result = 42
```
#### Configuration
| Parameter | Type | Description |
|-----------|------|-------------|
| `source` | string | Lua source code (required) |
| `method` | string | Function to call in returned table |
| `args` | any[] | Arguments passed to function |
| `modules` | string[] | Allowed builtin modules |
| `imports` | table | Registry entries to import |
| `context` | table | Values available as `ctx` |
| `allow_classes` | string[] | Additional module classes |
| `custom_modules` | table | Custom tables as modules |
| `limits` | table | Execution limits for this run |
#### Step Limit
`limits.max_steps` bounds how long one `runner.run` may execute:
```lua
local result, err = runner.run({
source = user_source,
method = "main",
limits = {max_steps = 500}
})
```
A step is one turn of the eval scheduler: the program advances until it yields or finishes, and each resume consumes one step. Pure computation between yields counts as one step no matter how long it runs, so the limit bounds scheduling turns, not CPU time.
When the count exceeds the limit the run stops and returns `errors.INTERNAL` with `eval exceeded maximum step limit`.
`max_steps = 0` means unlimited. Omitting `limits` inherits the host default:
```yaml
## .wippy.yaml
lua:
eval:
max_steps: 10000 # default budget for runs without limits.max_steps
# 0 = unlimited; a negative value fails boot
```
`limits` applies to `runner.run` only; `runner.compile` accepts no limits. `limits` must be a table containing only `max_steps`, and `max_steps` must be a non-negative integer — anything else returns `errors.INVALID` before the program runs.
#### Module Access
Whitelist allowed modules:
```lua
runner.run({
source = [[
local json = require("json")
return json.encode({hello = "world"})
]],
modules = {"json"}
})
```
Modules not in the list cannot be required.
#### Registry Imports
Import entries from the registry:
```lua
runner.run({
source = [[
local data = ...
local utils = require("utils")
return utils.format(data)
]],
imports = {
utils = "app.lib:utilities"
},
args = {{key = "value"}}
})
```
#### Privileged Imports
An import can be granted modules the eval'd code itself cannot see. Use the table form with `id` and `modules`:
```lua
runner.run({
source = [[
local pricing = require("pricing")
return pricing.quote(...)
]],
modules = {"json"},
imports = {
pricing = { id = "app.lib:pricing", modules = {"funcs"} }
},
})
```
The `pricing` library executes in its own scoped environment where `funcs` is available; the eval'd source cannot require or reach `funcs` directly. Granting a module to an import requires the caller to hold `eval.module` permission for that module — capabilities cannot be delegated beyond what the caller itself is allowed.
#### Custom Modules
Inject custom tables:
```lua
runner.run({
source = [[
return sdk.version
]],
custom_modules = {
sdk = {version = "1.0.0", api_key = "xxx"}
}
})
```
#### Context Values
Pass data accessible as `ctx`:
```lua
runner.run({
source = [[
return "Hello, " .. ctx.get("user")
]],
context = {user = "Alice"}
})
```
#### Compiling Programs
`runner.compile` validates source and reports its entrypoint and modules without running it:
```lua
local program, err = runner.compile([[
local function process(x)
return x * 2
end
return { process = process }
]], "process", {modules = {"json"}})
program:method() -- "process" (string)
program:modules() -- {"json"} (string[])
```
The options table accepts the same `modules` and `imports` fields as `runner.run`, and the same `eval.module` and `eval.import` permission checks apply. The compiled program is informational; execute by calling `runner.run` with the source and method.
#### Module Classes
Modules are categorized by capability:
| Class | Description | Default |
|-------|-------------|---------|
| `deterministic` | Pure functions | Allowed |
| `encoding` | Data encoding | Allowed |
| `time` | Time operations | Allowed |
| `nondeterministic` | Random, etc. | Allowed |
| `process` | Spawn, registry | Blocked |
| `storage` | File, database | Blocked |
| `network` | HTTP, sockets | Blocked |
#### Enabling Blocked Classes
```lua
runner.run({
source = [[
local http = require("http_client")
return http.get("https://api.example.com")
]],
modules = {"http_client"},
allow_classes = {"network"}
})
```
#### Permission Checks
The system checks permissions for:
- `eval.compile` - Before compilation
- `eval.run` - Before execution
- `eval.module` - For each module in whitelist, and for each module granted to a privileged import
- `eval.import` - For each registry import
- `eval.class` - For each allowed class
Configure in security policies.
### Compile Cache
Compiled programs are cached in an LRU keyed by source, method, modules, and allowed classes — repeated runs of identical code skip recompilation. Imports and context are bound at run time and do not affect the cache key.
```yaml
## .wippy.yaml
lua:
eval:
cache_size: 256 # entries; 0 or less disables caching (default: 256)
cache_ttl: 0 # expiry; 0 = no expiry (default: 0)
```
### Error Handling
```lua
local result, err = runner.run({...})
if err then
if err:kind() == errors.PERMISSION_DENIED then
-- Access denied by security policy
elseif err:kind() == errors.INVALID then
-- Invalid source or configuration
elseif err:kind() == errors.INTERNAL then
-- Execution or compilation error
end
end
```
#### Plugin System
```lua
local plugins = registry.find({meta = {type = "plugin"}})
for _, plugin in ipairs(plugins) do
local source = plugin:data().source
runner.run({
source = source,
method = "init",
modules = {"json", "time"},
context = {config = app_config}
})
end
```
#### Template Evaluation
```lua
local template = "Hello, {{name}}! You have {{count}} messages."
local compiled = expr.compile("name")
-- Fast repeated evaluation
for _, user in ipairs(users) do
local greeting = compiled:run({name = user.name})
end
```
#### User Scripts
```lua
local user_code = request:body()
local result, err = runner.run({
source = user_code,
modules = {"json", "text"}, -- Safe modules only
context = {data = input_data}
})
```
### See Also
- [Expression](lua/dynamic/expression.md) - Expression language reference
- [Exec](lua/dynamic/exec.md) - System command execution
- [Security](lua/security/security.md) - Security policies
---
# "Command Execution"
## Command Execution
Execute external commands and shell scripts with full control over I/O streams.
For executor configuration, see [Executor](system/exec.md).
### Loading
```lua
local exec = require("exec")
```
### Acquiring an Executor
Get a process executor resource by ID:
```lua
local executor, err = exec.get("app:exec")
if err then
return nil, err
end
-- Use executor
local proc = executor:exec("ls -la")
-- ...
-- Release when done
executor:release()
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `id` | string | Resource ID |
**Returns:** `Executor, error`
### Creating a Process
Create a new process with the specified command:
```lua
-- Simple command
local proc, err = executor:exec("echo 'Hello, World!'")
-- With working directory
local proc = executor:exec("npm install", {
work_dir = "/app/project"
})
-- With environment variables
local proc = executor:exec("python script.py", {
work_dir = "/scripts",
env = {
PYTHONPATH = "/app/lib",
DEBUG = "true",
API_KEY = api_key
}
})
-- Run shell script
local proc = executor:exec("./deploy.sh production", {
work_dir = "/app/scripts",
env = {
DEPLOY_ENV = "production"
}
})
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `cmd` | string | Executable and literal arguments |
| `options.work_dir` | string | Working directory |
| `options.env` | table | Environment variables |
| `options.pty` | table | Allocate a pseudo-terminal for the child |
| `options.process_group` | boolean | Start the child in its own process group so signals also reach descendants; unsupported on Windows |
**Returns:** `Process, error`
The process is created but not started.
#### Command Parsing
`cmd` is split into an executable and literal arguments using shell-like quoting: single and double quotes group a word, and a backslash escapes the following character. There is no shell, so no variable expansion, globbing, pipes, or redirection happens. An unclosed quote returns `errors.INVALID`.
```lua
-- One argument containing a space, passed literally
local proc = executor:exec("grep 'hello world' notes.txt")
-- $HOME is passed as the five characters $HOME, not expanded
local proc = executor:exec("echo $HOME")
```
To use shell features, invoke a shell explicitly:
```lua
local proc = executor:exec("/bin/sh -c 'ls *.log | wc -l'")
```
#### PTY Options
Allocating a PTY gives the child a real terminal: line editing, job control, and full-screen programs work as they do in a shell.
```lua
local proc = executor:exec("/bin/bash --noprofile --norc", {
pty = {width = 100, height = 30, term = "xterm-256color"},
})
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `width` | number | 80 | Initial PTY columns, 1 to 65535 |
| `height` | number | 24 | Initial PTY rows, 1 to 65535 |
| `term` | string | none | Child `TERM` value |
Width times height may not exceed 262,144 cells. A PTY-backed process merges the child's output into a single terminal stream; drive it with [resize](#resize) and [attach_terminal](#attach_terminal) rather than the stdin/stdout pipe methods.
### start / wait
Start the process and wait for completion.
```lua
local proc = executor:exec("./build.sh")
local ok, err = proc:start()
if err then
return nil, err
end
local exit_code, err = proc:wait()
if err then
return nil, err
end
if exit_code ~= 0 then
return nil, errors.new({ kind = errors.INTERNAL, message = "Build failed with exit code: " .. exit_code })
end
```
### stdout_stream / stderr_stream
Get streams to read process output.
```lua
local proc = executor:exec("./process-data.sh")
local stdout = proc:stdout_stream()
local stderr = proc:stderr_stream()
proc:start()
-- Read all stdout
local output = {}
while true do
local chunk = stdout:read(4096)
if not chunk then break end
table.insert(output, chunk)
end
local result = table.concat(output)
-- Check for errors
local err_output = {}
while true do
local chunk = stderr:read(4096)
if not chunk then break end
table.insert(err_output, chunk)
end
local exit_code = proc:wait()
stdout:close()
stderr:close()
if exit_code ~= 0 then
return nil, errors.new({ kind = errors.INTERNAL, message = table.concat(err_output) })
end
return result
```
### write_stdin
Write data to process stdin.
```lua
local proc = executor:exec("head -n 3")
local stdout = proc:stdout_stream()
proc:start()
proc:write_stdin("banana\napple\ncherry\n")
local lines = stdout:read()
proc:wait()
stdout:close()
```
Each call writes the given bytes and returns. Call `close_stdin()` when the child
must see EOF:
```lua
local proc = assert(executor:exec("sort"))
local stdout = assert(proc:stdout_stream())
assert(proc:start())
assert(proc:write_stdin("banana\napple\n"))
assert(proc:close_stdin())
local sorted = assert(stdout:read())
```
`close_stdin()` is idempotent. Later writes fail because the input side is
closed. PTY-backed processes do not expose this pipe operation.
### done
Use `done()` to observe exit without consuming the process handle:
```lua
local proc = assert(executor:exec("./worker"))
assert(proc:start())
local exits = assert(proc:done())
local status, open = exits:receive()
if open then
print(status.code, status.signal, status.error)
end
```
The returned channel delivers one exit record and then closes. Repeated calls
return the same channel. The record contains `code`, optional `signal`, and an
`error` only when the runtime could not observe the exit. A signal exit uses
`128 + signal` as its code. Unlike `wait()`, `done()` leaves the handle usable,
so streams, `signal()`, and `close()` remain available. `wait()` after delivery
returns the recorded code.
### signal / close
Send signals or release the process.
```lua
local proc = executor:exec("./long-running-server.sh")
proc:start()
-- ... later, need to stop it ...
-- Send SIGTERM and release the handle
proc:close()
-- Send SIGKILL and release the handle
proc:close(true)
-- Or send a specific signal and keep the handle
local SIGINT = 2
proc:signal(SIGINT)
```
`close(force?)` signals a started child with `SIGTERM`, or `SIGKILL` when `force` is true, then reaps it in the background so the call does not block. A child still running after a grace period is killed so the reap always completes. An unstarted handle is simply invalidated, and closing twice is not an error. When `process_group` is enabled, signals target the group and still reach descendants after the leader exits.
Streams acquired before reaping remain readable until their last writer closes,
including a descendant that inherited the pipe. After `close()`, process methods
report `process closed`; use `done()` when the exit matters and the handle must
remain usable.
### resize
Resize the PTY of a PTY-backed process. A pipe-backed process returns an error.
```lua
local ok, err = proc:resize(120, 40)
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `width` | number | Columns, 1 to 65535 |
| `height` | number | Rows, 1 to 65535 |
**Returns:** `boolean, error`
Use it to set the initial geometry before handing the process to a terminal session. Once a session owns the process, send it a `resize` event instead.
### attach_terminal
Attach an unstarted PTY-backed process to the calling process's terminal and return a `TerminalSession`.
```lua
local exec = require("exec")
local tty = require("tty")
local executor = assert(exec.get("app:exec"))
local proc = assert(executor:exec("/bin/bash --noprofile --norc", {
pty = {term = "xterm-256color"},
}))
local session = assert(proc:attach_terminal())
```
**Returns:** `TerminalSession, error`
The call consumes the process: the session becomes its sole lifecycle owner and the original handle can no longer be used. The session opens a surface on the current terminal port and owns PTY emulation, input encoding, resize, graceful and forced termination, and reaping. It needs a terminal port — a [terminal host](system/terminal.md) process, or a process spawned with a [viewport grant](lua/system/tty.md#viewport) — and fails when the port has no input controller or already has an open surface.
#### TerminalSession
| Method | Returns | Description |
|--------|---------|-------------|
| `send(event)` | `boolean, error` | Forward one canonical TTY event to the child |
| `done()` | channel | Channel that fires once when the child finishes |
| `status()` | `string, error` | `"running"` or `"done"`, with the failure error when it failed |
| `close()` | `boolean, error` | Request termination of a running child |
`send` accepts the key, mouse, resize, focus, and paste records described in [TTY](lua/system/tty.md#event-types). Sending after the child has finished returns an error.
```lua
local channel = require("channel")
local events = assert(tty.events())
assert(tty.start())
local done = session:done()
while true do
local selected = channel.select({
events:case_receive(),
done:case_receive(),
})
if not selected.ok or selected.channel == done then break end
if selected.value.type == "close" then break end
assert(session:send(selected.value))
end
assert(session:close())
```
### Permissions
Exec operations are subject to security policy evaluation.
| Action | Resource | Description |
|--------|----------|-------------|
| `exec.get` | Executor ID | Acquire an executor resource |
| `exec.run` | Command | Execute a specific command |
`exec.run` is evaluated against the raw command string, with the requested options as metadata:
| Key | Type | Description |
|-----|------|-------------|
| `work_dir` | string | Requested working directory, empty when unset |
| `env_names` | string[] | Names of the environment variables passed, sorted; values are not exposed |
| `pty.requested` | boolean | Whether a PTY was requested |
| `pty.width` | number | Resolved PTY columns, present when requested |
| `pty.height` | number | Resolved PTY rows, present when requested |
| `pty.term` | string | Requested `TERM` value, present when requested |
A policy can therefore allow plain commands while restricting the ones that ask for a terminal or a particular working directory.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Invalid ID | `errors.INVALID` | no |
| Permission denied | `errors.INVALID` | no |
| Process closed | `errors.INVALID` | no |
| Process not started | `errors.INVALID` | no |
| Already started | `errors.INVALID` | no |
| Unclosed quote in command | `errors.INVALID` | no |
| No PTY on the process | `errors.INVALID` | no |
| Terminal port unavailable | `errors.UNAVAILABLE` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
### See Also
- [Executor](system/exec.md) — executor configuration
- [TTY](lua/system/tty.md) — terminal events, surfaces, and viewports
- [Terminal UI](tutorials/tty.md) — a shell that hosts a PTY child in a viewport
---
# "Expression Language"
## Expression Language
The `expr` module compiles and evaluates [expr-lang](https://expr-lang.org/) expressions for filtering, validation, calculations, and rule evaluation without running Lua source code. This page is the canonical Lua API reference; its examples run inside an existing Wippy Lua process whose entry declares the `expr` module, but they are not standalone Wippy applications. See [Dynamic Evaluation](./eval.md) when choosing between expressions and capability-restricted Lua.
### Loading
```lua
local expr = require("expr")
```
### Caching
`expr.eval` keeps an internal LRU cache of compiled expressions (default capacity 1000). The cache is built into the module and requires no configuration.
### Evaluating Expressions
Evaluate an expression string and return its result. The function uses the internal cache of compiled expressions:
```lua
-- Simple math
local result, err = expr.eval("1 + 2 * 3")
if err then
return nil, err
end
-- result == 7
-- With variables
local total, total_err = expr.eval("price * quantity", {
price = 29.99,
quantity = 3
})
if total_err then
return nil, total_err
end
-- total == 89.97
-- Ternary operator
local label, label_err = expr.eval('score > 90 ? "A" : score > 80 ? "B" : "C"', {
score = 85
})
if label_err then
return nil, label_err
end
-- label == "B"
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `expression` | string | expr-lang syntax expression |
| `env` | `any` | Variable environment for the expression (optional; normally a table) |
**Returns:** `any, error`
### Compiling Expressions
Compile an expression into a reusable `Program` for repeated evaluation:
```lua
-- Compile once for repeated use
local discount_calc, err = expr.compile("price * (1 - discount_rate)")
if err then
return nil, err
end
-- Reuse with different inputs
local price1, run_err = discount_calc:run({price = 100, discount_rate = 0.1})
if run_err then
return nil, run_err
end
local price2, second_run_err = discount_calc:run({price = 50, discount_rate = 0.2})
if second_run_err then
return nil, second_run_err
end
-- price1 == 90 and price2 == 40
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `expression` | string | expr-lang syntax expression |
| `env` | `any` | Type-hint environment for compilation (optional; normally a table) |
**Returns:** `Program, error`
### Running Compiled Programs
Run a compiled expression with the provided environment:
```lua
-- Validation rule
local validator, compile_err = expr.compile("len(password) >= 8 and len(password) <= 128")
if compile_err then
return nil, compile_err
end
local valid, run_err = validator:run({password = "securepass123"})
if run_err then
return nil, run_err
end
-- valid == true
-- Pricing rule
local pricer, pricing_compile_err = expr.compile([[
base_price * quantity * (1 - bulk_discount) + shipping
]])
if pricing_compile_err then
return nil, pricing_compile_err
end
local order_total, pricing_run_err = pricer:run({
base_price = 25.00,
quantity = 10,
bulk_discount = 0.15,
shipping = 12.50
})
if pricing_run_err then
return nil, pricing_run_err
end
-- order_total == 225.00
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `env` | `any` | Variable environment for the expression (optional; normally a table) |
**Returns:** `any, error`
### Built-in Functions
Expr-lang includes built-in functions for common operations:
```lua
local maximum, max_err = expr.eval("max(1, 5, 3)")
if max_err then
return nil, max_err
end
local uppercase, upper_err = expr.eval('upper("hello")')
if upper_err then
return nil, upper_err
end
local total, sum_err = expr.eval("sum(values)", {values = {1, 2, 3, 4}})
if sum_err then
return nil, sum_err
end
-- maximum == 5, uppercase == "HELLO", and total == 10
```
Other built-ins include `min`, `abs`, `ceil`, `floor`, `len`, `lower`, and
`trim`. Expr-lang also provides operators such as `contains` for strings and
`in` for membership tests.
### Errors
| Condition | Kind | Retryable |
|-----------|------|-----------|
| Expression is empty | `errors.INVALID` | no |
| Expression syntax invalid | `errors.INTERNAL` | no |
| Expression evaluation fails | `errors.INTERNAL` | no |
| Result conversion fails | `errors.INTERNAL` | no |
See [Error Handling](lua/core/errors.md) for working with errors.
---
# "WebAssembly Runtime"
## WebAssembly Runtime
> The WASM runtime is an experimental extension. Configuration is stable, but runtime internals may change between releases.
Wippy registers WebAssembly modules alongside Lua code. Function entries join
the function registry and run through function pools. Process entries register
factories for persistent WASM actors: each PID owns one isolated module
instance and bounded mailbox until it exits. Both use the runtime scheduler and
security model.
**Classification: conceptual overview.** The Lua block contains independent call
patterns and assumes the named WASM entries and their WIT contracts are already
registered. See the Rust/WASM tutorial for a project with a compiled component.
### Entry Kinds
| Kind | Description |
|------|-------------|
| `function.wat` | Inline WebAssembly Text format function defined in YAML |
| `function.wasm` | Precompiled WASM binary loaded from a filesystem entry |
| `process.wasm` | Stateful WASM actor with one module instance per PID |
### How It Works
1. WASM modules are declared as registry entries in `_index.yaml`
2. At boot, `function.wat` and `function.wasm` entries are compiled, registered as functions, and placed into their configured function pools
3. Lua calls those function entries through `funcs.call()`
4. `process.wasm` entries register actor factories and are spawned under a process host; each PID keeps its module state across messages
5. Function arguments and return values are mapped between Lua tables and WIT types
6. Supported dispatcher-bridged operations, including clock polling and outgoing HTTP, yield so the scheduler can run other work
### Component Model
Wippy supports the WebAssembly Component Model with WIT (WebAssembly Interface Types). Component modules map these types between the host and guest:
- Records map to Lua tables with named fields
- Lists map to Lua arrays
- Results map to `(value, error)` return tuples
- Primitives (`s32`, `f64`, `string`, etc.) map directly
Raw/core WASM modules are also supported with explicit WIT signatures.
### Calling WASM from Lua
Call a WASM function by its registry ID through `funcs.call()`:
```lua
local funcs = require("funcs")
-- No arguments
local result, err = funcs.call("myns:answer_wat")
if err then return nil, err end
-- With arguments
local computed, compute_err = funcs.call("myns:compute", 6, 7)
if compute_err then return nil, compute_err end
-- With complex data
local users = {
{id = 1, name = "Alice", tags = {"admin"}, active = true},
{id = 2, name = "Bob", tags = {"user"}, active = false},
}
local transformed, err = funcs.call("myns:transform_users", users)
if err then return nil, err end
```
### Security
WASM executions inherit the caller's security context by default:
- Actor identity is inherited
- Scope is inherited
- Request context is inherited
Host capabilities are opt-in through explicit imports. Each entry declares the host profiles it needs, such as `funcs`, `wippy:actor`, `wasi1`, `wasi:cli`, or `wasi:filesystem`, limiting the module's access surface. Enabling a profile does not bypass runtime security checks on operations such as process messaging, function calls, sockets, or outgoing HTTP.
A guest that imports `funcs` can call back into the registry. Each call is policy-checked as `funcs.call` against the target ID, so the reachable set is exactly what the inherited scope already permits. Socket dials are authorized the same way, by the [network service](system/network.md), against the `socket.*` permissions.
### See Also
- [Functions](wasm/functions.md) - WASM function entry configuration
- [Host Functions](wasm/hosts.md) - Available WASI and Wippy host interfaces
- [Processes](wasm/processes.md) - Running WASM as long-lived processes
- [Rust/WASM Tutorial](../tutorials/rust-wasm.md) - Build and register a component
---
# "WASM Functions"
## WASM Functions
Use `function.wat` for inline WebAssembly Text source and `function.wasm` for precompiled binaries.
**Classification: function configuration reference.** WAT blocks are small
registry examples. Precompiled examples assume an external component build, a
filesystem entry, exported methods matching the guest WIT, and a SHA-256 digest
calculated from the exact binary. Real-looking sample hashes are illustrative.
### Inline WAT Functions
Define a WAT function directly in `_index.yaml`:
```yaml
entries:
- name: answer
kind: function.wat
source: |
(module
(func (export "answer") (result i32)
i32.const 42
)
)
wit: |
answer: func() -> s32;
method: answer
pool:
type: inline
```
For larger WAT sources, use a file reference:
```yaml
- name: answer
kind: function.wat
source: file://answer.wat
wit: |
answer: func() -> s32;
method: answer
pool:
type: inline
```
#### WAT Configuration Fields
| Field | Required | Description |
|-------|----------|-------------|
| `source` | Yes | Inline WAT source or `file://` reference |
| `method` | Yes | Exported function name to call |
| `wit` | No | WIT signature for raw/core modules |
| `pool` | No | Worker pool configuration |
| `transport` | No | Input/output mapping (default: `payload`) |
| `imports` | No | Host imports to enable (e.g., `wasi:cli`, `wasi:io`) |
| `wasi` | No | WASI configuration (args, env, mounts) |
| `options.limits` | No | Execution limits (`limits` remains a deprecated compatibility spelling) |
### Precompiled WASM Functions
Load compiled `.wasm` binaries from a filesystem entry:
```yaml
entries:
- name: assets
kind: fs.directory
directory: ./wasm
- name: compute
kind: function.wasm
fs: myns:assets
path: /compute.wasm
hash: sha256:292b796376f8b4cc360acf2ea6b82d1084871c3607a079f30b446da8e5c984a4
method: compute
pool:
type: lazy
max_size: 4
```
#### WASM Configuration Fields
| Field | Required | Description |
|-------|----------|-------------|
| `fs` | Yes | Filesystem entry ID containing the binary |
| `path` | Yes | Path to `.wasm` file within the filesystem |
| `hash` | Yes | SHA-256 hash for integrity verification (`sha256:...`) |
| `method` | Yes | Exported function name to call |
| `wit` | No | WIT signature for raw/core modules |
| `pool` | No | Worker pool configuration |
| `transport` | No | Input/output mapping (default: `payload`) |
| `imports` | No | Host imports to enable |
| `wasi` | No | WASI configuration |
| `options.limits` | No | Execution limits (`limits` remains a deprecated compatibility spelling) |
### Worker Pools
Each WASM function uses a pool of pre-compiled instances. The pool type controls concurrency and resource usage.
| Type | Description |
|------|-------------|
| `inline` | Mutex-serialized. Synchronous and asyncified calls reuse one warm instance; retained-memory policy or an execution failure can trigger replacement. |
| `lazy` | Zero idle workers. Scales on demand up to `max_size`. |
| `static` | Fixed number of workers with request queue. |
| `adaptive` | Auto-scaling elastic pool. |
#### Pool Configuration
```yaml
pool:
type: static
size: 4 # Total pool size
workers: 2 # Worker threads
buffer: 16 # Request queue buffer (default: workers * 64)
```
```yaml
pool:
type: lazy
max_size: 8 # Maximum concurrent instances
```
```yaml
pool:
type: adaptive
max_size: 16 # Upper scaling bound
```
The 100-worker default applies only to the implicitly selected pool (when no `type` is set). When you explicitly set `type: lazy` or `type: adaptive` without `max_size`, the default maximum is 16 workers.
#### Worker Classes and Core Affinity
Setting `pool.worker_class` routes the function to a dedicated pool of OS-thread-pinned workers instead of the shared pool types above (`type` is ignored when set; conventional name: `wasm`):
```yaml
pool:
worker_class: wasm
workers: 8 # optional; defaults to reserved cores, else min(NumCPU, 4)
```
Core isolation is opted into per runtime in `.wippy.yaml`:
```yaml
scheduler:
wasm_isolation:
enabled: true # default: false
reserved_cores: 2 # cores reserved for WASM pools (default: 1)
```
With isolation enabled, the actor scheduler and the pinned WASM pools run on disjoint CPU sets (`sched_setaffinity`, Linux only — other platforms size the pools but do not bind threads). Long-running WASM calls then cannot starve actor scheduling.
### Transports
Transports control how input and output are mapped between the runtime and the WASM module.
| Transport | Description |
|-----------|-------------|
| `payload` | Maps runtime payloads directly to WASM call arguments (default) |
| `wasi-http` | Maps HTTP request/response context to WASM arguments and results |
#### Payload Transport
The default transport passes arguments directly. Lua values are transcoded to Go types, then lowered to WIT types:
```yaml
- name: compute
kind: function.wasm
fs: myns:assets
path: /compute.wasm
hash: sha256:...
method: compute
pool:
type: inline
```
```lua
-- Arguments passed directly as WASM function parameters
local result, err = funcs.call("myns:compute", 6, 7)
if err then return nil, err end
-- result: 42
```
#### WASI HTTP Transport
The `wasi-http` transport maps HTTP requests to WASM and writes results back to the HTTP response. Use this to expose WASM functions as HTTP endpoints:
```yaml
- name: greet_wasm
kind: function.wasm
fs: myns:assets
path: /greet.wasm
hash: sha256:...
method: greet
transport: wasi-http
pool:
type: inline
- name: greet_endpoint
kind: http.endpoint
meta:
router: myns:api
method: POST
path: /api/greet
func: greet_wasm
```
### Execution Limits
The `options.limits` block bounds a function's execution time, its warm-worker memory, and the sockets it may open:
```yaml
options:
limits:
max_execution_ms: 5000
max_retained_memory_bytes: 134217728
retained_memory_check_interval: 32
max_open_sockets: 8
socket_timeout_ms: 5000
```
| Field | Default | Description |
|-------|---------|-------------|
| `max_execution_ms` | unlimited | Wall-clock budget for one call. When exceeded, the execution is cancelled and an error is returned. |
| `max_retained_memory_bytes` | `67108864` (64 MiB) | Post-call recycling trigger. A warm worker whose linear memory exceeds this is retired after the call instead of being reused. An explicit `0` disables retained-memory recycling. |
| `retained_memory_check_interval` | `16` with the built-in limit, every call with an explicit limit | Number of calls between post-call memory inspections. |
| `max_open_sockets` | `16` | Concurrently open connections per instance for the `socket` host. |
| `socket_timeout_ms` | `30000` | Deadline for a `socket` dial and for each send/receive. |
Negative values are rejected at boot.
The root `limits` and `meta.options.limits` spellings are accepted temporarily
with a deprecation warning. Keep `pool` at the entry root; it has not moved
under `options`.
### WASI Configuration
Configure WASI capabilities for the guest module:
```yaml
wasi:
args: ["--verbose"]
cwd: "/app"
env:
- id: myns:api_key
name: API_KEY
required: true
- id: myns:debug_mode
name: DEBUG
mounts:
- fs: myns:data_files
guest: /data
read_only: true
- fs: myns:output
guest: /output
```
| Field | Description |
|-------|-------------|
| `args` | Command-line arguments passed to the guest |
| `cwd` | Working directory inside the guest (must be absolute) |
| `env` | Environment variables mapped from registry env entries |
| `mounts` | Filesystem mounts from registry filesystem entries |
Environment variables are resolved from the environment registry at call time. Required variables cause an error if not found.
Mount paths must be absolute and unique. Each mount maps a runtime filesystem entry to a guest directory path.
#### Data Transformation Pipeline
```yaml
entries:
- name: wasm_binaries
kind: fs.directory
directory: ./wasm
- name: transform_users
kind: function.wasm
fs: myns:wasm_binaries
path: /mapper.wasm
hash: sha256:7304fc7d19778605458ae5804dae9a7343dcd3f5fc22bcc9415e98b5047192dd
method: transform-users
pool:
type: lazy
max_size: 4
- name: filter_active
kind: function.wasm
fs: myns:wasm_binaries
path: /mapper.wasm
hash: sha256:7304fc7d19778605458ae5804dae9a7343dcd3f5fc22bcc9415e98b5047192dd
method: filter-active
pool:
type: lazy
max_size: 4
```
```lua
local funcs = require("funcs")
local users = {
{id = 1, name = "Alice", tags = {"admin", "dev"}, active = true},
{id = 2, name = "Bob", tags = {"user"}, active = false},
{id = 3, name = "Carol", tags = {"dev"}, active = true},
}
-- Transform: adds display field and tag count
local transformed, err = funcs.call("myns:transform_users", users)
if err then return nil, err end
-- Filter: returns only active users
local active, filter_err = funcs.call("myns:filter_active", users)
if filter_err then return nil, filter_err end
```
#### Async Sleep with WASI Clocks
WASM components that import `wasi:clocks`, `wasi:io` and `wasi:poll` can use clocks and polling. The async yield mechanism integrates with the Wippy dispatcher:
```yaml
- name: sleep_ms
kind: function.wasm
fs: myns:wasm_binaries
path: /sleep_test.wasm
hash: sha256:...
method: "test-sleep#sleep-ms"
imports:
- wasi:io
- wasi:poll
- wasi:clocks
pool:
type: inline
```
The `#` separator in the method field references an interface method: `test-sleep#sleep-ms` calls the `sleep-ms` function from the `test-sleep` interface.
### See Also
- [Overview](wasm/overview.md) - WebAssembly runtime overview
- [Host Functions](wasm/hosts.md) - Available host interfaces
- [Processes](wasm/processes.md) - Running WASM as processes
- [Entry Kinds](guides/entry-kinds.md) - All registry entry kinds
---
# "Host Functions"
## Host Functions
WASM modules access runtime capabilities through host function imports. Each import is declared explicitly per entry in the `imports` list.
### Import Types
| Import | Namespace | Module kind | Description |
|--------|-----------|-------------|-------------|
| `wasi:cli` | `wasi:cli/*` | component | Environment, exit, stdin/stdout/stderr, terminal |
| `wasi:io` | `wasi:io/error`, `wasi:io/streams` | component | Streams and error handling |
| `wasi:poll` | `wasi:io/poll` | component | Async polling / cooperative yielding |
| `wasi:clocks` | `wasi:clocks/*` | component | Wall clock and monotonic clock |
| `wasi:filesystem` | `wasi:filesystem/*` | component | File system access through mounted directories |
| `wasi:random` | `wasi:random/*` | component | Cryptographically secure and insecure random numbers |
| `wasi:sockets` | `wasi:sockets/*` | component | TCP/UDP networking and DNS resolution |
| `wasi:http` | `wasi:http/*` | component | Outgoing HTTP client requests |
| `funcs` | `wippy:runtime/funcs@0.1.0` | component | Calling registry functions from the guest |
| `wippy:actor` | `wippy:actor/process@0.1.0` | component | PID identity and bounded actor mailbox messaging |
| `wasi1` | `wasi_snapshot_preview1` | core | WASI Preview 1 compatibility imports |
| `socket` | `wippy:runtime/socket@0.1.0` | core | Instance-owned outbound TCP through integer-only imports |
The eight `wasi:*` profiles, `funcs`, and `wippy:actor` are component-only:
declaring one on a core module fails the entry. `wasi1` and `socket` expose core
imports.
Each profile resolves under its short name, under any of the interface namespaces it provides, and under a versioned namespace. The version suffix is stripped before lookup, so `wasi:io/poll`, `wasi:io/poll@0.2.3` and `wasi:poll` all select the same profile.
An import that resolves to no profile fails the entry with `unsupported wasm host import: `; a component-only profile on a core module fails with `wasm host import requires component module: `.
Enable imports in your entry configuration:
```yaml
- name: my_function
kind: function.wasm
fs: myns:assets
path: /module.wasm
hash: sha256:...
method: run
imports:
- wasi:cli
- wasi:io
- wasi:clocks
- wasi:filesystem
pool:
type: inline
```
Only declare the imports your module actually needs.
### WASI Imports
Each `wasi:*` import enables a group of related WASI Preview 2 interfaces.
#### wasi:clocks
**Interfaces:** `wasi:clocks/wall-clock`, `wasi:clocks/monotonic-clock`
Wall clock and monotonic clock for time operations. Monotonic clock integrates with the Wippy dispatcher for async sleep.
#### wasi:io
**Interfaces:** `wasi:io/error`, `wasi:io/streams`
Stream read/write operations and error handling. The `wasi:io/poll` interface is provided separately by the `wasi:poll` import.
#### wasi:poll
**Interfaces:** `wasi:io/poll`
Async polling. The poll interface enables cooperative yielding through the dispatcher.
#### wasi:cli
**Interfaces:** `wasi:cli/environment`, `wasi:cli/exit`, `wasi:cli/stdin`, `wasi:cli/stdout`, `wasi:cli/stderr`, `wasi:cli/terminal-stdin`, `wasi:cli/terminal-stdout`, `wasi:cli/terminal-stderr`
Access to environment variables, process exit codes, and standard I/O streams. Environment variables are mapped from the Wippy environment registry through WASI configuration.
#### wasi:filesystem
**Interfaces:** `wasi:filesystem/types`, `wasi:filesystem/preopens`
File system access through mounted directories. Mounts are configured per-entry and map Wippy filesystem entries to guest paths.
```yaml
wasi:
mounts:
- fs: myns:data
guest: /data
read_only: true
```
#### wasi:random
**Interfaces:** `wasi:random/random`, `wasi:random/insecure`, `wasi:random/insecure-seed`
Cryptographically secure and insecure random number generation.
#### wasi:sockets
**Interfaces:** `wasi:sockets/instance-network`, `wasi:sockets/ip-name-lookup`, `wasi:sockets/tcp`, `wasi:sockets/tcp-create-socket`, `wasi:sockets/udp`, `wasi:sockets/udp-create-socket`
TCP and UDP networking with DNS resolution. Socket operations suspend the guest and run through the dispatcher, which performs every dial, bind and lookup on the [network service](system/network.md).
#### wasi:http
**Interfaces:** `wasi:http/types`, `wasi:http/outgoing-handler`
Outgoing HTTP client requests from within WASM modules. Supports request/response types defined by the WASI HTTP specification.
### funcs
**Namespace:** `wippy:runtime/funcs@0.1.0`
Calls registry functions from a component guest. Two entry points are exposed:
```wit
interface funcs {
call-string: func(target: string, input: string) -> result;
call-bytes: func(target: string, input: list) -> result, string>;
}
```
`target` is a registry ID in `namespace:name` form. Every call is policy-checked as `funcs.call` against that target, so a guest can only reach functions the caller's scope already permits.
### wippy:actor
**Namespace:** `wippy:actor/process@0.1.0`
Provides messaging for a component running as `process.wasm`:
```wit
interface process {
use wasi:io/poll@0.2.8.{pollable};
type pid = string;
record payload { format: string, data: list }
record message { %from: pid, topic: string, payloads: list }
self: func() -> pid;
send: func(target: pid, topic: string, payloads: list)
-> result;
try-receive: func() -> result