Cloud Storage

Access S3-compatible object storage. Upload, download, list, and manage objects, presign download, upload and multipart-part URLs, and read objects with random access.

For storage configuration, see Cloud Storage.

Loading

local cloudstorage = require("cloudstorage")

Acquiring Storage

Acquire a cloud storage resource by registry ID:

local storage, err = cloudstorage.get("app.infra:files")
if err then
    return nil, err
end

local uploaded, upload_err = storage:upload_object("data/file.txt", "content")
storage:release()
if upload_err then return nil, upload_err end
return uploaded

Returns: Storage, error

Uploading Objects

Upload content from a string or file:

local json = require("json")

local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end

-- Upload string content
local body, encode_err = json.encode({
    date = "2024-01-15",
    total = 1234
})
if encode_err then
    storage:release()
    return nil, encode_err
end
local ok, err = storage:upload_object("reports/daily.json", body)
if err then
    storage:release()
    return nil, err
end

-- Upload from file
local fs = require("fs")
local vol, fs_err = fs.get("app:data")
if fs_err then
    storage:release()
    return nil, fs_err
end
local file, open_err = vol:open("/large-file.bin", "r")
if open_err then
    storage:release()
    return nil, open_err
end

local uploaded, file_upload_err = storage:upload_object("backups/large-file.bin", file)
local _, close_err = file:close()

storage:release()
if file_upload_err then
    if close_err then report_cleanup_error(close_err) end
    return nil, file_upload_err
end
if close_err then return nil, close_err end
return uploaded

Returns: boolean, error

Upload Options

Attach metadata or guard the write with an options table:

local uploaded, err = storage:upload_object("reports/daily.json", body, {
    content_type = "application/json",
    cache_control = "max-age=3600",
    metadata = { owner = "team-a", run_id = "1234" },  -- stored as x-amz-meta-*
    only_if_absent = true                              -- fail if the key already exists
})
if err then return nil, err end
return uploaded

A conditional write that fails its precondition returns a precondition_failed error.

Downloading Objects

Download an object to a file writer:

local fs = require("fs")
local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end
local vol, fs_err = fs.get("app:temp")
if fs_err then
    storage:release()
    return nil, fs_err
end

local file, open_err = vol:open("/downloaded.json", "w")
if open_err then
    storage:release()
    return nil, open_err
end
local ok, err = storage:download_object("reports/daily.json", file)
local _, close_err = file:close()
if err then
    if close_err then report_cleanup_error(close_err) end
    storage:release()
    return nil, err
end
if close_err then
    storage:release()
    return nil, close_err
end

-- Download partial content (first 1KB)
local partial, partial_open_err = vol:open("/partial.bin", "w")
if partial_open_err then
    storage:release()
    return nil, partial_open_err
end
local partial_ok, partial_err = storage:download_object("backups/large-file.bin", partial, {
    range = "bytes=0-1023"
})
local _, partial_close_err = partial:close()

storage:release()
if partial_err then
    if partial_close_err then report_cleanup_error(partial_close_err) end
    return nil, partial_err
end
if partial_close_err then return nil, partial_close_err end
return partial_ok

Returns: boolean, error

A failed precondition (if_match/if_none_match) returns a precondition_failed error.

Listing Objects

List objects with optional prefix filtering:

local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end

local result, err = storage:list_objects({
    prefix = "reports/2024/",
    max_keys = 100
})
if err then
    storage:release()
    return nil, err
end

for _, obj in ipairs(result.objects) do
    print(obj.key, obj.size, obj.etag)
end

-- Paginate through large results
local token = nil
repeat
    local page, page_err = storage:list_objects({
        prefix = "logs/",
        max_keys = 1000,
        continuation_token = token
    })
    if page_err then
        storage:release()
        return nil, page_err
    end
    for _, obj in ipairs(page.objects) do
        process(obj)
    end
    token = page.next_continuation_token
    if not page.is_truncated then break end
until false

storage:release()

Returns: table, error

Result contains objects, is_truncated, next_continuation_token. Each object has key, size, etag, storage_class, and optional last_modified, version_id, and owner.

Object Metadata

Fetch a single object's metadata without downloading its body:

local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end

local meta, err = storage:head_object("reports/daily.json")
if err then
    storage:release()
    return nil, err
end

print(meta.size, meta.etag, meta.content_type)
for k, v in pairs(meta.metadata) do
    print("meta", k, v)
end

storage:release()

Returns: table, error

Result fields:

A missing object returns a not_found error.

Deleting Objects

Remove multiple objects:

local storage, storage_err = cloudstorage.get("app.infra:files")
if storage_err then return nil, storage_err end

local deleted, err = storage:delete_objects({
    "temp/file1.txt",
    "temp/file2.txt",
    "temp/file3.txt"
})

storage:release()
if err then return nil, err end
return deleted

Returns: boolean, error

Every key is attempted. Deleting a key that does not exist is not an error. When the provider reports per-key failures, the call returns a single error naming each failed key and its provider error code.

Download URLs

Create a temporary URL that permits downloading an object without storage credentials. A client can use the URL until it expires.

local storage, err = cloudstorage.get("app.infra:files")
if err then
    return nil, err
end

local url, err = storage:presigned_get_url("reports/quarterly.pdf", {
    expiration = 3600
})

storage:release()

if err then
    return nil, err
end

-- Return URL to client for direct download
return {download_url = url}

Returns: string, error

Upload URLs

Create a temporary URL that permits uploading an object without storage credentials. A client can upload directly to storage until the URL expires.

local storage, err = cloudstorage.get("app.infra:files")
if err then
    return nil, err
end

local url, err = storage:presigned_put_url("uploads/user-123/avatar.jpg", {
    expiration = 600,
    content_type = "image/jpeg",
    content_length = 1024 * 1024
})

storage:release()

if err then
    return nil, err
end

-- Return URL to client for direct upload
return {upload_url = url}

Returns: string, error

Multipart Uploads

A single presigned PUT caps an object at 5 GiB. A presigned multipart upload splits a larger object into parts that a client uploads directly, then assembles them server-side. Multipart is a provider capability: S3 implements it, and providers without it return errors.UNAVAILABLE.

local storage = cloudstorage.get("app.infra:files")

local mp, err = storage:create_multipart_upload("backups/huge.zip", {
    content_type = "application/zip",
    metadata = { source = "uploader" },
})
if err then return nil, err end

local urls, err = storage:presigned_part_urls("backups/huge.zip", mp.upload_id, {
    count = 3,
    expiration = 900,
})
if err then
    storage:abort_multipart_upload("backups/huge.zip", mp.upload_id)
    return nil, err
end

-- The client PUTs each url and returns the ETag from the response headers.
local done, err = storage:complete_multipart_upload("backups/huge.zip", mp.upload_id, {
    { part_number = 1, etag = etag1 },
    { part_number = 2, etag = etag2 },
    { part_number = 3, etag = etag3 },
})

storage:release()

create_multipart_upload

Start a multipart upload for a key.

Returns: table, error - the table carries upload_id, which identifies the upload for every later part, complete and abort call.

Conditional writes (if_match, if_none_match, only_if_absent) are not part of the multipart protocol and are not accepted here.

presigned_part_urls

Generate presigned PUT URLs for parts of an in-progress upload. Each URL is uploaded to with a plain HTTP PUT; the uploader must keep the ETag response header of each part for complete_multipart_upload.

Exactly one of parts or count is required, and a single call presigns at most 1000 URLs - presign in pages for very large objects.

Returns: table, error - an array of { part_number, url }.

Every part except the last must be at least 5 MiB; the provider enforces this at completion time.

complete_multipart_upload

Assemble the final object from its uploaded parts. Parts may be reported in any order and are sorted by part number before completion.

Returns: table, error - etag, plus version_id and location when the provider reports them. An unknown upload ID returns errors.NOT_FOUND.

abort_multipart_upload

Discard an in-progress upload and free its stored parts.

Returns: boolean, error

An upload that is never completed keeps its parts stored, and billed, until it is aborted. Abort on every failure path, and configure a bucket lifecycle rule as a backstop - see Cloud Storage.

Ranged Readers

open_reader opens random access over an object using ranged GETs - no local staging and no full download. Its main consumer is archive.open, which reads multi-GB archives straight out of object storage with bounded memory.

local archive = require("archive")
local storage = cloudstorage.get("app.infra:files")

local reader, err = storage:open_reader("uploads/huge.zip", {
    block_size = 8 * 1024 * 1024,
    cache_blocks = 4,
})
if err then return nil, err end

local r = assert(archive.open(reader))
for e in r:entries() do
    print(e.name, e.size)
end
r:close()
reader:close()

storage:release()

block_size * cache_blocks may not exceed 256 MiB. A missing object returns errors.NOT_FOUND.

Returns: Reader, error

The object's ETag is pinned when the reader opens and sent as If-Match on every ranged read, so an object overwritten mid-read fails the read with the provider's precondition error instead of serving a mix of two object generations; archive surfaces it as errors.INTERNAL. A provider that cannot supply an ETag returns errors.UNAVAILABLE; the reader never serves an unpinned object.

Cache-miss reads perform blocking network IO in the calling task and serialize concurrent readers, so sequential per-entry access - the archive pattern - is the intended shape.

Reader Methods

The reader is closed automatically at task scope if it is not closed explicitly.

Storage Methods

Permissions

Security policy evaluation applies to cloud storage operations.

Errors

See Error Handling for working with errors.