#Operating worker pools

Workers are the external-I/O boundary for WWKG. They fetch remote RDF, execute read-only SQL, call statement services, and stream results back to an authenticated submitter. A host does not perform those network calls locally.

#Choose the deployment model

Run wwkg-worker as a long-running daemon for production pools today. Give each worker a stable node identity, persistent data directory, reachable iroh/QUIC endpoint, and DNS or local discovery record. Long-running workers support the Git checkout cache, private-network placement, and jobs whose duration or result size exceeds a serverless provider’s limits.

Lambda and Cloud Run are not interchangeable container hosts for the daemon. Their invocation ingress is HTTP, while worker submission is authenticated iroh QUIC. WWKG deliberately does not publish a serverless image until the signed, streaming HTTP job transport in ADR 0027 is available. Do not wrap the daemon behind an unauthenticated HTTP proxy or put a claimed submitter node ID in a request header.

#Pool topology

Use multiple workers and advertise only the job kinds and access scopes each worker can actually execute. Submission ranks eligible workers by current load, then price. A private worker should advertise a specific scope such as acme-prod-vpc; the workspace egress policy must explicitly allow that scope. Public workers are appropriate only for public targets and credentials that do not depend on private-network reachability.

The worker publishes only the capabilities it enforces. Configure them in its data-directory config.toml; invalid or empty limits stop startup:

The task registry uses max_concurrency and rejects excess jobs immediately; workers do not retain caller streams and credentials in a local queue. Job dispatch rejects a requested duration above max_job_duration before resolving credentials or constructing an external client. The same duration, access profile, concurrency, pricing, and tags are published in NodeProfile; do not advertise a private scope the process cannot actually reach.

Place database workers in the smallest network segment that can reach the database. Database-side authentication and read-only roles remain mandatory: WWKG authorization is an additional boundary, not a replacement. PostgreSQL SQL jobs also start a server-enforced read-only transaction and require TLS.

#Identity, policy, and credentials

Configure the worker’s persistent node identity through the normal WWKG node identity flow. Configure trusted authorization issuers and the worker-local allow list in the worker data directory. Startup fails when those files are invalid; there is no permissive default for accepting jobs.

External-account bindings belong to the initiating wallet or organisation authority. The workspace stores policy, never broad external secrets. After an egress grant permits a binding, the authority supplies either a short-lived, exact-target credential or an opaque reference to worker-local material.

Worker-local credentials belong in the worker credential configuration and its secret-loading mechanism. Protect the data directory and OS credential store as production secrets. Do not commit plaintext credentials, place them in workspace triples, or copy a broad account token into JobSubmit.

Paid pools configure the pricing that the worker both advertises and meters:

Paid pricing requires a wallet- or individual-backed operator payee. Startup fails for another currency or a paid organisation operator whose billing wallet cannot yet be derived. A paid job without a matching PaymentIntent fails with PAYMENT_REQUIRED or INVALID_PAYMENT; a ceiling below the fixed or measured cost fails with PAYMENT_BUDGET_EXCEEDED. Free workers reject payment intents instead of silently accepting contradictory settlement terms.

#Combined authorization failures

Worker execution has separate deny points. Treat their codes differently:

• workspace egress denied: the authoritative workspace policy did not grant the principal, job kind, exact target, recipient, credential scopes, worker access scope, or requested limits. The job is not submitted.
• INVALID_AUTHORIZATION: the worker rejected the signed grant because its signature, issuer, time window, nonce, submitter, worker audience, parameters, target, credentials, recipient, or delegation binding was invalid. Do not retry the same authorization.
• WORKER_DENIED: the authorization was valid, but this worker operator’s local admission policy rejected it. Select an explicitly eligible worker or change that worker’s policy; do not weaken workspace policy as a workaround.
• ADMISSION_REJECTED: authorization passed but the worker’s bounded task capacity was unavailable. This is the only one of these four outcomes that may be retried on another eligible worker.

The worker checks authorization before job-kind lookup, credential resolution, task admission, or external client construction. Logs should retain the job ID, authenticated submitter node, selected worker, and machine-readable failure code, but never credential material.

#Capacity and lifecycle

Set per-job duration, byte, and row ceilings in workspace egress policy. The signed authorization carries those ceilings to the worker, which enforces them during streaming. Size the pool for queue depth rather than allowing unbounded local queues. Drain a worker by removing it from discovery or admission, wait for active tasks to finish, and then stop it; killing the process may leave the caller with an incomplete stream, which is a hard job failure.

Monitor task and journal events for rejection codes, latency, bytes, rows, queue depth, and worker health. Repeated WORKER_DENIED means selection and operator policy disagree. Repeated workspace denials mean the requested action has no matching authority grant. Repeated invalid authorizations are a security signal, not a capacity problem.

Workers publish live running and queued admission counts to the node catalog; the local worker queue is deliberately disabled, so queue_depth remains zero and capacity overflow returns ADMISSION_REJECTED for immediate rerouting.