Reusable Sandboxes: A Sandbox Doesn't Have to Be for Exactly One Request

Every handler in this series has had the same lifecycle.

When a request comes in to the service, the host spins up a sandbox, calls handle with the request and its body, the handler sends a response, handle returns, and the sandbox is gone. The very first post quoted the doc comment that says so:

/// Handle the given request.
///
/// This function is called once per sandbox. When it returns, the
/// sandbox exits. To opt into receiving multiple requests in a single
/// sandbox, use `http-downstream.next-request`.
///
/// To send a response for the given `request`, use `send-downstream`, or to
/// stream the response body after the response has been initiated, use
/// `send-downstream-streaming`.
handle: func(request: request, body: body) -> result;

The sentence after "the sandbox exits" has been waiting since then.

As it turns out, a handler doesn't have to return after one request. It can ask for the next one.

Handling additional requests in a single sandbox

To expose this behavior, the ABI provides the two functions next-request and await-request, which live in the http-downstream interface. Guest code that incorporates these is expected to call them in a loop, as so:

The last time we saw the http-downstream interface, we skipped past these functions and their related record:

interface http-downstream {
  use types.{error, ip-address};
  use http-req.{
    request, client-cert-verify-result, error-with-detail, cache-override, pending-request,
    request-with-body,
  };

  /// Configuration for `next-request`.
  record next-request-options {
    timeout-ms: option<u64>,

    /// Additional options may be added in the future via this resource type.
    extra: option<borrow<extra-next-request-options>>,
  }

  /// Extensibility for `next-request-options`
  resource extra-next-request-options {}

  ...

  /// Prepares to accept a new downstream request.
  ///
  /// By default, each sandbox accepts a single downstream request,
  /// passed in as an argument to the `http-incoming.handle` call. The
  /// `next-request` function enables a sandbox to accept additional
  /// downstream requests. `next-request` returns a `pending-request`, which
  /// can be passed to `await-request` to wait for the request to become
  /// available and return the `request`.
  ///
  /// When using this function, be mindful of two considerations:
  ///
  ///  - When a single sandbox accepts multiple requests, its state
  ///    isn't automatically cleared between requests. Applications are
  ///    therefore responsible for preventing sensitive data from leaking from
  ///    one request to another.
  ///
  ///  - There is no guarantee that a single sandbox will receive all
  ///    requests from a given client or from a given location. When it is
  ///    necessary to preserve state between multiple requests, store it outside
  ///    of the sandbox.
  next-request: func(
    options: next-request-options,
  ) -> result<pending-request, error>;

  /// Waits until the next request is available, and then returns the resulting
  /// request and body.
  ///
  /// Returns `ok(none)` if there are no more requests for this sandbox.
  await-request: func(
    pending: pending-request,
  ) -> result<option<request-with-body>, error>;

  ...
}

next-request registers your interest for obtaining another request in this sandbox, and returns right away with a pending-request. That's another name for the same pollable that Two Requests at Once passed to select-with-timeout, and the last alias of it this series hadn't used yet.

await-request takes this pending-request, blocks until something happens, and gives you back either of the following:

• a request-with-body, a tuple that holds the same request and body you'd get from handle
• none, meaning this sandbox won't get any more requests

The next-request-options record has one real field, a timeout, and an extra resource with no methods. That's the escape hatch bot-category used, here on a record rather than a variant: room for future options without changing the record's definition.

The doc comment spends most of its words on two warnings, which are about opposite problems. The first is for the situation where too much survives between requests. Anything your program left in memory is still there when the next request arrives, possibly from somebody else. The second is that you shouldn't count on too much to survive between requests. The next request might never come to this sandbox, so you can't use it to keep anything a client needs to find again.

A loop, where there used to be a return

Full working code: full example on GitHub (opens in a new window).

The example serves a request, then asks for another and waits up to five seconds for it. Every response says which sandbox handled it, which request it was, and how many requests that sandbox has handled so far, counted in a static variable that would keep its value between requests.

imports, and helpers for reading the request ID and sending a text response (23 lines)
mod bindings;

use std::sync::atomic::{AtomicU32, Ordering};

use bindings::{
    exports::fastly::compute::http_incoming,
    fastly::compute::{compute_runtime, http_body, http_downstream, http_req, http_resp},
};

fn request_id(request: &http_req::Request) -> String {
    match http_downstream::downstream_client_request_id(request, 128) {
        Ok(id) => id,
        Err(e) => format!("Err({e:?})"),
    }
}

fn respond(text: String) -> Result<(), ()> {
    let response = http_resp::Response::new().map_err(|_| ())?;
    response.insert_header("content-type", b"text/plain").map_err(|_| ())?;
    let body = http_body::new().map_err(|_| ())?;
    http_body::write(&body, text.as_bytes()).map_err(|_| ())?;
    http_resp::send_downstream(response, body).map_err(|_| ())
}

// Survives from one request to the next, for as long as the sandbox does.
static HANDLED: AtomicU32 = AtomicU32::new(0);

fn serve_one(request: &http_req::Request) -> Result<(), ()> {
    let count = HANDLED.fetch_add(1, Ordering::SeqCst) + 1;
    respond(format!(
        "sandbox {}\nrequest {}\nrequests this sandbox has handled: {count}\n",
        compute_runtime::get_sandbox_id(),
        request_id(request),
    ))
}

struct ReusableSandboxes;

impl http_incoming::Guest for ReusableSandboxes {
    fn handle(request: http_incoming::Request, _request_body: http_body::Body) -> Result<(), ()> {
        let mut request = request;

        loop {
            serve_one(&request)?;

            // Offer to take another request, and wait up to five seconds for one.
            let options = http_downstream::NextRequestOptions { timeout_ms: Some(5_000), extra: None };
            let pending = match http_downstream::next_request(&options) {
                Ok(pending) => pending,
                Err(e) => {
                    println!("next-request: Err({e:?})");
                    return Ok(());
                }
            };

            match http_downstream::await_request(pending) {
                Ok(Some((next, _body))) => request = next,
                Ok(None) => {
                    println!("await-request: no more requests for this sandbox");
                    return Ok(());
                }
                Err(e) => {
                    println!("await-request: Err({e:?})");
                    return Ok(());
                }
            }
        }
    }
}

bindings::export!(ReusableSandboxes with_types_in bindings);

handle still has the signature it always had. What changed is that it doesn't return while requests keep arriving. The response to each request goes out (http_resp.send_downstream) before the loop asks for the next one.

That order is very important. Fastly's sandbox lifecycle documentation (opens in a new window) lays out the sequence: the request handler runs to completion, the sandbox is checked against its limits, and only then does it wait for an additional request.

In practice, the host enforces that sequence rather than assuming it. If you ask too early, the call fails, which Viceroy reports as "cannot accept requests w/o handling the outstanding one."

That accounts for one of the two error branches in the loop above. The other belongs to next-request: if you're already holding a pending-request you haven't awaited, it fails, because a sandbox can only have one offer outstanding at a time. Viceroy's wording for that one is "Too many pending downstream request handles have been created."

Reusable sandboxes, locally

There's nothing additional that needs to be configured for testing reusable sandboxes locally. fastly compute serve is enough.

Send eight requests one after another:

for i in $(seq 1 8); do curl -s http://127.0.0.1:7676/; done

sandbox 00000000000000000000000000000000
request 00000000000000000000000000000000
requests this sandbox has handled: 1
sandbox 00000000000000000000000000000000
request 00000000000000000000000000000001
requests this sandbox has handled: 2
sandbox 00000000000000000000000000000000
request 00000000000000000000000000000002
requests this sandbox has handled: 3
sandbox 00000000000000000000000000000000
request 00000000000000000000000000000003
requests this sandbox has handled: 4
sandbox 00000000000000000000000000000000
request 00000000000000000000000000000004
requests this sandbox has handled: 5
sandbox 00000000000000000000000000000000
request 00000000000000000000000000000005
requests this sandbox has handled: 6
sandbox 00000000000000000000000000000006
request 00000000000000000000000000000006
requests this sandbox has handled: 1
sandbox 00000000000000000000000000000006
request 00000000000000000000000000000007
requests this sandbox has handled: 2

The first six requests went to one sandbox, which we can see from the counter: HANDLED kept its value between them. This is the scenario mentioned by the first warning in the doc comment, which we triggered on purpose here. The request ID changed every time, but the sandbox ID didn't. That's the distinction drawn by get-sandbox-id's doc comment. As it recommends, use downstream-client-request-id when you need something that identifies a request, now that one sandbox can handle several.

After the sixth, the sandbox was done. It appears that Viceroy lets a sandbox take five requests beyond its first. The next await-request returned none, and the program logged this to the terminal running the server:

await-request: no more requests for this sandbox

The seventh request got a fresh sandbox with its counter back at 1.

If we wait for longer than the five-second timeout, and send one more:

sandbox 00000000000000000000000000000008
request 00000000000000000000000000000008
requests this sandbox has handled: 1

The sandbox had been waiting in await-request and gave up after the timeout, returning none to the program. The reason for the none was different from the previous sandbox, but to the program it looks no different. Once its handler returned, the next request started over.

Viceroy also appears to cap any timeout you ask for at ten seconds.

On a real service, reuse is a maybe

Locally, reuse is deterministic. There's one server, so a waiting sandbox is always where the next request lands. On a real service, however, none of that carries over.

First, a real service needs reuse turned on before it reuses anything. There's no self-service switch: Fastly support enrolls the service.

Second, even with the setting on, reuse is opportunistic. A waiting sandbox gets a request only if that request arrives at the same cache node.

I tested by making multiple batches, each ten requests, to a real service. Run one after another, these batches reused a sandbox once, then not at all, then six times. We can see that the second warning in the doc comment is describing actual, ordinary behavior.

///  - There is no guarantee that a single sandbox will receive all
///    requests from a given client or from a given location. When it is
///    necessary to preserve state between multiple requests, store it outside
///    of the sandbox.

Third, while you can specify a timeout-ms to wait for the next request, Fastly's documentation (opens in a new window) says there is "a platform-defined maximum number of requests a sandbox can execute (this value is not publicly disclosed and may change)".

It's also worth noting the cost: a sandbox waiting for a request isn't using vCPU time, but it's still holding its memory and still accruing wall-clock time, and a service billed on memory over time pays for the wait whether a request arrives or not.

Beyond the WIT

handle is still a function that runs once. Reuse doesn't change the export at all. What it changes is who owns the server loop: before, the host called your handler per request, and now the host calls you once and your program asks for requests one at a time. The loop that a web framework normally hides from you is sitting in your handle.

That loop carries responsibilities the export never had. The WIT states none of them as types: you have to respond before asking for the next request; you have to decide when to stop, because the host will happily let you wait; and anything global you touch in one iteration is still there in the next.

A binding is the natural owner of all three.

The Rust SDK's Serve shows one possible way to model this. Rather than using the #[fastly::main] macro, you write an ordinary main() and build a Serve from fastly::http::serve, setting whichever limits you want through with_max_requests, with_max_lifetime, with_max_memory, and with_timeout. Then you pass your per-request handler to run, and return what comes back.

From there Serve owns the loop: it calls the handler, sends the response, and only then registers for the next request. The limits are options on it rather than conditions each program writes for itself, and the loop ends once one of them is reached. What run returns isn't a bare result either. It's a ServeSummary, which counts the requests served and splits the elapsed time between your handler and the waiting, and into_result collapses that into the Result main wants. A program built this way never touches the SDK's equivalents of next-request or await-request at all.

That first row is actually a little stranger than the diagram makes it look. The first request and its body are parameters to your handler. Every request after them is a return value from await-request. They are two mechanisms with nothing in common that deliver the same pair of things, and which one your code goes through depends only on how many requests have already gone by.

Underneath, both are doing the same thing. They update a slot in the sandbox's state machine. A sandbox has exactly one current downstream request and one current downstream response. handle's arguments fill that pair, and await-request replaces them. The rest of the interface is built around it: almost every function in http-downstream takes a borrow<request> so it can say which request it means, and the handful that don't are wired to the slot in the state machine. Likewise, send-downstream takes a response and a body and no request at all, because there's only ever one request it could be answering.

The loop's one ordering rule comes out of that same pair of slots. You respond before asking for the next request because await-request won't overwrite the request in front while the response behind it is still unsent. The rule isn't etiquette about finishing what you started. It's the only order two slots have room for.

By the way, the Rust SDK's model is only an example of one way to model this. Fastly's other SDKs actually implement this differently, in a way that matches its language's expectations. These sources are public, and the interested reader is encouraged to read them against each other.

The state problem is the harder one to design for, because a binding can't make a language forget its globals. What it can do is make the intended state visible. For example, the Rust SDK's Serve has a run_with_context variant that passes the same mutable context value to every call of the handler. That doesn't stop a handler from writing a static anyway. But it gives the state that's meant to be shared a declared place to live, which means everything else is presumed per request. For a program that may serve two unrelated clients with the same memory, that's a line worth drawing where a reviewer can see it.

Next: an interface where Viceroy can't help you at all. Image Optimizer transforms images on a real service, but has no local way of testing.