HTTP Cache: Serving Stale Content

A cache lookup answers a yes-or-no question: is there something here you can serve? http-cache has a flag for yes, and the absence of it for no. It also has a third answer, and the third answer is it depends on what happens next.

The flag that reads its neighbors

Here is the whole state a lookup can come back with:

flags lookup-state {
  /// A cached object was found
  found,
  /// The cached object is valid to use (implies found)
  usable,
  /// The cached object is stale (but may or may not be valid to use)
  stale,
  /// This client is requested to insert or revalidate an object
  must-insert-or-update,
  /// The cached object is only usable if revalidation has failed.
  /// If usable-if-error is set, either must-insert-or-update will be set
  /// (in which case the client must revalidate before use)
  /// or usable will be set (in which case revalidation has already failed).
  usable-if-error,
}

Four of those five describe the object. usable-if-error describes the other flags.

Its doc comment never describes a property of the stored response. Instead it sends you to check which of two companion flags came back alongside it, and the answer means opposite things in the two cases. With must-insert-or-update next to it, you may not serve the stale copy and you owe the cache a revalidation. With usable next to it, the revalidation already happened, already failed, and now you may.

The flag is the same in both cases and so is the object. What differs is the permission, and the permission is written in the neighbors.

Two windows that look alike

That state can only arise if the stored response reserved room for it. write-options has two fields that do, and they sit next to each other:

/// The maximum duration after `max-age` during which the response may be delivered stale
/// while being revalidated, in nanoseconds.
///
/// If this field is not set, the default value is zero.
stale-while-revalidate-ns: option<duration-ns>,

/// The maximum duration after `max-age` during which the response may be delivered stale
/// if synchronous revalidation produces an error.
///
/// If this field is not set, the default value is zero.
stale-if-error-ns: option<duration-ns>,

Identical first lines, and then they diverge on four words. One extends the life of the response while being revalidated. The other extends it if synchronous revalidation produces an error. Fastly's documentation on serving stale content (opens in a new window) covers both windows and the delivery behavior they buy, which is the background this post assumes rather than repeats.

That difference is where usable and usable-if-error come from. Under stale-while-revalidate the stale copy is servable immediately and the revalidation happens around it, so the lookup sets usable. Under stale-if-error it is not servable yet, because the whole point is to try the origin first and fall back only on failure, so the lookup withholds usable and sets usable-if-error instead. Two windows, two different answers to "may I serve this right now," and the flags carry the answers.

Both default to zero, which is worth pausing on. A response stored without thinking about either field has no window at all, so the moment it goes stale it stops being available for any purpose. Neither flag ever appears. Both mechanisms are opt-in, and the opt-in is a pair of fields most callers assembling a write-options will leave alone.

Discharging with nothing in hand

The obligation is the problem. A lookup that says must-insert-or-update has handed you a claim, and the interface expects you to settle it. But you get here precisely because the backend fetch failed, so you have no response to settle it with.

/// Fulfill an obligation to provide a response to the cache by selecting a stale-if-error response.
///
/// A guest that is obligated to insert/update the cache may not be able to produce an acceptable
/// response (e.g. unreachable backend, 5xx response). If the cache contains a response in the
/// stale-if-error period, the guest may prefer to use that response rather than returning an error.
/// If so, they can call transaction-choose-stale, after which the cache handle will reflect the stale
/// response (via get-found-response, get-state, etc).
///
...
transaction-choose-stale: func() -> result<_, error>;

The six ways to discharge an obligation introduced this one as the option you reach for when the backend is down. What that description leaves out is that calling it changes the lookup state you are holding, and the change is the whole point.

Watching it flip

Full working code: full example on GitHub (opens in a new window).

The example is the guest pipeline from last time with one path added. Where it would normally send the suggested backend request, a fail=1 in the URL makes it skip the send entirely and go straight to transaction-choose-stale, which is what a before-send hook returning an error amounts to from the ABI's side: an obligation, and nothing to discharge it with.

if uri.contains("fail=1") {
    log.push_str("send:                      skipped, simulating a failed send\n");
    match entry.transaction_choose_stale() {
        Ok(()) => {
            log.push_str("discharged with:           transaction-choose-stale\n");
            log.push_str(&format!("state after:               {:?}\n", entry.get_state()));
            match entry.get_found_response(1) {
                Ok(Some((r, b))) => {
                    log.push_str(&format!("stale status:              {:?}\n", r.get_status()));
                    log.push_str(&format!("stale body bytes:          {}\n", drain(&b)));
                }
                Ok(None) => log.push_str("stale response:            none\n"),
                Err(e) => log.push_str(&format!("get-found-response:        Err({e:?})\n")),
            }
        }
        Err(e) => log.push_str(&format!("transaction-choose-stale:  Err({e:?})\n")),
    }
    return Ok(());
}

Seed the cache with a response that reserves an error window, max-age=1 with stale-if-error=60:

origin cache-control:      max-age=1,stale-if-error=60
storage-action:            StorageAction::Insert
suggested sie-ns:          Some(60000000000)
discharged with:           transaction-insert (9 bytes)

Sixty seconds of error window arrives as 60000000000 nanoseconds, so the field is being read off the origin's Cache-Control and converted, not defaulted.

First, for contrast, the same seeding with stale-while-revalidate=60 instead. Wait for it to go stale, then look it up:

suggested swr-ns:          Some(60000000000)
suggested sie-ns:          Some(0)
...
lookup state:              LookupState(FOUND | USABLE | STALE | MUST_INSERT_OR_UPDATE)

The entry is stale, a revalidation is owed, and usable is set all the same. No usable-if-error appears, because no error window was reserved.

Those last two flags arriving together are a trap. usable says you may serve the stale copy immediately, which is the entire reason stale-while-revalidate exists. must-insert-or-update says you still owe the cache a revalidation. Read only the first and you write the obvious code: the object is usable, so serve it and return. Nothing errors. The obligation is simply dropped, the stored object never refreshes, and every request for the rest of the window repeats the pattern, serving progressively staler content while the revalidation that was supposed to happen never happens.

The Rust SDK checks the second flag on exactly this path. Having pulled a usable response out of the cache, it asks whether must-insert-or-update is set anyway, and when it is, starts the backend fetch before handing the stale response back. The comment says why: this request "may be the collapse winner responsible for kicking off a background revalidation while still returning the stale response immediately." The fetch gets parked on the outgoing response as a BackgroundRevalidation, a type whose entire job is to finish after the client has already been served. Dropping it "finishes the fetch, applies the resulting cache candidate in the background, and then discards the response."

The example in this post does none of that. In the stale-while-revalidate case it takes the ordinary path, fetching and inserting before it responds, which discharges the obligation correctly and throws away the latency the window existed to buy. That is a fair summary of what serving stale properly costs to implement by hand.

Now the same shape with the windows swapped. Wait for the second to pass, then ask again with a send that fails:

lookup state:              LookupState(STALE | MUST_INSERT_OR_UPDATE | USABLE_IF_ERROR)
send:                      skipped, simulating a failed send
discharged with:           transaction-choose-stale
state after:               LookupState(FOUND | USABLE | STALE | USABLE_IF_ERROR)
stale status:              Ok(200)
stale body bytes:          9

Those two state lines are the doc comment, running.

Before the call: stale, must-insert-or-update, usable-if-error, and conspicuously not usable. That is the first of the two cases the doc comment describes, the one where "the client must revalidate before use." The stale copy is sitting right there and you are not allowed to serve it.

After the call: must-insert-or-update is gone and usable has appeared. That is the second case, "revalidation has already failed." Nothing about the stored object changed—same status, same nine bytes—but the permission did. One call moved the entry from the first sentence of that doc comment to the second.

And the obligation is settled without anything being written. The doc comment says so in the part elided above: transaction-choose-stale "does not change the cached item. The next lookup will again collapse and/or get an obligation to revalidate." Everyone collapsed behind you gets the stale response, and the next request through will try the origin again.

When there's nothing to choose

Call it without a stale copy in reach and it fails:

lookup state:              LookupState(MUST_INSERT_OR_UPDATE)
send:                      skipped, simulating a failed send
transaction-choose-stale:  Err(Error::AuxiliaryError)

A cold lookup, an obligation, and no error window to fall back on. error.auxiliary-error is the generic cache-layer failure, described in types as indicating "the underlying cache entry or cache replace entry is no longer available," so it doesn't distinguish "there is no stale copy" from anything else that could have gone wrong here. You get a failure, not a diagnosis.

Which means the fallback is not something you can attempt speculatively and read the result of. If you want to know whether serving stale is available, the flags told you before you called: usable-if-error was either in the lookup state or it wasn't.

Beyond the WIT

Every other decision in this interface is offered as a suggestion you can decline. This one isn't offered at all. Nothing calls transaction-choose-stale on your behalf, and nothing tells you that now would be the moment.

That's because the moment is defined by something the cache can't see. The cache knows the entry is stale and that an error window is open. What it doesn't know is whether your backend fetch failed, because the fetch is yours—that's the whole shape of guest mode. So the one piece of information that decides this branch lives on the guest's side of the boundary, and the interface has no way to ask for it.

For a binding, that lands the responsibility somewhere specific and easy to miss. The cache-related calls are not where stale-on-error gets handled; the error path is. Whatever your SDK does when a backend send returns an error has to know it might be holding a cache transaction with an open error window, and has to check the flags before it converts that error into a 502 for the caller. A binding that treats send failures and cache transactions as separate concerns will never reach this function, and the failure mode is silent: origins go down, a perfectly good stale copy sits in cache with permission to be served, and every request 502s anyway.

The same shape turns up on the opposite path, which is what makes it a category rather than a special case. Stale-on-error gets dropped in the error handler. Stale-while-revalidate gets dropped in the success handler, by code that received a usable response and reasonably stopped thinking about the cache. Both failures are silent, both leave the cache behaving exactly as designed, and neither shows up in a test that asks whether the right bytes came back, because the right bytes did come back.

The flags are the API for both checks, which is why usable-if-error is shaped the way it is, and why must-insert-or-update can arrive alongside usable at all. They aren't describing the object for your benefit. They're telling whichever handler is holding the transaction what it still owes.

Next: the options records both cache interfaces use, set side by side, and what purging looks like from the HTTP side.