Core Cache: Insert and Replace

EDIT (2026-08-29): lookup-options and replace-options now set always-use-requested-range to true in the example, with a note on why. The reading-a-hit post explains what that flag decides; the short version is that leaving it at its default makes a ranged read's result depend on timing.

The map post named cache as the low-level one: a byte-string key, a body you stream into, metadata you attach at write time, and nothing that knows what HTTP is. Enough with the description; let's have a look at the code.

The smallest thing this interface can do is put an object in and take it back out again. Neither operation needs a transaction, neither one waits on anybody, and between them they use two of the three entry points this interface has.

Writing without ceremony

interface cache {
  ...

  /// Performs a non-request-collapsing cache insertion (or update).
  ///
  /// The returned handle is to a streaming body that is used for writing the object into
  /// the cache.
  insert: func(
    key: list<u8>,
    options: write-options,
  ) -> result<body, error>;

  ...
}

Surprising as it may seem, insert doesn't actually take the bytes you want stored. Instead, it hands you a body and lets you stream into it, which is the same body resource you've been writing response and request bodies into since The Body Isn't Part of the Request (or the Response). The cache doesn't get a new way to carry bytes; it reuses the one the whole ABI already has.

The key is list<u8>, not string. That's the same byte-safety hedge header values get, and for the same reason: a cache key is whatever you decide it is, and nothing requires it to be text.

write-options is a record of ten fields, plus the extra escape hatch every options record in this ABI carries, and it gets a post of its own later in this series. Only two of the ten aren't optional: max-age-ns, which the doc comment calls required outright, and sensitive-data, a bare bool that has to be given a value whether you care about it or not.

Reading without joining a queue

/// The outcome of a cache lookup (either bare or as part of a cache transaction)
resource entry {
  /// Performs a non-request-collapsing cache lookup.
  ///
  /// Returns a result without waiting for any request collapsing that may be ongoing.
  lookup: static func(
    key: list<u8>,
    options: lookup-options,
  ) -> result<entry, error>;

  ...
}

Two things here.

First, lookup is a static function on entry rather than a method. It has to be—a method needs an entry to be called on, and looking one up is precisely how you get the first one. open on the stores is spelled exactly the same way, a static func hanging off the resource it produces, so this is a shape the ABI already reaches for whenever a resource has to come from somewhere.

Second, that doc comment goes out of its way to say what lookup doesn't do. It doesn't collapse requests, and it returns "without waiting for any request collapsing that may be ongoing." Both insert and lookup carry a variant of that sentence, and the reason is that this interface's other half is built entirely around the waiting they skip. That half is the next post.

For now, "non-request-collapsing" means what it sounds like: if fifty instances all miss on the same key at the same time, lookup lets all fifty of them find out immediately and go fetch it themselves.

Lastly, this function takes a second parameter called lookup-options. For now, one of its fields is worth setting before you understand it: always pass always-use-requested-range as true. We'll be taking this one apart properly when we get to reading data from a cache entry.

The other way to write

insert is not the only way to put bytes under a key. There's a second write path with its own resource:

/// A replace operation.
resource replace-entry {
  /// The entrypoint to the replace API.
  ///
  /// This operation always participates in request collapsing and may return stale objects.
  replace: static func(
    key: list<u8>,
    options: replace-options,
  ) -> result<replace-entry, error>;

  ...
}

Note what changed in the doc comment: this one always participates in request collapsing. insert and lookup opt out; replace can't.

The interesting part is what a replace-entry lets you do while you hold it. It isn't a write handle—it's a handle on the object you're about to overwrite, and it carries most of the same getters a found entry does: get-age-ns, get-hits, get-length, get-max-age-ns, get-stale-while-revalidate-ns, get-state, get-user-metadata, and get-body. You can read the outgoing object before deciding what the incoming one should be.

Writing through it is a separate call, and it takes the handle rather than borrowing it:

/// Replace an object in the cache with the given metadata
///
/// The returned handle is to a streaming body that is used for writing the object into
/// the cache.
replace-insert: func(
  handle: replace-entry,
  options: write-options,
) -> result<body, error>;

handle: replace-entry with no borrow<> around it means the handle is consumed. Once you've called replace-insert, there's nothing left to close, which is why close-replace-entry exists for the other path—the one where you looked at the existing object and decided not to replace it after all.

/// Closes an ongoing replace interaction with the cache.
///
/// ...
close-replace-entry: func(handle: replace-entry) -> result<_, error>;

Choosing how to race

replace-options carries a field the lookup path has no equivalent for:

enum replace-strategy {
  /// Immediately start the replace and do not wait for any other pending requests for the same
  /// object, including insert requests.
  ///
  /// With this strategy a replace will race all other pending requests to update the object.
  ///
  /// The existing object will be accessible until this replace finishes providing the replacement
  /// object.
  ///
  /// This is the default replace strategy.
  immediate,

  /// Immediate, but remove the existing object immediately
  ///
  /// Requests for the same object that arrive after this replace starts will wait until this
  /// replace starts providing the replacement object.
  immediate-force-miss,

  /// Join the wait list behind other pending requests before starting this request.
  ///
  /// With this strategy this replace request will wait for an in-progress replace or insert
  /// request before starting.
  ///
  /// This strategy allows implementing a counter, but may cause timeouts if too many requests
  /// are waiting for in-progress and waiting updates to complete.
  wait,
}

There are three strategies, and the doc comments are unusually direct about the tradeoffs. immediate keeps the old object readable throughout, so nobody sees a miss. immediate-force-miss yanks it, so everyone who arrives mid-replace blocks until the new one starts streaming. wait queues behind whatever's already in flight—and the doc comment volunteers both what that buys you ("allows implementing a counter") and how it can hurt you ("may cause timeouts if too many requests are waiting").

So the difference between them is entirely about what other instances of your service on the same POP experience while your write is in progress. None of it is about the shape of the data; all of it is about what your neighbors experience.

Reading it directly

Full working code: full example on GitHub (opens in a new window).

mod bindings;

use bindings::{
    exports::fastly::compute::http_incoming,
    fastly::compute::{cache, http_body, http_resp},
};

write_options and lookup_options, ten fields and only two of them required (19 lines)
fn write_options(max_age_ns: u64) -> cache::WriteOptions<'static> {
    cache::WriteOptions {
        max_age_ns,
        request_headers: None,
        vary_rule: None,
        initial_age_ns: None,
        stale_while_revalidate_ns: None,
        surrogate_keys: None,
        length: None,
        user_metadata: None,
        edge_max_age_ns: None,
        sensitive_data: false,
        extra: None,
    }
}

fn lookup_options() -> cache::LookupOptions<'static> {
    cache::LookupOptions { request_headers: None, always_use_requested_range: true, extra: None }
}

read_entry_body, drains an entry's body into a String (15 lines)
fn read_entry_body(entry: &cache::Entry) -> String {
    let options = cache::GetBodyOptions { from: None, to: None, extra: None };
    let Ok(body) = entry.get_body(&options) else {
        return "<no body>".to_string();
    };
    let mut out = Vec::new();
    while let Ok(chunk) = http_body::read(&body, 1024) {
        if chunk.is_empty() {
            break;
        }
        out.extend_from_slice(&chunk);
    }
    let _ = http_body::close(body);
    String::from_utf8_lossy(&out).into_owned()
}

struct CoreCacheInsertReplace;

impl http_incoming::Guest for CoreCacheInsertReplace {
    fn handle(_request: http_incoming::Request, _request_body: http_body::Body) -> Result<(), ()> {
        let mut lines = String::new();
        let key = b"core-cache-demo".to_vec();

        // Write an object in.
        // No transaction, no obligation, no handshake - the write_options parameter is the max-age in nanoseconds
        let writing = cache::insert(&key, &write_options(60_000_000_000)).map_err(|_| ())?;
        http_body::write(&writing, b"first value").map_err(|_| ())?;
        http_body::close(writing).map_err(|_| ())?;

        // Read it back without joining anyone else's request collapsing.
        let found = cache::Entry::lookup(&key, &lookup_options()).map_err(|_| ())?;
        lines.push_str(&format!("state:  {:?}\n", found.get_state()));
        lines.push_str(&format!("body:   {}\n", read_entry_body(&found)));
        lines.push_str(&format!("age-ns: {:?}\n", found.get_age_ns()));
        lines.push_str(&format!("hits:   {:?}\n", found.get_hits()));
        cache::close_entry(found).map_err(|_| ())?;

        // Replace: the other write path, the one that can read what it's overwriting.
        let replace_options = cache::ReplaceOptions {
            request_headers: None,
            replace_strategy: Some(cache::ReplaceStrategy::Immediate),
            always_use_requested_range: true,
            extra: None,
        };
        match cache::ReplaceEntry::replace(&key, &replace_options) {
            Ok(replacing) => {
                lines.push_str(&format!("\nreplace state:  {:?}\n", replacing.get_state()));
                let writing = cache::replace_insert(replacing, &write_options(60_000_000_000))
                    .map_err(|_| ())?;
                http_body::write(&writing, b"second value").map_err(|_| ())?;
                http_body::close(writing).map_err(|_| ())?;

                let after = cache::Entry::lookup(&key, &lookup_options()).map_err(|_| ())?;
                lines.push_str(&format!("after replace: {}\n", read_entry_body(&after)));
                cache::close_entry(after).map_err(|_| ())?;
            }
            Err(e) => lines.push_str(&format!("\nreplace: Err({e:?})\n")),
        }

send the response, unchanged (5 lines)
        let response = http_resp::Response::new().map_err(|_| ())?;
        response.insert_header("content-type", b"text/plain").map_err(|_| ())?;
        let out_body = http_body::new().map_err(|_| ())?;
        http_body::write(&out_body, lines.as_bytes()).map_err(|_| ())?;
        http_resp::send_downstream(response, out_body).map_err(|_| ())?;

        Ok(())
    }
}

bindings::export!(CoreCacheInsertReplace with_types_in bindings);

There were two details the compiler insisted on before this would build, both of which say something about the ABI.

cache::WriteOptions needs a lifetime parameter. That's request-headers: option<borrow<request>> showing through: the record can hold a borrowed request handle, so Rust won't let you pretend it's plain owned data (even when you pass None). And replace_insert takes replacing by value rather than by reference, which is the handle: replace-entry signature enforcing itself—the handle is gone after the call.

On the local development environment:

state:  Ok(LookupState(FOUND | USABLE))
body:   first value
age-ns: Ok(Some(27750))
hits:   Err(Error::Unsupported)

replace: Err(Error::Unsupported)

The insert-and-look-it-back-up path works end to end. get-state comes back with two flags set at once, which is a shape this series hasn't met before and which gets a post of its own once we start reading what's actually in a hit. The age is a real elapsed measurement in nanoseconds, so it differs every run.

Two calls report Error::Unsupported, but there's actually a distinction here. get-hits is a single getter that Viceroy hasn't implemented. replace is the entry point to an entire API, so everything downstream of it—replace-insert, close-replace-entry, every getter on replace-entry—is unreachable locally too. This is the same category of gap the KV Store's blocking functions turned out to have: nothing in compute.wit marks these as second-class, and the gap exists one layer down in a specific host's implementation state. Unlike that case, I couldn't find an open Viceroy issue or PR to point at, so treat this as observed behavior at the time of writing rather than a documented roadmap item.

Beyond the WIT

Three entry points into one interface, at three different levels. insert is a free function on cache. lookup is a static function on entry. replace is a static function on replace-entry. Conceptually they're siblings—write, read, overwrite—and structurally they're nothing alike.

Fastly's own Rust SDK doesn't preserve that. fastly::cache::core (opens in a new window) flattens all three into top-level free functions, insert(), lookup(), and replace(), each returning a builder—InsertBuilder, LookupBuilder, ReplaceBuilder—that you configure and then execute. The ABI's option records become the SDK's builder chains, and the ABI's three structural levels become one.

Both halves of that are decisions any binding faces. The levels are the easier call: the ABI's placement is an artifact of which resource each function needs to produce, not a statement about how callers think, and flattening it costs nothing. The records are harder. A record with one required field and nine optional ones is genuinely awkward in a language without named arguments or struct update syntax, and a builder solves that—but it also means the required field can be omitted at the type level and only caught at runtime, which is exactly the guarantee the record was giving you for free. And the obvious ergonomic escape makes it worse rather than better: ..Default::default() is exactly what would let a caller leave max_age_ns out and silently cache with a zero lifetime. The syntax that fixes the awkwardness is the syntax that removes the guarantee. The SDK went the builder way, consistently, across all three.

That one is worth deciding deliberately rather than by reflex, because whichever way you go, you'll be living with it on write-options—and that record is about to get a lot more interesting.

Next: we stop skipping the queue, and find out what request collapsing, along with the obligation to insert that comes with it, actually buys you.