Where Viceroy and Compute Disagree

Publishing to Compute covered getting a component onto a real service: the [setup] half of fastly.toml, the checks at upload and activation, and the wait for the edge. That's the mechanics. What a program sees once it's running is a separate question. A real Compute service and local Viceroy implement the same compute.wit, but they still don't always give the same answer.

Most of the differences have already turned up somewhere in this series, one post at a time. Here they are in one place, sorted by how they fail, along with the limits, which hadn't been measured until now.

The limits

Fastly publishes (current) Compute's resource limits (opens in a new window). A small test service measured five of them on both hosts, and found another that isn't on the list:

The heap limit is the only one both hosts enforce, and they report it differently. Viceroy's response body is the trap's full backtrace. The real service's is Internal Server Error.

Neither host stopped at the documented CPU figure. get-vcpu-ms from compute-runtime confirmed the busy loop was really using the CPU, 998 ms of it in a 1-second loop on the real service. What the real service did enforce was a ceiling of about 60 seconds for a request that kept the CPU busy, and about 130 seconds for one that spent its time waiting. Both ended in a 503 from the platform, not an error the program could see. These readings come from one service on one POP, so take them as what happened there, not as specifications.

Backend requests are the limit a program can see coming. The 33rd send-uncached returned error.limit-exceeded, the variant whose doc comment says it's "returned when an attempt to allocate a resource has exceeded the maximum number of resources permitted." Under Viceroy it never comes.

Viceroy's issue tracker has an open request for this whole table, fastly/Viceroy#407 (opens in a new window): "Compute enforces a few limits on code executing, which are only partially replicated in Viceroy."

Refused before it runs

Three problems in this series stopped a real deploy before it ran, with no counterpart under Viceroy:

• A UTF-16 string encoding. The Same Tally, in MoonBit built a component with --encoding utf16, MoonBit's default. Viceroy ran it, and the upload failed with "Failed to extract metadata from package."
• Three linear memories. The Same Tally, in JavaScript composed three components, each with its own memory, and activation refused anything over two.
• A WASI version the service world doesn't have. From Core Module to Component's @0.2.4 build and the JavaScript build's @0.2.12 imports both ran in Viceroy, and activation refused each one with a "missing import named" error for wasi:io/poll at that version.

A component that imports an interface the host doesn't provide at all is the one case where both hosts agree. The service World's wasi:http build failed in Viceroy and failed activation on a real service, with the same message from both.

Missing from Viceroy, and saying so

Some functions don't work under Viceroy, and it tells you:

• The HTTP Cache in guest mode. Every function in http-cache returns unsupported, so the HTTP Cache posts ran their examples on a real service.
• Core Cache's replace API. Core Cache: Insert and Replace found replace answering unsupported, and with it everything downstream of it.
• Image Optimizer. Image Optimizer: "error.unsupported is the only answer Viceroy has."

These are the easy ones. A test that calls one fails loudly the first time it runs locally.

Answering, but not with the platform's answer

These are the dangerous ones. Viceroy returns a successful, well-formed result, and nothing about it says the answer is made up:

• Edge rate limiting. Edge Rate Limiting found all six functions implemented as constants, so counters never count.
• Next-Gen WAF. Next-Gen WAF: "Viceroy's verdict is a constant," and it always says allow.
• Health checks. Health Checks: "Viceroy doesn't run the checks," so a backend's health stays unknown.
• Device detection. Device Detection: the classification data lives on Fastly's infrastructure, "not in anything viceroy ships."
• The readthrough cache. On a real service, send and send-async go through Fastly's cache. Request Handoff got an x-cache: HIT from one, which is why the later examples use the uncached variants. Viceroy doesn't cache those sends, so a local run can't show whether a response was served from cache. fastly/Viceroy#496 (opens in a new window) asks for it.

A test that passes locally against any of these has proven only that the call doesn't fail.

Both answer, differently

Then there are the cases where both hosts really do the work and still disagree:

• Log endpoint names. Realtime Logging: names a real service refuses with invalid-syntax open fine under Viceroy.
• Framing headers. Framing Headers found the two hosts handling a mismatched Content-Length differently, and neither fully matching the doc comment (fastly/Viceroy#179 (opens in a new window)).
• Error detail. Send Errors: Viceroy never fills in a send error's detail (fastly/Viceroy#727 (opens in a new window)).
• ACL results. ACLs: the same match, pretty-printed by Viceroy and compact on a real service, so parse it instead of comparing text.
• Health-check rules. Health Checks: Viceroy enforces rules between settings that a real service doesn't check when the setter is called.
• Shielding. Shielding: Viceroy needs its shield sites listed in fastly.toml, and hands back what's listed there.

And a few places where both hosts agree, which is worth knowing too. Neither delivers trailers on an incoming request (Write-Front and Trailers), and both treat a select-with-timeout of 0 as "don't wait" (Two Pollables, One World).

Beyond the WIT

Every row in this post is about something the WIT doesn't say. compute.wit declares that send-uncached can return limit-exceeded, and it doesn't say the limit is 32. It declares that a pollable can be waited on, and it doesn't say how long a request may wait. It says nothing about string encodings or how many memories a component may contain, because those are properties of the host, not of the interface. Viceroy and a real service implement the same interface faithfully, and they're different hosts.

For a binding, that changes what its tests can claim. A test suite run only under Viceroy proves that the binding calls the ABI correctly, and for the loud failures, that it handles unsupported. It can't prove anything about limits, about the inert functions, or about the cases where the hosts disagree, and those are the ones a binding's users will hit in production.

The workable arrangement is a suite that knows which host it's running against. Each test says which hosts it applies to, and an assertion that's known to fail on one host is marked as an expected failure there, so it starts failing loudly the day that host closes the gap. Viceroy runs on every change, because it's fast and free. A real test service runs less often, and it's the only place a test like "the 33rd request fails" can mean anything.

Next: which of these has bitten you? The map is the start of a list, and the next entries are the ones nobody has found yet.