The Wasm Component Model on Fastly Compute

Send Errors: Asking a Failed Backend Call What Went Wrong

Every backend call in this series has handled failure the same way: .map_err(|_| ())?. The request didn't work, the handler gave up, and whatever the platform said about why went straight into the discard pile.

Calling a Backend by Hand noticed that send returns a richer error type than everything around it, said it "exists to say more," and moved on. This post goes back for it. It's the first of a few that return to parts of the HTTP interfaces the series walked past on the way to everything else.

What a failed send hands back

Three functions in http-req fail with error-with-detail instead of the plain error: send, send-uncached, and await-response, the one that finishes what send-async started. Here's the record:

WIT
/// An `error` code, optionally with extra request error information.
record error-with-detail {
    detail: option<send-error-detail>,
    error: error,
}

error is the same types.error variant every other function returns, so a caller that only knows about that type still gets something. detail is optional, and it's where the information lives.

send-error-detail

WIT
/// Information about errors encountered by sent requests.
variant send-error-detail {
    /// The system encountered a timeout when trying to find an IP address for the backend
    /// hostname.
    dns-timeout,
    /// The system encountered a DNS error when trying to find an IP address for the backend
    /// hostname.
    dns-error(dns-error-detail),1
    /// The system cannot determine which backend to use, or the specified backend was invalid.
    destination-not-found,
    /// The system considers the backend to be unavailable, for example when recent attempts to
    /// communicate with it may have failed, or a health check may indicate that it is down.
    destination-unavailable,
    /// The system cannot find a route to the next-hop IP address.
    destination-ip-unroutable,
    /// The system's connection to the backend was refused.
    connection-refused,
    /// The system's connection to the backend was closed before a complete response was
    /// received.
    connection-terminated,
    /// The system's attempt to open a connection to the backend timed out.
    connection-timeout,
    /// The system is configured to limit the number of connections it has to the backend, and
    /// that limit has been exceeded.
    connection-limit-reached,
    /// The system encountered an error when verifying the certificate presented by the backend.
    tls-certificate-error,
    /// The system encountered an error with the backend TLS configuration.
    tls-configuration-error,
    /// The system received an incomplete response to the request from the backend.
    http-incomplete-response,
    /// The system received a response to the request whose header section was considered too
    /// large.
    http-response-header-section-too-large,
    /// The system received a response to the request whose body was considered too large.
    http-response-body-too-large,
    /// The system reached a configured time limit waiting for the complete response.
    http-response-timeout,
    /// The system received a response to the request whose status code or reason phrase was
    /// invalid.
    http-response-status-invalid,
    /// The process of negotiating an upgrade of the HTTP version between the system and the
    /// backend failed.
    http-upgrade-failed,
    /// The system encountered an HTTP protocol error when communicating with the backend.
    ///
    /// This error will only be used when a more specific one is not defined.
    http-protocol-error,
    /// An invalid cache key was provided for the request.
    http-request-cache-key-invalid,
    /// An invalid URI was provided for the request.
    http-request-uri-invalid,
    /// The system encountered an unexpected internal error.
    internal-error,
    /// The system received a TLS alert from the backend.
    tls-alert-received(tls-alert-received-detail),1
    /// The system encountered a TLS error when communicating with the backend, either during
    /// the handshake or afterwards.
    tls-protocol-error,
    /// The system received an HTTP/2 error frame and code from the backend.
    h2-error(h2-error-detail),1
    /// Additional error details may be added in the future via this resource type.
    extra(extra-send-error-detail),2
}

It's a long list, and it follows the life of a request. The name has to resolve (dns-*), the platform has to be willing and able to go there (destination-*), a connection has to open (connection-*), TLS has to agree (tls-*), and then the response has to arrive whole, on time, and well-formed (http-response-*). A few cases are about the request you built rather than the network: an invalid URI, an invalid cache key.

Most cases are bare names. Three carry a record, holding codes that some protocol already defines for that failure:

WIT
/// Variant fields for `send-error.dns-error`.
record dns-error-detail {
    rcode: option<u16>,
    info-code: option<u16>,
}

/// Variant fields for `send-error.tls-alert-received`.
record tls-alert-received-detail {
    id: option<u8>,
}

/// Variant fields for `send-error.h2-error`.
record h2-error-detail {
    frame-type: u8,
    error-code: u32,
}

rcode is the DNS response code, and info-code is an extended DNS error (opens in a new window) code. id is a TLS alert (opens in a new window) number. The HTTP/2 pair are a frame type and error code (opens in a new window). None of them is a Fastly invention, so the registries that define them are what you look them up in. The DNS and TLS fields are all option, since the platform may know that a lookup failed without having a code to report.

Those doc comments say send-error.dns-error, and nothing in compute.wit is called send-error. The variant is send-error-detail. It's harmless, but it means all three doc comments point at a type you won't find.

The last case, extra, holds a handle to a resource with nothing in it. Hold on to that one: it's there for a future version of this variant, and it's the subject of Beyond the WIT below.

Making things fail on purpose

Full working code: full example on GitHub (opens in a new window).

The example registers a dynamic backend per case, each set up to fail in one particular way, sends a GET through it, and reports what came back. The cases:

Case Target What's wrong
ok http-me.fastly.dev Nothing, for comparison
dns no-such-host.invalid .invalid is reserved and can never resolve
reserved-ip 192.0.2.1:80 A documentation-only address
silent-port example.com:81 A port that drops connection attempts without answering
expired-cert expired.badssl.com A certificate that expired in 2015
old-tls http-me.fastly.dev tls-max-version capped at TLS 1.0
slow-origin http-me.fastly.dev /wait=3000 takes three seconds to answer, with a one-second first-byte-timeout

Here's the part that handles a failed send-uncached:

Rust

mod bindings;

use std::time::Instant;

use bindings::{
    exports::fastly::compute::http_incoming,
    fastly::compute::{backend, http_body, http_req, http_resp},
};

/// One way for a backend request to go wrong, set up on purpose.
struct Case {
    name: &'static str,
    target: &'static str,
    uri: &'static str,
    use_tls: bool,
    tweak: fn(&backend::DynamicBackendOptions),
}

fn no_tweak(_: &backend::DynamicBackendOptions) {}

const CASES: &[Case] = &[
    // A request that works, for comparison.
    Case {
        name: "ok",
        target: "http-me.fastly.dev",
        uri: "/anything",
        use_tls: true,
        tweak: no_tweak,
    },
    // .invalid is reserved (RFC 2606), so this name can never resolve.
    Case {
        name: "dns",
        target: "no-such-host.invalid",
        uri: "/",
        use_tls: true,
        tweak: no_tweak,
    },
    // 192.0.2.1 is TEST-NET-1 (RFC 5737), reserved for documentation.
    Case {
        name: "reserved-ip",
        target: "192.0.2.1:80",
        uri: "/",
        use_tls: false,
        tweak: |o| o.connect_timeout(1_000),
    },
    // A real host, on a port that drops connection attempts unanswered.
    Case {
        name: "silent-port",
        target: "example.com:81",
        uri: "/",
        use_tls: false,
        tweak: |o| o.connect_timeout(1_000),
    },
    // badssl.com serves deliberately broken certificates. cert-hostname
    // turns on the hostname check, as its doc comment recommends.
    Case {
        name: "expired-cert",
        target: "expired.badssl.com",
        uri: "/",
        use_tls: true,
        tweak: |o| o.cert_hostname("expired.badssl.com"),
    },
    // Offer nothing newer than TLS 1.0 to a server that won't speak it.
    Case {
        name: "old-tls",
        target: "http-me.fastly.dev",
        uri: "/",
        use_tls: true,
        tweak: |o| o.tls_max_version(0x0301),
    },
    // http-me waits 3s before answering; we only give it 1s.
    Case {
        name: "slow-origin",
        target: "http-me.fastly.dev",
        uri: "/wait=3000",
        use_tls: true,
        tweak: |o| o.first_byte_timeout(1_000),
    },
];

fn describe(detail: &http_req::SendErrorDetail) -> String {
    use http_req::SendErrorDetail as D;
    match detail {
        D::DnsError(d) => format!("dns-error (rcode {:?}, info-code {:?})", d.rcode, d.info_code),3
        D::TlsAlertReceived(d) => format!("tls-alert-received (id {:?})", d.id),3
        D::H2Error(d) => format!(
            "h2-error (frame-type {}, error-code {})",
            d.frame_type, d.error_code
        ),3
        D::Extra(_) => "extra (a case this guest's bindings predate)".to_string(),
        other => format!("{other:?}"),
    }
}

fn run(case: &Case) -> String {
    let options = backend::DynamicBackendOptions::new();
    options.use_tls(case.use_tls);
    (case.tweak)(&options);

    let backend = match backend::register_dynamic_backend(case.name, case.target, options) {
        Ok(b) => b,
        Err(e) => return format!("register-dynamic-backend failed: {e:?}"),
    };


    let request = http_req::Request::new().expect("request.new");
    request.set_method("GET").expect("set-method");
    request.set_uri(case.uri).expect("set-uri");
    let body = http_body::new().expect("body.new");

    let started = Instant::now();
    let result = http_req::send_uncached(request, body, &backend);
    let ms = started.elapsed().as_millis();

    match result {
        Ok((response, _body)) => {
            let status = response.get_status().expect("get-status");
            format!("ok, status {status} ({ms} ms)")
        }
        Err(http_req::ErrorWithDetail { detail, error }) => {
            let detail = match &detail {
                Some(d) => describe(d),
                None => "none".to_string(),
            };
            format!("error: {error:?}, detail: {detail} ({ms} ms)")
        }
    }
}

The three cases with payloads get their fields spelled out, and everything else falls through to the Debug output wit-bindgen generates.

The handler, not shown, calls run once per case and writes one line per result into the response. The example uses send-uncached so that the readthrough cache stays out of the picture and every attempt really goes to the network.

Running it

Under Viceroy, with fastly compute serve and a plain curl http://127.0.0.1:7676/:

Terminal output
ok               ok, status 200 (41 ms)
dns              error: Error::GenericError, detail: none (2 ms)
reserved-ip      error: Error::GenericError, detail: none (15000 ms)
silent-port      error: Error::GenericError, detail: none (15001 ms)
expired-cert     error: Error::GenericError, detail: none (379 ms)
old-tls          ok, status 200 (32 ms)
slow-origin      error: Error::GenericError, detail: none (1001 ms)

Every failure is generic-error, and detail is none every time. That isn't a gap in the example. Viceroy doesn't populate send-error-detail at all, on either of its ABI surfaces. It's a known, open issue, fastly/Viceroy#727 (opens in a new window), and until it's fixed a local run can't tell one failure from another.

Two smaller differences show up in the timings. Viceroy doesn't apply connect-timeout, so the two connection cases ran until the default 15-second first-byte timeout caught them, and it doesn't apply the TLS version cap, so old-tls succeeded. Both are among the dynamic backend options tracked as unimplemented in fastly/Viceroy#342 (opens in a new window).

The same code, published to a real Compute service:

Terminal output
ok               ok, status 200 (29 ms)
dns              error: Error::GenericError, detail: dns-error (rcode Some(3), info-code None) (1 ms)
reserved-ip      error: Error::GenericError, detail: SendErrorDetail::ConnectionRefused (0 ms)
silent-port      error: Error::GenericError, detail: SendErrorDetail::ConnectionTimeout (1000 ms)
expired-cert     error: Error::GenericError, detail: SendErrorDetail::TlsCertificateError (443 ms)
old-tls          error: Error::GenericError, detail: tls-alert-received (id Some(70)) (10 ms)
slow-origin      error: Error::GenericError, detail: SendErrorDetail::HttpResponseTimeout (1000 ms)

Now every failure has a name:

  1. dns is dns-error with rcode 3, which the DNS registry calls NXDOMAIN: the name doesn't exist.
  2. reserved-ip is connection-refused, in under a millisecond. That's too fast for a round trip to anywhere, so the refusal came from the platform itself and not from anything at 192.0.2.1. In practice, that's what an address from a reserved documentation range gets.
  3. silent-port is connection-timeout, at 1,000 milliseconds, exactly the connect-timeout the case set. Nothing answered, so nothing refused.
  4. expired-cert is tls-certificate-error. This case sets cert-hostname, which the WIT recommends whenever you use TLS; the backend-tuning post later in this run looks at the TLS options properly.
  5. old-tls is tls-alert-received with alert 70, which the TLS registry names protocol_version: the server wouldn't speak TLS 1.0 and said so.
  6. slow-origin is http-response-timeout, at one second. The connection opened and the request went out, but no response started arriving in time.

And error is generic-error on every single line. On a real service, as locally, the outer error says nothing more than "it failed". The information is entirely in detail.

Fastly's Rust SDK hands this to you as a SendError (opens in a new window), whose root_cause() returns a SendErrorCause (opens in a new window). That enum has one variant per case above, with the payload fields as named struct fields (DnsError { rcode, info_code }, TlsAlertReceived { alert_id }). It also adds variants the ABI doesn't have, such as FanoutNotEnabled and IoError, because a SendError covers everything that can go wrong in the SDK's send, not only what the host reported. SendError::into_sent_req() gives you back the request's headers and metadata, since the body was consumed by the attempt.

Beyond the WIT

The first lesson for a binding is which field to build on. It's tempting to map types.error to an exception or error type and attach the detail as an afterthought, because error is the field every function has. For a send, that's backwards. error was generic-error for six different failures, and detail told them apart. A binding's send error should be keyed on the detail, and fall back to error only when the detail is none.

The second is what to do with extra. Your bindings were generated from one version of compute.wit, and the platform your code runs on may be newer. If Fastly ever starts reporting a failure that didn't exist when you built, it arrives as extra, and your code has to decide what that means with nothing to go on.

There are two defensible answers. The Rust SDK folds extra into its InternalError cause, so an unrecognized failure is reported as a failure and nothing more, and the application carries on handling it like any other. The other answer is to throw an error that says the bindings are out of date, on the reasoning that a host reporting something the guest can't name is a sign the guest should be rebuilt, and a quiet InternalError hides that. The first answer suits code in production that needs to keep serving. The second is better for finding out that your bindings are stale before a user does.

Either way, extra only helps because it was there from the start. Why a WIT author reserves an empty resource instead of just adding a case later is a story about how an ABI changes without breaking the code built against it, and compute.wit has several of these placeholders. We'll get to that story when one of them finally gets filled in.

Next: headers. Every post so far has set headers one at a time and only read the ones it already knew the names of, and HTTP has always allowed more than that.