Health Checks: Rules the Defaults Break

Two of healthcheck-options' setters state rules about each other. interval-ms must be "less than the current timeout_ms," and timeout-ms must "not be less than the current interval_ms." Their defaults are an interval of 15,000 milliseconds and a timeout of 5,000, which breaks both rules at once.

Backend Options tuned a dynamic backend and read its settings back. The one getter it had nothing to say about was is-healthy, which reports on what these setters configure, and it answered unknown for backends without a health check. A failing health check is also one way to get destination-unavailable from Send Errors, whose doc comment includes the case where "a health check may indicate that it is down."

Adding a health check

A health check is a request Fastly sends to a backend on a schedule, to find out whether it's working before your traffic does. dynamic-backend-options takes one:

/// Adds a created healthcheck builder to this dynamic backend to be used when checking if the backend
/// is healthy. A healthcheck can be created with the default HealthcheckBuilder::new(host: String)
/// and additional options can be set through that.
healthcheck: func(value: healthcheck-options);

That doc comment names HealthcheckBuilder::new(host: String), which is the Rust SDK's type, not anything in the WIT. The WIT's own version is healthcheck-options, which Dynamic Backends by Hand quoted as far as its constructor and its host, method, and path setters. The rest of it sets how the check is judged and how often it runs:

/// Options for `register-dynamic-backend` with healthchecks
resource healthcheck-options {
  ...

  /// Set the number of most recent health check queries to keep.
  /// Must not be greater than 15.
  ///
  /// Defaults to 5.
  window: func(value: u32) -> result<_, error>;

  /// Set the number of health checks that must be successful within the window
  /// to be considered healthy. Must not be greater than the current window.
  ///
  /// Defaults to 3.
  threshold: func(value: u32) -> result<_, error>;

  /// Set the number of successes to assume are successful when beginning a health check.
  /// Must not be greater than the current window.
  ///
  /// Defaults to 4.
  initial: func(value: u32) -> result<_, error>;

  /// Set the interval in milliseconds where a health check should be performed.
  /// Times must be 1 second to 1 hour inclusive. Must be less
  /// than the current timeout_ms.
  ///
  /// Defaults to 15000ms.
  interval-ms: func(value: u64) -> result<_, error>;

  /// Set the time in milliseconds in which to perform the health check.
  /// Note that querying the health check renews the timer. Must not be less
  /// than the current interval_ms.
  ///
  /// Defaults to 5000ms.
  timeout-ms: func(value: u64) -> result<_, error>;
}

Its result goes to is-healthy on the backend:

enum backend-health {
  unknown,
  healthy,
  unhealthy,
}

/// Returns the health of the backend if configured and currently known.
///
/// For backends without a configured healthcheck, this will always return
/// `backend-health.unknown`.
is-healthy: func() -> result<backend-health, error>;

Unlike the dynamic-backend-options setters, these return result, and the doc comments state rules to break. Several depend on "the current" value of another setting, which means the order you call them in matters. The interval and timeout rules are two of those, and they're the two the defaults break.

Which rules are enforced

Full working code: full example on GitHub (opens in a new window).

The example's /rules path breaks each rule on a fresh healthcheck-options and reports what happened:

The first row confirms what Dynamic Backends by Hand found: neither host rejects a method with a space in it. The two hosts agree on the absolute limits, a window of at most 15 and an interval of at least a second. They disagree about the rules that compare one setting to another: Viceroy enforces the window's relationship to the threshold and the initial count, and the real service doesn't, at least not when the setter is called. Neither enforces the interval and timeout rules, which is just as well, given that the defaults break them.

The example's own health check ran straight into the ordering rule under Viceroy. It wanted a window of 2, a threshold of 1, and an initial count of 0. Setting the window first failed, because the threshold was still at its default of 3. Lowering the threshold and the initial count first, and then the window, works on both hosts.

Watching is-healthy

The /health path registers two health-checked backends, one checking a path that answers 200 and one checking a path that answers 503, and watches is-healthy for six seconds. Under Viceroy, both registrations succeeded, both backends stayed unknown the whole time, and a request through the "unhealthy" one went through fine: Viceroy doesn't run the checks. On the real service, both registrations failed with unsupported. The same backend without a health check registers normally.

That isn't a mistake in the example. The Rust SDK's documentation for its dynamic-backend health checks says the feature "is not available to all customers / on all services," and suggests contacting Fastly to try it. On a static backend, a health check is configured on the service itself, and is-healthy reports its result. Without one, as the /static output in Backend Options showed, it's unknown.

Beyond the WIT

The two hosts took one list of rules and enforced different parts of it at different moments. The absolute limits were checked when the setter was called, everywhere. The rules relating one setting to another were checked when the setter was called in Viceroy, and not at that point on the real service. The rules that conflict with the defaults weren't checked by anyone.

For a binding, setters with order-dependent rules make a poor interface to pass straight through. A caller who wants a smaller window has to know to lower the threshold first, and one who wants a larger threshold has to raise the window first. The order depends on the direction of the change, which is exactly the kind of thing an API should hide.

The Rust SDK's HealthcheckBuilder goes partway around it. Its setters don't call the host. They store plain values in the guest, so a program can set them in any order. What happens next depends on the build. Built for Fastly's older, pre-component ABI, the SDK checks the finished configuration as a whole (the window against its limit, the threshold and the initial count against the window, the interval and the timeout against their ranges) and passes everything to the host at once. Built as a component, it skips that check and calls the WIT setters one at a time, in a fixed order: the window first, then the threshold, then the initial count.

A fixed order can't be right for every configuration, because the safe order depends on whether the window is growing or shrinking. A binding that holds the values in the guest has everything it needs to do better: compare the finished configuration against the documented rules, report a broken rule by name, and only then call the host setters in whichever order is safe for those particular values, instead of surfacing an invalid-argument from whichever setter happened to run first.

Next: compute.wit changed while this series was being written, and one of the changes finally fills in one of the empty extra resources.