The Same Tally, in MoonBit

MoonBit (opens in a new window) is a general-purpose language from the International Digital Economy Academy (IDEA), optimized for WebAssembly. If you've read Rust, MoonBit code will look familiar: static types, structs and enums, pattern matching, and methods declared as fn Type::name. What it leaves out is Rust's ownership model. MoonBit has a garbage collector (opens in a new window), so there are no lifetimes or borrows to satisfy. Besides JavaScript and native code, it has two Wasm backends: wasm-gc, built on the Wasm GC proposal, and wasm, which uses linear memory the way Rust's output does. This post uses wasm. wit-bindgen, the generator every Rust example in this series has used, has a MoonBit backend too.

Resources Aren't Host-Only ended on a claim: the tally component "could have been written in anything that targets components, and the service wouldn't change." Here's tally in MoonBit, with the Rust service left exactly as it was. The service really doesn't change. Getting the result onto a real Compute service takes one more step, though, and that step says something about strings.

One more import

Full working code: full example on GitHub (opens in a new window).

The tally interface and the service world are the same as last time. The world tally builds against has one new line. Here's the whole file, so you don't have to go back for it:

package local:main;

/// A running total, owned by whichever component implements this interface.
interface tally {
  resource tally {
    constructor(name: string);
    /// Adds `n` and returns the new total.
    add: func(n: u32) -> u32;
  }
}

/// The component that implements `tally`, plus the one WASI interface its log lines need.
world tally-provider {
  import wasi:cli/stderr@0.2.6;
  export tally;
}

/// A Compute service that uses `tally` without implementing it.
world service {
  import tally;
  import fastly:compute/http-body@0.1.0;
  import fastly:compute/http-resp@0.1.0;
  export fastly:compute/http-incoming@0.1.0;
}

tally-provider exports the resource, and service imports it and holds handles.

The Rust tally printed its log lines with eprintln!, and never had to ask for stderr. Rust's standard library brought it along, with twelve other WASI interfaces besides. MoonBit's Wasm output doesn't reach for WASI by itself, so if the component is going to write to stderr, the world has to import it. wasm-tools component wit shows the result:

  import wasi:io/error@0.2.6;
  import wasi:io/streams@0.2.6;
  import wasi:cli/stderr@0.2.6;
  export local:main/tally;

Three imports. wasi:cli/stderr hands back an output-stream, which is defined in wasi:io/streams. That interface uses both wasi:io/error and wasi:io/poll, but only error made the list: blocking-write-and-flush can fail with a stream-error, which carries an error, and nothing the component calls involves a pollable. That's the trimming Reading WIT Back Out of a Component found, where a component records the exact slice of each interface it uses.

The rep is yours

The Rust bindings asked for a type to stand behind the resource, and Tally was a struct holding the name and the total. wit-bindgen moonbit generates something else. Here's how the file it writes for the tally interface starts:

the generated header (2 lines)
// Generated by `wit-bindgen` 0.62.0. DO NOT EDIT!

///|
pub(all) struct Tally(Int32)

///|
/// Creates a new resource with the given `rep` as its representation and returning the handle to this resource.
pub fn Tally::new(rep : Int32) -> Tally {
  Tally::Tally(wasmExportResourceNewTally(rep))
}

///|
fn wasmExportResourceNewTally(rep : Int32) -> Int32 = "[export]local:main/tally" "[resource-new]tally"

If you've been reading the Rust in this series, most of this will look familiar. These are the pieces that MoonBit does its own way:

• ///| comes before every top-level item. moon fmt puts it there, and it's a comment, so it changes nothing about what the code does.
• struct Tally(Int32) is a struct with one unnamed field, like a Rust tuple struct, and Tally::Tally(...) constructs one. Int32 is a 32-bit integer, which MoonBit also spells Int.
• pub(all) lets other packages construct a Tally, not only read one.
• fn Tally::new is a method on Tally. MoonBit has no impl blocks, so the type's name goes in front of the method's.
• The last function has no body. One written as = "module" "name" is imported from outside the Wasm module, and the two strings are the module and field names of that import.

So Tally is a number, and Tally::new is the Component Model's resource.new: you hand it a rep, and it adds an entry to this component's handle table and hands back the index, as last time walked through. The file wraps resource.drop and resource.rep in two more methods built like this one, and then it ends with three functions it doesn't write:

///|
/// Destructor of the resource.
declare pub fn Tally::dtor(_self : Tally) -> Unit

///|
declare pub fn Tally::tally(name : String) -> Tally

///|
/// Adds `n` and returns the new total.
declare pub fn Tally::add(self : Tally, n : UInt32) -> UInt32

declare gives a function a signature and no body, and some other file in the package has to supply the body. These three are yours to write: the destructor, the constructor (named after the resource, Tally::tally), and add. They do the job the GuestTally trait did in Rust. In Rust, though, wit-bindgen chose the rep for you. Here, choosing it is your job, and so is turning it back into an object:

The reps below start at 1 and count up, and states is the map:

///|
/// What a tally is, on this side of the boundary. The component that holds
/// a handle never sees this; it only ever sees a number.
priv struct State {
  name : String
  mut total : UInt
}

///|
/// From each resource's rep to its state.
let states : Map[Int, State] = Map([])

///|
let next_rep : Ref[Int] = { val: 0 }

priv keeps State inside this package. A field can't be changed unless it's marked mut, and UInt is the short name for UInt32. Map([]) makes an empty map. A top-level let can't be reassigned, so the counter lives in a Ref, a box whose val field can be. { val: 0 } makes one.

The log lines go to stderr through the wasi:cli/stderr import:

///|
fn log(line : String) -> Unit {
  let bytes = @utf8.encode(line + "\n")
  let stream = @stderr.get_stderr()
  stream.blocking_write_and_flush(bytes.to_fixedarray()) |> ignore
  stream.drop()
}

A name starting with @ is another package. @utf8 is the UTF-8 package from MoonBit's core library, and @stderr is the bindings wit-bindgen generated for wasi:cli/stderr. x |> f is a pipe that calls f(x), and ignore throws away whatever it's given, so |> ignore discards the write's result the way let _ = would in Rust. The last line, stream.drop(), is there for a reason that comes up in Beyond the WIT.

With that, the three functions declare asked for:

///|
pub fn Tally::tally(name : String) -> Tally {
  next_rep.val += 1
  let rep = next_rep.val
  states[rep] = { name, total: 0 }
  log("tally: created \{name}")
  Tally::new(rep)
}

///|
pub fn Tally::add(self : Tally, n : UInt) -> UInt {
  let Tally(rep) = self
  let state = states[rep]
  state.total += n
  state.total
}

///|
pub fn Tally::dtor(self : Tally) -> Unit {
  let Tally(rep) = self
  let state = states[rep]
  log("tally: dropped \{state.name} at \{state.total}")
  states.remove(rep)
}

{ name, total: 0 } builds a State, which MoonBit works out from the map's type, and "\{name}" puts name into the string. let Tally(rep) = self takes the struct apart, as it would in Rust. Unlike in Rust, let state = states[rep] doesn't copy anything. A MoonBit struct is a garbage-collected object and a variable holds a reference to it, so state.total += n changes the state inside the map.

Look at the highlighted line. The constructor returns a Tally holding a handle, but the Tally that arrives in add and in the destructor holds the rep. The generated glue passes along whatever number the canonical ABI gives it, and when a component's own resource comes back in, through a method call or a drop, the canonical ABI gives it the rep. If you start the reps at 101 instead of 1 and log both numbers, you can see it (the created lines are left out here):

tally: rep 101, handle 1
tally: rep 102, handle 1
tally: add got 101
tally: add got 102
tally: add got 102

Both handles are 1. The constructor returns an owned handle, so each entry left tally's table as soon as it was returned, and the next constructor reused that slot. The reps are the numbers that stay attached to each tally, and they're what add receives.

It's one type holding two kinds of number. Here that's harmless, since the constructor never looks inside the Tally it returns, but a function that took a Tally from both places would have to know which one it had.

Same service, same log

Although we hadn't discussed it so far in this series, every component built in it has gone through two stages under the hood: first a core module, then a component made from it. In Rust, that happened automatically. On wasm32-wasip2, rustc hands linking to wasm-component-ld, which links a core module with wasm-ld and then uses wit-component to read the WIT embedded in that module and turn it into a component, all in one command. In MoonBit, both stages are in front of us, so let's take a deeper look.

Building the composed component takes one step per tool:

1. cargo build -p service builds the Rust service, unchanged from last time.
2. moon build --target wasm builds tally as a core Wasm module.
3. wasm-tools component embed attaches the tally-provider world to that module, in a component-type custom section like the ones Reading WIT Back Out of a Component found wit-bindgen adding for Rust. wasm-tools component new reads it back out and turns the module into a component.
4. wac plug composes the two, the same as last time.

The result imports eighteen interfaces, all in the service world from The service World. The service sends the same response as the all-Rust version:

bytes reached 42

And the log matches the all-Rust one, line for line:

tally: created requests
tally: created bytes
tally: dropped bytes at 42
service: dropped its handle to bytes
tally: dropped requests at 1

drop(bytes) in Rust ran Tally::dtor in MoonBit before the service printed its next line, exactly the way it ran Rust's Drop. Nothing on the service side can tell the difference. The MoonBit component is also small: a release build of it is 19 KB, against 79 KB for the Rust one.

The one objection

Step 3 hides a choice. MoonBit's tutorial for building components (opens in a new window) runs wasm-tools component embed with --encoding utf16, and that's what the generated glue expects. Built that way, the component ran in Viceroy without complaint. fastly compute publish stopped at the upload:

ERROR: the Fastly API returned 400 Bad Request: Bad request (Failed to extract metadata from package).

Stripping the component's custom sections didn't change that, and a composed Rust-and-Rust component, published the same way as this one, deployed fine. The MoonBit component, rebuilt with --encoding utf8, uploaded and deployed. I couldn't find string encodings mentioned anywhere in Fastly's documentation, so this is what the upload did, not a rule anyone has written down.

The encoding is part of the canonical ABI. When a string crosses from one component to another, the runtime lifts it out of the caller's memory and lowers it into the callee's. Each side's functions carry a string-encoding option, utf8, utf16, or latin1+utf16, saying how strings look in that component's memory. MoonBit strings are UTF-16, so with utf16 the runtime transcodes the Rust service's UTF-8 as it copies the constructor's name into tally's memory, and the generated glue can relabel those bytes as a String in place. With utf8, the runtime copies the bytes over unchanged and they need decoding, so the glue that receives name gets one hand-patched line:

pub fn wasmExportConstructorTally(p0 : Int32, p1 : Int32) -> Int32 {
  let result = @utf8.decode_lossy(mbt_ffi_ptr2bytes(p0, p1))

  let result0 : Tally = Tally::tally(result)

  let Tally(handle) = result0
  let ret = handle
  return ret
}

mbt_ffi_ptr2bytes relabels that allocation as Bytes instead of a String, and @utf8.decode_lossy makes a String from it. With that, the composed component deploys, and a real Compute service answers bytes reached 42. fastly log-tail shows the five lines Viceroy printed, word for word.

Beyond the WIT

For a binding, a string encoding is a choice made once per component, and the canonical ABI was designed so that each language can choose its own. Rust and C use utf8. A language whose strings are UTF-16, like MoonBit, Java, C#, or JavaScript, can use utf16 and never transcode its strings at all. When two components that chose differently call each other, the runtime converts between them at the boundary, and neither side has to know what the other chose.

Compute's upload accepted utf8 and refused utf16, so for now a binding for a UTF-16 language has to give up that advantage. It has to choose utf8 and transcode in the guest: decode on the way in, the way the patched line does, and encode on the way out. That costs a copy per string crossing the boundary, which is what utf16 was there to save.

The rep-versus-handle split is the other thing a binding has to get right, and the generated code above shows how easy it is to blur. The generated entry point for the destructor is wasmExportTallyDtor(handle : Int32), and the number the canonical ABI passes it is a rep, not a handle. A binding that keeps one type for both, as this one does, works as long as nothing uses the number for the wrong job. A binding that hands those numbers to user code needs two types.

The states map also settles the garbage-collection question from last time. The State objects stay alive because states holds them, and they go away when the destructor removes them. A destructor that forgot states.remove would leak, but it couldn't leave a valid handle pointing at nothing. Making the map something you write by hand turns that failure mode into one you can see.

The importing side leaks too, and MoonBit makes that leak easy to write. If you write log the obvious way, as @stderr.get_stderr().blocking_write_and_flush(...), every line leaks a handle. get-stderr returns an owned output-stream, and nothing drops it. In Rust, the binding's Drop would have released it at the end of the statement. MoonBit's binding is a struct around the handle with a drop method you have to call, and no finalizer. With that version of log, the experiment above prints handles 2 and 4 instead of 1 and 1, because the leaked streams are holding slots 1 and 3. Resources Aren't Host-Only said a finalizer can clean up a lost handle, but that only works if the binding has one. A binding for a garbage-collected language has to choose between an explicit drop that's easy to forget and a finalizer that runs whenever the collector gets to it.

MoonBit's import list makes one more difference concrete. The Rust components in this series import WASI @0.2.6 because that's the point release the pinned Rust toolchain settles on when it builds a component, and Hello, World described what a real service does with a component built by a newer one. MoonBit's compiler doesn't choose a WASI version at all. tally's three imports are @0.2.6 because the WIT in wit/deps is, and they'd stay that way through any number of moon upgrades. A binding for a language whose standard library isn't built on WASI picks its WASI imports, versions included, from the same vendored WIT as compute.wit. The flip side is that nothing picks them for it, so keeping them in step with Compute's service world is the binding's job.

Next: MoonBit's tally imports three interfaces, because it asked for one. Rust's standard library asks for a lot more on your behalf, and the next post finds the one call that's enough to ask for something the service world doesn't have.