ttrpc#

The containerd ttrpc protocol in pure OCaml: "GRPC for low-memory environments".

ttrpc is protobuf RPC over a raw stream socket -- a fixed 10-byte frame header, then the Request / Response protobuf messages with the google.rpc.Status error model. It is designed for low-latency, same-host connections (unix, vsock or tcp sockets) where HTTP/2 is too heavy, and is the protocol of containerd shims, the Kata agent and the vAccel RPC agent.

Two packages:

• ttrpc -- the I/O-free core: Packet framing, typed Messages, and pure Client / Server state machines (bytes in, RPC events and bytes out; no sockets, clocks passed as arguments).
• ttrpc-eio -- the Eio adapter: a client multiplexing concurrent unary calls over one connected flow, and a server dispatching typed handlers, one fiber per call.

Unary calls only; protocol version 1.2 streaming is not implemented.

Installation#

Install with opam:

$ opam install ttrpc

If opam cannot find the package, it may not yet be released in the public opam-repository. Add the overlay repository, then install it:

$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
$ opam update
$ opam install ttrpc

The pure machines#

The state machines run without any I/O backend, so a complete call fits in memory: the client produces the request bytes, the server consumes them and surfaces a call event, and the reply travels back the same way.

# let client, id, request =
    Result.get_ok
      (Ttrpc.Client.call (Ttrpc.Client.v ())
         ~now:(Mtime.of_uint64_ns 0L)
         ~service:"test.Echo" ~meth:"Echo" "ping")
val client : Ttrpc.Client.t = <abstr>
val id : int = 1
val request : Rope.t = <abstr>
# let pp_events = List.map (Fmt.str "%a" Ttrpc.Event.pp)
val pp_events : Ttrpc.Event.t list -> string list = <fun>
# let server, calls =
    let server, _auto, events =
      Result.get_ok
        (Ttrpc.Server.incoming (Ttrpc.Server.v ()) (Rope.to_string request))
    in
    (server, pp_events events)
val server : Ttrpc.Server.t = <abstr>
val calls : string list = ["call stream=1 test.Echo/Echo len=4 timeout=none"]
# let _server, reply = Result.get_ok (Ttrpc.Server.reply server ~id (Ok "pong"))
val _server : Ttrpc.Server.t = <abstr>
val reply : Rope.t = <abstr>
# let replies =
    let _client, events =
      Result.get_ok (Ttrpc.Client.incoming client (Rope.to_string reply))
    in
    pp_events events
val replies : string list = ["reply stream=1 OK(0) len=4"]

The Eio adapter#

ttrpc-eio wires the machines to a connected flow. A method is described by its service-qualified name and hand-written nox-protobuf codecs; Ttrpc_eio.Client.call then issues typed unary calls (with optional deadlines), and Ttrpc_eio.Server dispatches registered handlers, one fiber per call. test/eio/test_ttrpc_eio.ml runs a typed echo client against the server over a socketpair -- the end-to-end example.

Spec#

• PROTOCOL.md -- the frame format, message types, flags and stream rules. The test suite cites its sections.
• request.proto -- the Request / Response schemas.

test/interop/ttrpc-go/ replays wire traces recorded from a real containerd/ttrpc client and server, and asserts our encoders produce byte-identical frames (REGEN=1 dune build @traces refreshes the traces against ttrpc-go).

Related work#

• containerd/ttrpc -- the reference Go implementation.
• containerd/ttrpc-rust -- the Rust implementation used by the vAccel RPC agent.
• nox-grpc -- gRPC over HTTP/2 in this tree; ttrpc reuses its Grpc.Status code set so statuses flow between the two layers.

License#

ISC. See LICENSE.md.