http-proto#

Pure-OCaml HTTP/1.1, HTTP/2, and WebSocket codecs for OCaml 5.x.

The libraries parse and serialise the protocols and run the per-connection protocol state machines. They are sans-I/O: they perform no reads or writes themselves. You feed them bytes received from a transport and drain the bytes they want to send, so they work under any runtime (Eio, Lwt, Miou, blocking Unix, or an in-memory test harness).

The package has no dependencies outside the OCaml standard library and no C code.

Bodies are handled with stdlib Bigarray bigstrings end to end: a connection reads from a bigstring and writes a vector of bigstring/bytes regions, so a body moves between the application and the socket without intermediate copies.

Scope: a codec library, not a server#

This repository is a codec library. It does not provide a server: protocol negotiation, the accept loop, socket setup, timeouts, TLS, routing, logging, and process lifecycle are written by the consuming application on top of these libraries.

Layout#

One opam package (http-proto) with four wrapped libraries. Each library's public modules:

lib/    the libraries
test/   testo / qcheck-core suite
bench/  throughput benchmarks

Build and test#

dune build
dune runtest
dune exec --profile release bench/bench.exe

The sans-I/O model#

Each connection — Http1.Server_connection, Http2.Server_connection, Websocket.Connection — exposes the same operations:

• read t bigstring ~off ~len — hand it bytes received from the transport, in a Http_core.Bigstring.t (an Eio Cstruct.buffer maps directly; otherwise blit into one).
• next_write_operation t — what to send next: Writev iovs (a list of Http_core.Iovec.t regions for one vectored write), Yield (nothing to send right now), or Close.
• report_write_result t n — how many bytes of the last Writev you sent (handles partial writes; the connection advances a cursor across the iovecs).
• read_eof t — the peer closed its side.
• is_closed t — the connection is finished and fully flushed.

The connection owns its buffers (Http_core.Iobuf), the parser, and the serialiser. You own the socket and the loop.

Usage#

HTTP/1.1 over a blocking socket#

Http1.Server_connection.create services each request with a request -> response handler. The request body is delivered whole as an owned bigstring (req.body : Http_core.Bigstring.t); the response body is Fixed s (a string), Bigstring b (sent by reference, no copy), or a Stream.

For large uploads, use Http1.Server_connection.create_stream instead. It invokes a reqd -> unit handler as soon as the request head arrives, before the body is read. Register schedule_read callbacks to receive owned body chunks as they are parsed, and call respond when ready:

open Http_core
module H1 = Http1.Server_connection

let upload reqd =
  let received = ref 0 in
  H1.schedule_read reqd
    ~on_data:(fun chunk -> received := !received + Bigstring.length chunk)
    ~on_eof:(fun () ->
      H1.respond reqd
        {
          H1.status = Status.ok;
          headers = Headers.empty;
          body = H1.Fixed (string_of_int !received);
        })

open Http_core
module H1 = Http1.Server_connection

let handle (req : H1.request) : H1.response =
  ignore req;
  {
    H1.status = Status.ok;
    headers = Headers.of_list [ ("content-type", "text/plain") ];
    body = H1.Fixed "hello\n";
  }

(* Drive one connection on a blocking Unix socket. The library does no I/O;
   this read/write loop is yours to write (or replace with a runtime). A real
   runtime (Eio) hands [read] its receive Cstruct's bigarray and writes the
   bigstring iovecs directly; the stdlib [Unix] has no bigarray I/O, so this
   example copies at the boundary. *)
let serve_connection fd =
  let conn = H1.create handle in
  let inbuf = Bytes.create 65536 in
  (* write the iovecs in order, stopping at the first short write; return the
     bytes actually written so the connection advances its cursor *)
  let rec write_iovs written = function
    | [] -> written
    | iov :: rest ->
        let b, off, len =
          match (iov : Iovec.t) with
          | Iovec.Bytes (b, off, len) -> (b, off, len)
          | Iovec.Bigstring (b, off, len) ->
              (Bytes.unsafe_of_string (Bigstring.sub_string b off len), 0, len)
        in
        let n = Unix.write fd b off len in
        if n < len then written + n else write_iovs (written + n) rest
  in
  let rec flush () =
    match H1.next_write_operation conn with
    | H1.Writev iovs ->
        H1.report_write_result conn (write_iovs 0 iovs);
        flush ()
    | H1.Yield | H1.Close -> ()
  in
  let rec loop () =
    flush ();
    if H1.is_closed conn then ()
    else
      let n = Unix.read fd inbuf 0 (Bytes.length inbuf) in
      if n = 0 then (
        H1.read_eof conn;
        flush ())
      else (
        let src = Bigstring.of_string (Bytes.sub_string inbuf 0 n) in
        ignore (H1.read conn src ~off:0 ~len:n);
        loop ())
  in
  loop ()

HTTP/2 (streaming, push-based)#

Http2.Server_connection is streaming. Its handler is reqd -> unit, invoked the moment a request's headers arrive. The request body is pushed to callbacks registered with schedule_read (a sans-I/O connection cannot block to pull), and the response is sent with respond. A response body is a whole string, a bigstring sent by reference, or a producer drained under flow control:

type chunk = Data of string | End of Headers.t   (* End carries trailers *)
type body  =
  | Fixed of string
  | Bigstring of Http_core.Bigstring.t            (* sent by reference, no copy *)
  | Stream of (unit -> chunk)

module H2 = Http2.Server_connection

(* echo: collect the request body, then send it straight back *)
let handler reqd =
  let buf = Buffer.create 256 in
  H2.schedule_read reqd
    ~on_data:(fun chunk ->
      (* chunk is an owned [Http_core.Bigstring.t] for this delivery *)
      Buffer.add_string buf
        (Http_core.Bigstring.sub_string chunk 0 (Http_core.Bigstring.length chunk)))
    ~on_eof:(fun _trailers ->
      H2.respond reqd
        {
          H2.status = Http_core.Status.ok;
          headers = Http_core.Headers.empty;
          body = H2.Fixed (Buffer.contents buf);
        })

let conn = H2.create handler

on_data is called with each body chunk (an owned bigstring) in order; on_eof once at END_STREAM with any trailers. A Stream producer is called repeatedly until it returns End, whose trailers (use Headers.empty for none) are emitted as a final HEADERS frame. Read the request head with H2.meth reqd, H2.path reqd, H2.headers reqd, etc. The connection is driven over a socket with the same read / next_write_operation loop as HTTP/1.

WebSocket#

The handshake is ordinary HTTP/1.1: detect the Upgrade with Websocket.Handshake.upgrade_key, answer 101 with Websocket.Handshake.response_headers, then hand the post-handshake bytes (Http1.Server_connection.upgraded_input) to a Websocket.Connection for the data phase. The connection reassembles fragmented messages, answers Ping with Pong, echoes Close, and validates RSV bits, opcodes, fragmentation, UTF-8 and close codes.

module WS = Websocket.Connection

let echo conn kind payload =
  match kind with
  | `Text -> WS.send_text conn payload
  | `Binary -> WS.send_binary conn payload

let ws = WS.create echo   (* then drive it like the other connections *)

Configuration#

create takes optional bounds. The connection enforces them; the caller chooses the values (defaults shown).

HTTP/1 (Http1.Server_connection.create):

HTTP/2 (Http2.Server_connection.create):

Behaviour notes#

• HTTP/1.1 parsing requires strict CRLF line endings (bare CR or LF is rejected), requires exactly one valid Host field, rejects whitespace before the field-name colon and obs-fold continuation lines, and rejects messages that carry both Transfer-Encoding and Content-Length. Line and header-value scans are byte-at-a-time over the input bigstring, allocation-free.
• Responses with no body by definition (HEAD, 1xx, 204, 304, and a 2xx to CONNECT) are serialised without a body.
• Before serialising, each response field is checked: names must be non-empty tokens (no :, no control bytes), values carry no CR/LF/NUL and no leading/trailing whitespace. Fields that fail are dropped.
• HTTP/2 maintains the stream state machine, connection/stream flow control, HEADERS + CONTINUATION reassembly, and HPACK with a dynamic table; closed streams are dropped from the stream table.
• WebSocket validates reserved bits, opcodes, fragmentation, control-frame constraints, UTF-8 (text payloads and close reasons), and close codes.

Not implemented#

• HTTP/2 server PUSH_PROMISE (rejected) and PRIORITY scheduling (PRIORITY frames are parsed and ignored).
• WebSocket permessage-deflate.
• TLS and ALPN, and h1/h2 protocol negotiation — transport/server concerns, outside this library.

Testing#

dune runtest runs the unit, property, allocation, and regression suites in test/:

• test/test_conformance.ml — drives the sans-I/O connections in memory to cover protocol/limit/streaming behaviour (mirrors h2spec / Autobahn cases).
• test/test_fuzz.ml — randomised (qcheck) robustness checks: every parser is total, bounded, and produces the same output whether the byte stream is fed whole or split at an arbitrary point.
• test/test_smuggling.ml — a request-smuggling corpus fed to Http1.Server_connection, asserting desync/smuggling vectors are rejected and that a malformed chunked body closes the connection rather than serving a pipelined request.

fuzz/ holds an AFL-compatible harness for the same parser entry points, for coverage-guided fuzzing under afl-fuzz (see fuzz/README.md).