nandi.uk
Fork of daniellemaywood.uk/gleam — Wasm codegen work
1<!--
2 SPDX-License-Identifier: Apache-2.0
3 SPDX-FileCopyrightText: 2021 The Gleam contributors
4-->
5
6# Compiler documentation
7
8Hello! Welcome to the documentation for the Gleam compiler. I hope you have fun
9with the project!
10
11<!-- vscode-markdown-toc -->
12* [Project structure](#project-structure)
13* [Compilation flow](#compilation-flow)
14* [Testing](#testing)
15 * [Running the tests](#running-the-tests)
16 * [Snapshot testing](#snapshot-testing)
17
18<!-- vscode-markdown-toc-config
19 numbering=false
20 autoSave=true
21 /vscode-markdown-toc-config -->
22<!-- /vscode-markdown-toc -->
23
24## <a name='Projectstructure'></a>Project structure
25
26The project is made up of several Rust crates (projects):
27
28- `compiler-core`: This project parses, analyses, and compiles Gleam projects.
29 It is entirely pure and has no IO so that is provided by the other Rust crates
30 that wrap this one.
31- `language-server`: This project contains code for the Gleam language server,
32 including autocomplete, code actions, hover and other features.
33- `compiler-cli`: A command line interface that wraps the core compiler and
34 language server, providing IO to files and to the console.
35- `compiler-wasm`: A JavaScript interface to the core compiler via web assembly.
36 Suitable for running in a web browser.
37
38In addition to the Rust code there are these components:
39
40- `Makefile`: A makefile that defines shortcut commands for common tasks when
41 working in the Gleam codebase. Run `make help` to view them.
42- `test`: A collection of (mostly) Gleam projects that serve as integration tests for the
43 compiler.
44- `deny.toml`: Configuration for the `cargo-deny` tool which is used to ensure
45 that the Rust libraries the compiler depends on adhere to our expectations.
46- `containers`: A collection of docker-files used to produce OCI containers for
47 each Gleam release.
48- `.github/workflows`: GitHub Actions workflow definitions, used to build, test,
49 and release new versions of the project.
50- `docs`: You're looking at it pal.
51
52## <a name='Compilationflow'></a>Compilation flow
53
54The process for compiling Gleam modules within the compiler looks roughly like
55this:
56
57```txt
58 Gleam source code .cache binaries
59 ▼ ▼
60┌────────────────────┐ ┌───────────────────────┐
61│ Parser │ │ Metadata deserializer │
62└────────────────────┘ └───────────────────────┘
63 │ │
64 Untyped AST Module metadata
65 └─────────┐ ┌────────┘ │
66 ▼ ▼ │
67 ┌─────────────────────┐ │
68 │ Dependency sorter │ │
69 └─────────────────────┘ │
70 │ │
71 Untyped AST │
72 (sorted by deps) │
73 ▼ │
74 ┌───────────────────┐ │
75 │ Type checker │◄─────┘
76 └───────────────────┘
77 │
78 ┌────── Typed AST ──────┐
79 ▼ ▼
80┌────────────────────┐ ┌─────────────────────┐
81│ Code generator │ │ Metadata serializer │
82└────────────────────┘ └─────────────────────┘
83 │ │
84 │ ▼
85 Erlang or JavaScript .cache binaries
86 printing algebra
87 ▼
88┌────────────────────┐
89│ Pretty printer │
90└────────────────────┘
91 │
92 ▼
93 Erlang or JavaScript
94 source code
95```
96
97## <a name='Testing'></a>Testing
98
99We like automated tests! They're a great way to verify that the compiler is
100doing what we expect it do as we make changes.
101
102### <a name='Runningthetests'></a>Running the tests
103
104- `make test`: Run all the tests. Worth doing before committing any changes.
105- `make test-watch`: Run the Rust unit tests when files are saved.
106- `make language-test`: Run the cross-platform language integration tests in
107 `test/language`.
108- `make language-test-watch`: Run said tests when files are saved.
109- `make javascript-prelude-test`: Run the unit tests for the JavaScript code
110 that implements the Gleam prelude when compiling to JavaScript.
111- `make javascript-prelude-test-watch`: Run said tests when files are saved.
112
113The `*-watch` commands require the [`watchexec`][watchexec] program to be
114installed.
115
116### <a name='Snapshottesting'></a>Snapshot testing
117
118The compiler makes heavy use of snapshot testing using the
119[`cargo-insta`][cargo-insta] tool.
120
121[cargo-insta]: https://github.com/mitsuhiko/insta
122[watchexec]: https://github.com/watchexec/watchexec
123
124If you're not familiar with snapshot testing instead of writing an input and an
125expected output (as in normal example based tests) you write only the input. The
126snapshot testing tool can then be used to mark any new outputs as accepted and
127saved into the repository as snap files. If an output for one of the tests
128changes then it is considered a failed test and the programmer has the option to
129either reject the new version (as it is an incorrect result) or accept it as the
130new correct output.
131
132This style of testing saves us huge amounts of time as manually updating all the
133expected output when we make changes to the output format of the compiler or
134error messaging is time-consuming and very dull. With snapshot testing it takes
135seconds.
136
137```sh
138# Run the tests
139make test
140
141# Interactively verify changes to the snapshots
142cargo insta review
143```