This repository has no description
1# Jetstream
2
3Jetstream is a streaming service that consumes an ATProto `com.atproto.sync.subscribeRepos` stream and converts it into lightweight, friendly JSON.
4
5Jetstream converts the CBOR-encoded MST blocks produced by the ATProto firehose and translates them into JSON objects that are easier to interface with using standard tooling available in programming languages.
6
7### Public Instances
8
9As of writing, there are 4 official public Jetstream instances operated by Bluesky.
10
11| Hostname | Region |
12| --------------------------------- | ------- |
13| `jetstream1.us-east.bsky.network` | US-East |
14| `jetstream2.us-east.bsky.network` | US-East |
15| `jetstream1.us-west.bsky.network` | US-West |
16| `jetstream2.us-west.bsky.network` | US-West |
17
18Connect to these instances over WSS: `wss://jetstream2.us-west.bsky.network/subscribe`
19
20We will monitor and operate these instances and do our best to keep them available for public use by developers.
21
22Feel free to have multiple connections to Jetstream instances if needed. We encourage you to make use of Jetstream wherever you may consider using the `com.atproto.sync.subscribeRepos` firehose if you don't need the features of the full sync protocol.
23
24Because cursors for Jetstream are time-based (unix microseconds), you can use the same cursor for multiple instances to get roughly the same data.
25
26When switching between instances, it may be prudent to rewind your cursor a few seconds for gapless playback if you process events idempotently.
27
28## Running Jetstream
29
30To run Jetstream, make sure you have docker and docker compose installed and run `make up` in the repo root.
31
32This will pull the latest built image from GHCR and start a Jetstream instance at `http://localhost:6008`
33
34- To build Jetstream from source via Docker and start it up, run `make rebuild`
35
36Once started, you can connect to the event stream at: `ws://localhost:6008/subscribe`
37
38Prometheus metrics are exposed at `http://localhost:6009/metrics`
39
40A [Grafana Dashboard](#dashboard-preview) for Jetstream is available at `./grafana-dashboard.json` and should be easy to copy/paste into Grafana's dashboard import prompt.
41
42- This dashboard has a few device-specific graphs for disk and network usage that require NodeExporter and may need to be tuned to your setup.
43
44## Consuming Jetstream
45
46To consume Jetstream you can use any websocket client
47
48Connect to `ws://localhost:6008/subscribe` to start the stream
49
50The following Query Parameters are supported:
51
52- `wantedCollections` - An array of [Collection NSIDs](https://atproto.com/specs/nsid) to filter which records you receive on your stream (default empty = all collections)
53 - `wantedCollections` supports NSID path prefixes i.e. `app.bsky.graph.*`, or `app.bsky.*`. The prefix before the `.*` must pass NSID validation and Jetstream **does not** support incomplete prefixes i.e. `app.bsky.graph.fo*`.
54 - Regardless of desired collections, all subscribers receive Account and Identity events.
55 - You can specify at most 100 wanted collections/prefixes.
56- `wantedDids` - An array of Repo DIDs to filter which records you receive on your stream (Default empty = all repos)
57 - You can specify at most 10,000 wanted DIDs.
58- `maxMessageSizeBytes` - The maximum size of a payload that this client would like to receive. Zero means no limit, negative values are treated as zero. (Default "0" or empty = no maximum size)
59- `cursor` - A unix microseconds timestamp cursor to begin playback from
60 - An absent cursor or a cursor from the future will result in live-tail operation
61 - When reconnecting, use the `time_us` from your most recently processed event and maybe provide a negative buffer (i.e. subtract a few seconds) to ensure gapless playback
62- `compress` - Set to `true` to enable `zstd` [compression](#compression)
63- `requireHello` - Set to `true` to pause replay/live-tail until the server receives a [`SubscriberOptionsUpdatePayload`](#options-updates) over the socket in a [Subscriber Sourced Message](#subscriber-sourced-messages)
64
65### Examples
66
67A simple example that hits the public instance looks like:
68
69```bash
70$ websocat wss://jetstream2.us-east.bsky.network/subscribe\?wantedCollections=app.bsky.feed.post
71```
72
73A maximal example using all parameters looks like:
74
75```bash
76$ websocat "ws://localhost:6008/subscribe?wantedCollections=app.bsky.feed.post&wantedCollections=app.bsky.feed.like&wantedCollections=app.bsky.graph.follow&wantedDids=did:plc:q6gjnaw2blty4crticxkmujt&cursor=1725519626134432"
77```
78
79### Example events:
80
81Jetstream events have 3 `kinds`s (so far):
82
83- `commit`: a Commit to a repo which involves either a create, update, or delete of a record
84- `identity`: an Identity update for a DID which indicates that you may want to purge an identity cache and revalidate the DID doc and handle
85- `account`: an Account event that indicates a change in account status i.e. from `active` to `deactivated`, or to `takendown` if the PDS has taken down the repo.
86
87Jetstream Commits have 3 `operations`:
88
89- `create`: Create a new record with the contents provided
90- `update`: Update an existing record and replace it with the contents provided
91- `delete`: Delete an existing record with the DID, Collection, and RKey provided
92
93#### A like committed to a repo
94
95```json
96{
97 "did": "did:plc:eygmaihciaxprqvxpfvl6flk",
98 "time_us": 1725911162329308,
99 "kind": "commit",
100 "commit": {
101 "rev": "3l3qo2vutsw2b",
102 "operation": "create",
103 "collection": "app.bsky.feed.like",
104 "rkey": "3l3qo2vuowo2b",
105 "record": {
106 "$type": "app.bsky.feed.like",
107 "createdAt": "2024-09-09T19:46:02.102Z",
108 "subject": {
109 "cid": "bafyreidc6sydkkbchcyg62v77wbhzvb2mvytlmsychqgwf2xojjtirmzj4",
110 "uri": "at://did:plc:wa7b35aakoll7hugkrjtf3xf/app.bsky.feed.post/3l3pte3p2e325"
111 }
112 },
113 "cid": "bafyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi"
114 }
115}
116```
117
118#### A deleted follow record
119
120```json
121{
122 "did": "did:plc:rfov6bpyztcnedeyyzgfq42k",
123 "time_us": 1725516666833633,
124 "kind": "commit",
125 "commit": {
126 "rev": "3l3f6nzl3cv2s",
127 "operation": "delete",
128 "collection": "app.bsky.graph.follow",
129 "rkey": "3l3dn7tku762u"
130 }
131}
132```
133
134#### An identity update
135
136```json
137{
138 "did": "did:plc:ufbl4k27gp6kzas5glhz7fim",
139 "time_us": 1725516665234703,
140 "kind": "identity",
141 "identity": {
142 "did": "did:plc:ufbl4k27gp6kzas5glhz7fim",
143 "handle": "yohenrique.bsky.social",
144 "seq": 1409752997,
145 "time": "2024-09-05T06:11:04.870Z"
146 }
147}
148```
149
150#### An account becoming active
151
152```json
153{
154 "did": "did:plc:ufbl4k27gp6kzas5glhz7fim",
155 "time_us": 1725516665333808,
156 "kind": "account",
157 "account": {
158 "active": true,
159 "did": "did:plc:ufbl4k27gp6kzas5glhz7fim",
160 "seq": 1409753013,
161 "time": "2024-09-05T06:11:04.870Z"
162 }
163}
164```
165
166### Compression
167
168Jetstream supports `zstd`-based compression of messages. Jetstream uses a custom dictionary for compression that can be found in `pkg/models/zstd_dictionary` and is required to decode compressed messages from the server.
169
170`zstd` compressed Jetstream messages are ~56% smaller on average than the raw JSON version of the Jetstream firehose.
171
172The provided client library uses compression by default, using an embedded copy of the Dictionary from the `models` package.
173
174To request a compressed stream, pass the `Socket-Encoding: zstd` header through when initiating the websocket _or_ pass `compress=true` in the query string.
175
176### Subscriber Sourced messages
177
178Subscribers can send Text messages to Jetstream over the websocket using the `SubscriberSourcedMessage` framing below:
179
180```go
181type SubscriberSourcedMessage struct {
182 Type string `json:"type"`
183 Payload json.RawMessage `json:"payload"`
184}
185```
186
187The supported message types are as follows:
188
189- `options_update`
190
191#### Options Updates
192
193A client can update their `wantedCollections` and `wantedDids` after connecting to the socket by sending a Subscriber Sourced Message.
194
195To send an Options Update, provide the string `options_update` in the `type` field and a `SubscriberOptionsUpdatePayload` in the `payload` field.
196
197The shape for a `SubscriberOptionsUpdatePayload` is as follows:
198
199```go
200type SubscriberOptionsUpdateMsg struct {
201 WantedCollections []string `json:"wantedCollections"`
202 WantedDIDs []string `json:"wantedDids"`
203 MaxMessageSizeBytes int `json:"maxMessageSizeBytes"`
204}
205```
206
207If either array is empty, the relevant filter will be disabled (i.e. sending empty `wantedDids` will mean a client gets messages for all DIDs again).
208
209Some limitations apply around the size of the message: right now the message can be at most 10MB in size and can contain up to 100 collection filters _and_ up to 10,000 DID filters.
210
211Additionally, a client can connect with `?requireHello=true` in the query params to pause replay/live-tail until the first Options Update message is sent by the client over the socket.
212
213Invalid Options Updates in `requireHello` mode or normal operating mode will result in the client being disconnected.
214
215An example Subscriber Sourced Message with an Options Update payload is as follows:
216
217```json
218{
219 "type": "options_update",
220 "payload": {
221 "wantedCollections": ["app.bsky.feed.post"],
222 "wantedDids": ["did:plc:q6gjnaw2blty4crticxkmujt"],
223 "maxMessageSizeBytes": 1000000
224 }
225}
226```
227
228The above payload will filter such that a client receives only posts, and only from a the specified DID.
229
230### Dashboard Preview
231
232
233
234# Firehose Docs
235
236Usage
237
238The firehose class will spin up a websocket connection to com.atproto.sync.subscribeRepos on a given repo host (by default the Relay run by Bluesky).
239
240Each event will be parsed, authenticated, and then passed on to the supplied handleEvt which can handle indexing.
241
242On Commit events, the firehose will verify signatures and repo proofs to ensure that the event is authentic. This can be disabled with the unauthenticatedCommits flag. Similarly on Identity events, the firehose will fetch the latest DID document for the repo and do bidirectional verification on the associated handle. This can be disabled with the unauthenticatedHandles flag.
243
244Events of a certain type can be excluded using the excludeIdentity/excludeAccount/excludeCommit flags. And repo writes can be filtered down to specific collections using filterCollections. By default, all events are parsed and passed through to the handler. Note that this filtered currently happens client-side, though it is likely we will introduce server-side methods for doing so in the future.
245
246Non-fatal errors that are encountered will be passed to the required onError handler. In most cases, these can just be logged.
247
248When using the firehose class, events are processed serially. Each event must be finished being handled before the next one is parsed and authenticated.
249
250```ts
251import { Firehose } from '@atproto/sync';
252import { IdResolver } from '@atproto/identity';
253
254const idResolver = new IdResolver();
255const firehose = new Firehose({
256 idResolver,
257 service: 'wss://bsky.network',
258 handleEvt: async (evt) => {
259 if (evt.event === 'identity') {
260 // ...
261 } else if (evt.event === 'account') {
262 // ...
263 } else if (evt.event === 'create') {
264 // ...
265 } else if (evt.event === 'update') {
266 // ...
267 } else if (evt.event === 'delete') {
268 // ...
269 }
270 },
271 onError: (err) => {
272 console.error(err);
273 },
274 filterCollections: ['com.myexample.app'],
275});
276firehose.start();
277
278// on service shutdown
279await firehose.destroy();
280```
281
282For more robust indexing pipelines, it's recommended to use the supplied MemoryRunner class. This provides an in-memory partitioned queue. As events from a given repo must be processed in order, this allows events to be processed concurrently while still processing events from any given repo serially.
283
284The MemoryRunner also tracks an internal cursor based on the last finished consecutive work. This ensures that no events are dropped, although it does mean that some events may occassionally be replayed (if the websocket drops and reconnects) and therefore it's recommended that any indexing logic is idempotent. An optional setCursor parameter may be supplied to the MemoryRunner which can be used to persistently store the most recently processed cursor.
285
286```ts
287import { Firehose, MemoryRunner } from '@atproto/sync';
288import { IdResolver } from '@atproto/identity';
289
290const idResolver = new IdResolver();
291const runner = new MemoryRunner({
292 setCursor: (cursor) => {
293 // persist cursor
294 },
295});
296const firehose = new Firehose({
297 idResolver,
298 runner,
299 service: 'wss://bsky.network',
300 handleEvt: async (evt) => {
301 // ...
302 },
303 onError: (err) => {
304 console.error(err);
305 },
306});
307firehose.start();
308
309// on service shutdown
310await firehose.destroy();
311await runner.destroy();
312```
313
314# Firehose Object
315
316```ts
317import { CID } from 'multiformats/cid';
318import type { ClientOptions } from 'ws';
319import { Deferrable, createDeferrable, wait } from '@atproto/common';
320import {
321 DidDocument,
322 IdResolver,
323 parseToAtprotoDocument,
324} from '@atproto/identity';
325import {
326 RepoVerificationError,
327 cborToLexRecord,
328 formatDataKey,
329 parseDataKey,
330 readCar,
331 readCarWithRoot,
332 verifyProofs,
333} from '@atproto/repo';
334import { AtUri } from '@atproto/syntax';
335import { Subscription } from '@atproto/xrpc-server';
336import {
337 AccountEvt,
338 AccountStatus,
339 CommitEvt,
340 CommitMeta,
341 Event,
342 IdentityEvt,
343 SyncEvt,
344} from '../events';
345import { EventRunner } from '../runner';
346import { didAndSeqForEvt } from '../util';
347import {
348 type Account,
349 type Commit,
350 type Identity,
351 type RepoEvent,
352 RepoOp,
353 type Sync,
354 isAccount,
355 isCommit,
356 isIdentity,
357 isSync,
358 isValidRepoEvent,
359} from './lexicons';
360
361export type FirehoseOptions = ClientOptions & {
362 idResolver: IdResolver;
363
364 handleEvent: (evt: Event) => Awaited<void>;
365 onError: (err: Error) => void;
366 getCursor?: () => Awaited<number | undefined>;
367
368 runner?: EventRunner; // should only set getCursor *or* runner
369
370 service?: string;
371 subscriptionReconnectDelay?: number;
372
373 unauthenticatedCommits?: boolean;
374 unauthenticatedHandles?: boolean;
375
376 filterCollections?: string[];
377 excludeIdentity?: boolean;
378 excludeAccount?: boolean;
379 excludeCommit?: boolean;
380 excludeSync?: boolean;
381};
382
383export class Firehose {
384 private sub: Subscription<RepoEvent>;
385 private abortController: AbortController;
386 private destoryDefer: Deferrable;
387 private matchCollection: ((col: string) => boolean) | null = null;
388
389 constructor(public opts: FirehoseOptions) {
390 this.destoryDefer = createDeferrable();
391 this.abortController = new AbortController();
392 if (this.opts.getCursor && this.opts.runner) {
393 throw new Error('Must set only `getCursor` or `runner`');
394 }
395 if (opts.filterCollections) {
396 const exact = new Set<string>();
397 const prefixes: string[] = [];
398
399 for (const pattern of opts.filterCollections) {
400 if (pattern.endsWith('.*')) {
401 prefixes.push(pattern.slice(0, -2));
402 } else {
403 exact.add(pattern);
404 }
405 }
406 this.matchCollection = (col: string): boolean => {
407 if (exact.has(col)) return true;
408 for (const prefix of prefixes) {
409 if (col.startsWith(prefix)) return true;
410 }
411 return false;
412 };
413 }
414 this.sub = new Subscription({
415 ...opts,
416 service: opts.service ?? 'wss://bsky.network',
417 method: 'com.atproto.sync.subscribeRepos',
418 signal: this.abortController.signal,
419 getParams: async () => {
420 const getCursorFn = () =>
421 this.opts.runner?.getCursor() ?? this.opts.getCursor;
422 if (!getCursorFn) {
423 return undefined;
424 }
425 const cursor = await getCursorFn();
426 return { cursor };
427 },
428 validate: (value: unknown) => {
429 try {
430 return isValidRepoEvent(value);
431 } catch (err) {
432 this.opts.onError(new FirehoseValidationError(err, value));
433 }
434 },
435 });
436 }
437
438 async start() {
439 try {
440 for await (const evt of this.sub) {
441 if (this.opts.runner) {
442 const parsed = didAndSeqForEvt(evt);
443 if (!parsed) {
444 continue;
445 }
446 this.opts.runner.trackEvent(parsed.did, parsed.seq, async () => {
447 const parsed = await this.parseEvt(evt);
448 for (const write of parsed) {
449 try {
450 await this.opts.handleEvent(write);
451 } catch (err) {
452 this.opts.onError(new FirehoseHandlerError(err, write));
453 }
454 }
455 });
456 } else {
457 await this.processEvt(evt);
458 }
459 }
460 } catch (err) {
461 if (err && err['name'] === 'AbortError') {
462 this.destoryDefer.resolve();
463 return;
464 }
465 this.opts.onError(new FirehoseSubscriptionError(err));
466 await wait(this.opts.subscriptionReconnectDelay ?? 3000);
467 return this.start();
468 }
469 }
470
471 private async parseEvt(evt: RepoEvent): Promise<Event[]> {
472 try {
473 if (isCommit(evt) && !this.opts.excludeCommit) {
474 return this.opts.unauthenticatedCommits
475 ? await parseCommitUnauthenticated(evt, this.matchCollection)
476 : await parseCommitAuthenticated(
477 this.opts.idResolver,
478 evt,
479 this.matchCollection,
480 );
481 } else if (isAccount(evt) && !this.opts.excludeAccount) {
482 const parsed = parseAccount(evt);
483 return parsed ? [parsed] : [];
484 } else if (isIdentity(evt) && !this.opts.excludeIdentity) {
485 const parsed = await parseIdentity(
486 this.opts.idResolver,
487 evt,
488 this.opts.unauthenticatedHandles,
489 );
490 return parsed ? [parsed] : [];
491 } else if (isSync(evt) && !this.opts.excludeSync) {
492 const parsed = await parseSync(evt);
493 return parsed ? [parsed] : [];
494 } else {
495 return [];
496 }
497 } catch (err) {
498 this.opts.onError(new FirehoseParseError(err, evt));
499 return [];
500 }
501 }
502
503 private async processEvt(evt: RepoEvent) {
504 const parsed = await this.parseEvt(evt);
505 for (const write of parsed) {
506 try {
507 await this.opts.handleEvent(write);
508 } catch (err) {
509 this.opts.onError(new FirehoseHandlerError(err, write));
510 }
511 }
512 }
513
514 async destroy(): Promise<void> {
515 this.abortController.abort();
516 await this.destoryDefer.complete;
517 }
518}
519
520export const parseCommitAuthenticated = async (
521 idResolver: IdResolver,
522 evt: Commit,
523 matchCollection?: ((col: string) => boolean) | null,
524 forceKeyRefresh = false,
525): Promise<CommitEvt[]> => {
526 const did = evt.repo;
527 const ops = maybeFilterOps(evt.ops, matchCollection);
528 if (ops.length === 0) {
529 return [];
530 }
531 const claims = ops.map((op) => {
532 const { collection, rkey } = parseDataKey(op.path);
533 return {
534 collection,
535 rkey,
536 cid: op.action === 'delete' ? null : op.cid,
537 };
538 });
539 const key = await idResolver.did.resolveAtprotoKey(did, forceKeyRefresh);
540 const verifiedCids: Record<string, CID | null> = {};
541 try {
542 const results = await verifyProofs(evt.blocks, claims, did, key);
543 results.verified.forEach((op) => {
544 const path = formatDataKey(op.collection, op.rkey);
545 verifiedCids[path] = op.cid;
546 });
547 } catch (err) {
548 if (err instanceof RepoVerificationError && !forceKeyRefresh) {
549 return parseCommitAuthenticated(idResolver, evt, matchCollection, true);
550 }
551 throw err;
552 }
553 const verifiedOps: RepoOp[] = ops.filter((op) => {
554 if (op.action === 'delete') {
555 return verifiedCids[op.path] === null;
556 } else {
557 return op.cid !== null && op.cid.equals(verifiedCids[op.path]);
558 }
559 });
560 return formatCommitOps(evt, verifiedOps, {
561 skipCidVerification: true, // already checked via verifyProofs()
562 });
563};
564
565export const parseCommitUnauthenticated = async (
566 evt: Commit,
567 matchCollection?: ((col: string) => boolean) | null,
568): Promise<CommitEvt[]> => {
569 const ops = maybeFilterOps(evt.ops, matchCollection);
570 return formatCommitOps(evt, ops);
571};
572
573const maybeFilterOps = (
574 ops: RepoOp[],
575 matchCollection?: ((col: string) => boolean) | null,
576): RepoOp[] => {
577 if (!matchCollection) return ops;
578 return ops.filter((op) => {
579 const { collection } = parseDataKey(op.path);
580 return matchCollection(collection);
581 });
582};
583
584const formatCommitOps = async (
585 evt: Commit,
586 ops: RepoOp[],
587 options?: { skipCidVerification: boolean },
588) => {
589 const car = await readCar(evt.blocks, options);
590
591 const evts: CommitEvt[] = [];
592
593 for (const op of ops) {
594 const uri = AtUri.make(evt.repo, op.path);
595
596 const meta: CommitMeta = {
597 seq: evt.seq,
598 time: evt.time,
599 commit: evt.commit,
600 blocks: car.blocks,
601 rev: evt.rev,
602 uri,
603 did: uri.host,
604 collection: uri.collection,
605 rkey: uri.rkey,
606 };
607
608 if (op.action === 'create' || op.action === 'update') {
609 if (!op.cid) continue;
610 const recordBytes = car.blocks.get(op.cid);
611 if (!recordBytes) continue;
612 const record = cborToLexRecord(recordBytes);
613 evts.push({
614 ...meta,
615 event: op.action as 'create' | 'update',
616 cid: op.cid,
617 record,
618 });
619 }
620
621 if (op.action === 'delete') {
622 evts.push({
623 ...meta,
624 event: 'delete',
625 });
626 }
627 }
628
629 return evts;
630};
631
632export const parseSync = async (evt: Sync): Promise<SyncEvt | null> => {
633 const car = await readCarWithRoot(evt.blocks);
634
635 return {
636 event: 'sync',
637 seq: evt.seq,
638 time: evt.time,
639 did: evt.did,
640 cid: car.root,
641 rev: evt.rev,
642 blocks: car.blocks,
643 };
644};
645
646export const parseIdentity = async (
647 idResolver: IdResolver,
648 evt: Identity,
649 unauthenticated = false,
650): Promise<IdentityEvt | null> => {
651 const res = await idResolver.did.resolve(evt.did);
652 const handle =
653 res && !unauthenticated
654 ? await verifyHandle(idResolver, evt.did, res)
655 : undefined;
656
657 return {
658 event: 'identity',
659 seq: evt.seq,
660 time: evt.time,
661 did: evt.did,
662 handle,
663 didDocument: res ?? undefined,
664 };
665};
666
667const verifyHandle = async (
668 idResolver: IdResolver,
669 did: string,
670 didDoc: DidDocument,
671): Promise<string | undefined> => {
672 const { handle } = parseToAtprotoDocument(didDoc);
673 if (!handle) {
674 return undefined;
675 }
676 const res = await idResolver.handle.resolve(handle);
677 return res === did ? handle : undefined;
678};
679
680export const parseAccount = (evt: Account): AccountEvt | undefined => {
681 if (evt.status && !isValidStatus(evt.status)) return;
682 return {
683 event: 'account',
684 seq: evt.seq,
685 time: evt.time,
686 did: evt.did,
687 active: evt.active,
688 status: evt.status as AccountStatus | undefined,
689 };
690};
691
692const isValidStatus = (str: string): str is AccountStatus => {
693 return ['takendown', 'suspended', 'deleted', 'deactivated'].includes(str);
694};
695
696export class FirehoseValidationError extends Error {
697 constructor(
698 err: unknown,
699 public value: unknown,
700 ) {
701 super('error in firehose event lexicon validation', { cause: err });
702 }
703}
704
705export class FirehoseParseError extends Error {
706 constructor(
707 err: unknown,
708 public event: RepoEvent,
709 ) {
710 super('error in parsing and authenticating firehose event', { cause: err });
711 }
712}
713
714export class FirehoseSubscriptionError extends Error {
715 constructor(err: unknown) {
716 super('error on firehose subscription', { cause: err });
717 }
718}
719
720export class FirehoseHandlerError extends Error {
721 constructor(
722 err: unknown,
723 public event: Event,
724 ) {
725 super('error in firehose event handler', { cause: err });
726 }
727}
728```