nandi.uk / semble
This repository has no description
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

docs: update multi-worker coordination documentation for event saga processing

Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat>

author
Wesley Finck co-author
aider (anthropic/claude-sonnet-4-20250514) date (Oct 23, 2025, 11:28 AM -0700) commit 42d38d3a parent 28791486
+226 -19
+226 -19
docs/architecture/event_system_overview.md
··· 262 262 263 263 ## Multi-Worker Coordination 264 264 265 + ### Scaling Feed Workers 266 + 267 + When you scale feed workers horizontally (multiple instances processing the feeds queue), the saga coordination becomes critical to prevent duplicate activities and ensure proper event aggregation. 268 + 269 + #### Worker Scaling Scenarios 270 + 271 + **Single Feed Worker:** 272 + ``` 273 + ┌─────────────────┐ 274 + │ Feed Worker 1 │ 275 + │ │ 276 + │ ┌─────────────┐ │ 277 + │ │CardCollection│ │ 278 + │ │Saga │ │ 279 + │ └─────────────┘ │ 280 + └─────────────────┘ 281 + │ 282 + v 283 + ┌─────────────┐ 284 + │ Redis │ 285 + │ Queue+State │ 286 + └─────────────┘ 287 + ``` 288 + 289 + **Multiple Feed Workers (Production Scale):** 290 + ``` 291 + ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ 292 + │ Feed Worker 1 │ │ Feed Worker 2 │ │ Feed Worker 3 │ 293 + │ │ │ │ │ │ 294 + │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ 295 + │ │CardCollection│ │ │ │CardCollection│ │ │ │CardCollection│ │ 296 + │ │Saga │ │ │ │Saga │ │ │ │Saga │ │ 297 + │ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │ 298 + └─────────────────┘ └─────────────────┘ └─────────────────┘ 299 + │ │ │ 300 + └───────────────────────┼───────────────────────┘ 301 + v 302 + ┌─────────────────┐ 303 + │ Redis │ 304 + │ ┌─────────────┐ │ 305 + │ │feeds Queue │ │ 306 + │ │- Event A │ │ 307 + │ │- Event B │ │ 308 + │ │- Event C │ │ 309 + │ └─────────────┘ │ 310 + │ ┌─────────────┐ │ 311 + │ │Saga State │ │ 312 + │ │+ Locks │ │ 313 + │ └─────────────┘ │ 314 + └─────────────────┘ 315 + ``` 316 + 265 317 ### Redis-Based Distributed Locking 266 318 267 - When multiple feed workers process events for the same card/user: 319 + The saga uses Redis-based distributed locking to coordinate between multiple workers processing events for the same card/user combination: 320 + 321 + #### Lock Acquisition Strategy 322 + 323 + ```typescript 324 + // CardCollectionSaga.acquireLock() 325 + const lockKey = `saga:feed:lock:${cardId}-${actorId}`; 326 + const lockTtl = 13; // seconds (aggregation window + buffer) 327 + const result = await redis.set(lockKey, '1', 'EX', lockTtl, 'NX'); 328 + return result === 'OK'; // true if lock acquired, false if already held 329 + ``` 330 + 331 + #### Detailed Multi-Worker Flow 332 + 333 + **Scenario: 3 workers, same card/user events arrive simultaneously** 334 + 335 + ``` 336 + Time: T0 (Events arrive in queue) 337 + ┌─────────────────────────────────────────────────────────────┐ 338 + │ Redis feeds Queue: │ 339 + │ - CardAddedToLibraryEvent (card-123, user-abc) │ 340 + │ - CardAddedToCollectionEvent (card-123, user-abc, coll-1) │ 341 + │ - CardAddedToCollectionEvent (card-123, user-abc, coll-2) │ 342 + └─────────────────────────────────────────────────────────────┘ 343 + 344 + Time: T0+5ms (Workers pick up events) 345 + Worker 1: Gets CardAddedToLibraryEvent 346 + Worker 2: Gets CardAddedToCollectionEvent (coll-1) 347 + Worker 3: Gets CardAddedToCollectionEvent (coll-2) 348 + 349 + Time: T0+10ms (Lock competition) 350 + Worker 1: SET saga:feed:lock:card-123-user-abc 1 EX 13 NX → OK ✓ 351 + Worker 2: SET saga:feed:lock:card-123-user-abc 1 EX 13 NX → null ✗ 352 + Worker 3: SET saga:feed:lock:card-123-user-abc 1 EX 13 NX → null ✗ 353 + 354 + Time: T0+15ms (Lock holder processes) 355 + Worker 1: 356 + - Creates new pending activity 357 + - Sets hasLibraryEvent = true 358 + - Schedules flush in 3000ms 359 + - Releases lock: DEL saga:feed:lock:card-123-user-abc 360 + 361 + Worker 2 & 3: Exit gracefully (lock acquisition failed) 362 + 363 + Time: T0+20ms (Workers retry - BullMQ automatic retry) 364 + Worker 2: SET saga:feed:lock:card-123-user-abc 1 EX 13 NX → OK ✓ 365 + Worker 3: SET saga:feed:lock:card-123-user-abc 1 EX 13 NX → null ✗ 366 + 367 + Time: T0+25ms (Second lock holder processes) 368 + Worker 2: 369 + - Finds existing pending activity 370 + - Merges: adds coll-1 to collectionIds 371 + - Sets hasCollectionEvents = true 372 + - Releases lock 373 + 374 + Time: T0+30ms (Third worker gets chance) 375 + Worker 3: SET saga:feed:lock:card-123-user-abc 1 EX 13 NX → OK ✓ 376 + Worker 3: 377 + - Finds existing pending activity 378 + - Merges: adds coll-2 to collectionIds 379 + - Releases lock 380 + 381 + Time: T0+3000ms (Flush timer fires) 382 + Worker 1's timer: 383 + - Acquires lock again 384 + - Creates single aggregated activity: 385 + { 386 + cardId: "card-123", 387 + actorId: "user-abc", 388 + collectionIds: ["coll-1", "coll-2"], 389 + hasLibraryEvent: true, 390 + hasCollectionEvents: true 391 + } 392 + - Calls AddActivityToFeedUseCase 393 + - Cleans up saga state 394 + - Releases lock 395 + ``` 396 + 397 + #### Race Condition Prevention 398 + 399 + **Problem Without Locking:** 400 + ``` 401 + Worker 1: Reads pending state → null 402 + Worker 2: Reads pending state → null 403 + Worker 1: Creates activity A 404 + Worker 2: Creates activity B 405 + Result: Duplicate activities! ❌ 406 + ``` 268 407 408 + **Solution With Distributed Locking:** 269 409 ``` 270 - Time: T0 271 - Worker 1: Receives CardAddedToLibraryEvent 272 - Worker 2: Receives CardAddedToCollectionEvent (same card/user) 410 + Worker 1: Acquires lock → processes → releases 411 + Worker 2: Waits for lock → finds existing state → merges 412 + Result: Single aggregated activity ✅ 413 + ``` 414 + 415 + #### Lock Timeout and Recovery 416 + 417 + ```typescript 418 + // Lock TTL calculation 419 + const AGGREGATION_WINDOW_MS = 3000; 420 + const PROCESSING_BUFFER_MS = 10000; 421 + const lockTtl = Math.ceil((AGGREGATION_WINDOW_MS + PROCESSING_BUFFER_MS) / 1000); 273 422 274 - Time: T0+10ms 275 - Worker 1: Acquires lock "saga:feed:lock:card-123-did:user:abc" 276 - Worker 2: Attempts lock, fails, exits gracefully 423 + // If a worker crashes while holding the lock: 424 + // 1. Redis automatically expires the lock after TTL 425 + // 2. Other workers can acquire the lock and continue processing 426 + // 3. Flush timer in crashed worker becomes irrelevant 427 + ``` 277 428 278 - Time: T0+50ms 279 - Worker 1: Creates pending activity, schedules flush 280 - Worker 1: Releases lock 429 + #### State Store Keys and Data 281 430 282 - Time: T0+100ms 283 - Worker 2: Retries, acquires lock 284 - Worker 2: Finds existing pending activity, merges collection info 285 - Worker 2: Releases lock 431 + ``` 432 + Lock Key: "saga:feed:lock:card-123-user-abc" 433 + Value: "1" 434 + TTL: 13 seconds 286 435 436 + Pending Activity Key: "saga:feed:pending:card-123-user-abc" 437 + Value: { 438 + "cardId": "card-123", 439 + "actorId": "user-abc", 440 + "collectionIds": ["coll-1", "coll-2"], 441 + "timestamp": "2024-01-15T10:30:00.000Z", 442 + "hasLibraryEvent": true, 443 + "hasCollectionEvents": true 444 + } 445 + TTL: 8 seconds (aggregation window + buffer) 446 + ``` 447 + 448 + ### Worker Failure Scenarios 449 + 450 + #### Worker Crashes After Lock Acquisition 451 + 452 + ``` 453 + Time: T0 454 + Worker 1: Acquires lock, starts processing 455 + Worker 1: CRASHES 💥 (before releasing lock) 456 + 457 + Time: T0+13s 458 + Redis: Lock expires automatically 459 + Worker 2: Can now acquire lock and continue processing 460 + ``` 461 + 462 + #### Worker Crashes During Flush 463 + 464 + ``` 287 465 Time: T0+3000ms 288 466 Worker 1: Flush timer fires, acquires lock 289 - Worker 1: Creates single aggregated feed activity 290 - Worker 1: Cleans up saga state 467 + Worker 1: CRASHES 💥 (before completing flush) 468 + 469 + Time: T0+3013s 470 + Redis: Lock expires 471 + Worker 2: Timer fires, acquires lock, completes flush 472 + ``` 473 + 474 + #### Network Partition 475 + 476 + ``` 477 + Worker 1: Loses Redis connection 478 + Worker 1: Cannot acquire/release locks 479 + Worker 2 & 3: Continue processing normally 480 + Worker 1: Reconnects, participates in future events 291 481 ``` 292 482 293 - ### State Store Keys 483 + ### Performance Characteristics 484 + 485 + #### Lock Contention 486 + 487 + - **Low contention**: Different card/user combinations → parallel processing 488 + - **High contention**: Same card/user → sequential processing (by design) 489 + - **Lock duration**: ~1-5ms per event (very brief) 490 + 491 + #### Throughput Impact 294 492 295 493 ``` 296 - Pending Activity: "saga:feed:pending:card-123-did:user:abc" 297 - Distributed Lock: "saga:feed:lock:card-123-did:user:abc" 494 + Single Worker: 1000 events/sec 495 + Multiple Workers (different cards): 3000 events/sec (3x scaling) 496 + Multiple Workers (same card): 1000 events/sec (sequential by design) 497 + ``` 498 + 499 + #### Memory Usage 500 + 501 + ``` 502 + Per Pending Activity: ~200 bytes in Redis 503 + Lock Overhead: ~50 bytes per lock 504 + Cleanup: Automatic via TTL expiration 298 505 ``` 299 506 300 507 ## Queue Routing