nandi.uk
This repository has no description
19 kB
524 lines
1# URL Semantic Search Architecture
2
3This document outlines the URL semantic search system that provides similarity-based URL discovery using vector embeddings and semantic search capabilities.
4
5## System Overview
6
7The URL semantic search system consists of:
8
91. **Search Worker** - Processes `CardAddedToLibraryEvent` to index URLs for search
102. **Vector Database Interface** - Abstracts vector storage and similarity search
113. **Search Service** - Coordinates indexing and querying operations
124. **Use Cases** - Command and query operations for search functionality
13
14## Architecture Components
15
16### Core Components
17
18```
19┌─────────────────────────────────────────────────────────────┐
20│ Search System │
21│ │
22│ ┌─────────────────┐ ┌─────────────────┐ │
23│ │ IndexUrlForSearch│ │GetSimilarUrlsFor│ │
24│ │ UseCase │ │ UrlUseCase │ │
25│ └─────────────────┘ └─────────────────┘ │
26│ │ │ │
27│ v v │
28│ ┌─────────────────────────────────────────────────────────┐│
29│ │ SearchService ││
30│ └─────────────────────────────────────────────────────────┘│
31│ │ │ │
32│ v v │
33│ ┌─────────────────┐ ┌─────────────────┐ │
34│ │ IVectorDatabase │ │ IMetadataService│ │
35│ │ Interface │ │ │ │
36│ └─────────────────┘ └─────────────────┘ │
37└─────────────────────────────────────────────────────────────┘
38```
39
40### Event Flow
41
42```
43CardAddedToLibraryEvent
44 │
45 v
46┌─────────────────┐
47│ Search Worker │
48│ │
49│ ┌─────────────┐ │
50│ │CardAddedTo │ │
51│ │LibraryEvent │ │
52│ │Handler │ │
53│ └─────────────┘ │
54│ │ │
55│ v │
56│ ┌─────────────┐ │
57│ │IndexUrlFor │ │
58│ │SearchUseCase│ │
59│ └─────────────┘ │
60└─────────────────┘
61 │
62 v
63┌─────────────────┐
64│ SearchService │
65│ │
66│ 1. Get metadata │
67│ 2. Generate │
68│ embeddings │
69│ 3. Store in │
70│ vector DB │
71└─────────────────┘
72```
73
74## Data Types
75
76### URL View (Search Result)
77
78```typescript
79export interface UrlView {
80 url: string;
81 metadata: {
82 url: string;
83 title?: string;
84 description?: string;
85 author?: string;
86 thumbnailUrl?: string;
87 };
88 urlLibraryCount: number;
89 urlInLibrary?: boolean;
90}
91```
92
93### Search Parameters
94
95```typescript
96export interface GetSimilarUrlsForUrlParams extends PaginatedParams {
97 url: string;
98}
99
100export interface GetSimilarUrlsForUrlResponse {
101 urls: UrlView[];
102 pagination: Pagination;
103}
104```
105
106### Vector Database Interface
107
108```typescript
109export interface IVectorDatabase {
110 indexUrl(params: IndexUrlParams): Promise<Result<void>>;
111 findSimilarUrls(
112 params: FindSimilarUrlsParams,
113 ): Promise<Result<UrlSearchResult[]>>;
114 deleteUrl(url: string): Promise<Result<void>>;
115}
116
117export interface IndexUrlParams {
118 url: string;
119 title?: string;
120 description?: string;
121 author?: string;
122 content: string; // Combined text for embedding
123}
124
125export interface FindSimilarUrlsParams {
126 url: string;
127 limit: number;
128 threshold?: number; // Similarity threshold
129}
130
131export interface UrlSearchResult {
132 url: string;
133 similarity: number;
134 metadata: {
135 title?: string;
136 description?: string;
137 author?: string;
138 };
139}
140```
141
142## Deployment Contexts
143
144### 1. Production (Fly.io)
145
146```
147┌─────────────────┐ ┌─────────────────┐
148│ Web Process │ │ Search Worker │
149│ │ │ Process │
150│ ┌─────────────┐ │ │ ┌─────────────┐ │
151│ │GetSimilar │ │ │ │EventHandler │ │
152│ │UrlsUseCase │ │ │ │ │ │
153│ └─────────────┘ │ │ └─────────────┘ │
154│ │ │ │ │ │
155│ v │ │ v │
156│ ┌─────────────┐ │ │ ┌─────────────┐ │
157│ │SearchService│ │ │ │IndexUrlFor │ │
158│ │ │ │ │ │SearchUseCase│ │
159│ └─────────────┘ │ │ └─────────────┘ │
160│ │ │ │ │ │
161└────────┼────────┘ └────────┼────────┘
162 │ │
163 v v
164 ┌─────────────────────────────────┐
165 │ Redis │
166 │ ┌─────────────────────────────┐│
167 │ │ search Queue ││
168 │ │ ┌─────────────────────────┐ ││
169 │ │ │CardAddedToLibraryEvent │ ││
170 │ │ └─────────────────────────┘ ││
171 │ └─────────────────────────────┘│
172 └─────────────────────────────────┘
173 │ │
174 v v
175 ┌─────────────┐ ┌─────────────┐
176 │Vector DB │ │PostgreSQL │
177 │(Pinecone/ │ │(URL metadata│
178 │ Weaviate) │ │ + counts) │
179 └─────────────┘ └─────────────┘
180```
181
182**Configuration:**
183
184- `USE_IN_MEMORY_EVENTS=false`
185- `VECTOR_DB_URL` configured (Pinecone, Weaviate, etc.)
186- Separate search worker process
187
188### 2. Local Development
189
190```
191┌─────────────────────────────────────┐
192│ Combined Process │
193│ │
194│ ┌─────────────┐ ┌─────────────┐ │
195│ │GetSimilar │ │EventHandler │ │
196│ │UrlsUseCase │ │ │ │
197│ └─────────────┘ └─────────────┘ │
198│ │ │ │
199│ v v │
200│ ┌─────────────┐ ┌─────────────┐ │
201│ │SearchService│ │IndexUrlFor │ │
202│ │ │ │SearchUseCase│ │
203│ └─────────────┘ └─────────────┘ │
204│ │ │ │
205└────────┼──────────────────┼─────────┘
206 │ │
207 v v
208 ┌─────────────────────────────────┐
209 │ Local Redis │
210 │ ┌─────────────────────────────┐│
211 │ │ search Queue ││
212 │ └─────────────────────────────┘│
213 └─────────────────────────────────┘
214 │ │
215 v v
216 ┌─────────────┐ ┌─────────────┐
217 │Local Vector │ │PostgreSQL │
218 │DB (Docker) │ │(Docker) │
219 └─────────────┘ └─────────────┘
220```
221
222**Configuration:**
223
224- `USE_IN_MEMORY_EVENTS=false`
225- Local vector DB via Docker (Weaviate/Qdrant)
226- Both web app and search worker in same process
227
228### 3. Local Mock Development
229
230```
231┌─────────────────────────────────────┐
232│ Single Process │
233│ │
234│ ┌─────────────┐ ┌─────────────┐ │
235│ │GetSimilar │ │EventHandler │ │
236│ │UrlsUseCase │ │ │ │
237│ └─────────────┘ └─────────────┘ │
238│ │ │ │
239│ v v │
240│ ┌─────────────┐ ┌─────────────┐ │
241│ │SearchService│ │IndexUrlFor │ │
242│ │ │ │SearchUseCase│ │
243│ └─────────────┘ └─────────────┘ │
244│ │ │ │
245│ └──────────────────┘ │
246│ │
247│ ┌─────────────────────────────────┐ │
248│ │ InMemoryVectorDatabase │ │
249│ │ - Map-based storage │ │
250│ │ - Simple cosine similarity │ │
251│ │ - No external dependencies │ │
252│ └─────────────────────────────────┘ │
253└─────────────────────────────────────┘
254```
255
256**Configuration:**
257
258- `USE_IN_MEMORY_EVENTS=true`
259- `USE_MOCK_VECTOR_DB=true`
260- No external vector DB required
261- Simple in-memory similarity using basic text matching
262
263## Search Worker Implementation
264
265### Event Handler
266
267```typescript
268export class CardAddedToLibraryEventHandler
269 implements IEventHandler<CardAddedToLibraryEvent>
270{
271 constructor(private indexUrlForSearchUseCase: IndexUrlForSearchUseCase) {}
272
273 async handle(event: CardAddedToLibraryEvent): Promise<Result<void>> {
274 // Only index URL cards
275 const cardResult = await this.getCardDetails(event.cardId);
276 if (cardResult.isErr() || !cardResult.value.isUrlCard) {
277 return ok(undefined);
278 }
279
280 return this.indexUrlForSearchUseCase.execute({
281 url: cardResult.value.url,
282 cardId: event.cardId.getStringValue(),
283 });
284 }
285}
286```
287
288### Search Service
289
290```typescript
291export class SearchService {
292 constructor(
293 private vectorDatabase: IVectorDatabase,
294 private metadataService: IMetadataService,
295 private cardQueryRepository: ICardQueryRepository,
296 ) {}
297
298 async indexUrl(url: string): Promise<Result<void>> {
299 // 1. Get metadata
300 const metadataResult = await this.metadataService.fetchMetadata(url);
301 if (metadataResult.isErr()) {
302 return err(metadataResult.error);
303 }
304
305 // 2. Prepare content for embedding
306 const content = this.prepareContentForEmbedding(metadataResult.value);
307
308 // 3. Index in vector database
309 return this.vectorDatabase.indexUrl({
310 url: url.value,
311 title: metadataResult.value.title,
312 description: metadataResult.value.description,
313 author: metadataResult.value.author,
314 content,
315 });
316 }
317
318 async findSimilarUrls(
319 url: string,
320 options: { limit: number; threshold?: number },
321 ): Promise<Result<UrlView[]>> {
322 // 1. Find similar URLs from vector DB
323 const similarResult = await this.vectorDatabase.findSimilarUrls({
324 url,
325 limit: options.limit,
326 threshold: options.threshold,
327 });
328
329 if (similarResult.isErr()) {
330 return err(similarResult.error);
331 }
332
333 // 2. Enrich with library counts and user context
334 const enrichedUrls = await this.enrichUrlsWithContext(similarResult.value);
335
336 return ok(enrichedUrls);
337 }
338}
339```
340
341## Queue Configuration
342
343The search queue is added to the existing queue routing:
344
345```typescript
346// BullMQEventPublisher.getTargetQueues()
347switch (eventName) {
348 case EventNames.CARD_ADDED_TO_LIBRARY:
349 return [QueueNames.FEEDS, QueueNames.SEARCH, QueueNames.ANALYTICS];
350 case EventNames.CARD_ADDED_TO_COLLECTION:
351 return [QueueNames.FEEDS];
352 default:
353 return [QueueNames.FEEDS];
354}
355```
356
357## Vector Database Implementations
358
359### Production: Pinecone/Weaviate
360
361```typescript
362export class PineconeVectorDatabase implements IVectorDatabase {
363 constructor(private client: PineconeClient) {}
364
365 async indexUrl(params: IndexUrlParams): Promise<Result<void>> {
366 // Generate embeddings and upsert to Pinecone
367 }
368
369 async findSimilarUrls(
370 params: FindSimilarUrlsParams,
371 ): Promise<Result<UrlSearchResult[]>> {
372 // Query Pinecone for similar vectors
373 }
374}
375```
376
377### Local Development: Weaviate/Qdrant (Docker)
378
379```typescript
380export class WeaviateVectorDatabase implements IVectorDatabase {
381 constructor(private client: WeaviateClient) {}
382
383 async indexUrl(params: IndexUrlParams): Promise<Result<void>> {
384 // Index in local Weaviate instance
385 }
386
387 async findSimilarUrls(
388 params: FindSimilarUrlsParams,
389 ): Promise<Result<UrlSearchResult[]>> {
390 // Query local Weaviate instance
391 }
392}
393```
394
395### Local Mock: In-Memory
396
397```typescript
398export class InMemoryVectorDatabase implements IVectorDatabase {
399 private urls: Map<string, IndexedUrl> = new Map();
400
401 async indexUrl(params: IndexUrlParams): Promise<Result<void>> {
402 // Store in memory with simple text-based similarity
403 this.urls.set(params.url, {
404 url: params.url,
405 content: params.content,
406 metadata: { title: params.title, description: params.description },
407 });
408 return ok(undefined);
409 }
410
411 async findSimilarUrls(
412 params: FindSimilarUrlsParams,
413 ): Promise<Result<UrlSearchResult[]>> {
414 // Simple text similarity using keyword matching
415 const results = Array.from(this.urls.values())
416 .filter(
417 (indexed) =>
418 this.calculateSimilarity(params.url, indexed.content) > 0.3,
419 )
420 .map((indexed) => ({
421 url: indexed.url,
422 similarity: this.calculateSimilarity(params.url, indexed.content),
423 metadata: indexed.metadata,
424 }))
425 .sort((a, b) => b.similarity - a.similarity)
426 .slice(0, params.limit);
427
428 return ok(results);
429 }
430
431 private calculateSimilarity(query: string, content: string): number {
432 // Simple keyword-based similarity for mocking
433 const queryWords = query.toLowerCase().split(/\W+/);
434 const contentWords = content.toLowerCase().split(/\W+/);
435 const intersection = queryWords.filter((word) =>
436 contentWords.includes(word),
437 );
438 return (
439 intersection.length / Math.max(queryWords.length, contentWords.length)
440 );
441 }
442}
443```
444
445## Environment Variables
446
447```bash
448# Vector database configuration
449VECTOR_DB_TYPE=pinecone|weaviate|qdrant|mock
450VECTOR_DB_URL=https://your-vector-db-endpoint
451VECTOR_DB_API_KEY=your-api-key
452
453# Search configuration
454SEARCH_SIMILARITY_THRESHOLD=0.7
455SEARCH_DEFAULT_LIMIT=20
456
457# Mock configuration
458USE_MOCK_VECTOR_DB=true|false
459```
460
461## Local Development Commands
462
463### `npm run dev` (Redis + BullMQ + Vector DB)
464
465- Uses real vector database (Docker)
466- Separate search worker process
467- Full production-like behavior
468
469### `npm run dev:mock` (In-Memory)
470
471- Uses `InMemoryVectorDatabase`
472- Simple text-based similarity
473- No external dependencies
474
475### `npm run search-worker` (Dedicated Search Worker)
476
477- Runs only the search worker process
478- For scaling search processing separately
479
480## Performance Considerations
481
482### Indexing Strategy
483
484- **Immediate Indexing**: Index URLs as soon as they're added to library
485- **Batch Processing**: For high-volume scenarios, batch index operations
486- **Deduplication**: Avoid re-indexing the same URL multiple times
487
488### Query Optimization
489
490- **Caching**: Cache similar URL results for popular queries
491- **Pagination**: Implement cursor-based pagination for large result sets
492- **Filtering**: Pre-filter results by library counts or user context
493
494### Scaling
495
496- **Horizontal Scaling**: Multiple search worker instances
497- **Vector DB Sharding**: Partition URLs across multiple vector DB instances
498- **CDN Caching**: Cache search results at CDN level for popular queries
499
500## Integration Points
501
502### With Existing Systems
503
5041. **Card System**: Listens to `CardAddedToLibraryEvent`
5052. **Metadata Service**: Reuses existing URL metadata fetching
5063. **Query Repository**: Enriches results with library counts
5074. **Profile Service**: Adds user context to results
508
509### API Endpoints
510
511```typescript
512// GET /api/search/similar-urls?url=https://example.com&limit=10
513app.get('/api/search/similar-urls', async (req, res) => {
514 const result = await getSimilarUrlsForUrlUseCase.execute({
515 url: req.query.url,
516 limit: parseInt(req.query.limit) || 10,
517 page: parseInt(req.query.page) || 1,
518 });
519
520 res.json(result);
521});
522```
523
524This architecture provides a scalable, testable URL semantic search system that integrates seamlessly with the existing event-driven architecture while supporting different deployment contexts from local development to production.