URL Semantic Search Architecture#

This document outlines the URL semantic search system that provides similarity-based URL discovery using vector embeddings and semantic search capabilities.

System Overview#

The URL semantic search system consists of:

1. Search Worker - Processes CardAddedToLibraryEvent to index URLs for search
2. Vector Database Interface - Abstracts vector storage and similarity search
3. Search Service - Coordinates indexing and querying operations
4. Use Cases - Command and query operations for search functionality

Architecture Components#

Core Components#

┌─────────────────────────────────────────────────────────────┐
│                    Search System                            │
│                                                             │
│  ┌─────────────────┐    ┌─────────────────┐                │
│  │ IndexUrlForSearch│    │GetSimilarUrlsFor│                │
│  │    UseCase      │    │   UrlUseCase    │                │
│  └─────────────────┘    └─────────────────┘                │
│           │                       │                        │
│           v                       v                        │
│  ┌─────────────────────────────────────────────────────────┐│
│  │              SearchService                              ││
│  └─────────────────────────────────────────────────────────┘│
│           │                       │                        │
│           v                       v                        │
│  ┌─────────────────┐    ┌─────────────────┐                │
│  │ IVectorDatabase │    │ IMetadataService│                │
│  │   Interface     │    │                 │                │
│  └─────────────────┘    └─────────────────┘                │
└─────────────────────────────────────────────────────────────┘

Event Flow#

CardAddedToLibraryEvent
         │
         v
┌─────────────────┐
│  Search Worker  │
│                 │
│ ┌─────────────┐ │
│ │CardAddedTo  │ │
│ │LibraryEvent │ │
│ │Handler      │ │
│ └─────────────┘ │
│        │        │
│        v        │
│ ┌─────────────┐ │
│ │IndexUrlFor  │ │
│ │SearchUseCase│ │
│ └─────────────┘ │
└─────────────────┘
         │
         v
┌─────────────────┐
│ SearchService   │
│                 │
│ 1. Get metadata │
│ 2. Generate     │
│    embeddings   │
│ 3. Store in     │
│    vector DB    │
└─────────────────┘

Data Types#

URL View (Search Result)#

export interface UrlView {
  url: string;
  metadata: {
    url: string;
    title?: string;
    description?: string;
    author?: string;
    thumbnailUrl?: string;
  };
  urlLibraryCount: number;
  urlInLibrary?: boolean;
}

Search Parameters#

export interface GetSimilarUrlsForUrlParams extends PaginatedParams {
  url: string;
}

export interface GetSimilarUrlsForUrlResponse {
  urls: UrlView[];
  pagination: Pagination;
}

Vector Database Interface#

export interface IVectorDatabase {
  indexUrl(params: IndexUrlParams): Promise<Result<void>>;
  findSimilarUrls(
    params: FindSimilarUrlsParams,
  ): Promise<Result<UrlSearchResult[]>>;
  deleteUrl(url: string): Promise<Result<void>>;
}

export interface IndexUrlParams {
  url: string;
  title?: string;
  description?: string;
  author?: string;
  content: string; // Combined text for embedding
}

export interface FindSimilarUrlsParams {
  url: string;
  limit: number;
  threshold?: number; // Similarity threshold
}

export interface UrlSearchResult {
  url: string;
  similarity: number;
  metadata: {
    title?: string;
    description?: string;
    author?: string;
  };
}

Deployment Contexts#

1. Production (Fly.io)#

┌─────────────────┐    ┌─────────────────┐
│   Web Process   │    │ Search Worker   │
│                 │    │   Process       │
│ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │GetSimilar   │ │    │ │EventHandler │ │
│ │UrlsUseCase  │ │    │ │             │ │
│ └─────────────┘ │    │ └─────────────┘ │
│        │        │    │        │        │
│        v        │    │        v        │
│ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │SearchService│ │    │ │IndexUrlFor  │ │
│ │             │ │    │ │SearchUseCase│ │
│ └─────────────┘ │    │ └─────────────┘ │
│        │        │    │        │        │
└────────┼────────┘    └────────┼────────┘
         │                      │
         v                      v
    ┌─────────────────────────────────┐
    │           Redis                 │
    │  ┌─────────────────────────────┐│
    │  │     search Queue            ││
    │  │ ┌─────────────────────────┐ ││
    │  │ │CardAddedToLibraryEvent  │ ││
    │  │ └─────────────────────────┘ ││
    │  └─────────────────────────────┘│
    └─────────────────────────────────┘
         │                      │
         v                      v
    ┌─────────────┐    ┌─────────────┐
    │Vector DB    │    │PostgreSQL   │
    │(Pinecone/   │    │(URL metadata│
    │ Weaviate)   │    │ + counts)   │
    └─────────────┘    └─────────────┘

Configuration:

• USE_IN_MEMORY_EVENTS=false
• VECTOR_DB_URL configured (Pinecone, Weaviate, etc.)
• Separate search worker process

2. Local Development#

┌─────────────────────────────────────┐
│         Combined Process            │
│                                     │
│ ┌─────────────┐    ┌─────────────┐  │
│ │GetSimilar   │    │EventHandler │  │
│ │UrlsUseCase  │    │             │  │
│ └─────────────┘    └─────────────┘  │
│        │                  │         │
│        v                  v         │
│ ┌─────────────┐    ┌─────────────┐  │
│ │SearchService│    │IndexUrlFor  │  │
│ │             │    │SearchUseCase│  │
│ └─────────────┘    └─────────────┘  │
│        │                  │         │
└────────┼──────────────────┼─────────┘
         │                  │
         v                  v
    ┌─────────────────────────────────┐
    │        Local Redis              │
    │  ┌─────────────────────────────┐│
    │  │     search Queue            ││
    │  └─────────────────────────────┘│
    └─────────────────────────────────┘
         │                  │
         v                  v
    ┌─────────────┐    ┌─────────────┐
    │Local Vector │    │PostgreSQL   │
    │DB (Docker)  │    │(Docker)     │
    └─────────────┘    └─────────────┘

Configuration:

• USE_IN_MEMORY_EVENTS=false
• Local vector DB via Docker (Weaviate/Qdrant)
• Both web app and search worker in same process

3. Local Mock Development#

┌─────────────────────────────────────┐
│         Single Process              │
│                                     │
│ ┌─────────────┐    ┌─────────────┐  │
│ │GetSimilar   │    │EventHandler │  │
│ │UrlsUseCase  │    │             │  │
│ └─────────────┘    └─────────────┘  │
│        │                  │         │
│        v                  v         │
│ ┌─────────────┐    ┌─────────────┐  │
│ │SearchService│    │IndexUrlFor  │  │
│ │             │    │SearchUseCase│  │
│ └─────────────┘    └─────────────┘  │
│        │                  │         │
│        └──────────────────┘         │
│                                     │
│ ┌─────────────────────────────────┐ │
│ │    InMemoryVectorDatabase       │ │
│ │  - Map-based storage            │ │
│ │  - Simple cosine similarity     │ │
│ │  - No external dependencies     │ │
│ └─────────────────────────────────┘ │
└─────────────────────────────────────┘

Configuration:

• USE_IN_MEMORY_EVENTS=true
• USE_MOCK_VECTOR_DB=true
• No external vector DB required
• Simple in-memory similarity using basic text matching

Search Worker Implementation#

Event Handler#

export class CardAddedToLibraryEventHandler
  implements IEventHandler<CardAddedToLibraryEvent>
{
  constructor(private indexUrlForSearchUseCase: IndexUrlForSearchUseCase) {}

  async handle(event: CardAddedToLibraryEvent): Promise<Result<void>> {
    // Only index URL cards
    const cardResult = await this.getCardDetails(event.cardId);
    if (cardResult.isErr() || !cardResult.value.isUrlCard) {
      return ok(undefined);
    }

    return this.indexUrlForSearchUseCase.execute({
      url: cardResult.value.url,
      cardId: event.cardId.getStringValue(),
    });
  }
}

Search Service#

export class SearchService {
  constructor(
    private vectorDatabase: IVectorDatabase,
    private metadataService: IMetadataService,
    private cardQueryRepository: ICardQueryRepository,
  ) {}

  async indexUrl(url: string): Promise<Result<void>> {
    // 1. Get metadata
    const metadataResult = await this.metadataService.fetchMetadata(url);
    if (metadataResult.isErr()) {
      return err(metadataResult.error);
    }

    // 2. Prepare content for embedding
    const content = this.prepareContentForEmbedding(metadataResult.value);

    // 3. Index in vector database
    return this.vectorDatabase.indexUrl({
      url: url.value,
      title: metadataResult.value.title,
      description: metadataResult.value.description,
      author: metadataResult.value.author,
      content,
    });
  }

  async findSimilarUrls(
    url: string,
    options: { limit: number; threshold?: number },
  ): Promise<Result<UrlView[]>> {
    // 1. Find similar URLs from vector DB
    const similarResult = await this.vectorDatabase.findSimilarUrls({
      url,
      limit: options.limit,
      threshold: options.threshold,
    });

    if (similarResult.isErr()) {
      return err(similarResult.error);
    }

    // 2. Enrich with library counts and user context
    const enrichedUrls = await this.enrichUrlsWithContext(similarResult.value);

    return ok(enrichedUrls);
  }
}

Queue Configuration#

The search queue is added to the existing queue routing:

// BullMQEventPublisher.getTargetQueues()
switch (eventName) {
  case EventNames.CARD_ADDED_TO_LIBRARY:
    return [QueueNames.FEEDS, QueueNames.SEARCH, QueueNames.ANALYTICS];
  case EventNames.CARD_ADDED_TO_COLLECTION:
    return [QueueNames.FEEDS];
  default:
    return [QueueNames.FEEDS];
}

Vector Database Implementations#

Production: Pinecone/Weaviate#

export class PineconeVectorDatabase implements IVectorDatabase {
  constructor(private client: PineconeClient) {}

  async indexUrl(params: IndexUrlParams): Promise<Result<void>> {
    // Generate embeddings and upsert to Pinecone
  }

  async findSimilarUrls(
    params: FindSimilarUrlsParams,
  ): Promise<Result<UrlSearchResult[]>> {
    // Query Pinecone for similar vectors
  }
}

Local Development: Weaviate/Qdrant (Docker)#

export class WeaviateVectorDatabase implements IVectorDatabase {
  constructor(private client: WeaviateClient) {}

  async indexUrl(params: IndexUrlParams): Promise<Result<void>> {
    // Index in local Weaviate instance
  }

  async findSimilarUrls(
    params: FindSimilarUrlsParams,
  ): Promise<Result<UrlSearchResult[]>> {
    // Query local Weaviate instance
  }
}

Local Mock: In-Memory#

export class InMemoryVectorDatabase implements IVectorDatabase {
  private urls: Map<string, IndexedUrl> = new Map();

  async indexUrl(params: IndexUrlParams): Promise<Result<void>> {
    // Store in memory with simple text-based similarity
    this.urls.set(params.url, {
      url: params.url,
      content: params.content,
      metadata: { title: params.title, description: params.description },
    });
    return ok(undefined);
  }

  async findSimilarUrls(
    params: FindSimilarUrlsParams,
  ): Promise<Result<UrlSearchResult[]>> {
    // Simple text similarity using keyword matching
    const results = Array.from(this.urls.values())
      .filter(
        (indexed) =>
          this.calculateSimilarity(params.url, indexed.content) > 0.3,
      )
      .map((indexed) => ({
        url: indexed.url,
        similarity: this.calculateSimilarity(params.url, indexed.content),
        metadata: indexed.metadata,
      }))
      .sort((a, b) => b.similarity - a.similarity)
      .slice(0, params.limit);

    return ok(results);
  }

  private calculateSimilarity(query: string, content: string): number {
    // Simple keyword-based similarity for mocking
    const queryWords = query.toLowerCase().split(/\W+/);
    const contentWords = content.toLowerCase().split(/\W+/);
    const intersection = queryWords.filter((word) =>
      contentWords.includes(word),
    );
    return (
      intersection.length / Math.max(queryWords.length, contentWords.length)
    );
  }
}

Environment Variables#

# Vector database configuration
VECTOR_DB_TYPE=pinecone|weaviate|qdrant|mock
VECTOR_DB_URL=https://your-vector-db-endpoint
VECTOR_DB_API_KEY=your-api-key

# Search configuration
SEARCH_SIMILARITY_THRESHOLD=0.7
SEARCH_DEFAULT_LIMIT=20

# Mock configuration
USE_MOCK_VECTOR_DB=true|false

Local Development Commands#

npm run dev (Redis + BullMQ + Vector DB)#

• Uses real vector database (Docker)
• Separate search worker process
• Full production-like behavior

npm run dev:mock (In-Memory)#

• Uses InMemoryVectorDatabase
• Simple text-based similarity
• No external dependencies

npm run search-worker (Dedicated Search Worker)#

• Runs only the search worker process
• For scaling search processing separately

Performance Considerations#

Indexing Strategy#

• Immediate Indexing: Index URLs as soon as they're added to library
• Batch Processing: For high-volume scenarios, batch index operations
• Deduplication: Avoid re-indexing the same URL multiple times

Query Optimization#

• Caching: Cache similar URL results for popular queries
• Pagination: Implement cursor-based pagination for large result sets
• Filtering: Pre-filter results by library counts or user context

Scaling#

• Horizontal Scaling: Multiple search worker instances
• Vector DB Sharding: Partition URLs across multiple vector DB instances
• CDN Caching: Cache search results at CDN level for popular queries

Integration Points#

With Existing Systems#

1. Card System: Listens to CardAddedToLibraryEvent
2. Metadata Service: Reuses existing URL metadata fetching
3. Query Repository: Enriches results with library counts
4. Profile Service: Adds user context to results

API Endpoints#

// GET /api/search/similar-urls?url=https://example.com&limit=10
app.get('/api/search/similar-urls', async (req, res) => {
  const result = await getSimilarUrlsForUrlUseCase.execute({
    url: req.query.url,
    limit: parseInt(req.query.limit) || 10,
    page: parseInt(req.query.page) || 1,
  });

  res.json(result);
});

This architecture provides a scalable, testable URL semantic search system that integrates seamlessly with the existing event-driven architecture while supporting different deployment contexts from local development to production.