nandi.uk
This repository has no description
1# URL Metadata Strategy (DDD)
2
3This document outlines the domain-driven approach for fetching, aggregating, and caching URL metadata from multiple external sources.
4
5## Problem Statement
6
7We need to:
8
91. Fetch metadata from multiple sources (Citoid, Iframely, etc.)
102. Cache results to avoid redundant API calls
113. Aggregate data from different sources intelligently
124. Maintain clean domain boundaries and testability
13
14## Domain Model
15
16### Value Objects
17
18**UrlMetadata**
19
20- Immutable value object containing normalized metadata fields
21- Derived from raw API responses through transformation
22- Used by the domain layer for business logic
23
24**RawMetadataResponse**
25
26- Immutable value object containing the original API response
27- Includes `source` field to track which service provided the data
28- Includes `retrievedAt` timestamp for cache invalidation
29- Preserves the exact JSON structure returned by each API
30
31**MetadataSource**
32
33- Enumeration of available metadata sources (CITOID, IFRAMELY, etc.)
34
35### Domain Services
36
37**MetadataAggregationService**
38
39- Coordinates fetching from multiple sources
40- Implements intelligent merging strategies
41- Handles fallback logic when sources fail
42
43### Infrastructure Services
44
45**IMetadataProvider** (Interface)
46
47- Contract for individual metadata sources
48- Returns raw API responses wrapped in RawMetadataResponse
49- Implemented by CitoidMetadataService, IframelyMetadataService, etc.
50
51**IRawMetadataRepository** (Interface)
52
53- Stores and retrieves raw API responses by URL and source
54- Enables querying for existing raw data to determine what's missing
55- Preserves original API response structure for future reprocessing
56
57**IMetadataTransformer** (Interface)
58
59- Transforms raw API responses into domain UrlMetadata objects
60- Source-specific implementations handle different API response formats
61- Enables reprocessing of stored raw data as domain models evolve
62
63## Architecture Pattern: Strategy + Repository + Aggregation
64
65```
66┌─────────────────────────────────────────────────────────────┐
67│ Application Layer │
68│ GetUrlMetadataUseCase │
69└─────────────────────┬───────────────────────────────────────┘
70 │
71┌─────────────────────▼───────────────────────────────────────┐
72│ Domain Layer │
73│ MetadataAggregationService │
74│ ├── IMetadataRepository (cache check/store) │
75│ ├── IMetadataProvider[] (multiple sources) │
76│ └── Aggregation Logic │
77└─────────────────────┬───────────────────────────────────────┘
78 │
79┌─────────────────────▼───────────────────────────────────────┐
80│ Infrastructure Layer │
81│ ├── DrizzleMetadataRepository │
82│ ├── CitoidMetadataService │
83│ ├── IframelyMetadataService │
84│ └── OpenGraphMetadataService │
85└─────────────────────────────────────────────────────────────┘
86```
87
88## Implementation Strategy
89
90### 1. Provider Interface
91
92```typescript
93export interface IMetadataProvider {
94 readonly source: MetadataSource;
95 fetchRawMetadata(url: URL): Promise<Result<RawMetadataResponse>>;
96 isAvailable(): Promise<boolean>;
97}
98```
99
100### 2. Repository Interface
101
102```typescript
103export interface IRawMetadataRepository {
104 findByUrl(url: URL): Promise<Result<RawMetadataResponse[]>>; // All cached raw responses for URL
105 findByUrlAndSource(
106 url: URL,
107 source: MetadataSource,
108 ): Promise<Result<RawMetadataResponse | null>>;
109 save(rawResponse: RawMetadataResponse): Promise<Result<void>>;
110 isStale(rawResponse: RawMetadataResponse, maxAge: Duration): boolean;
111}
112```
113
114### 3. Transformer Interface
115
116```typescript
117export interface IMetadataTransformer {
118 readonly source: MetadataSource;
119 transform(rawResponse: RawMetadataResponse): Result<UrlMetadata>;
120}
121```
122
123### 4. Aggregation Service
124
125```typescript
126export class MetadataAggregationService {
127 constructor(
128 private readonly rawRepository: IRawMetadataRepository,
129 private readonly providers: IMetadataProvider[],
130 private readonly transformers: Map<MetadataSource, IMetadataTransformer>,
131 private readonly maxCacheAge: Duration = Duration.days(7),
132 ) {}
133
134 async getMetadata(
135 url: URL,
136 sources?: MetadataSource[],
137 ): Promise<Result<UrlMetadata>> {
138 // 1. Check cache for existing raw responses
139 const cachedRaw = await this.rawRepository.findByUrl(url);
140
141 // 2. Determine which sources need fresh data
142 const sourcesToFetch = this.determineSourcesToFetch(cachedRaw, sources);
143
144 // 3. Fetch raw responses from required sources in parallel
145 const freshRawResults = await this.fetchRawFromSources(url, sourcesToFetch);
146
147 // 4. Cache new raw results
148 await this.cacheRawResults(freshRawResults);
149
150 // 5. Transform all available raw responses to domain objects
151 const allRawResponses = [
152 ...(cachedRaw.isOk() ? cachedRaw.value : []),
153 ...freshRawResults,
154 ];
155 const transformedMetadata = this.transformRawResponses(allRawResponses);
156
157 // 6. Aggregate transformed metadata
158 return this.aggregateMetadata(transformedMetadata);
159 }
160
161 private transformRawResponses(
162 rawResponses: RawMetadataResponse[],
163 ): UrlMetadata[] {
164 return rawResponses
165 .map((raw) => {
166 const transformer = this.transformers.get(raw.source);
167 return transformer ? transformer.transform(raw) : null;
168 })
169 .filter((result) => result?.isOk())
170 .map((result) => result!.value);
171 }
172
173 private aggregateMetadata(metadataList: UrlMetadata[]): Result<UrlMetadata> {
174 // Intelligent merging logic:
175 // - Prefer academic sources (Citoid) for scholarly content
176 // - Prefer social media optimized sources (Iframely) for rich media
177 // - Combine fields from multiple sources (e.g., best title, description, image)
178 }
179}
180```
181
182### 5. Use Case
183
184```typescript
185export class GetUrlMetadataUseCase {
186 constructor(
187 private readonly aggregationService: MetadataAggregationService,
188 ) {}
189
190 async execute(
191 url: string,
192 preferredSources?: MetadataSource[],
193 ): Promise<Result<UrlMetadata>> {
194 const urlResult = URL.create(url);
195 if (urlResult.isErr()) {
196 return err(urlResult.error);
197 }
198
199 return this.aggregationService.getMetadata(
200 urlResult.value,
201 preferredSources,
202 );
203 }
204}
205```
206
207## Caching Strategy
208
209### Cache Key Structure
210
211- Raw responses: `raw_metadata:{url_hash}:{source}`
212- Transformed metadata: `url_metadata:{url_hash}` (optional, for performance)
213
214### Cache Invalidation
215
216- Time-based: 7 days default, configurable per source
217- Manual: When user requests fresh metadata
218- Source-specific: Different TTL for different providers
219
220### Partial Cache Hits
221
222- If we have Citoid data but need Iframely data, only fetch from Iframely
223- Aggregate cached + fresh data intelligently
224
225## Data Aggregation Rules
226
227### Field Priority (Configurable)
228
2291. **Title**: Citoid > Iframely > OpenGraph
2302. **Description**: Iframely > Citoid > OpenGraph
2313. **Author**: Citoid > Iframely
2324. **Image**: Iframely > OpenGraph > Citoid
2335. **Published Date**: Citoid > Iframely
234
235### Conflict Resolution
236
237- Prefer more recent data when timestamps differ significantly
238- Prefer more complete data (fewer null fields)
239- Allow manual source preference overrides
240
241## Benefits of This Approach
242
2431. **Domain Purity**: Business logic stays in domain layer
2442. **Testability**: Easy to mock providers and repository
2453. **Flexibility**: Easy to add new metadata sources
2464. **Performance**: Intelligent caching reduces API calls
2475. **Reliability**: Fallback between sources when one fails
2486. **Configurability**: Source preferences can be adjusted per use case
249
250## Future Enhancements
251
2521. **Source Health Monitoring**: Track success rates and response times
2532. **Dynamic Source Selection**: Choose sources based on URL patterns
2543. **Batch Processing**: Fetch metadata for multiple URLs efficiently
2554. **User Preferences**: Allow users to prefer certain sources
2565. **Metadata Enrichment**: Combine multiple sources for richer data
257
258## Raw Data Storage Benefits
259
2601. **Data Preservation**: Original API responses preserved exactly as returned
2612. **Reprocessing Capability**: Can re-transform data as domain models evolve
2623. **Debugging**: Easy to inspect what each API actually returned
2634. **Audit Trail**: Complete history of API interactions
2645. **Schema Evolution**: Domain objects can change without losing source data
2656. **Multi-version Support**: Can support multiple versions of transformers
266
267## Implementation Order
268
2691. Define interfaces (`IMetadataProvider`, `IRawMetadataRepository`, `IMetadataTransformer`)
2702. Implement raw metadata repository with caching
2713. Create `RawMetadataResponse` value object
2724. Refactor existing `CitoidMetadataService` to return raw responses
2735. Implement `CitoidMetadataTransformer` to convert raw to domain objects
2746. Implement `MetadataAggregationService`
2757. Add additional providers and transformers (Iframely, OpenGraph)
2768. Implement intelligent aggregation logic
2779. Add monitoring and health checks