A self-hosted household ledger.
propose csv-backfill-import change
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
author
graham.systems
co-author
Claude Opus 4.8
date
(Jul 16, 2026, 6:29 PM -0700)
commit
8e7c1e71
8e7c1e712cc9c9aa92bd67f469acc4cf6992a0ec
parent
f39d0059
f39d0059c6f2e57fd2837478670a9773d8e480e5
change-id
nmkpzsos
nmkpzsosvvsmvtspmowzpnqxrysoymop
graham.systems
co-author
+785
+2
openspec/changes/csv-backfill-import/.openspec.yaml
+2
openspec/changes/csv-backfill-import/.openspec.yaml
+281
openspec/changes/csv-backfill-import/design.md
+281
openspec/changes/csv-backfill-import/design.md
···
1
+
## Context
2
+
3
+
Quantum's ingestion spine is: fetch bytes → archive them verbatim in `raw_syncs`
4
+
→ normalize as a pure, replayable function of the archived payload → upsert. That
5
+
shape is load-bearing and this change preserves it.
6
+
7
+
The relevant current state:
8
+
9
+
- `transactions` is keyed `UNIQUE (account_id, sfin_id)` with `sfin_id TEXT NOT
10
+
NULL`. Identity comes from the provider; nothing in the codebase synthesizes it.
11
+
- `ingestTransactions` (`src/lib/server/services/sync.ts:183`) is an
12
+
**authoritative** writer. It assumes its input is the complete truth for the
13
+
account over a window: it reconciles new posted rows against pending rows within
14
+
±5 days, and it soft-removes every pending row absent from the feed
15
+
(`sync.ts:291-301`).
16
+
- `accounts.connection_id` is `NOT NULL`, and `account-management` requires that
17
+
accounts originate only from sync.
18
+
- `normalizePayload` (`normalize.ts:61`) is pure and emits `NormalizedAccount[]`.
19
+
- `applyRulesToUncategorized` (`rules.ts:168`) is unscoped and database-wide.
20
+
- Reports scope on `t.pending = 0 AND t.removed_at IS NULL` (`reports.ts:7`) and
21
+
order by `COALESCE(posted, transacted_at, created_at)`.
22
+
23
+
The user-facing problem is one account whose SimpleFIN feed reaches back only
24
+
about a week. Critically, its synced history still **accumulates**: each daily sync
25
+
tops up the last seven days, so over time the account holds synced rows across a
26
+
widening stretch. A backfill CSV and the synced data therefore overlap across a
27
+
*region*, not at a boundary line.
28
+
29
+
## Goals / Non-Goals
30
+
31
+
**Goals:**
32
+
33
+
- Close a historical gap in one existing account from a bank's CSV export.
34
+
- Never corrupt synced data. An import must not be able to delete, reconcile, or
35
+
overwrite anything sync owns.
36
+
- Re-importing the same or an overlapping file must be a no-op, not a duplication.
37
+
- Surface plausible cross-source duplicates for human judgment before commit.
38
+
- Make a botched import reversible as a unit.
39
+
- Preserve the archive-then-replay property for CSV payloads.
40
+
41
+
**Non-Goals:**
42
+
43
+
- Creating accounts from a CSV. `account-management` forbids it, and a
44
+
connection-less account would have no balance snapshots, which would silently
45
+
break the net-worth report. Out of scope, deliberately.
46
+
- Importing a category column. Provenance for "Mint thought so in 2024" is an
47
+
unresolved question and `categorization_events.source` is a closed enum. Rules
48
+
cover the need for now.
49
+
- Saved per-account mapping profiles. Worth it only if imports recur; this is a
50
+
one-time gap fill.
51
+
- Fuzzy merchant matching. Duplicate detection stays on amount and date.
52
+
53
+
## Decisions
54
+
55
+
### 1. CSV is an additive writer, not a reuse of `ingestTransactions`
56
+
57
+
**Decision:** A new, insert-only ingestion path. It never reconciles, never
58
+
sweeps, never updates an existing row.
59
+
60
+
**Why:** Reusing `ingestTransactions` looks attractive — it's the same table and
61
+
the same `NormalizedTransaction` shape — but it would be a live bug. Its stale
62
+
sweep soft-deletes every pending row in the account not present in the feed; a
63
+
2023 backfill contains none of today's pending rows, so importing would silently
64
+
remove all of them. Its ±5-day reconciliation could also let an old CSV row hijack
65
+
a live pending row.
66
+
67
+
The seam is one level below the function: sync and CSV share the table and the
68
+
insert, but not the authority. Sync is authoritative for a window; CSV is additive
69
+
into a gap. Encoding that distinction as two writers is both safer and less code
70
+
than parameterizing one writer with a `mode` flag, which would leave the dangerous
71
+
paths one boolean away from a backfill.
72
+
73
+
**Alternative considered:** Add `authoritative: boolean` to `ingestTransactions`.
74
+
Rejected — the flag makes the hazardous branches reachable from the CSV path by
75
+
mistake, and the shared code would be only the INSERT statement.
76
+
77
+
### 2. Synthetic identity: content hash + occurrence index
78
+
79
+
**Decision:** `sfin_id = 'csv:' + hash(account_id, date, amount_cents,
80
+
description, occurrence_index)`, where `occurrence_index` is the row's ordinal
81
+
among identical rows within the same file.
82
+
83
+
**Why:** `sfin_id` is `NOT NULL` under a unique key, so CSV must invent one. A
84
+
plain content hash would collapse two genuinely distinct identical transactions
85
+
(two $4.75 coffees on one Tuesday) into a single row. The occurrence index keeps
86
+
them distinct while staying deterministic.
87
+
88
+
This yields idempotency with a pleasing property: because the grouping key *is*
89
+
the row's content, rows within a group are interchangeable. If a second export
90
+
lists the same group in a different order, or contains a superset of it, the set
91
+
of derived ids is unchanged for the rows that already exist, and only genuinely
92
+
new rows insert. Re-import of an identical or overlapping file is therefore a
93
+
no-op that never even reaches duplicate review.
94
+
95
+
**Alternative considered:** Rename `sfin_id` to `source_id` and add a `source`
96
+
discriminator. Honest, but it touches every query in the codebase, and `import_id`
97
+
(decision 5) already records true origin. Namespacing the value with a `csv:`
98
+
prefix keeps the lie small and greppable.
99
+
100
+
### 3. Archive on upload, decide later
101
+
102
+
**Decision:** The verbatim file is written to a new `imports` row at upload time
103
+
with `status = 'draft'`, before any parsing. The column mapping and the duplicate
104
+
decisions are stored on that same row. Wizard steps re-parse the archived bytes.
105
+
106
+
**Why:** This does double duty. It honors the existing "archive before
107
+
normalization" spine, and it solves where a multi-step wizard keeps its state
108
+
without stuffing a CSV into a session. Replay determinism requires the mapping and
109
+
the decisions to be archived alongside the bytes — bytes alone do not determine
110
+
the outcome, since a human chose the mapping and adjudicated the duplicates.
111
+
112
+
**Alternative considered:** A separate `raw_imports` (bytes) plus `imports`
113
+
(record). Rejected as over-normalization for a two-person app; the payload column
114
+
inside `imports` is written once and never mutated, which preserves the property
115
+
that matters.
116
+
117
+
### 4. Duplicate detection: amount and date, never description
118
+
119
+
**Decision:** A CSV row is flagged when the target account already holds a
120
+
non-removed transaction with the identical `amount_cents` and an effective date
121
+
within ±1 day. Descriptions are displayed side by side but never matched on.
122
+
Flagged rows default to **skip**.
123
+
124
+
**Why:** The two sources render the same merchant differently — SimpleFIN gives
125
+
the bank's raw string (`AMZN Mktp US*2K4LM9QR3`), the CSV gives its own rendering
126
+
(`Amazon`). Matching on description would catch almost nothing, which is the worst
127
+
outcome: a duplicate check that appears to run and silently passes everything. So
128
+
description is evidence for the human, not a filter.
129
+
130
+
Amount + date alone is deliberately loose and will throw occasional false
131
+
positives (a second identical coffee). That is what the review step is for; a
132
+
loose matcher plus a human beats a tight matcher that misses.
133
+
134
+
The ±1 day window rather than same-day: a CSV's single date column is often the
135
+
transaction date while SimpleFIN's `posted` can trail it, so strict same-day would
136
+
miss the very rows most at risk.
137
+
138
+
**Skip as the default is principled, not arbitrary.** Where both sources claim a
139
+
transaction, the synced row is strictly better: it carries a real SimpleFIN id so
140
+
it stays idempotent under future syncs, it holds the bank's own description, and
141
+
it already carries categorization history. The CSV row is a photocopy of a record
142
+
already held. Declining it loses nothing. The human's job in the wizard is to
143
+
catch the false positives and flip those few to keep — a much lighter chore than
144
+
adjudicating every row from scratch.
145
+
146
+
### 5. `import_id` on transactions: undo and origin in one column
147
+
148
+
**Decision:** `transactions.import_id INTEGER NULL REFERENCES imports (id)`.
149
+
NULL means synced.
150
+
151
+
**Why:** One nullable column serves three needs. Undo becomes "soft-remove where
152
+
`import_id = ?`". The ledger's source filter becomes `import_id IS NULL` /
153
+
`IS NOT NULL`. The origin line in the history panel joins through it. Retrofitting
154
+
this later would mean re-deriving hashes to work out which rows came from which
155
+
file.
156
+
157
+
Undo **soft-removes** (`removed_at = now`) rather than hard-deleting, matching the
158
+
existing stale-pending precedent and the house rule that history is never deleted.
159
+
Hard deletion would also collide with the append-only categorization event log,
160
+
since events reference `transaction_id`.
161
+
162
+
Undo sets `imports.status = 'undone'`. Re-importing a corrected file after an undo
163
+
generally produces different hashes (a fixed mapping parses different values), so
164
+
new rows insert cleanly. In the case where the same mapping is re-imported, the
165
+
insert path finds the soft-removed row by unique key and revives it
166
+
(`removed_at = NULL`, new `import_id`) rather than erroring.
167
+
168
+
### 6. Which date column to write
169
+
170
+
**Decision:** Map the CSV's date to `posted`, and set `pending = 0`. If the file
171
+
exposes a distinct transaction date, map it to `transacted_at`; otherwise leave it
172
+
NULL.
173
+
174
+
**Why:** Reports scope on `pending = 0` (`reports.ts:7`), not on `posted IS NOT
175
+
NULL`, so either choice lands in reports. But a row with `pending = 0` and `posted
176
+
IS NULL` is a state sync has never produced, and inventing novel states for
177
+
downstream queries to encounter is the riskier path. A CSV export is by definition
178
+
a statement of settled history, so `pending = 0` with `posted` set matches the
179
+
shape sync emits for posted rows, and every existing query behaves identically.
180
+
181
+
The residual imprecision — the file's single date column may be the transaction
182
+
date rather than the post date — is absorbed by the ±1 day duplicate window.
183
+
184
+
### 7. Amount parsing: a CSV pre-pass, not a change to `parseAmountToCents`
185
+
186
+
**Decision:** Normalize CSV negative conventions (`(12.34)`, `12.34-`, currency
187
+
symbols) to a signed decimal string, then hand off to the existing
188
+
`parseAmountToCents`. Support a single signed column, and separate debit/credit
189
+
columns, as distinct mapping modes.
190
+
191
+
**Why:** `parseAmountToCents` (`normalize.ts:48`) is carefully float-free and its
192
+
regex `^([+-]?)(\d*)(?:\.(\d+))?$` is a deliberate contract for SimpleFIN's
193
+
numeric strings. Loosening it to accept accountant's parentheses would weaken
194
+
validation on the sync path to serve the CSV path. A pre-pass keeps each source's
195
+
tolerances where they belong.
196
+
197
+
### 8. Categorization comes free
198
+
199
+
**Decision:** Call `applyRulesToUncategorized(db)` once after commit.
200
+
201
+
**Why:** It's already unscoped and database-wide (`rules.ts:168`), and it already
202
+
skips transactions whose latest event is `manual`. Backfilled history categorizes
203
+
itself against existing rules with no new code and no new event source.
204
+
205
+
### 9. Surfacing: filter and popover, no ledger badge
206
+
207
+
**Decision:** `LedgerFilters` gains `source?: 'synced' | 'imported'`; `LedgerRow`
208
+
gains origin; the existing history panel gains an origin line. No per-row badge.
209
+
The import record lives in Settings, not the main nav.
210
+
211
+
**Why:** Within a backfilled range every row is imported and above it none are, so
212
+
a per-row badge would be present on 100% of rows where it appears — carrying no
213
+
information exactly where it's densest, while competing with the existing
214
+
categorization badge, which answers a different question (*who chose this
215
+
category*, not *where did this row come from*).
216
+
217
+
A seam marker at the import boundary was considered and rejected on stronger
218
+
grounds: there is no boundary. Because the short-history account accumulates
219
+
synced rows daily, synced and imported rows genuinely interleave across the same
220
+
dates. A seam would be a drawn line where none exists.
221
+
222
+
Origin interest is episodic — it matters during and shortly after an import, then
223
+
it is just history. A filter answers it on demand; the popover answers it for a
224
+
specific row where the question is actually asked.
225
+
226
+
## Risks / Trade-offs
227
+
228
+
- **A wrong column mapping silently imports garbage** (e.g. dates off by a
229
+
century, amounts inverted) → Undo via `import_id`, plus a preview step that
230
+
states the parsed date range and row count before commit, where an inverted or
231
+
misparsed column is obvious at a glance.
232
+
233
+
- **The overlap region is wide, so duplicate review could have real volume** →
234
+
Skip-by-default means clicking through without reading yields the safe outcome
235
+
(synced data retained, no double-counting). The cost of inattention is a missing
236
+
photocopy, not corrupted reporting.
237
+
238
+
- **A false positive silently drops a real transaction** (two genuinely distinct
239
+
identical charges on one day, one already synced) → This is the flip side of
240
+
skip-by-default and is the accepted trade. The wizard shows both rows side by
241
+
side specifically to make it catchable, and the import record retains the raw
242
+
file, so a missed row can be recovered by re-importing with a corrected
243
+
decision.
244
+
245
+
- **`sfin_id` now holds values that are not SimpleFIN ids** → Namespaced `csv:`
246
+
prefix makes them greppable, and `import_id` is the authoritative origin field.
247
+
Accepted as a small, contained dishonesty rather than a codebase-wide rename.
248
+
249
+
- **Abandoned drafts accumulate** → Low volume by nature; drafts are invisible to
250
+
every surface except the import record. Not worth a reaper.
251
+
252
+
- **Two writers to `transactions` could drift apart** → The CSV writer is
253
+
insert-only and deliberately shares no code with `ingestTransactions` beyond the
254
+
row shape. The risk is a future schema change updating one and not the other;
255
+
mitigated by both paths being covered by tests over the same table.
256
+
257
+
## Migration Plan
258
+
259
+
One additive migration (`005_csv_import.sql`):
260
+
261
+
- `CREATE TABLE imports` — `id`, `account_id` (FK, NOT NULL), `filename`,
262
+
`uploaded_at`, `payload` (verbatim, never mutated), `mapping` (JSON), `decisions`
263
+
(JSON), `status` CHECK in (`draft`, `committed`, `undone`), `committed_at`,
264
+
`undone_at`.
265
+
- `ALTER TABLE transactions ADD COLUMN import_id INTEGER REFERENCES imports (id)`
266
+
— nullable, NULL meaning synced, so every existing row is correct without a
267
+
backfill.
268
+
- Index on `transactions (import_id)` for undo and the source filter.
269
+
270
+
No changes to existing columns and no data rewrite, so the migration is
271
+
forward-only and safe to re-run against a populated database. Rollback is dropping
272
+
the table and column; imported rows would need removal first, which the undo flow
273
+
already performs.
274
+
275
+
## Open Questions
276
+
277
+
- Should the preview step hard-refuse a file whose parsed range extends into the
278
+
future or predates the account's opening, as a cheap guard against a
279
+
catastrophically wrong date mapping — or is stating the range enough?
280
+
- Does the import record in Settings need to list the individual skipped rows, or
281
+
is a count sufficient?
+93
openspec/changes/csv-backfill-import/proposal.md
+93
openspec/changes/csv-backfill-import/proposal.md
···
1
+
## Why
2
+
3
+
One connected account only returns about a week of transaction history from
4
+
SimpleFIN, while another returns the full 90 days. The short-history account
5
+
therefore has a permanent hole in its past that no sync will ever fill — the
6
+
lookback window advances, it does not reach back. Reports that span more than a
7
+
week are silently wrong for that account, and no amount of waiting fixes it.
8
+
9
+
The bank does publish that history as a CSV export. This change lets a user pour
10
+
that file into the account it belongs to, once, to close the gap.
11
+
12
+
## What Changes
13
+
14
+
- A CSV import flow, reached from Settings: upload a file, attribute it to one
15
+
existing account, confirm the column mapping, review potential duplicates,
16
+
commit.
17
+
- Imports are **additive only**. A CSV never reconciles pending transactions,
18
+
never marks accounts inactive, never removes rows, and never overwrites a
19
+
synced transaction. It inserts, or it does nothing.
20
+
- Imported transactions get a **synthetic stable id** derived from their content,
21
+
so re-importing the same file (or an overlapping range from a second file) is
22
+
a no-op rather than a duplication.
23
+
- **Duplicate review**: rows matching an existing transaction in the same account
24
+
by amount and near-identical date are surfaced for human judgment before
25
+
commit, defaulting to skip. Descriptions are shown side by side to inform the
26
+
decision but are not used to match, because the two sources render the same
27
+
merchant differently.
28
+
- **Undo**: every imported transaction records which import produced it, so a
29
+
botched mapping can be reversed as a unit.
30
+
- Raw uploaded bytes are archived verbatim before any normalization, alongside
31
+
the mapping and duplicate decisions, preserving the existing "normalization is
32
+
a replayable pure function of an archived payload" property.
33
+
- The ledger gains a **source filter** (synced / imported), and a transaction's
34
+
origin appears in its existing history panel. No new per-row ledger chrome:
35
+
within a backfilled range every row is imported, so a per-row badge would carry
36
+
no information where it appeared most.
37
+
- Rules run over imported transactions after commit, so backfilled history
38
+
categorizes itself against rules that already exist.
39
+
40
+
Not in this change: creating accounts from a CSV, importing a category column
41
+
from a CSV, and saved per-account mapping profiles. See Impact.
42
+
43
+
## Capabilities
44
+
45
+
### New Capabilities
46
+
- `csv-import`: Uploading a CSV of historical transactions, archiving it
47
+
verbatim, mapping its columns, detecting and adjudicating potential duplicates
48
+
against existing data, additively ingesting the survivors into one existing
49
+
account, recording the import as a reversible unit, and undoing it.
50
+
51
+
### Modified Capabilities
52
+
- `reporting`: The transaction ledger requirement gains a source filter (synced
53
+
vs. imported) that composes with the existing filters, and the ledger surfaces
54
+
a transaction's origin in its detail/history panel.
55
+
56
+
## Impact
57
+
58
+
**Affected specs**: new `csv-import`; modified `reporting` (transaction ledger
59
+
requirement).
60
+
61
+
**Deliberately unaffected**: `account-management` requires that "the system SHALL
62
+
create account records only from sync data, never via in-app creation." This
63
+
change honors that rule rather than repealing it — a CSV must be attributed to an
64
+
account that sync already discovered. Consequently no CSV-only account exists, no
65
+
balance snapshots are invented, and the net-worth report is untouched.
66
+
`simplefin-sync` is likewise unchanged; its reconciliation and stale-sweep
67
+
behavior remain the exclusive authority of sync.
68
+
69
+
**Affected code**:
70
+
- `migrations/` — new migration: a `raw_imports` archive table; a nullable
71
+
`import_id` FK on `transactions`.
72
+
- `src/lib/server/services/sync.ts` — `ingestTransactions` is *not* reused. It
73
+
treats its input as authoritative for the account (its stale-pending sweep would
74
+
soft-delete every pending row absent from a backfill CSV, and its ±5-day
75
+
reconciliation could let an old CSV row hijack a live pending row). CSV needs a
76
+
separate, insert-only writer.
77
+
- `src/lib/server/services/normalize.ts` — `parseAmountToCents` is reusable but
78
+
its regex rejects the parenthesized-negative and trailing-minus conventions
79
+
common in bank CSV exports; it needs a CSV-flavored pre-pass rather than a
80
+
change to its own contract.
81
+
- `src/lib/server/services/ledger.ts` — `LedgerFilters` gains `source`;
82
+
`LedgerRow` gains origin.
83
+
- `src/lib/server/services/rules.ts` — `applyRulesToUncategorized` is called
84
+
post-commit. It is already unscoped and database-wide, so it needs no change.
85
+
- New service for CSV parsing/mapping/duplicate detection; new Settings routes for
86
+
the import wizard and the import record.
87
+
88
+
**Risk**: The overlap between a CSV's range and existing synced data is a region,
89
+
not a boundary — the short-history account accumulates synced rows daily, so both
90
+
sources can claim the same stretch of time. Duplicate review is therefore the
91
+
central safeguard, not an edge case. Where both sources claim a transaction, the
92
+
synced row wins: it carries a real SimpleFIN id, reconciles correctly under future
93
+
syncs, and already holds categorization history.
+236
openspec/changes/csv-backfill-import/specs/csv-import/spec.md
+236
openspec/changes/csv-backfill-import/specs/csv-import/spec.md
···
1
+
## ADDED Requirements
2
+
3
+
### Requirement: Import attribution to an existing account
4
+
5
+
The system SHALL require every CSV import to be attributed to exactly one account
6
+
that already exists in the database. The system SHALL NOT create, rename, or
7
+
otherwise modify an account as a result of an import, preserving the
8
+
`account-management` rule that accounts originate only from sync data. Accounts in
9
+
state `HIDDEN` SHALL NOT be offered as import targets.
10
+
11
+
#### Scenario: User selects a target account
12
+
13
+
- **WHEN** a user begins a CSV import
14
+
- **THEN** the system requires them to choose one existing non-hidden account, and
15
+
every transaction ingested from that file is attached to that account
16
+
17
+
#### Scenario: No account selected
18
+
19
+
- **WHEN** a user attempts to proceed without choosing a target account
20
+
- **THEN** the import does not proceed and no rows are ingested
21
+
22
+
### Requirement: Verbatim archival before normalization
23
+
24
+
The system SHALL store the uploaded file's bytes verbatim in an `imports` record
25
+
before any parsing, mapping, or normalization occurs. The archived payload SHALL
26
+
never be mutated or deleted by the application. The system SHALL store the chosen
27
+
column mapping and the duplicate decisions on the same record, so that the
28
+
outcome of an import is a deterministic function of the archived record alone.
29
+
30
+
#### Scenario: File archived on upload
31
+
32
+
- **WHEN** a user uploads a CSV file
33
+
- **THEN** an `imports` row is committed containing the exact uploaded bytes with
34
+
status `draft`, before any row is parsed
35
+
36
+
#### Scenario: Import is replayable
37
+
38
+
- **WHEN** the import logic is re-run over an archived record with its stored
39
+
mapping and decisions
40
+
- **THEN** it produces the same set of transactions as the original run
41
+
42
+
#### Scenario: Unparseable file
43
+
44
+
- **WHEN** an uploaded file cannot be parsed as CSV
45
+
- **THEN** the archived record is retained, the failure is stated plainly with the
46
+
reason, and no transactions are ingested
47
+
48
+
### Requirement: Column mapping
49
+
50
+
The system SHALL detect a CSV's date, amount, and description columns from its
51
+
header row where possible, and SHALL require the user to confirm or correct the
52
+
mapping before commit. The system SHALL support amounts expressed as a single
53
+
signed column and as separate debit and credit columns. The system SHALL accept
54
+
the negative conventions common to bank exports, including parenthesized values
55
+
and trailing minus signs, and SHALL convert amounts to integer cents without
56
+
floating-point arithmetic. The system SHALL map the file's date to the
57
+
transaction's posted timestamp and record imported transactions as not pending;
58
+
when the file exposes a distinct transaction date, the system SHALL map it to the
59
+
transaction date.
60
+
61
+
#### Scenario: Headers auto-detected
62
+
63
+
- **WHEN** a CSV's header row contains recognizable date, amount, and description
64
+
columns
65
+
- **THEN** the system pre-selects them and presents the mapping for confirmation
66
+
67
+
#### Scenario: User corrects a wrong guess
68
+
69
+
- **WHEN** the auto-detected mapping is wrong
70
+
- **THEN** the user can reassign any field to any column before committing
71
+
72
+
#### Scenario: Parenthesized negative
73
+
74
+
- **WHEN** an amount column contains `(12.34)`
75
+
- **THEN** it is ingested as `-1234` integer cents
76
+
77
+
#### Scenario: Separate debit and credit columns
78
+
79
+
- **WHEN** a file expresses amounts as separate debit and credit columns
80
+
- **THEN** the user can map both, and each row resolves to a single signed integer
81
+
cent amount
82
+
83
+
#### Scenario: Imported rows appear in reports
84
+
85
+
- **WHEN** an imported transaction is committed
86
+
- **THEN** it is recorded as not pending with its posted timestamp set, and
87
+
appears in monthly reports for the month of that timestamp
88
+
89
+
### Requirement: Stable synthetic identity
90
+
91
+
The system SHALL derive a stable, deterministic identifier for each imported
92
+
transaction from its content — the target account, date, amount, and description —
93
+
combined with an occurrence index distinguishing rows that are otherwise
94
+
identical within the same file. Identifiers SHALL be namespaced so they are
95
+
distinguishable from provider-supplied identifiers. Importing a file whose rows
96
+
have already been ingested under the same identifiers SHALL NOT create duplicate
97
+
rows.
98
+
99
+
#### Scenario: Same file imported twice
100
+
101
+
- **WHEN** a user imports a file and then imports the identical file again
102
+
- **THEN** no new transactions are created and the second import reports that
103
+
every row already exists
104
+
105
+
#### Scenario: Overlapping files
106
+
107
+
- **WHEN** a user imports a January–March file and then a February–April file into
108
+
the same account
109
+
- **THEN** the February–March rows are recognized as already ingested and only the
110
+
April rows are added
111
+
112
+
#### Scenario: Genuinely identical transactions
113
+
114
+
- **WHEN** a file contains two rows with the same date, amount, and description,
115
+
representing two real transactions
116
+
- **THEN** both are ingested as separate transactions
117
+
118
+
### Requirement: Potential duplicate review
119
+
120
+
The system SHALL, before commit, identify each candidate row that would create a
121
+
transaction resembling one the target account already holds — matching on
122
+
identical amount and an effective date within one day, against non-removed
123
+
transactions regardless of their origin. The system SHALL NOT match on
124
+
description, because the same transaction is rendered differently by different
125
+
sources. The system SHALL present each flagged pair to the user with both
126
+
descriptions shown, and SHALL default the flagged row to being skipped. The user
127
+
SHALL be able to override any flagged row to be imported.
128
+
129
+
#### Scenario: Cross-source duplicate flagged
130
+
131
+
- **WHEN** a candidate row has the same amount and date as an existing synced
132
+
transaction in the target account, but a differently worded description
133
+
- **THEN** the row is flagged for review, both descriptions are shown side by
134
+
side, and it defaults to skipped
135
+
136
+
#### Scenario: Default skip retains the synced row
137
+
138
+
- **WHEN** a user commits an import without changing any duplicate decision
139
+
- **THEN** every flagged row is skipped, the existing transactions are left
140
+
untouched, and no duplicate is created
141
+
142
+
#### Scenario: User keeps a false positive
143
+
144
+
- **WHEN** a flagged row is in fact a distinct transaction and the user marks it to
145
+
be imported
146
+
- **THEN** it is ingested as a new transaction alongside the existing one
147
+
148
+
#### Scenario: Preview before commit
149
+
150
+
- **WHEN** a mapped file is ready for review
151
+
- **THEN** the system states the parsed date range, the number of rows to be
152
+
imported, the number already present, and the number flagged as potential
153
+
duplicates
154
+
155
+
### Requirement: Additive-only ingestion
156
+
157
+
An import SHALL only insert transactions. The system SHALL NOT, as a result of an
158
+
import, modify or remove any existing transaction, reconcile pending transactions,
159
+
change any account's state, or write balance snapshots. Reconciliation and
160
+
feed-authority behavior SHALL remain exclusive to sync.
161
+
162
+
#### Scenario: Pending transactions untouched
163
+
164
+
- **WHEN** an import is committed into an account that holds pending transactions
165
+
absent from the CSV
166
+
- **THEN** those pending transactions remain unchanged and are not removed
167
+
168
+
#### Scenario: Existing transaction not overwritten
169
+
170
+
- **WHEN** a candidate row resolves to an identifier already present in the account
171
+
- **THEN** the existing transaction is left exactly as it was
172
+
173
+
#### Scenario: No balance snapshots
174
+
175
+
- **WHEN** an import is committed
176
+
- **THEN** no balance snapshot is written and the net worth report is unaffected
177
+
178
+
#### Scenario: Account state unaffected
179
+
180
+
- **WHEN** an import is committed into an `ACTIVE` account
181
+
- **THEN** the account's state and `last_successful_data_at` are unchanged
182
+
183
+
### Requirement: Rule application after import
184
+
185
+
The system SHALL apply existing categorization rules to imported transactions once
186
+
the import is committed, using the same rule precedence and the same `rule`
187
+
categorization event source as sync. Imported transactions SHALL NOT introduce a
188
+
new categorization event source.
189
+
190
+
#### Scenario: Backfilled history categorizes itself
191
+
192
+
- **WHEN** an import is committed and existing rules match some of the imported
193
+
transactions
194
+
- **THEN** those transactions are categorized with `rule` events recording the
195
+
winning rule, exactly as if they had arrived by sync
196
+
197
+
#### Scenario: Manual decisions respected
198
+
199
+
- **WHEN** rules are applied after an import
200
+
- **THEN** transactions whose latest categorization event is `manual` are not
201
+
re-categorized
202
+
203
+
### Requirement: Import record and undo
204
+
205
+
The system SHALL record every import and surface the record in Settings, showing
206
+
the file name, target account, parsed date range, counts of rows imported and
207
+
skipped, and the commit time. Each imported transaction SHALL record which import
208
+
produced it. The system SHALL allow a committed import to be undone as a unit,
209
+
removing the transactions it produced without deleting any categorization event
210
+
history. An undone import SHALL be re-importable after correcting its mapping.
211
+
212
+
#### Scenario: Import listed in settings
213
+
214
+
- **WHEN** a user opens the import record in Settings
215
+
- **THEN** each import is listed with its file name, account, date range, counts,
216
+
and commit time
217
+
218
+
#### Scenario: Undo removes only that import's rows
219
+
220
+
- **WHEN** a user undoes an import
221
+
- **THEN** exactly the transactions that import produced are removed from the
222
+
ledger and reports, no synced transaction is affected, and the import is marked
223
+
undone
224
+
225
+
#### Scenario: Event history survives undo
226
+
227
+
- **WHEN** an imported transaction had been categorized and its import is undone
228
+
- **THEN** the transaction no longer appears in the ledger or reports, and its
229
+
categorization events remain in the append-only log
230
+
231
+
#### Scenario: Re-import after a corrected mapping
232
+
233
+
- **WHEN** a user undoes an import made with a wrong mapping and re-imports the
234
+
same file with a corrected mapping
235
+
- **THEN** the corrected transactions are ingested and the undone import's rows do
236
+
not reappear
+32
openspec/changes/csv-backfill-import/specs/reporting/spec.md
+32
openspec/changes/csv-backfill-import/specs/reporting/spec.md
···
1
+
## MODIFIED Requirements
2
+
3
+
### Requirement: Transaction ledger
4
+
The system SHALL provide a ledger view of transactions filterable by account, category (including uncategorized), month, pending status, and source (synced vs. imported), and searchable by free text. Text search SHALL match case-insensitively against the raw description, payee, and memo fields and against the effective displayed name produced by rule display-name overlays, so search finds what the user sees. Search SHALL compose with all structured filters, and the source filter SHALL compose with all other filters. The ledger shows date, account, displayed name (rule overlay applied when present), amount, category, and a provenance indicator (rule vs. person). The ledger SHALL NOT mark a transaction's source on the row itself; a transaction's origin — synced from a connection, or imported from a named file — SHALL be shown in its history panel. The ledger is the surface for manual categorization and for creating rules from transactions.
5
+
6
+
#### Scenario: Filter to uncategorized
7
+
- **WHEN** a user filters the ledger to uncategorized transactions
8
+
- **THEN** only transactions with no current category are listed, ready for manual assignment
9
+
10
+
#### Scenario: Provenance indicator
11
+
- **WHEN** a categorized transaction is displayed
12
+
- **THEN** the row indicates whether the category came from a rule, a person (with their identity), or reconciliation carry-forward
13
+
14
+
#### Scenario: Search matches a renamed transaction
15
+
- **WHEN** a rule renames "ACH TRANSFER 4417" to display as "Rent" and a user searches the ledger for "rent"
16
+
- **THEN** the transaction is found, even though no raw field contains "rent"
17
+
18
+
#### Scenario: Search composes with filters
19
+
- **WHEN** a user searches for "netflix" with a month filter active
20
+
- **THEN** only that month's transactions matching the text (raw fields or displayed name) are listed
21
+
22
+
#### Scenario: Filter to imported transactions
23
+
- **WHEN** a user filters the ledger by source to imported transactions
24
+
- **THEN** only transactions produced by a CSV import are listed
25
+
26
+
#### Scenario: Source filter composes
27
+
- **WHEN** a user filters by source to imported with an account and month filter active
28
+
- **THEN** only that account's imported transactions in that month are listed
29
+
30
+
#### Scenario: Origin shown in history
31
+
- **WHEN** a user opens the history panel of a transaction that came from a CSV import
32
+
- **THEN** the panel states that it was imported and names the file and import date, distinguishing it from a transaction synced from a connection
+141
openspec/changes/csv-backfill-import/tasks.md
+141
openspec/changes/csv-backfill-import/tasks.md
···
1
+
## 1. Schema
2
+
3
+
- [ ] 1.1 Add `migrations/005_csv_import.sql` creating the `imports` table: `id`,
4
+
`account_id` (NOT NULL, FK to accounts), `filename`, `uploaded_at`, `payload`
5
+
(verbatim bytes, never mutated), `mapping` (JSON, nullable), `decisions` (JSON,
6
+
nullable), `status` NOT NULL DEFAULT `'draft'` CHECK in (`draft`, `committed`,
7
+
`undone`), `committed_at`, `undone_at`.
8
+
- [ ] 1.2 In the same migration, `ALTER TABLE transactions ADD COLUMN import_id
9
+
INTEGER REFERENCES imports (id)` (nullable; NULL means synced) and add an index
10
+
on `transactions (import_id)`.
11
+
- [ ] 1.3 Extend `src/lib/server/db.test.ts` to assert the migration applies to a
12
+
populated database and that existing transactions read back with
13
+
`import_id IS NULL`.
14
+
15
+
## 2. CSV parsing and mapping (pure)
16
+
17
+
- [ ] 2.1 Add `@std/csv` (JSR) as a dependency. Do not hand-roll parsing —
18
+
quoted delimiters, embedded newlines, and BOMs are the failure modes.
19
+
- [ ] 2.2 Create `src/lib/server/services/csv-import.ts` with a pure
20
+
`parseCsv(payloadText)` returning header row plus data rows, tolerating a BOM
21
+
and CRLF line endings.
22
+
- [ ] 2.3 Implement pure `detectMapping(headers)` returning best-guess column
23
+
assignments for date, amount (single signed column or debit/credit pair), and
24
+
description, plus optional payee/memo and a distinct transaction-date column.
25
+
- [ ] 2.4 Implement pure `normalizeCsvAmount(raw)` handling parenthesized
26
+
negatives `(12.34)`, trailing minus `12.34-`, currency symbols, and thousands
27
+
separators, emitting a signed decimal string for the existing
28
+
`parseAmountToCents`. Do not loosen `parseAmountToCents` itself — its strict
29
+
regex is the sync path's contract.
30
+
- [ ] 2.5 Implement pure `normalizeCsv(payloadText, mapping, accountId)` producing
31
+
candidate rows with `posted` set from the file's date, `pending = 0`,
32
+
`transacted_at` set only when the mapping names a distinct transaction-date
33
+
column, and amounts as integer cents.
34
+
- [ ] 2.6 Implement the synthetic identity: `csv:` + hash over (accountId, date,
35
+
amountCents, description) plus an occurrence index among identical rows within
36
+
the file.
37
+
- [ ] 2.7 Unit-test 2.2–2.6 in `csv-import.test.ts`: BOM/CRLF, both amount modes,
38
+
each negative convention, date parsing, and that two identical rows receive
39
+
distinct ids while a reordered or superset file re-derives the same id set.
40
+
41
+
## 3. Duplicate detection
42
+
43
+
- [ ] 3.1 Implement `findPotentialDuplicates(db, accountId, candidates)` matching
44
+
each candidate against non-removed transactions in the account by identical
45
+
`amount_cents` and effective date within ±1 day (compare against
46
+
`COALESCE(posted, transacted_at)`). Never match on description.
47
+
- [ ] 3.2 Return, per candidate, one of: `new`, `already-present` (its synthetic
48
+
id already exists — never surfaced to the user), or `flagged` with the existing
49
+
transaction's date, amount, and description for side-by-side display.
50
+
- [ ] 3.3 Test: a cross-source duplicate with differently worded descriptions is
51
+
flagged; a row one day off is flagged; a row two days off is not; an
52
+
already-ingested row reports `already-present` rather than `flagged`.
53
+
54
+
## 4. Import lifecycle and additive ingestion
55
+
56
+
- [ ] 4.1 Implement `createDraftImport(db, accountId, filename, payload)` writing
57
+
the `imports` row with verbatim payload and `status = 'draft'` before any
58
+
parsing.
59
+
- [ ] 4.2 Implement `saveMapping` and `saveDecisions` persisting to the draft row,
60
+
so the archived record alone determines the outcome.
61
+
- [ ] 4.3 Implement `commitImport(db, importId)` — an **insert-only** writer.
62
+
It MUST NOT reconcile, MUST NOT sweep stale pending rows, MUST NOT update
63
+
existing transactions, MUST NOT write balance snapshots, and MUST NOT touch
64
+
account state. Do not call or extend `ingestTransactions`; its stale-pending
65
+
sweep (`sync.ts:291-301`) would soft-delete every pending row absent from the
66
+
CSV.
67
+
- [ ] 4.4 In `commitImport`, skip candidates marked skipped and those whose
68
+
synthetic id already exists; revive a soft-removed row matching the unique key
69
+
by clearing `removed_at` and reassigning `import_id`; stamp `import_id` on every
70
+
inserted row; set `status = 'committed'` and `committed_at`. Run the whole
71
+
commit in one transaction.
72
+
- [ ] 4.5 Call `applyRulesToUncategorized(db)` after the commit transaction
73
+
succeeds. No new categorization event source.
74
+
- [ ] 4.6 Test: importing into an account holding pending rows leaves them
75
+
untouched; no snapshots are written; account state and
76
+
`last_successful_data_at` are unchanged; re-importing the same file inserts
77
+
nothing; an overlapping file inserts only the new rows; committed rows carry
78
+
`import_id` and are picked up by rules.
79
+
80
+
## 5. Undo
81
+
82
+
- [ ] 5.1 Implement `undoImport(db, importId)` soft-removing (`removed_at = now`)
83
+
every transaction with that `import_id`, setting `status = 'undone'` and
84
+
`undone_at`, in one transaction. Never hard-delete — categorization events
85
+
reference `transaction_id` and the event log is append-only.
86
+
- [ ] 5.2 Test: undo removes exactly that import's rows from ledger and report
87
+
scope, leaves synced rows untouched, preserves categorization events for the
88
+
removed rows, and a corrected re-import afterwards does not resurrect the old
89
+
rows.
90
+
91
+
## 6. Ledger surfacing
92
+
93
+
- [ ] 6.1 Add `source?: 'synced' | 'imported'` to `LedgerFilters` in
94
+
`src/lib/server/services/ledger.ts`, translating to `t.import_id IS NULL` /
95
+
`IS NOT NULL`, composing with every existing filter.
96
+
- [ ] 6.2 Add origin to `LedgerRow` (import id, file name, import date; null when
97
+
synced) via a join through `import_id`.
98
+
- [ ] 6.3 Test in `ledger.test.ts`: the source filter composes with account,
99
+
month, category, pending, and text search.
100
+
- [ ] 6.4 Add the source filter control to the ledger page alongside the existing
101
+
filters. Add no per-row badge — within a backfilled range every row is imported,
102
+
so a per-row mark carries no information where it appears.
103
+
- [ ] 6.5 Add an origin line to the top of the existing history panel: "Imported
104
+
from `<file>` · `<date>`" or "Synced from SimpleFIN".
105
+
106
+
## 7. Import wizard (Settings)
107
+
108
+
- [ ] 7.1 Add a Settings route for the import wizard. Step 1: file upload plus
109
+
target-account picker (non-hidden accounts only), archiving on submit and
110
+
redirecting to the draft's id.
111
+
- [ ] 7.2 Step 2: mapping confirmation, pre-filled from `detectMapping`, every
112
+
field reassignable, with a few parsed sample rows rendered so a wrong guess is
113
+
visible.
114
+
- [ ] 7.3 Step 3: preview stating the parsed date range, rows to import, rows
115
+
already present, and rows flagged — the guard against a catastrophically wrong
116
+
date mapping.
117
+
- [ ] 7.4 Step 4: duplicate review listing each flagged row against its existing
118
+
match, both descriptions shown, defaulting to skip, each flippable to keep.
119
+
- [ ] 7.5 Commit action, then a result summary linking to the ledger filtered to
120
+
that account and source.
121
+
- [ ] 7.6 Style per DESIGN.md: tabular numerals on money and dates, calm empty and
122
+
error states (one sentence plus one action), no new red unless data is at risk.
123
+
124
+
## 8. Import record (Settings)
125
+
126
+
- [ ] 8.1 Add a Settings section listing imports: file name, account, parsed date
127
+
range, rows imported, rows skipped, commit time, status. Not in the main nav.
128
+
- [ ] 8.2 Add the undo action with a confirmation stating exactly how many
129
+
transactions will be removed.
130
+
- [ ] 8.3 Render an empty state for the no-imports-yet case.
131
+
132
+
## 9. Verification
133
+
134
+
- [ ] 9.1 Run `deno task test` and `deno task check`.
135
+
- [ ] 9.2 Drive the wizard end to end against a real bank CSV export in dev:
136
+
confirm the gap closes in the ledger, the month reports for backfilled months
137
+
change as expected, and the net worth chart is unchanged (no snapshots written).
138
+
- [ ] 9.3 Verify a sync runs cleanly after an import: pending rows still
139
+
reconcile, and no imported row is disturbed.
140
+
- [ ] 9.4 Exercise undo on a real import and confirm the ledger and reports return
141
+
to their pre-import state.