A self-hosted household ledger.
csv backfill import: additive-only writer, duplicate review, undo, ledger source filter
Fills the permanent history gap on accounts whose SimpleFIN feed reaches back
only a week. Imports attach to an account sync already discovered; they only
insert, never reconcile, sweep, or overwrite anything sync owns.
Deliberately does not reuse ingestTransactions: it treats its feed as
authoritative and its stale-pending sweep would soft-delete every pending row
absent from a backfill CSV.
Date-only values anchor at noon UTC, not midnight — formatDay renders local, so
midnight displayed a day early across the Americas and a month's first day would
have read as the previous month while reporting under the correct one. The
duplicate window follows, counting whole UTC calendar days rather than 86400s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
author
graham.systems
co-author
Claude Opus 4.8
date
(Jul 16, 2026, 6:29 PM -0700)
commit
fc96b968
fc96b968944b783e6f6a90182b4f69886747f5d6
parent
8e7c1e71
8e7c1e712cc9c9aa92bd67f469acc4cf6992a0ec
change-id
rlpxpono
rlpxponosqowlmnmmlzkrtvrxvqxqryy
graham.systems
co-author
+2674
-52
+10
deno.lock
+10
deno.lock
···
11
11
"npm:@sveltejs/kit@^2.63.0": "2.69.2_@sveltejs+vite-plugin-svelte@7.2.0__svelte@5.56.4__vite@8.1.4___@types+node@26.1.1__@types+node@26.1.1_svelte@5.56.4_typescript@6.0.3_vite@8.1.4__@types+node@26.1.1_@types+node@26.1.1",
12
12
"npm:@sveltejs/vite-plugin-svelte@^7.1.2": "7.2.0_svelte@5.56.4_vite@8.1.4__@types+node@26.1.1_@types+node@26.1.1",
13
13
"npm:@types/node@^26.1.1": "26.1.1",
14
+
"npm:csv-parse@^5.6.0": "5.6.0",
14
15
"npm:poline@~0.13.1": "0.13.1",
15
16
"npm:svelte-check@*": "4.7.2_svelte@5.56.4_typescript@6.0.3",
16
17
"npm:svelte-check@^4.6.0": "4.7.2_svelte@5.56.4_typescript@6.0.3",
···
18
19
"npm:typescript@^6.0.3": "6.0.3",
19
20
"npm:vite@*": "8.1.4_@types+node@26.1.1",
20
21
"npm:vite@^8.0.16": "8.1.4_@types+node@26.1.1"
22
+
},
23
+
"jsr": {
24
+
"@std/streams@1.0.17": {
25
+
"integrity": "7859f3d9deed83cf4b41f19223d4a67661b3d3819e9fc117698f493bf5992140"
26
+
}
21
27
},
22
28
"npm": {
23
29
"@atproto-labs/did-resolver@0.3.5": {
···
645
651
"integrity": "sha512-es1U2+YTtzpwkxVLwAFdSpaIMyQaq0PBgm3YD1W3Qpsn1NAmO3KSgZfu+oGSWVu6NvLHoHCV/aYcsE5wiB7ALg==",
646
652
"scripts": true
647
653
},
654
+
"csv-parse@5.6.0": {
655
+
"integrity": "sha512-l3nz3euub2QMg5ouu5U09Ew9Wf6/wQ8I++ch1loQ0ljmzhmfZYrH9fflS22i/PQEvsPvxCwxgz5q7UB8K1JO4Q=="
656
+
},
648
657
"deepmerge@4.3.1": {
649
658
"integrity": "sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A=="
650
659
},
···
1052
1061
"npm:@sveltejs/kit@^2.63.0",
1053
1062
"npm:@sveltejs/vite-plugin-svelte@^7.1.2",
1054
1063
"npm:@types/node@^26.1.1",
1064
+
"npm:csv-parse@^5.6.0",
1055
1065
"npm:poline@~0.13.1",
1056
1066
"npm:svelte-check@^4.6.0",
1057
1067
"npm:svelte@^5.56.1",
+28
migrations/005_csv_import.sql
+28
migrations/005_csv_import.sql
···
1
+
-- CSV backfill import (csv-backfill-import). A connected account whose SimpleFIN
2
+
-- feed reaches back only a week has a permanent hole in its past that no sync can
3
+
-- fill; the bank publishes that history as a CSV. Imports are additive only: they
4
+
-- attach to an account sync already discovered, and never reconcile, sweep, or
5
+
-- overwrite anything sync owns.
6
+
7
+
-- One row per uploaded file. `payload` is verbatim and never mutated, mirroring
8
+
-- raw_syncs. `mapping` and `decisions` live here too because the outcome of an
9
+
-- import is a function of the bytes AND the human's choices — bytes alone would
10
+
-- not replay deterministically.
11
+
CREATE TABLE imports (
12
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
13
+
account_id TEXT NOT NULL REFERENCES accounts (id),
14
+
filename TEXT,
15
+
uploaded_at TEXT NOT NULL,
16
+
payload TEXT NOT NULL, -- verbatim uploaded bytes; never mutated
17
+
mapping TEXT, -- JSON: field -> column, chosen by the user
18
+
decisions TEXT, -- JSON: synthetic id -> 'keep' | 'skip'
19
+
status TEXT NOT NULL DEFAULT 'draft' CHECK (status IN ('draft', 'committed', 'undone')),
20
+
committed_at TEXT,
21
+
undone_at TEXT
22
+
);
23
+
24
+
-- NULL means synced, so every pre-existing row is already correct and no backfill
25
+
-- is needed. Carries origin for the ledger source filter and scopes undo.
26
+
ALTER TABLE transactions ADD COLUMN import_id INTEGER REFERENCES imports (id);
27
+
28
+
CREATE INDEX idx_transactions_import ON transactions (import_id);
+36
-5
openspec/changes/csv-backfill-import/design.md
+36
-5
openspec/changes/csv-backfill-import/design.md
···
181
181
The residual imprecision — the file's single date column may be the transaction
182
182
date rather than the post date — is absorbed by the ±1 day duplicate window.
183
183
184
+
**Date order is part of the mapping.** `03/04/2026` is March 4th at one bank and
185
+
April 3rd at another, and the file never says which; guessing wrong silently shifts
186
+
an entire import by months. The format (`iso` / `mdy` / `dmy`) is therefore a
187
+
stored, user-confirmable field of the mapping, archived like every other mapping
188
+
choice so replay stays deterministic. It is inferred from the data where the data
189
+
settles it — any component above 12 can only be a day — and falls back to
190
+
month-first when every row is ambiguous. ISO dates are unambiguous and always
191
+
parse as themselves regardless of the declared format. The preview's stated date
192
+
range is the human backstop against a wrong inference.
193
+
184
194
### 7. Amount parsing: a CSV pre-pass, not a change to `parseAmountToCents`
185
195
186
196
**Decision:** Normalize CSV negative conventions (`(12.34)`, `12.34-`, currency
···
272
282
the table and column; imported rows would need removal first, which the undo flow
273
283
already performs.
274
284
285
+
## Resolved During Implementation
286
+
287
+
- **Hard-refuse implausible date ranges?** No. The preview states the parsed range
288
+
and shows the first rows as parsed, which makes a wrong date mapping obvious at a
289
+
glance, and undo makes a mistake cheap. A hard refusal would need a notion of
290
+
"the account's opening date" that does not exist, and would block legitimate
291
+
imports at the edges. Stating the range is enough.
292
+
- **List individual skipped rows in the import record?** No — counts only. The
293
+
skipped rows are, by construction, transactions the ledger already contains; the
294
+
interesting artifact is the archived file, which is retained.
295
+
- **Date-only values are anchored at noon UTC, not midnight.** Found by running a
296
+
real file: `formatDay` renders in local time, so UTC-midnight rows displayed a day
297
+
early across the Americas, and a month's first day would have displayed in the
298
+
previous month while still reporting under the correct one. Noon holds the
299
+
calendar day for UTC-12..+11 — every timezone these two users will see.
300
+
(`formatMonth` already anchors to the 15th for the same reason.) Not universal:
301
+
UTC+12 and beyond would still shift. Documented in the tests; the honest fix, if
302
+
it ever matters, is rendering date-only rows in UTC.
303
+
- **The duplicate window counts whole UTC calendar days, not ±86400 seconds.** A
304
+
consequence of the above: a CSV row sits at noon while a synced row sits at
305
+
whatever hour the bank posted it, so a seconds-based window would mean "within 24
306
+
hours" and would miss rows one calendar day apart (noon vs. midnight is 36 hours).
307
+
The spec says "within one day" and now the code means it.
308
+
275
309
## Open Questions
276
310
277
-
- Should the preview step hard-refuse a file whose parsed range extends into the
278
-
future or predates the account's opening, as a cheap guard against a
279
-
catastrophically wrong date mapping — or is stating the range enough?
280
-
- Does the import record in Settings need to list the individual skipped rows, or
281
-
is a count sufficient?
311
+
- Abandoned drafts are visible in the import record as "not finished". If they
312
+
accumulate enough to be noise, they want either a reaper or a hide.
+19
-4
openspec/changes/csv-backfill-import/specs/csv-import/spec.md
+19
-4
openspec/changes/csv-backfill-import/specs/csv-import/spec.md
···
50
50
The system SHALL detect a CSV's date, amount, and description columns from its
51
51
header row where possible, and SHALL require the user to confirm or correct the
52
52
mapping before commit. The system SHALL support amounts expressed as a single
53
-
signed column and as separate debit and credit columns. The system SHALL accept
54
-
the negative conventions common to bank exports, including parenthesized values
55
-
and trailing minus signs, and SHALL convert amounts to integer cents without
56
-
floating-point arithmetic. The system SHALL map the file's date to the
53
+
signed column and as separate debit and credit columns. The system SHALL treat the
54
+
interpretation of ambiguous numeric dates (whether `03/04/2026` is March 4th or
55
+
April 3rd) as a user-confirmable part of the mapping, inferring it from the data
56
+
where the data settles it and defaulting to month-first otherwise. The system
57
+
SHALL accept the negative conventions common to bank exports, including
58
+
parenthesized values and trailing minus signs, and SHALL convert amounts to
59
+
integer cents without floating-point arithmetic. The system SHALL map the file's date to the
57
60
transaction's posted timestamp and record imported transactions as not pending;
58
61
when the file exposes a distinct transaction date, the system SHALL map it to the
59
62
transaction date.
···
68
71
69
72
- **WHEN** the auto-detected mapping is wrong
70
73
- **THEN** the user can reassign any field to any column before committing
74
+
75
+
#### Scenario: Date order inferred from the data
76
+
77
+
- **WHEN** a file's date column contains `13/04/2026`, which can only be
78
+
day-first
79
+
- **THEN** the whole file is interpreted day-first
80
+
81
+
#### Scenario: Ambiguous date order is confirmable
82
+
83
+
- **WHEN** every date in a file is ambiguous, such as `03/04/2026`
84
+
- **THEN** the system defaults to month-first, states the resulting date range
85
+
before commit, and allows the user to select day-first instead
71
86
72
87
#### Scenario: Parenthesized negative
73
88
+43
-40
openspec/changes/csv-backfill-import/tasks.md
+43
-40
openspec/changes/csv-backfill-import/tasks.md
···
1
1
## 1. Schema
2
2
3
-
- [ ] 1.1 Add `migrations/005_csv_import.sql` creating the `imports` table: `id`,
3
+
- [x] 1.1 Add `migrations/005_csv_import.sql` creating the `imports` table: `id`,
4
4
`account_id` (NOT NULL, FK to accounts), `filename`, `uploaded_at`, `payload`
5
5
(verbatim bytes, never mutated), `mapping` (JSON, nullable), `decisions` (JSON,
6
6
nullable), `status` NOT NULL DEFAULT `'draft'` CHECK in (`draft`, `committed`,
7
7
`undone`), `committed_at`, `undone_at`.
8
-
- [ ] 1.2 In the same migration, `ALTER TABLE transactions ADD COLUMN import_id
8
+
- [x] 1.2 In the same migration, `ALTER TABLE transactions ADD COLUMN import_id
9
9
INTEGER REFERENCES imports (id)` (nullable; NULL means synced) and add an index
10
10
on `transactions (import_id)`.
11
-
- [ ] 1.3 Extend `src/lib/server/db.test.ts` to assert the migration applies to a
11
+
- [x] 1.3 Extend `src/lib/server/db.test.ts` to assert the migration applies to a
12
12
populated database and that existing transactions read back with
13
13
`import_id IS NULL`.
14
14
15
15
## 2. CSV parsing and mapping (pure)
16
16
17
-
- [ ] 2.1 Add `@std/csv` (JSR) as a dependency. Do not hand-roll parsing —
18
-
quoted delimiters, embedded newlines, and BOMs are the failure modes.
19
-
- [ ] 2.2 Create `src/lib/server/services/csv-import.ts` with a pure
17
+
- [x] 2.1 Add `csv-parse` (npm) as a dependency. Do not hand-roll parsing —
18
+
quoted delimiters, embedded newlines, and BOMs are the failure modes. An npm
19
+
package rather than JSR's `@std/csv`: Vite-under-Deno resolves a JSR specifier
20
+
fine, but `svelte-check` resolves Node-style and cannot, which would have left
21
+
`deno task check` permanently red.
22
+
- [x] 2.2 Create `src/lib/server/services/csv-import.ts` with a pure
20
23
`parseCsv(payloadText)` returning header row plus data rows, tolerating a BOM
21
24
and CRLF line endings.
22
-
- [ ] 2.3 Implement pure `detectMapping(headers)` returning best-guess column
25
+
- [x] 2.3 Implement pure `detectMapping(headers)` returning best-guess column
23
26
assignments for date, amount (single signed column or debit/credit pair), and
24
27
description, plus optional payee/memo and a distinct transaction-date column.
25
-
- [ ] 2.4 Implement pure `normalizeCsvAmount(raw)` handling parenthesized
28
+
- [x] 2.4 Implement pure `normalizeCsvAmount(raw)` handling parenthesized
26
29
negatives `(12.34)`, trailing minus `12.34-`, currency symbols, and thousands
27
30
separators, emitting a signed decimal string for the existing
28
31
`parseAmountToCents`. Do not loosen `parseAmountToCents` itself — its strict
29
32
regex is the sync path's contract.
30
-
- [ ] 2.5 Implement pure `normalizeCsv(payloadText, mapping, accountId)` producing
33
+
- [x] 2.5 Implement pure `normalizeCsv(payloadText, mapping, accountId)` producing
31
34
candidate rows with `posted` set from the file's date, `pending = 0`,
32
35
`transacted_at` set only when the mapping names a distinct transaction-date
33
36
column, and amounts as integer cents.
34
-
- [ ] 2.6 Implement the synthetic identity: `csv:` + hash over (accountId, date,
37
+
- [x] 2.6 Implement the synthetic identity: `csv:` + hash over (accountId, date,
35
38
amountCents, description) plus an occurrence index among identical rows within
36
39
the file.
37
-
- [ ] 2.7 Unit-test 2.2–2.6 in `csv-import.test.ts`: BOM/CRLF, both amount modes,
40
+
- [x] 2.7 Unit-test 2.2–2.6 in `csv-import.test.ts`: BOM/CRLF, both amount modes,
38
41
each negative convention, date parsing, and that two identical rows receive
39
42
distinct ids while a reordered or superset file re-derives the same id set.
40
43
41
44
## 3. Duplicate detection
42
45
43
-
- [ ] 3.1 Implement `findPotentialDuplicates(db, accountId, candidates)` matching
46
+
- [x] 3.1 Implement `findPotentialDuplicates(db, accountId, candidates)` matching
44
47
each candidate against non-removed transactions in the account by identical
45
48
`amount_cents` and effective date within ±1 day (compare against
46
49
`COALESCE(posted, transacted_at)`). Never match on description.
47
-
- [ ] 3.2 Return, per candidate, one of: `new`, `already-present` (its synthetic
50
+
- [x] 3.2 Return, per candidate, one of: `new`, `already-present` (its synthetic
48
51
id already exists — never surfaced to the user), or `flagged` with the existing
49
52
transaction's date, amount, and description for side-by-side display.
50
-
- [ ] 3.3 Test: a cross-source duplicate with differently worded descriptions is
53
+
- [x] 3.3 Test: a cross-source duplicate with differently worded descriptions is
51
54
flagged; a row one day off is flagged; a row two days off is not; an
52
55
already-ingested row reports `already-present` rather than `flagged`.
53
56
54
57
## 4. Import lifecycle and additive ingestion
55
58
56
-
- [ ] 4.1 Implement `createDraftImport(db, accountId, filename, payload)` writing
59
+
- [x] 4.1 Implement `createDraftImport(db, accountId, filename, payload)` writing
57
60
the `imports` row with verbatim payload and `status = 'draft'` before any
58
61
parsing.
59
-
- [ ] 4.2 Implement `saveMapping` and `saveDecisions` persisting to the draft row,
62
+
- [x] 4.2 Implement `saveMapping` and `saveDecisions` persisting to the draft row,
60
63
so the archived record alone determines the outcome.
61
-
- [ ] 4.3 Implement `commitImport(db, importId)` — an **insert-only** writer.
64
+
- [x] 4.3 Implement `commitImport(db, importId)` — an **insert-only** writer.
62
65
It MUST NOT reconcile, MUST NOT sweep stale pending rows, MUST NOT update
63
66
existing transactions, MUST NOT write balance snapshots, and MUST NOT touch
64
67
account state. Do not call or extend `ingestTransactions`; its stale-pending
65
68
sweep (`sync.ts:291-301`) would soft-delete every pending row absent from the
66
69
CSV.
67
-
- [ ] 4.4 In `commitImport`, skip candidates marked skipped and those whose
70
+
- [x] 4.4 In `commitImport`, skip candidates marked skipped and those whose
68
71
synthetic id already exists; revive a soft-removed row matching the unique key
69
72
by clearing `removed_at` and reassigning `import_id`; stamp `import_id` on every
70
73
inserted row; set `status = 'committed'` and `committed_at`. Run the whole
71
74
commit in one transaction.
72
-
- [ ] 4.5 Call `applyRulesToUncategorized(db)` after the commit transaction
75
+
- [x] 4.5 Call `applyRulesToUncategorized(db)` after the commit transaction
73
76
succeeds. No new categorization event source.
74
-
- [ ] 4.6 Test: importing into an account holding pending rows leaves them
77
+
- [x] 4.6 Test: importing into an account holding pending rows leaves them
75
78
untouched; no snapshots are written; account state and
76
79
`last_successful_data_at` are unchanged; re-importing the same file inserts
77
80
nothing; an overlapping file inserts only the new rows; committed rows carry
···
79
82
80
83
## 5. Undo
81
84
82
-
- [ ] 5.1 Implement `undoImport(db, importId)` soft-removing (`removed_at = now`)
85
+
- [x] 5.1 Implement `undoImport(db, importId)` soft-removing (`removed_at = now`)
83
86
every transaction with that `import_id`, setting `status = 'undone'` and
84
87
`undone_at`, in one transaction. Never hard-delete — categorization events
85
88
reference `transaction_id` and the event log is append-only.
86
-
- [ ] 5.2 Test: undo removes exactly that import's rows from ledger and report
89
+
- [x] 5.2 Test: undo removes exactly that import's rows from ledger and report
87
90
scope, leaves synced rows untouched, preserves categorization events for the
88
91
removed rows, and a corrected re-import afterwards does not resurrect the old
89
92
rows.
90
93
91
94
## 6. Ledger surfacing
92
95
93
-
- [ ] 6.1 Add `source?: 'synced' | 'imported'` to `LedgerFilters` in
96
+
- [x] 6.1 Add `source?: 'synced' | 'imported'` to `LedgerFilters` in
94
97
`src/lib/server/services/ledger.ts`, translating to `t.import_id IS NULL` /
95
98
`IS NOT NULL`, composing with every existing filter.
96
-
- [ ] 6.2 Add origin to `LedgerRow` (import id, file name, import date; null when
99
+
- [x] 6.2 Add origin to `LedgerRow` (import id, file name, import date; null when
97
100
synced) via a join through `import_id`.
98
-
- [ ] 6.3 Test in `ledger.test.ts`: the source filter composes with account,
101
+
- [x] 6.3 Test in `ledger.test.ts`: the source filter composes with account,
99
102
month, category, pending, and text search.
100
-
- [ ] 6.4 Add the source filter control to the ledger page alongside the existing
103
+
- [x] 6.4 Add the source filter control to the ledger page alongside the existing
101
104
filters. Add no per-row badge — within a backfilled range every row is imported,
102
105
so a per-row mark carries no information where it appears.
103
-
- [ ] 6.5 Add an origin line to the top of the existing history panel: "Imported
106
+
- [x] 6.5 Add an origin line to the top of the existing history panel: "Imported
104
107
from `<file>` · `<date>`" or "Synced from SimpleFIN".
105
108
106
109
## 7. Import wizard (Settings)
107
110
108
-
- [ ] 7.1 Add a Settings route for the import wizard. Step 1: file upload plus
111
+
- [x] 7.1 Add a Settings route for the import wizard. Step 1: file upload plus
109
112
target-account picker (non-hidden accounts only), archiving on submit and
110
113
redirecting to the draft's id.
111
-
- [ ] 7.2 Step 2: mapping confirmation, pre-filled from `detectMapping`, every
114
+
- [x] 7.2 Step 2: mapping confirmation, pre-filled from `detectMapping`, every
112
115
field reassignable, with a few parsed sample rows rendered so a wrong guess is
113
116
visible.
114
-
- [ ] 7.3 Step 3: preview stating the parsed date range, rows to import, rows
117
+
- [x] 7.3 Step 3: preview stating the parsed date range, rows to import, rows
115
118
already present, and rows flagged — the guard against a catastrophically wrong
116
119
date mapping.
117
-
- [ ] 7.4 Step 4: duplicate review listing each flagged row against its existing
120
+
- [x] 7.4 Step 4: duplicate review listing each flagged row against its existing
118
121
match, both descriptions shown, defaulting to skip, each flippable to keep.
119
-
- [ ] 7.5 Commit action, then a result summary linking to the ledger filtered to
122
+
- [x] 7.5 Commit action, then a result summary linking to the ledger filtered to
120
123
that account and source.
121
-
- [ ] 7.6 Style per DESIGN.md: tabular numerals on money and dates, calm empty and
124
+
- [x] 7.6 Style per DESIGN.md: tabular numerals on money and dates, calm empty and
122
125
error states (one sentence plus one action), no new red unless data is at risk.
123
126
124
127
## 8. Import record (Settings)
125
128
126
-
- [ ] 8.1 Add a Settings section listing imports: file name, account, parsed date
129
+
- [x] 8.1 Add a Settings section listing imports: file name, account, parsed date
127
130
range, rows imported, rows skipped, commit time, status. Not in the main nav.
128
-
- [ ] 8.2 Add the undo action with a confirmation stating exactly how many
131
+
- [x] 8.2 Add the undo action with a confirmation stating exactly how many
129
132
transactions will be removed.
130
-
- [ ] 8.3 Render an empty state for the no-imports-yet case.
133
+
- [x] 8.3 Render an empty state for the no-imports-yet case.
131
134
132
135
## 9. Verification
133
136
134
-
- [ ] 9.1 Run `deno task test` and `deno task check`.
135
-
- [ ] 9.2 Drive the wizard end to end against a real bank CSV export in dev:
137
+
- [x] 9.1 Run `deno task test` and `deno task check`.
138
+
- [x] 9.2 Drive the wizard end to end against a real bank CSV export in dev:
136
139
confirm the gap closes in the ledger, the month reports for backfilled months
137
140
change as expected, and the net worth chart is unchanged (no snapshots written).
138
-
- [ ] 9.3 Verify a sync runs cleanly after an import: pending rows still
141
+
- [x] 9.3 Verify a sync runs cleanly after an import: pending rows still
139
142
reconcile, and no imported row is disturbed.
140
-
- [ ] 9.4 Exercise undo on a real import and confirm the ledger and reports return
143
+
- [x] 9.4 Exercise undo on a real import and confirm the ledger and reports return
141
144
to their pre-import state.
+2
-1
package.json
+2
-1
package.json
···
16
16
"@atproto/jwk-jose": "^0.2.4",
17
17
"@atproto/oauth-client-node": "^0.4.8",
18
18
"@atproto/oauth-types": "^0.7.5",
19
-
"@fontsource-variable/inter": "^5.2.8"
19
+
"@fontsource-variable/inter": "^5.2.8",
20
+
"csv-parse": "^5.6.0"
20
21
},
21
22
"devDependencies": {
22
23
"@sveltejs/adapter-node": "^5.5.0",
+67
src/lib/server/db.test.ts
+67
src/lib/server/db.test.ts
···
47
47
db.close();
48
48
});
49
49
50
+
Deno.test('csv import migration applies to a populated database', () => {
51
+
// Stage only the migrations before 005, populate, then upgrade — the path an
52
+
// existing install actually takes.
53
+
const stageDir = Deno.makeTempDirSync();
54
+
for (const entry of Deno.readDirSync(MIGRATIONS_DIR)) {
55
+
const version = Number(entry.name.match(/^(\d+)_/)?.[1]);
56
+
if (Number.isFinite(version) && version < 5) {
57
+
Deno.copyFileSync(join(MIGRATIONS_DIR, entry.name), join(stageDir, entry.name));
58
+
}
59
+
}
60
+
61
+
const dbPath = join(Deno.makeTempDirSync(), 'test.db');
62
+
const db = openDatabase(dbPath, stageDir);
63
+
64
+
const now = new Date().toISOString();
65
+
db.prepare('INSERT INTO connections (access_url, claimed_at) VALUES (?, ?)').run(
66
+
'https://example.test',
67
+
now
68
+
);
69
+
db.prepare(
70
+
`INSERT INTO accounts (id, connection_id, name, currency, state, created_at)
71
+
VALUES ('acct-1', 1, 'Checking', 'USD', 'ACTIVE', ?)`
72
+
).run(now);
73
+
db.prepare(
74
+
`INSERT INTO transactions (account_id, sfin_id, posted, amount_cents, description, created_at)
75
+
VALUES ('acct-1', 'sfin-1', 1700000000, -1234, 'COFFEE', ?)`
76
+
).run(now);
77
+
78
+
const applied = runMigrations(db, MIGRATIONS_DIR);
79
+
if (applied !== 1) throw new Error(`expected only 005 to apply, got ${applied}`);
80
+
81
+
const row = db.prepare("SELECT import_id FROM transactions WHERE sfin_id = 'sfin-1'").get() as {
82
+
import_id: number | null;
83
+
};
84
+
if (row.import_id !== null) {
85
+
throw new Error(`pre-existing rows must read back as synced, got ${row.import_id}`);
86
+
}
87
+
88
+
db.close();
89
+
});
90
+
91
+
Deno.test('imports status is a closed enum', () => {
92
+
const dbPath = join(Deno.makeTempDirSync(), 'test.db');
93
+
const db = openDatabase(dbPath, MIGRATIONS_DIR);
94
+
const now = new Date().toISOString();
95
+
db.prepare('INSERT INTO connections (access_url, claimed_at) VALUES (?, ?)').run(
96
+
'https://example.test',
97
+
now
98
+
);
99
+
db.prepare(
100
+
`INSERT INTO accounts (id, connection_id, name, currency, state, created_at)
101
+
VALUES ('acct-1', 1, 'Checking', 'USD', 'ACTIVE', ?)`
102
+
).run(now);
103
+
104
+
let threw = false;
105
+
try {
106
+
db.prepare(
107
+
`INSERT INTO imports (account_id, uploaded_at, payload, status)
108
+
VALUES ('acct-1', ?, 'raw', 'bogus')`
109
+
).run(now);
110
+
} catch {
111
+
threw = true;
112
+
}
113
+
if (!threw) throw new Error('expected status CHECK violation');
114
+
db.close();
115
+
});
116
+
50
117
function join(...parts: string[]) {
51
118
return parts.join('/');
52
119
}
+312
src/lib/server/services/csv-import.test.ts
+312
src/lib/server/services/csv-import.test.ts
···
1
+
/// <reference lib="deno.ns" />
2
+
import {
3
+
detectDateFormat,
4
+
detectMapping,
5
+
normalizeCsv,
6
+
normalizeCsvAmount,
7
+
parseCsv,
8
+
parseCsvDate,
9
+
type CsvMapping
10
+
} from './csv-import.ts';
11
+
import { parseAmountToCents } from './normalize.ts';
12
+
13
+
const SIGNED_CSV = `Date,Description,Amount
14
+
2026-07-02,Trader Joe's,-88.14
15
+
2026-07-03,Paycheck,2500.00
16
+
`;
17
+
18
+
function signedMapping(overrides: Partial<CsvMapping> = {}): CsvMapping {
19
+
return {
20
+
date: 'Date',
21
+
dateFormat: 'iso',
22
+
amountMode: 'signed',
23
+
amount: 'Amount',
24
+
description: 'Description',
25
+
...overrides
26
+
};
27
+
}
28
+
29
+
Deno.test('parseCsv strips a BOM and tolerates CRLF', () => {
30
+
const parsed = parseCsv('Date,Description,Amount\r\n2026-07-02,Coffee,-4.75\r\n');
31
+
if (parsed.headers[0] !== 'Date') throw new Error(`BOM leaked: ${JSON.stringify(parsed.headers[0])}`);
32
+
if (parsed.rows.length !== 1) throw new Error(`expected 1 row, got ${parsed.rows.length}`);
33
+
});
34
+
35
+
Deno.test('parseCsv handles quoted delimiters and embedded newlines', () => {
36
+
const parsed = parseCsv('Date,Description,Amount\n2026-07-02,"Shop, Inc.\nStore #4",-10.00\n');
37
+
if (parsed.rows[0][1] !== 'Shop, Inc.\nStore #4') {
38
+
throw new Error(`quoting mishandled: ${JSON.stringify(parsed.rows[0][1])}`);
39
+
}
40
+
if (parsed.rows[0][2] !== '-10.00') throw new Error('column alignment broken by quoting');
41
+
});
42
+
43
+
Deno.test('parseCsv rejects an empty file', () => {
44
+
let threw = false;
45
+
try {
46
+
parseCsv('\n\n');
47
+
} catch {
48
+
threw = true;
49
+
}
50
+
if (!threw) throw new Error('expected an empty file to throw');
51
+
});
52
+
53
+
Deno.test('normalizeCsvAmount handles bank negative conventions', () => {
54
+
// Thousands separators pass through by design — parseAmountToCents strips them.
55
+
const cases: [string, string][] = [
56
+
['(12.34)', '-12.34'],
57
+
['12.34-', '-12.34'],
58
+
['-12.34', '-12.34'],
59
+
['$1,234.56', '1,234.56'],
60
+
['($1,234.56)', '-1,234.56'],
61
+
['12.34', '12.34'],
62
+
['', ''],
63
+
[' ', '']
64
+
];
65
+
for (const [input, expected] of cases) {
66
+
const actual = normalizeCsvAmount(input);
67
+
if (actual !== expected) {
68
+
throw new Error(`${JSON.stringify(input)} -> ${JSON.stringify(actual)}, want ${expected}`);
69
+
}
70
+
}
71
+
});
72
+
73
+
Deno.test('normalizeCsvAmount output is accepted by parseAmountToCents', () => {
74
+
// The handoff is the point: this pre-pass exists so parseAmountToCents can stay
75
+
// strict for the sync path.
76
+
const cases: [string, number][] = [
77
+
['(12.34)', -1234],
78
+
['12.34-', -1234],
79
+
['$1,234.56', 123456],
80
+
['($1,234.56)', -123456],
81
+
['12.34', 1234]
82
+
];
83
+
for (const [input, expected] of cases) {
84
+
const actual = parseAmountToCents(normalizeCsvAmount(input));
85
+
if (actual !== expected) {
86
+
throw new Error(`${JSON.stringify(input)} -> ${actual} cents, want ${expected}`);
87
+
}
88
+
}
89
+
});
90
+
91
+
Deno.test('parenthesized negative reaches integer cents', () => {
92
+
const { candidates, errors } = normalizeCsv(
93
+
'Date,Description,Amount\n2026-07-02,Coffee,(12.34)\n',
94
+
signedMapping(),
95
+
'acct-1'
96
+
);
97
+
if (errors.length) throw new Error(errors.join('; '));
98
+
if (candidates[0].amountCents !== -1234) {
99
+
throw new Error(`got ${candidates[0].amountCents}, want -1234`);
100
+
}
101
+
});
102
+
103
+
Deno.test('detectDateFormat reads the order off the data', () => {
104
+
if (detectDateFormat(['2026-07-02']) !== 'iso') throw new Error('iso not detected');
105
+
// 13 can only be a day, so day comes first.
106
+
if (detectDateFormat(['03/04/2026', '13/04/2026']) !== 'dmy') throw new Error('dmy not detected');
107
+
if (detectDateFormat(['03/04/2026', '04/13/2026']) !== 'mdy') throw new Error('mdy not detected');
108
+
// Wholly ambiguous input falls back to month-first.
109
+
if (detectDateFormat(['03/04/2026']) !== 'mdy') throw new Error('ambiguous should default to mdy');
110
+
});
111
+
112
+
Deno.test('parseCsvDate respects the mapped order', () => {
113
+
const mdy = parseCsvDate('03/04/2026', 'mdy');
114
+
const dmy = parseCsvDate('03/04/2026', 'dmy');
115
+
if (mdy !== Date.UTC(2026, 2, 4, 12) / 1000) throw new Error('mdy parsed wrong');
116
+
if (dmy !== Date.UTC(2026, 3, 3, 12) / 1000) throw new Error('dmy parsed wrong');
117
+
// ISO is unambiguous and ignores the declared order.
118
+
if (parseCsvDate('2026-07-02', 'dmy') !== Date.UTC(2026, 6, 2, 12) / 1000) {
119
+
throw new Error('iso should win regardless of format');
120
+
}
121
+
});
122
+
123
+
Deno.test('a date-only value survives local rendering from UTC-12 to UTC+11', () => {
124
+
// The reason for anchoring at noon: formatDay renders in local time, so a
125
+
// UTC-midnight stamp would read as the previous day across the Americas, and a
126
+
// month's first day would display in the month before the one it reports under.
127
+
// Noon buys 12 hours of slack each way, which covers every timezone the two
128
+
// users of this app will ever be in. It is not universal: at UTC+12 and beyond
129
+
// (New Zealand, Kiribati) noon crosses into the next day. Documented, not fixed
130
+
// — the honest fix is rendering date-only rows in UTC, which is worth doing only
131
+
// if that ever matters.
132
+
const posted = parseCsvDate('2026-07-01', 'iso');
133
+
for (let offset = -12; offset <= 11; offset++) {
134
+
const local = new Date((posted + offset * 3600) * 1000);
135
+
if (local.getUTCDate() !== 1 || local.getUTCMonth() !== 6) {
136
+
throw new Error(`UTC${offset >= 0 ? '+' : ''}${offset} shifts the calendar day`);
137
+
}
138
+
}
139
+
});
140
+
141
+
Deno.test('parseCsvDate rejects nonsense', () => {
142
+
for (const bad of ['', 'not a date', '99/99/2026']) {
143
+
let threw = false;
144
+
try {
145
+
parseCsvDate(bad, 'mdy');
146
+
} catch {
147
+
threw = true;
148
+
}
149
+
if (!threw) throw new Error(`expected ${JSON.stringify(bad)} to throw`);
150
+
}
151
+
});
152
+
153
+
Deno.test('detectMapping finds signed-column headers', () => {
154
+
const mapping = detectMapping(parseCsv(SIGNED_CSV));
155
+
if (!mapping) throw new Error('expected a mapping');
156
+
if (mapping.amountMode !== 'signed' || mapping.amount !== 'Amount') {
157
+
throw new Error(`wrong amount mapping: ${JSON.stringify(mapping)}`);
158
+
}
159
+
if (mapping.date !== 'Date' || mapping.description !== 'Description') {
160
+
throw new Error(`wrong column mapping: ${JSON.stringify(mapping)}`);
161
+
}
162
+
if (mapping.dateFormat !== 'iso') throw new Error('expected iso');
163
+
});
164
+
165
+
Deno.test('detectMapping finds a debit/credit pair', () => {
166
+
const mapping = detectMapping(
167
+
parseCsv('Date,Description,Debit,Credit\n07/02/2026,Coffee,4.75,\n')
168
+
);
169
+
if (!mapping) throw new Error('expected a mapping');
170
+
if (mapping.amountMode !== 'debit-credit') throw new Error('expected debit-credit mode');
171
+
if (mapping.debit !== 'Debit' || mapping.credit !== 'Credit') {
172
+
throw new Error(`wrong pair: ${JSON.stringify(mapping)}`);
173
+
}
174
+
});
175
+
176
+
Deno.test('detectMapping gives up rather than guessing', () => {
177
+
if (detectMapping(parseCsv('Foo,Bar,Baz\n1,2,3\n')) !== null) {
178
+
throw new Error('expected null for unrecognizable headers');
179
+
}
180
+
});
181
+
182
+
Deno.test('debit and credit columns resolve to one signed amount', () => {
183
+
const mapping: CsvMapping = {
184
+
date: 'Date',
185
+
dateFormat: 'mdy',
186
+
amountMode: 'debit-credit',
187
+
debit: 'Debit',
188
+
credit: 'Credit',
189
+
description: 'Description'
190
+
};
191
+
const { candidates, errors } = normalizeCsv(
192
+
'Date,Description,Debit,Credit\n07/02/2026,Coffee,4.75,\n07/03/2026,Refund,,20.00\n',
193
+
mapping,
194
+
'acct-1'
195
+
);
196
+
if (errors.length) throw new Error(errors.join('; '));
197
+
if (candidates[0].amountCents !== -475) throw new Error(`debit -> ${candidates[0].amountCents}`);
198
+
if (candidates[1].amountCents !== 2000) throw new Error(`credit -> ${candidates[1].amountCents}`);
199
+
});
200
+
201
+
Deno.test('a bad row is reported without sinking the file', () => {
202
+
const { candidates, errors } = normalizeCsv(
203
+
'Date,Description,Amount\n2026-07-02,Coffee,-4.75\nnot-a-date,Broken,-1.00\n',
204
+
signedMapping(),
205
+
'acct-1'
206
+
);
207
+
if (candidates.length !== 1) throw new Error(`expected 1 good row, got ${candidates.length}`);
208
+
if (errors.length !== 1 || !errors[0].includes('Line 3')) {
209
+
throw new Error(`expected a line-3 error, got ${JSON.stringify(errors)}`);
210
+
}
211
+
});
212
+
213
+
Deno.test('imported rows carry posted and never pend', () => {
214
+
const { candidates } = normalizeCsv(SIGNED_CSV, signedMapping(), 'acct-1');
215
+
if (candidates[0].posted !== Date.UTC(2026, 6, 2, 12) / 1000) throw new Error('posted not set');
216
+
if (candidates[0].transactedAt !== null) throw new Error('transactedAt should be null');
217
+
});
218
+
219
+
Deno.test('a distinct transaction-date column maps separately', () => {
220
+
const { candidates, errors } = normalizeCsv(
221
+
'Post Date,Transaction Date,Description,Amount\n2026-07-04,2026-07-02,Coffee,-4.75\n',
222
+
signedMapping({ date: 'Post Date', transactedDate: 'Transaction Date' }),
223
+
'acct-1'
224
+
);
225
+
if (errors.length) throw new Error(errors.join('; '));
226
+
if (candidates[0].posted !== Date.UTC(2026, 6, 4, 12) / 1000) throw new Error('wrong posted');
227
+
if (candidates[0].transactedAt !== Date.UTC(2026, 6, 2, 12) / 1000) {
228
+
throw new Error('wrong transactedAt');
229
+
}
230
+
});
231
+
232
+
Deno.test('identity is stable across runs', () => {
233
+
const a = normalizeCsv(SIGNED_CSV, signedMapping(), 'acct-1');
234
+
const b = normalizeCsv(SIGNED_CSV, signedMapping(), 'acct-1');
235
+
const idsA = a.candidates.map((c) => c.syntheticId).join(',');
236
+
const idsB = b.candidates.map((c) => c.syntheticId).join(',');
237
+
if (idsA !== idsB) throw new Error('ids must be deterministic');
238
+
});
239
+
240
+
Deno.test('identity is namespaced and account-scoped', () => {
241
+
const { candidates } = normalizeCsv(SIGNED_CSV, signedMapping(), 'acct-1');
242
+
if (!candidates[0].syntheticId.startsWith('csv:')) throw new Error('missing csv: namespace');
243
+
244
+
const other = normalizeCsv(SIGNED_CSV, signedMapping(), 'acct-2');
245
+
if (candidates[0].syntheticId === other.candidates[0].syntheticId) {
246
+
throw new Error('the same row in a different account must not share an id');
247
+
}
248
+
});
249
+
250
+
Deno.test('two genuinely identical rows stay distinct', () => {
251
+
const { candidates } = normalizeCsv(
252
+
'Date,Description,Amount\n2026-07-02,Coffee,-4.75\n2026-07-02,Coffee,-4.75\n',
253
+
signedMapping(),
254
+
'acct-1'
255
+
);
256
+
if (candidates.length !== 2) throw new Error(`expected 2 rows, got ${candidates.length}`);
257
+
if (candidates[0].syntheticId === candidates[1].syntheticId) {
258
+
throw new Error('identical rows must get distinct ids via the occurrence index');
259
+
}
260
+
});
261
+
262
+
Deno.test('cosmetic description re-rendering does not mint a new identity', () => {
263
+
const a = normalizeCsv('Date,Description,Amount\n2026-07-02,Trader Joe\'s,-88.14\n', signedMapping(), 'acct-1');
264
+
const b = normalizeCsv(
265
+
'Date,Description,Amount\n2026-07-02, TRADER JOE\'S ,-88.14\n',
266
+
signedMapping(),
267
+
'acct-1'
268
+
);
269
+
if (a.candidates[0].syntheticId !== b.candidates[0].syntheticId) {
270
+
throw new Error('folded description should yield the same id');
271
+
}
272
+
});
273
+
274
+
Deno.test('a reordered group re-derives the same id set', () => {
275
+
// Rows within a content group are interchangeable, so export order must not
276
+
// change the set of ids — this is what makes an overlapping re-import a no-op.
277
+
const first = normalizeCsv(
278
+
'Date,Description,Amount\n2026-07-02,Coffee,-4.75\n2026-07-02,Bagel,-3.00\n2026-07-02,Coffee,-4.75\n',
279
+
signedMapping(),
280
+
'acct-1'
281
+
);
282
+
const reordered = normalizeCsv(
283
+
'Date,Description,Amount\n2026-07-02,Coffee,-4.75\n2026-07-02,Coffee,-4.75\n2026-07-02,Bagel,-3.00\n',
284
+
signedMapping(),
285
+
'acct-1'
286
+
);
287
+
const setA = new Set(first.candidates.map((c) => c.syntheticId));
288
+
const setB = new Set(reordered.candidates.map((c) => c.syntheticId));
289
+
if (setA.size !== setB.size || [...setA].some((id) => !setB.has(id))) {
290
+
throw new Error('id set must be independent of row order within a group');
291
+
}
292
+
});
293
+
294
+
Deno.test('a superset re-export only adds genuinely new ids', () => {
295
+
const first = normalizeCsv(
296
+
'Date,Description,Amount\n2026-07-02,Coffee,-4.75\n2026-07-02,Coffee,-4.75\n',
297
+
signedMapping(),
298
+
'acct-1'
299
+
);
300
+
const superset = normalizeCsv(
301
+
'Date,Description,Amount\n2026-07-02,Coffee,-4.75\n2026-07-02,Coffee,-4.75\n2026-07-02,Coffee,-4.75\n',
302
+
signedMapping(),
303
+
'acct-1'
304
+
);
305
+
const before = new Set(first.candidates.map((c) => c.syntheticId));
306
+
const after = superset.candidates.map((c) => c.syntheticId);
307
+
const added = after.filter((id) => !before.has(id));
308
+
if (added.length !== 1) throw new Error(`expected exactly 1 new id, got ${added.length}`);
309
+
for (const id of before) {
310
+
if (!after.includes(id)) throw new Error('previously ingested ids must survive a superset');
311
+
}
312
+
});
+309
src/lib/server/services/csv-import.ts
+309
src/lib/server/services/csv-import.ts
···
1
+
// Pure parsing and normalization of a bank CSV export. No I/O, no DB: a function
2
+
// from archived payload text + mapping to typed candidate rows, mirroring
3
+
// normalize.ts so an import can be replayed from its archived record.
4
+
//
5
+
// CSV import is an ADDITIVE writer. Nothing here reconciles, sweeps, or mutates.
6
+
7
+
import { parse } from 'csv-parse/sync';
8
+
import { createHash } from 'node:crypto';
9
+
import { parseAmountToCents } from './normalize.ts';
10
+
11
+
/**
12
+
* `03/04/2026` is March 4th at one bank and April 3rd at another, and the file
13
+
* never says which. The format is part of the mapping so the user's answer is
14
+
* archived and the import replays deterministically.
15
+
*/
16
+
export type CsvDateFormat = 'iso' | 'mdy' | 'dmy';
17
+
18
+
export type CsvAmountMode = 'signed' | 'debit-credit';
19
+
20
+
export interface CsvMapping {
21
+
date: string;
22
+
/** Distinct transaction date, when the file separates it from the post date. */
23
+
transactedDate?: string | null;
24
+
dateFormat: CsvDateFormat;
25
+
amountMode: CsvAmountMode;
26
+
/** `signed` mode: one column carrying the sign. */
27
+
amount?: string | null;
28
+
/** `debit-credit` mode: two columns of unsigned magnitudes. */
29
+
debit?: string | null;
30
+
credit?: string | null;
31
+
description: string;
32
+
payee?: string | null;
33
+
memo?: string | null;
34
+
}
35
+
36
+
export interface CsvCandidate {
37
+
/** Deterministic content-derived id; namespaced to never collide with SimpleFIN's. */
38
+
syntheticId: string;
39
+
posted: number; // Unix seconds, UTC midnight of the file's date
40
+
transactedAt: number | null;
41
+
amountCents: number;
42
+
description: string;
43
+
payee: string | null;
44
+
memo: string | null;
45
+
}
46
+
47
+
export interface ParsedCsv {
48
+
headers: string[];
49
+
rows: string[][];
50
+
}
51
+
52
+
export interface NormalizedCsv {
53
+
candidates: CsvCandidate[];
54
+
/** Row-level failures, kept as messages so one bad row doesn't sink the file. */
55
+
errors: string[];
56
+
}
57
+
58
+
/**
59
+
* Parse CSV text into a header row and data rows. Quoting, embedded newlines,
60
+
* CRLF, and the UTF-8 BOM that Excel-exported bank files carry are all delegated
61
+
* to csv-parse — hand-rolling those is where CSV handling quietly eats data.
62
+
* Ragged rows are tolerated: bank exports pad or truncate trailing columns, and a
63
+
* stray comma must not sink an otherwise good file.
64
+
*/
65
+
export function parseCsv(payloadText: string): ParsedCsv {
66
+
const rows = parse(payloadText, {
67
+
bom: true,
68
+
skip_empty_lines: true,
69
+
relax_column_count: true,
70
+
relax_quotes: true
71
+
}) as string[][];
72
+
const nonEmpty = rows.filter((row) => row.some((cell) => cell.trim() !== ''));
73
+
if (nonEmpty.length === 0) throw new Error('The file has no rows.');
74
+
const [headers, ...data] = nonEmpty;
75
+
return { headers: headers.map((h) => h.trim()), rows: data };
76
+
}
77
+
78
+
const DATE_RE = /^(date|posted|post date|posting date|transaction date|trans date)$/i;
79
+
const TXN_DATE_RE = /^(transaction date|trans date)$/i;
80
+
const AMOUNT_RE = /^(amount|value|transaction amount)$/i;
81
+
const DEBIT_RE = /^(debit|withdrawal|withdrawals|money out|paid out)$/i;
82
+
const CREDIT_RE = /^(credit|deposit|deposits|money in|paid in)$/i;
83
+
const DESC_RE = /^(description|details|name|merchant|payee|narrative|transaction)$/i;
84
+
const PAYEE_RE = /^(payee|merchant|name)$/i;
85
+
const MEMO_RE = /^(memo|note|notes|reference)$/i;
86
+
87
+
function findHeader(headers: string[], re: RegExp): string | null {
88
+
return headers.find((h) => re.test(h.trim())) ?? null;
89
+
}
90
+
91
+
/**
92
+
* Infer the date format from the data itself: a component above 12 can only be a
93
+
* day, which settles the order. When every row is ambiguous (all components ≤ 12)
94
+
* there is no signal, so fall back to month-first and let the user correct it —
95
+
* the preview's stated date range is the backstop.
96
+
*/
97
+
export function detectDateFormat(values: string[]): CsvDateFormat {
98
+
let sawIso = false;
99
+
for (const value of values) {
100
+
const text = value.trim();
101
+
if (/^\d{4}-\d{1,2}-\d{1,2}/.test(text)) {
102
+
sawIso = true;
103
+
continue;
104
+
}
105
+
const match = text.match(/^(\d{1,2})[/.-](\d{1,2})[/.-](\d{2,4})$/);
106
+
if (!match) continue;
107
+
if (Number(match[1]) > 12) return 'dmy';
108
+
if (Number(match[2]) > 12) return 'mdy';
109
+
}
110
+
return sawIso ? 'iso' : 'mdy';
111
+
}
112
+
113
+
/** Best-guess column assignment from a header row. Always user-confirmable. */
114
+
export function detectMapping(parsed: ParsedCsv): CsvMapping | null {
115
+
const { headers, rows } = parsed;
116
+
const date = findHeader(headers, DATE_RE);
117
+
const description = findHeader(headers, DESC_RE);
118
+
if (!date || !description) return null;
119
+
120
+
const amount = findHeader(headers, AMOUNT_RE);
121
+
const debit = findHeader(headers, DEBIT_RE);
122
+
const credit = findHeader(headers, CREDIT_RE);
123
+
124
+
const dateIndex = headers.indexOf(date);
125
+
const dateFormat = detectDateFormat(rows.map((r) => r[dateIndex] ?? ''));
126
+
127
+
// A distinct transaction-date column only counts when it isn't the one already
128
+
// serving as the post date.
129
+
const txnDate = findHeader(headers, TXN_DATE_RE);
130
+
const transactedDate = txnDate && txnDate !== date ? txnDate : null;
131
+
132
+
// Prefer an explicit signed column; fall back to a debit/credit pair.
133
+
const payee = findHeader(headers, PAYEE_RE);
134
+
const base = {
135
+
date,
136
+
transactedDate,
137
+
dateFormat,
138
+
description,
139
+
payee: payee && payee !== description ? payee : null,
140
+
memo: findHeader(headers, MEMO_RE)
141
+
};
142
+
if (amount) return { ...base, amountMode: 'signed', amount };
143
+
if (debit || credit) return { ...base, amountMode: 'debit-credit', debit, credit };
144
+
return null;
145
+
}
146
+
147
+
/**
148
+
* Normalize a bank CSV's negative conventions into a signed decimal string that
149
+
* parseAmountToCents accepts. That function's strict regex is the sync path's
150
+
* contract — widening it to swallow accountant's parentheses would weaken
151
+
* validation on SimpleFIN payloads to serve this one, so the tolerance lives here.
152
+
* Returns '' for a blank cell.
153
+
*/
154
+
export function normalizeCsvAmount(raw: string): string {
155
+
let text = String(raw ?? '').trim();
156
+
if (!text) return '';
157
+
158
+
let negative = false;
159
+
const parenthesized = text.match(/^\((.*)\)$/);
160
+
if (parenthesized) {
161
+
negative = true;
162
+
text = parenthesized[1].trim();
163
+
}
164
+
if (text.endsWith('-')) {
165
+
negative = !negative;
166
+
text = text.slice(0, -1).trim();
167
+
}
168
+
if (text.startsWith('-')) {
169
+
negative = !negative;
170
+
text = text.slice(1).trim();
171
+
}
172
+
173
+
// Drop currency symbols and stray spaces; parseAmountToCents handles commas.
174
+
text = text.replace(/[^\d.,]/g, '');
175
+
if (!text) return '';
176
+
return negative ? `-${text}` : text;
177
+
}
178
+
179
+
/**
180
+
* Parse a CSV date cell to Unix seconds at **noon UTC**. ISO always wins.
181
+
*
182
+
* Noon, not midnight: a CSV gives a calendar day with no time, but `formatDay`
183
+
* renders in local time, so a UTC-midnight stamp displays as the *previous* day
184
+
* anywhere west of Greenwich — and a July 1 row would read "Jun 30" while still
185
+
* counting in July's report, which buckets by UTC. Noon is the same calendar day
186
+
* in local time for every offset from -11 to +12, and still lands inside the
187
+
* right UTC month. `formatMonth` already anchors to the 15th for the same reason.
188
+
*/
189
+
export function parseCsvDate(raw: string, format: CsvDateFormat): number {
190
+
const text = String(raw ?? '').trim();
191
+
let year: number, month: number, day: number;
192
+
193
+
const iso = text.match(/^(\d{4})-(\d{1,2})-(\d{1,2})/);
194
+
if (iso) {
195
+
[year, month, day] = [Number(iso[1]), Number(iso[2]), Number(iso[3])];
196
+
} else {
197
+
const parts = text.match(/^(\d{1,2})[/.-](\d{1,2})[/.-](\d{2,4})$/);
198
+
if (!parts) throw new Error(`Unparseable date: ${JSON.stringify(raw)}`);
199
+
const first = Number(parts[1]);
200
+
const second = Number(parts[2]);
201
+
year = Number(parts[3]);
202
+
if (year < 100) year += 2000;
203
+
if (format === 'dmy') {
204
+
day = first;
205
+
month = second;
206
+
} else {
207
+
month = first;
208
+
day = second;
209
+
}
210
+
}
211
+
212
+
if (month < 1 || month > 12 || day < 1 || day > 31) {
213
+
throw new Error(`Unparseable date: ${JSON.stringify(raw)}`);
214
+
}
215
+
return Date.UTC(year, month - 1, day, 12) / 1000;
216
+
}
217
+
218
+
/**
219
+
* The grouping key IS the row's content, so rows within a group are
220
+
* interchangeable: a re-export listing them in a different order, or a superset
221
+
* of them, re-derives the same id set for rows already ingested and only genuinely
222
+
* new rows fall outside it. Description is folded (trimmed, whitespace-collapsed,
223
+
* lowercased) so cosmetic re-rendering doesn't mint a new identity.
224
+
*/
225
+
function contentKey(accountId: string, posted: number, amountCents: number, description: string) {
226
+
const folded = description.trim().replace(/\s+/g, ' ').toLowerCase();
227
+
return [accountId, String(posted), String(amountCents), folded].join('�');
228
+
}
229
+
230
+
function syntheticId(key: string, occurrence: number): string {
231
+
const digest = createHash('sha256').update(`${key}�${occurrence}`).digest('hex');
232
+
return `csv:${digest.slice(0, 32)}`;
233
+
}
234
+
235
+
function cell(row: string[], headers: string[], name: string | null | undefined): string {
236
+
if (!name) return '';
237
+
const index = headers.indexOf(name);
238
+
return index === -1 ? '' : (row[index] ?? '');
239
+
}
240
+
241
+
function resolveAmountCents(row: string[], headers: string[], mapping: CsvMapping): number {
242
+
if (mapping.amountMode === 'signed') {
243
+
const normalized = normalizeCsvAmount(cell(row, headers, mapping.amount));
244
+
if (!normalized) throw new Error('amount is blank');
245
+
return parseAmountToCents(normalized);
246
+
}
247
+
248
+
const debitText = normalizeCsvAmount(cell(row, headers, mapping.debit));
249
+
const creditText = normalizeCsvAmount(cell(row, headers, mapping.credit));
250
+
const debit = debitText ? parseAmountToCents(debitText) : 0;
251
+
const credit = creditText ? parseAmountToCents(creditText) : 0;
252
+
253
+
if (debit !== 0 && credit !== 0) throw new Error('both debit and credit are populated');
254
+
if (debit !== 0) return -Math.abs(debit);
255
+
if (credit !== 0) return Math.abs(credit);
256
+
throw new Error('neither debit nor credit is populated');
257
+
}
258
+
259
+
/**
260
+
* Pure: archived bytes + mapping + account -> candidate rows. Imported rows are
261
+
* settled history, so they carry `posted` and are never pending — matching the
262
+
* shape sync emits for posted rows, so every downstream query behaves identically.
263
+
*/
264
+
export function normalizeCsv(
265
+
payloadText: string,
266
+
mapping: CsvMapping,
267
+
accountId: string
268
+
): NormalizedCsv {
269
+
const { headers, rows } = parseCsv(payloadText);
270
+
const candidates: CsvCandidate[] = [];
271
+
const errors: string[] = [];
272
+
const seen = new Map<string, number>();
273
+
274
+
rows.forEach((row, index) => {
275
+
const line = index + 2; // 1-based, and the header occupies line 1
276
+
try {
277
+
const posted = parseCsvDate(cell(row, headers, mapping.date), mapping.dateFormat);
278
+
const amountCents = resolveAmountCents(row, headers, mapping);
279
+
const description = cell(row, headers, mapping.description).trim();
280
+
if (!description) throw new Error('description is blank');
281
+
282
+
const transactedRaw = cell(row, headers, mapping.transactedDate).trim();
283
+
const transactedAt = transactedRaw
284
+
? parseCsvDate(transactedRaw, mapping.dateFormat)
285
+
: null;
286
+
287
+
const key = contentKey(accountId, posted, amountCents, description);
288
+
const occurrence = seen.get(key) ?? 0;
289
+
seen.set(key, occurrence + 1);
290
+
291
+
const payee = cell(row, headers, mapping.payee).trim();
292
+
const memo = cell(row, headers, mapping.memo).trim();
293
+
294
+
candidates.push({
295
+
syntheticId: syntheticId(key, occurrence),
296
+
posted,
297
+
transactedAt,
298
+
amountCents,
299
+
description,
300
+
payee: payee || null,
301
+
memo: memo || null
302
+
});
303
+
} catch (err) {
304
+
errors.push(`Line ${line}: ${err instanceof Error ? err.message : String(err)}`);
305
+
}
306
+
});
307
+
308
+
return { candidates, errors };
309
+
}
+491
src/lib/server/services/imports.test.ts
+491
src/lib/server/services/imports.test.ts
···
1
+
/// <reference lib="deno.ns" />
2
+
import { openDatabase } from '../db.ts';
3
+
import {
4
+
commitImport,
5
+
createDraftImport,
6
+
findPotentialDuplicates,
7
+
previewImport,
8
+
saveDecisions,
9
+
saveMapping,
10
+
undoImport
11
+
} from './imports.ts';
12
+
import { normalizeCsv, type CsvMapping } from './csv-import.ts';
13
+
import { createRule } from './rules.ts';
14
+
import { categorizeManually } from './categorization.ts';
15
+
import { monthlyReport, netWorthSeries } from './reports.ts';
16
+
import type { DatabaseSync } from 'node:sqlite';
17
+
18
+
const MIGRATIONS_DIR = new URL('../../../../migrations', import.meta.url).pathname.replace(
19
+
/^\/([A-Za-z]:)/,
20
+
'$1'
21
+
);
22
+
23
+
/** The synced row sits just after UTC midnight on purpose: a CSV row is anchored at
24
+
* noon, so a naive ±86400s window would be hour-sensitive. The duplicate check
25
+
* must compare calendar days. */
26
+
const JULY_02_EARLY = Math.floor(Date.UTC(2026, 6, 2, 1, 30) / 1000);
27
+
/** Where a CSV row for that date lands. */
28
+
const JULY_02_NOON = Math.floor(Date.UTC(2026, 6, 2, 12) / 1000);
29
+
const JULY_03_NOON = Math.floor(Date.UTC(2026, 6, 3, 12) / 1000);
30
+
31
+
const MAPPING: CsvMapping = {
32
+
date: 'Date',
33
+
dateFormat: 'iso',
34
+
amountMode: 'signed',
35
+
amount: 'Amount',
36
+
description: 'Description'
37
+
};
38
+
39
+
/** The account holds a synced posted row, a synced pending row, and a category. */
40
+
function testDb(): DatabaseSync {
41
+
const db = openDatabase(`${Deno.makeTempDirSync()}/t.db`, MIGRATIONS_DIR);
42
+
const now = new Date().toISOString();
43
+
db.prepare("INSERT INTO users (did, handle, created_at) VALUES ('did:plc:test', 'tester', ?)").run(
44
+
now
45
+
);
46
+
db.prepare("INSERT INTO connections (access_url, claimed_at) VALUES ('https://x', ?)").run(now);
47
+
for (const [id, state] of [
48
+
['chk', 'ACTIVE'],
49
+
['ghost', 'HIDDEN']
50
+
]) {
51
+
db.prepare(
52
+
`INSERT INTO accounts (id, connection_id, name, currency, state, last_successful_data_at, created_at)
53
+
VALUES (?, 1, ?, 'USD', ?, ?, ?)`
54
+
).run(id, id, state, now, now);
55
+
}
56
+
db.prepare("INSERT INTO categories (name, kind, created_at) VALUES ('Dining','expense',?)").run(
57
+
now
58
+
);
59
+
const insert = db.prepare(
60
+
`INSERT INTO transactions
61
+
(account_id, sfin_id, posted, amount_cents, description, pending, created_at)
62
+
VALUES (?, ?, ?, ?, ?, ?, ?)`
63
+
);
64
+
// The bank's own rendering of the same July 2 purchase the CSV also contains.
65
+
insert.run('chk', 'sfin-amazon', JULY_02_EARLY, -5231, 'AMZN Mktp US*2K4LM9QR3', 0, now);
66
+
insert.run('chk', 'sfin-pending', null, -500, 'PENDING COFFEE', 1, now);
67
+
return db;
68
+
}
69
+
70
+
function draft(db: DatabaseSync, csv: string, accountId = 'chk'): number {
71
+
const id = createDraftImport(db, accountId, 'bank.csv', csv);
72
+
saveMapping(db, id, MAPPING);
73
+
return id;
74
+
}
75
+
76
+
Deno.test('createDraftImport archives bytes verbatim before parsing', () => {
77
+
const db = testDb();
78
+
// Garbage that could never parse still has to survive the upload.
79
+
const raw = 'not,really\nvalid ""csv';
80
+
const id = createDraftImport(db, 'chk', 'junk.csv', raw);
81
+
const row = db.prepare('SELECT payload, status FROM imports WHERE id = ?').get(id) as {
82
+
payload: string;
83
+
status: string;
84
+
};
85
+
if (row.payload !== raw) throw new Error('payload was not archived verbatim');
86
+
if (row.status !== 'draft') throw new Error(`expected draft, got ${row.status}`);
87
+
db.close();
88
+
});
89
+
90
+
Deno.test('a hidden account cannot be an import target', () => {
91
+
const db = testDb();
92
+
let threw = false;
93
+
try {
94
+
createDraftImport(db, 'ghost', 'x.csv', 'Date,Description,Amount\n');
95
+
} catch {
96
+
threw = true;
97
+
}
98
+
if (!threw) throw new Error('expected hidden account to be rejected');
99
+
db.close();
100
+
});
101
+
102
+
Deno.test('cross-source duplicate is flagged despite a different description', () => {
103
+
const db = testDb();
104
+
const csv = 'Date,Description,Amount\n2026-07-02,Amazon,-52.31\n';
105
+
const { candidates } = normalizeCsv(csv, MAPPING, 'chk');
106
+
const [result] = findPotentialDuplicates(db, 'chk', candidates);
107
+
if (result.status !== 'flagged') throw new Error(`expected flagged, got ${result.status}`);
108
+
if (result.match?.description !== 'AMZN Mktp US*2K4LM9QR3') throw new Error('wrong match');
109
+
if (result.match?.source !== 'synced') throw new Error('match should be synced');
110
+
db.close();
111
+
});
112
+
113
+
Deno.test('duplicate window spans one calendar day, not 24 hours', () => {
114
+
const db = testDb();
115
+
// The synced row is at 01:30 UTC on Jul 2; this candidate lands at noon Jul 3.
116
+
// That is 34.5 hours apart but one calendar day, and must still flag.
117
+
const near = normalizeCsv(
118
+
'Date,Description,Amount\n2026-07-03,Amazon,-52.31\n',
119
+
MAPPING,
120
+
'chk'
121
+
).candidates;
122
+
if (findPotentialDuplicates(db, 'chk', near)[0].status !== 'flagged') {
123
+
throw new Error('one calendar day off should be flagged regardless of the hours');
124
+
}
125
+
const far = normalizeCsv(
126
+
'Date,Description,Amount\n2026-07-04,Amazon,-52.31\n',
127
+
MAPPING,
128
+
'chk'
129
+
).candidates;
130
+
if (findPotentialDuplicates(db, 'chk', far)[0].status !== 'new') {
131
+
throw new Error('two calendar days off should not be flagged');
132
+
}
133
+
// And the day before, symmetrically.
134
+
const before = normalizeCsv(
135
+
'Date,Description,Amount\n2026-07-01,Amazon,-52.31\n',
136
+
MAPPING,
137
+
'chk'
138
+
).candidates;
139
+
if (findPotentialDuplicates(db, 'chk', before)[0].status !== 'flagged') {
140
+
throw new Error('the window must be symmetric');
141
+
}
142
+
db.close();
143
+
});
144
+
145
+
Deno.test('a different amount on the same day is not a duplicate', () => {
146
+
const db = testDb();
147
+
const { candidates } = normalizeCsv(
148
+
'Date,Description,Amount\n2026-07-02,Amazon,-52.30\n',
149
+
MAPPING,
150
+
'chk'
151
+
);
152
+
if (findPotentialDuplicates(db, 'chk', candidates)[0].status !== 'new') {
153
+
throw new Error('a one-cent difference is a different transaction');
154
+
}
155
+
db.close();
156
+
});
157
+
158
+
Deno.test('matches are consumed, so a real second charge stays importable', () => {
159
+
const db = testDb();
160
+
// Two identical charges in the file, one already synced: one dupe, one new.
161
+
const { candidates } = normalizeCsv(
162
+
'Date,Description,Amount\n2026-07-02,Amazon,-52.31\n2026-07-02,Amazon,-52.31\n',
163
+
MAPPING,
164
+
'chk'
165
+
);
166
+
const results = findPotentialDuplicates(db, 'chk', candidates);
167
+
const statuses = results.map((r) => r.status).sort();
168
+
if (statuses.join(',') !== 'flagged,new') {
169
+
throw new Error(`expected one flagged and one new, got ${statuses.join(',')}`);
170
+
}
171
+
db.close();
172
+
});
173
+
174
+
Deno.test('commit defaults flagged rows to skip and leaves the synced row alone', () => {
175
+
const db = testDb();
176
+
const id = draft(db, 'Date,Description,Amount\n2026-07-02,Amazon,-52.31\n2026-06-01,Shell,-40.00\n');
177
+
const result = commitImport(db, id);
178
+
179
+
if (result.skipped !== 1) throw new Error(`expected 1 skipped, got ${result.skipped}`);
180
+
if (result.inserted !== 1) throw new Error(`expected 1 inserted, got ${result.inserted}`);
181
+
182
+
const amazon = db
183
+
.prepare("SELECT COUNT(*) AS n FROM transactions WHERE account_id='chk' AND amount_cents=-5231 AND removed_at IS NULL")
184
+
.get() as { n: number };
185
+
if (amazon.n !== 1) throw new Error(`double-counted: ${amazon.n} rows at -5231`);
186
+
187
+
const synced = db
188
+
.prepare("SELECT description, import_id FROM transactions WHERE sfin_id='sfin-amazon'")
189
+
.get() as { description: string; import_id: number | null };
190
+
if (synced.description !== 'AMZN Mktp US*2K4LM9QR3' || synced.import_id !== null) {
191
+
throw new Error('the synced row must be untouched');
192
+
}
193
+
db.close();
194
+
});
195
+
196
+
Deno.test('a kept flagged row is imported alongside the existing one', () => {
197
+
const db = testDb();
198
+
const csv = 'Date,Description,Amount\n2026-07-02,Amazon,-52.31\n';
199
+
const id = draft(db, csv);
200
+
const { candidates } = normalizeCsv(csv, MAPPING, 'chk');
201
+
saveDecisions(db, id, { [candidates[0].syntheticId]: 'keep' });
202
+
203
+
const result = commitImport(db, id);
204
+
if (result.inserted !== 1) throw new Error(`expected 1 inserted, got ${result.inserted}`);
205
+
const rows = db
206
+
.prepare("SELECT COUNT(*) AS n FROM transactions WHERE account_id='chk' AND amount_cents=-5231 AND removed_at IS NULL")
207
+
.get() as { n: number };
208
+
if (rows.n !== 2) throw new Error(`expected both rows, got ${rows.n}`);
209
+
db.close();
210
+
});
211
+
212
+
Deno.test('import never disturbs pending rows, snapshots, or account state', () => {
213
+
const db = testDb();
214
+
const before = db.prepare("SELECT state, last_successful_data_at FROM accounts WHERE id='chk'").get() as {
215
+
state: string;
216
+
last_successful_data_at: string;
217
+
};
218
+
219
+
const id = draft(db, 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n');
220
+
commitImport(db, id);
221
+
222
+
// The stale-pending sweep in ingestTransactions would have removed this; the
223
+
// additive writer must not.
224
+
const pending = db
225
+
.prepare("SELECT removed_at FROM transactions WHERE sfin_id='sfin-pending'")
226
+
.get() as { removed_at: string | null };
227
+
if (pending.removed_at !== null) throw new Error('an import must never remove a pending row');
228
+
229
+
const snaps = db.prepare('SELECT COUNT(*) AS n FROM balance_snapshots').get() as { n: number };
230
+
if (snaps.n !== 0) throw new Error('an import must not write balance snapshots');
231
+
232
+
const after = db.prepare("SELECT state, last_successful_data_at FROM accounts WHERE id='chk'").get() as {
233
+
state: string;
234
+
last_successful_data_at: string;
235
+
};
236
+
if (after.state !== before.state || after.last_successful_data_at !== before.last_successful_data_at) {
237
+
throw new Error('an import must not touch account state');
238
+
}
239
+
db.close();
240
+
});
241
+
242
+
Deno.test('re-importing the same file inserts nothing', () => {
243
+
const db = testDb();
244
+
const csv = 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n2026-06-02,Coffee,-4.75\n';
245
+
commitImport(db, draft(db, csv));
246
+
const second = commitImport(db, draft(db, csv));
247
+
248
+
if (second.inserted !== 0) throw new Error(`expected 0 inserted, got ${second.inserted}`);
249
+
if (second.alreadyPresent !== 2) throw new Error(`expected 2 already-present, got ${second.alreadyPresent}`);
250
+
const total = db
251
+
.prepare("SELECT COUNT(*) AS n FROM transactions WHERE import_id IS NOT NULL AND removed_at IS NULL")
252
+
.get() as { n: number };
253
+
if (total.n !== 2) throw new Error(`expected 2 imported rows total, got ${total.n}`);
254
+
db.close();
255
+
});
256
+
257
+
Deno.test('an overlapping file adds only the new rows', () => {
258
+
const db = testDb();
259
+
commitImport(db, draft(db, 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n2026-06-02,Coffee,-4.75\n'));
260
+
const second = commitImport(
261
+
db,
262
+
draft(db, 'Date,Description,Amount\n2026-06-02,Coffee,-4.75\n2026-06-03,Bagel,-3.00\n')
263
+
);
264
+
if (second.inserted !== 1) throw new Error(`expected 1 inserted, got ${second.inserted}`);
265
+
if (second.alreadyPresent !== 1) throw new Error(`expected 1 already-present, got ${second.alreadyPresent}`);
266
+
db.close();
267
+
});
268
+
269
+
Deno.test('committed rows carry import_id and are categorized by existing rules', () => {
270
+
const db = testDb();
271
+
createRule(db, {
272
+
pattern: 'shell',
273
+
matchType: 'contains',
274
+
categoryId: 2,
275
+
createdByDid: 'did:plc:test'
276
+
});
277
+
278
+
const id = draft(db, 'Date,Description,Amount\n2026-06-01,SHELL OIL 4417,-40.00\n');
279
+
const result = commitImport(db, id);
280
+
if (result.ruleCategorized < 1) throw new Error('rules should have caught the imported row');
281
+
282
+
const row = db
283
+
.prepare("SELECT import_id, category_id FROM transactions WHERE description='SHELL OIL 4417'")
284
+
.get() as { import_id: number; category_id: number | null };
285
+
if (row.import_id !== id) throw new Error(`import_id not stamped: ${row.import_id}`);
286
+
if (row.category_id !== 2) throw new Error('imported row should be rule-categorized');
287
+
288
+
const event = db
289
+
.prepare(
290
+
`SELECT source FROM categorization_events WHERE transaction_id =
291
+
(SELECT id FROM transactions WHERE description='SHELL OIL 4417')`
292
+
)
293
+
.get() as { source: string };
294
+
if (event.source !== 'rule') throw new Error(`expected a rule event, got ${event.source}`);
295
+
db.close();
296
+
});
297
+
298
+
Deno.test('backfilled rows reach the month report — the point of the feature', () => {
299
+
// The whole path: a CSV fills a gap, existing rules categorize it, and the
300
+
// month that had no data now reports. Report totals only count categorized
301
+
// rows, so this exercises import -> rules -> report end to end.
302
+
const db = testDb();
303
+
db.prepare("INSERT INTO categories (name, kind, created_at) VALUES ('Salary','income',?)").run(
304
+
new Date().toISOString()
305
+
);
306
+
createRule(db, {
307
+
pattern: 'shell',
308
+
matchType: 'contains',
309
+
categoryId: 2,
310
+
createdByDid: 'did:plc:test'
311
+
});
312
+
createRule(db, {
313
+
pattern: 'paycheck',
314
+
matchType: 'contains',
315
+
categoryId: 3,
316
+
createdByDid: 'did:plc:test'
317
+
});
318
+
319
+
if (monthlyReport(db, '2026-06').expenseTotalCents !== 0) throw new Error('June should start empty');
320
+
321
+
const id = draft(db, 'Date,Description,Amount\n2026-06-15,SHELL OIL,-40.00\n2026-06-20,ACME PAYCHECK,1000.00\n');
322
+
commitImport(db, id);
323
+
324
+
const june = monthlyReport(db, '2026-06');
325
+
if (june.expenseTotalCents !== -4000) {
326
+
throw new Error(`imported expense missing from the report: ${june.expenseTotalCents}`);
327
+
}
328
+
if (june.incomeTotalCents !== 100000) {
329
+
throw new Error(`imported income missing from the report: ${june.incomeTotalCents}`);
330
+
}
331
+
332
+
// And they leave again with an undo.
333
+
undoImport(db, id);
334
+
const after = monthlyReport(db, '2026-06');
335
+
if (after.expenseTotalCents !== 0 || after.incomeTotalCents !== 0) {
336
+
throw new Error('undone rows must leave the report');
337
+
}
338
+
db.close();
339
+
});
340
+
341
+
Deno.test('an import writes no snapshots, so net worth is untouched', () => {
342
+
const db = testDb();
343
+
const before = netWorthSeries(db).length;
344
+
commitImport(db, draft(db, 'Date,Description,Amount\n2026-06-15,Shell,-40.00\n'));
345
+
if (netWorthSeries(db).length !== before) {
346
+
throw new Error('an import must not add points to the net worth series');
347
+
}
348
+
db.close();
349
+
});
350
+
351
+
Deno.test('preview states the range and the counts', () => {
352
+
const db = testDb();
353
+
const id = draft(
354
+
db,
355
+
'Date,Description,Amount\n2026-06-01,Shell,-40.00\n2026-07-02,Amazon,-52.31\nbad,Row,-1.00\n'
356
+
);
357
+
const preview = previewImport(db, id);
358
+
if (preview.rangeStart !== Math.floor(Date.UTC(2026, 5, 1, 12) / 1000)) {
359
+
throw new Error('wrong start');
360
+
}
361
+
if (preview.rangeEnd !== JULY_02_NOON) throw new Error('wrong end');
362
+
if (preview.newCount !== 1) throw new Error(`newCount ${preview.newCount}`);
363
+
if (preview.flaggedCount !== 1) throw new Error(`flaggedCount ${preview.flaggedCount}`);
364
+
if (preview.errors.length !== 1) throw new Error('the bad row should be reported');
365
+
db.close();
366
+
});
367
+
368
+
Deno.test('a draft cannot be committed twice', () => {
369
+
const db = testDb();
370
+
const id = draft(db, 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n');
371
+
commitImport(db, id);
372
+
let threw = false;
373
+
try {
374
+
commitImport(db, id);
375
+
} catch {
376
+
threw = true;
377
+
}
378
+
if (!threw) throw new Error('expected a committed import to refuse a second commit');
379
+
db.close();
380
+
});
381
+
382
+
Deno.test('undo removes exactly that import and spares synced rows', () => {
383
+
const db = testDb();
384
+
const id = draft(db, 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n2026-06-02,Coffee,-4.75\n');
385
+
commitImport(db, id);
386
+
387
+
const removed = undoImport(db, id);
388
+
if (removed !== 2) throw new Error(`expected 2 removed, got ${removed}`);
389
+
390
+
const live = db
391
+
.prepare('SELECT COUNT(*) AS n FROM transactions WHERE import_id = ? AND removed_at IS NULL')
392
+
.get(id) as { n: number };
393
+
if (live.n !== 0) throw new Error('undo should remove every row of the import');
394
+
395
+
const synced = db
396
+
.prepare("SELECT removed_at FROM transactions WHERE sfin_id='sfin-amazon'")
397
+
.get() as { removed_at: string | null };
398
+
if (synced.removed_at !== null) throw new Error('undo must not touch synced rows');
399
+
400
+
const status = db.prepare('SELECT status FROM imports WHERE id = ?').get(id) as { status: string };
401
+
if (status.status !== 'undone') throw new Error(`expected undone, got ${status.status}`);
402
+
db.close();
403
+
});
404
+
405
+
Deno.test('event history survives undo', () => {
406
+
const db = testDb();
407
+
const id = draft(db, 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n');
408
+
commitImport(db, id);
409
+
410
+
const txn = db.prepare("SELECT id FROM transactions WHERE description='Shell'").get() as {
411
+
id: number;
412
+
};
413
+
categorizeManually(db, txn.id, 2, 'did:plc:test');
414
+
undoImport(db, id);
415
+
416
+
const events = db
417
+
.prepare('SELECT COUNT(*) AS n FROM categorization_events WHERE transaction_id = ?')
418
+
.get(txn.id) as { n: number };
419
+
if (events.n === 0) throw new Error('the append-only event log must outlive the rows');
420
+
db.close();
421
+
});
422
+
423
+
Deno.test('re-import after a corrected mapping does not resurrect the old rows', () => {
424
+
const db = testDb();
425
+
// A day-first file misread as month-first: June 7 instead of July 6.
426
+
const csv = 'Date,Description,Amount\n06/07/2026,Shell,-40.00\n';
427
+
const wrong = createDraftImport(db, 'chk', 'bank.csv', csv);
428
+
saveMapping(db, wrong, { ...MAPPING, dateFormat: 'mdy' });
429
+
commitImport(db, wrong);
430
+
undoImport(db, wrong);
431
+
432
+
const right = createDraftImport(db, 'chk', 'bank.csv', csv);
433
+
saveMapping(db, right, { ...MAPPING, dateFormat: 'dmy' });
434
+
const result = commitImport(db, right);
435
+
if (result.inserted !== 1) throw new Error(`expected a fresh insert, got ${result.inserted}`);
436
+
437
+
const live = db
438
+
.prepare("SELECT posted FROM transactions WHERE removed_at IS NULL AND import_id IS NOT NULL")
439
+
.all() as { posted: number }[];
440
+
if (live.length !== 1) throw new Error(`expected exactly 1 live imported row, got ${live.length}`);
441
+
if (live[0].posted !== Math.floor(Date.UTC(2026, 6, 6, 12) / 1000)) {
442
+
throw new Error('the corrected date should be July 6');
443
+
}
444
+
db.close();
445
+
});
446
+
447
+
Deno.test('undo then re-import with the same mapping revives rather than duplicating', () => {
448
+
const db = testDb();
449
+
const csv = 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n';
450
+
const first = draft(db, csv);
451
+
commitImport(db, first);
452
+
undoImport(db, first);
453
+
454
+
const second = draft(db, csv);
455
+
const result = commitImport(db, second);
456
+
if (result.revived !== 1) throw new Error(`expected 1 revived, got ${result.revived}`);
457
+
458
+
const rows = db
459
+
.prepare("SELECT import_id FROM transactions WHERE description='Shell' AND removed_at IS NULL")
460
+
.all() as { import_id: number }[];
461
+
if (rows.length !== 1) throw new Error(`expected 1 live row, got ${rows.length}`);
462
+
if (rows[0].import_id !== second) throw new Error('the revived row should belong to the new import');
463
+
db.close();
464
+
});
465
+
466
+
Deno.test('only a committed import can be undone', () => {
467
+
const db = testDb();
468
+
const id = draft(db, 'Date,Description,Amount\n2026-06-01,Shell,-40.00\n');
469
+
let threw = false;
470
+
try {
471
+
undoImport(db, id);
472
+
} catch {
473
+
threw = true;
474
+
}
475
+
if (!threw) throw new Error('expected undoing a draft to be refused');
476
+
db.close();
477
+
});
478
+
479
+
Deno.test('JULY_03 nearby row exercises the window boundary', () => {
480
+
// Guards the constant itself: a row exactly one day out must still flag.
481
+
const db = testDb();
482
+
const { candidates } = normalizeCsv(
483
+
`Date,Description,Amount\n2026-07-03,Amazon,-52.31\n`,
484
+
MAPPING,
485
+
'chk'
486
+
);
487
+
const [result] = findPotentialDuplicates(db, 'chk', candidates);
488
+
if (result.candidate.posted !== JULY_03_NOON) throw new Error('fixture drift');
489
+
if (result.status !== 'flagged') throw new Error('boundary row should flag');
490
+
db.close();
491
+
});
+392
src/lib/server/services/imports.ts
+392
src/lib/server/services/imports.ts
···
1
+
// Lifecycle of a CSV backfill import: archive -> map -> review -> commit -> undo.
2
+
// The DB-touching half of CSV import; csv-import.ts holds the pure parsing, the
3
+
// same way sync.ts and normalize.ts are split.
4
+
//
5
+
// An import is an ADDITIVE writer. It only ever inserts (or revives a row it
6
+
// previously removed). It never reconciles pending transactions, never sweeps,
7
+
// never mutates a synced row, never writes balance snapshots, and never touches
8
+
// account state — all of that is exclusively sync's authority.
9
+
10
+
import type { DatabaseSync } from 'node:sqlite';
11
+
import { normalizeCsv, type CsvCandidate, type CsvMapping } from './csv-import.ts';
12
+
import { applyRulesToUncategorized } from './rules.ts';
13
+
14
+
/**
15
+
* A CSV's date is often the transaction date while SimpleFIN's posted date trails
16
+
* it by a day, so a strict same-day check would miss the very rows most at risk of
17
+
* being double-counted.
18
+
*
19
+
* Measured in whole UTC calendar days, not in seconds: a CSV row carries only a
20
+
* date (anchored at noon) while a synced row carries a real timestamp at whatever
21
+
* hour the bank posted it. Comparing raw seconds would make the window mean
22
+
* "within 24 hours", so whether two rows a calendar day apart matched would depend
23
+
* on the hours involved — noon-vs-midnight is 36 hours and would silently miss.
24
+
*/
25
+
const DUPLICATE_WINDOW_DAYS = 1;
26
+
27
+
/** Whole days since the epoch, UTC. Bank data is post-1970, so truncation is floor. */
28
+
const UTC_DAY = `CAST(COALESCE(posted, transacted_at) / 86400 AS INTEGER)`;
29
+
30
+
export type ImportStatus = 'draft' | 'committed' | 'undone';
31
+
32
+
export interface ImportRecord {
33
+
id: number;
34
+
accountId: string;
35
+
filename: string | null;
36
+
uploadedAt: string;
37
+
payload: string;
38
+
mapping: CsvMapping | null;
39
+
decisions: Record<string, 'keep' | 'skip'>;
40
+
status: ImportStatus;
41
+
committedAt: string | null;
42
+
undoneAt: string | null;
43
+
}
44
+
45
+
export interface ExistingMatch {
46
+
id: number;
47
+
effectiveAt: number | null;
48
+
amountCents: number;
49
+
description: string;
50
+
payee: string | null;
51
+
source: 'synced' | 'imported';
52
+
}
53
+
54
+
/** `already-present` rows are never surfaced — identity already made them a no-op. */
55
+
export type CandidateStatus = 'new' | 'already-present' | 'flagged';
56
+
57
+
export interface ClassifiedCandidate {
58
+
candidate: CsvCandidate;
59
+
status: CandidateStatus;
60
+
match: ExistingMatch | null;
61
+
}
62
+
63
+
interface ImportRow {
64
+
id: number;
65
+
account_id: string;
66
+
filename: string | null;
67
+
uploaded_at: string;
68
+
payload: string;
69
+
mapping: string | null;
70
+
decisions: string | null;
71
+
status: ImportStatus;
72
+
committed_at: string | null;
73
+
undone_at: string | null;
74
+
}
75
+
76
+
function hydrate(row: ImportRow): ImportRecord {
77
+
return {
78
+
id: row.id,
79
+
accountId: row.account_id,
80
+
filename: row.filename,
81
+
uploadedAt: row.uploaded_at,
82
+
payload: row.payload,
83
+
mapping: row.mapping ? (JSON.parse(row.mapping) as CsvMapping) : null,
84
+
decisions: row.decisions ? (JSON.parse(row.decisions) as Record<string, 'keep' | 'skip'>) : {},
85
+
status: row.status,
86
+
committedAt: row.committed_at,
87
+
undoneAt: row.undone_at
88
+
};
89
+
}
90
+
91
+
/** Archive the bytes before anything is parsed, so nothing is lost to a bad mapping. */
92
+
export function createDraftImport(
93
+
db: DatabaseSync,
94
+
accountId: string,
95
+
filename: string | null,
96
+
payload: string
97
+
): number {
98
+
const account = db
99
+
.prepare("SELECT id, state FROM accounts WHERE id = ? AND state != 'HIDDEN'")
100
+
.get(accountId) as { id: string } | undefined;
101
+
if (!account) throw new Error(`Unknown or hidden account: ${accountId}`);
102
+
103
+
const result = db
104
+
.prepare(
105
+
`INSERT INTO imports (account_id, filename, uploaded_at, payload, status)
106
+
VALUES (?, ?, ?, ?, 'draft')`
107
+
)
108
+
.run(accountId, filename, new Date().toISOString(), payload);
109
+
return Number(result.lastInsertRowid);
110
+
}
111
+
112
+
export function getImport(db: DatabaseSync, importId: number): ImportRecord | null {
113
+
const row = db.prepare('SELECT * FROM imports WHERE id = ?').get(importId) as
114
+
| ImportRow
115
+
| undefined;
116
+
return row ? hydrate(row) : null;
117
+
}
118
+
119
+
export function listImports(db: DatabaseSync): (ImportRecord & { rowCount: number })[] {
120
+
const rows = db
121
+
.prepare(
122
+
`SELECT i.*, (
123
+
SELECT COUNT(*) FROM transactions t
124
+
WHERE t.import_id = i.id AND t.removed_at IS NULL
125
+
) AS row_count
126
+
FROM imports i ORDER BY i.id DESC`
127
+
)
128
+
.all() as unknown as (ImportRow & { row_count: number })[];
129
+
return rows.map((r) => ({ ...hydrate(r), rowCount: r.row_count }));
130
+
}
131
+
132
+
function requireDraft(db: DatabaseSync, importId: number): ImportRecord {
133
+
const record = getImport(db, importId);
134
+
if (!record) throw new Error(`Unknown import: ${importId}`);
135
+
if (record.status !== 'draft') throw new Error(`Import ${importId} is already ${record.status}.`);
136
+
return record;
137
+
}
138
+
139
+
export function saveMapping(db: DatabaseSync, importId: number, mapping: CsvMapping): void {
140
+
requireDraft(db, importId);
141
+
db.prepare('UPDATE imports SET mapping = ? WHERE id = ?').run(JSON.stringify(mapping), importId);
142
+
}
143
+
144
+
export function saveDecisions(
145
+
db: DatabaseSync,
146
+
importId: number,
147
+
decisions: Record<string, 'keep' | 'skip'>
148
+
): void {
149
+
requireDraft(db, importId);
150
+
db.prepare('UPDATE imports SET decisions = ? WHERE id = ?').run(
151
+
JSON.stringify(decisions),
152
+
importId
153
+
);
154
+
}
155
+
156
+
/**
157
+
* Classify each candidate against what the account already holds.
158
+
*
159
+
* Matching is on amount and date only. The two sources render the same merchant
160
+
* differently — SimpleFIN gives the bank's raw string, the CSV its own — so
161
+
* matching on description would catch almost nothing while appearing to work.
162
+
* Description is evidence for the human, not a filter.
163
+
*
164
+
* Matches are consumed: if the file has two identical coffees and the account
165
+
* already holds one, exactly one candidate is flagged and the other is new.
166
+
*/
167
+
export function findPotentialDuplicates(
168
+
db: DatabaseSync,
169
+
accountId: string,
170
+
candidates: CsvCandidate[]
171
+
): ClassifiedCandidate[] {
172
+
const existing = db.prepare(
173
+
'SELECT id FROM transactions WHERE account_id = ? AND sfin_id = ? AND removed_at IS NULL'
174
+
);
175
+
const nearby = db.prepare(
176
+
`SELECT id, COALESCE(posted, transacted_at) AS effective_at, amount_cents,
177
+
description, payee, import_id
178
+
FROM transactions
179
+
WHERE account_id = ? AND removed_at IS NULL AND amount_cents = ?
180
+
AND ${UTC_DAY} BETWEEN ? AND ?
181
+
ORDER BY id`
182
+
);
183
+
184
+
const consumed = new Set<number>();
185
+
return candidates.map((candidate) => {
186
+
if (existing.get(accountId, candidate.syntheticId)) {
187
+
return { candidate, status: 'already-present' as const, match: null };
188
+
}
189
+
190
+
const day = Math.floor(candidate.posted / 86400);
191
+
const rows = nearby.all(
192
+
accountId,
193
+
candidate.amountCents,
194
+
day - DUPLICATE_WINDOW_DAYS,
195
+
day + DUPLICATE_WINDOW_DAYS
196
+
) as {
197
+
id: number;
198
+
effective_at: number | null;
199
+
amount_cents: number;
200
+
description: string;
201
+
payee: string | null;
202
+
import_id: number | null;
203
+
}[];
204
+
205
+
const hit = rows.find((r) => !consumed.has(r.id));
206
+
if (!hit) return { candidate, status: 'new' as const, match: null };
207
+
208
+
consumed.add(hit.id);
209
+
return {
210
+
candidate,
211
+
status: 'flagged' as const,
212
+
match: {
213
+
id: hit.id,
214
+
effectiveAt: hit.effective_at,
215
+
amountCents: hit.amount_cents,
216
+
description: hit.description,
217
+
payee: hit.payee,
218
+
source: hit.import_id === null ? ('synced' as const) : ('imported' as const)
219
+
}
220
+
};
221
+
});
222
+
}
223
+
224
+
export interface ImportPreview {
225
+
classified: ClassifiedCandidate[];
226
+
errors: string[];
227
+
/** Unix seconds; null when nothing parsed. */
228
+
rangeStart: number | null;
229
+
rangeEnd: number | null;
230
+
newCount: number;
231
+
alreadyPresentCount: number;
232
+
flaggedCount: number;
233
+
}
234
+
235
+
/**
236
+
* Re-parse the archived bytes with the stored mapping. Pure with respect to the
237
+
* archive: every wizard step can call this instead of holding state elsewhere.
238
+
*/
239
+
export function previewImport(db: DatabaseSync, importId: number): ImportPreview {
240
+
const record = getImport(db, importId);
241
+
if (!record) throw new Error(`Unknown import: ${importId}`);
242
+
if (!record.mapping) throw new Error(`Import ${importId} has no mapping yet.`);
243
+
244
+
const { candidates, errors } = normalizeCsv(record.payload, record.mapping, record.accountId);
245
+
const classified = findPotentialDuplicates(db, record.accountId, candidates);
246
+
const dates = candidates.map((c) => c.posted);
247
+
248
+
return {
249
+
classified,
250
+
errors,
251
+
rangeStart: dates.length ? Math.min(...dates) : null,
252
+
rangeEnd: dates.length ? Math.max(...dates) : null,
253
+
newCount: classified.filter((c) => c.status === 'new').length,
254
+
alreadyPresentCount: classified.filter((c) => c.status === 'already-present').length,
255
+
flaggedCount: classified.filter((c) => c.status === 'flagged').length
256
+
};
257
+
}
258
+
259
+
export interface CommitResult {
260
+
inserted: number;
261
+
revived: number;
262
+
skipped: number;
263
+
alreadyPresent: number;
264
+
ruleCategorized: number;
265
+
}
266
+
267
+
/**
268
+
* Insert-only. Deliberately shares nothing with ingestTransactions: that function
269
+
* treats its feed as authoritative for the account, and its stale-pending sweep
270
+
* would soft-delete every pending row absent from a backfill CSV.
271
+
*/
272
+
export function commitImport(db: DatabaseSync, importId: number): CommitResult {
273
+
const record = requireDraft(db, importId);
274
+
if (!record.mapping) throw new Error(`Import ${importId} has no mapping yet.`);
275
+
276
+
const { candidates } = normalizeCsv(record.payload, record.mapping, record.accountId);
277
+
const classified = findPotentialDuplicates(db, record.accountId, candidates);
278
+
const now = new Date().toISOString();
279
+
280
+
let inserted = 0;
281
+
let revived = 0;
282
+
let skipped = 0;
283
+
let alreadyPresent = 0;
284
+
285
+
db.exec('BEGIN');
286
+
try {
287
+
const findAny = db.prepare(
288
+
'SELECT id FROM transactions WHERE account_id = ? AND sfin_id = ?'
289
+
);
290
+
const revive = db.prepare(
291
+
`UPDATE transactions SET removed_at = NULL, import_id = ?, posted = ?, transacted_at = ?,
292
+
amount_cents = ?, description = ?, payee = ?, memo = ?, pending = 0
293
+
WHERE id = ?`
294
+
);
295
+
const insert = db.prepare(
296
+
`INSERT INTO transactions
297
+
(account_id, sfin_id, posted, transacted_at, amount_cents, description, payee, memo,
298
+
pending, import_id, created_at)
299
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0, ?, ?)`
300
+
);
301
+
302
+
for (const { candidate, status } of classified) {
303
+
if (status === 'already-present') {
304
+
alreadyPresent++;
305
+
continue;
306
+
}
307
+
// Flagged rows default to skip: where both sources claim a transaction the
308
+
// synced row is strictly better, so declining the CSV's copy loses nothing.
309
+
if (status === 'flagged' && record.decisions[candidate.syntheticId] !== 'keep') {
310
+
skipped++;
311
+
continue;
312
+
}
313
+
314
+
const prior = findAny.get(record.accountId, candidate.syntheticId) as
315
+
| { id: number }
316
+
| undefined;
317
+
if (prior) {
318
+
// Only reachable for a row removed by an earlier undo — a live row would
319
+
// have classified as already-present.
320
+
revive.run(
321
+
importId,
322
+
candidate.posted,
323
+
candidate.transactedAt,
324
+
candidate.amountCents,
325
+
candidate.description,
326
+
candidate.payee,
327
+
candidate.memo,
328
+
prior.id
329
+
);
330
+
revived++;
331
+
} else {
332
+
insert.run(
333
+
record.accountId,
334
+
candidate.syntheticId,
335
+
candidate.posted,
336
+
candidate.transactedAt,
337
+
candidate.amountCents,
338
+
candidate.description,
339
+
candidate.payee,
340
+
candidate.memo,
341
+
importId,
342
+
now
343
+
);
344
+
inserted++;
345
+
}
346
+
}
347
+
348
+
db.prepare("UPDATE imports SET status = 'committed', committed_at = ? WHERE id = ?").run(
349
+
now,
350
+
importId
351
+
);
352
+
db.exec('COMMIT');
353
+
} catch (err) {
354
+
db.exec('ROLLBACK');
355
+
throw err;
356
+
}
357
+
358
+
// Outside the transaction: appendCategorizationEvent manages its own. Backfilled
359
+
// history categorizes itself against rules that already exist, with the ordinary
360
+
// `rule` event source — an import introduces no new kind of provenance.
361
+
const ruleCategorized = applyRulesToUncategorized(db);
362
+
return { inserted, revived, skipped, alreadyPresent, ruleCategorized };
363
+
}
364
+
365
+
/**
366
+
* Reverse an import as a unit. Soft-removes: categorization events reference
367
+
* transaction ids and the event log is append-only, so history outlives the rows.
368
+
*/
369
+
export function undoImport(db: DatabaseSync, importId: number): number {
370
+
const record = getImport(db, importId);
371
+
if (!record) throw new Error(`Unknown import: ${importId}`);
372
+
if (record.status !== 'committed') {
373
+
throw new Error(`Only a committed import can be undone; ${importId} is ${record.status}.`);
374
+
}
375
+
376
+
const now = new Date().toISOString();
377
+
db.exec('BEGIN');
378
+
try {
379
+
const result = db
380
+
.prepare('UPDATE transactions SET removed_at = ? WHERE import_id = ? AND removed_at IS NULL')
381
+
.run(now, importId);
382
+
db.prepare("UPDATE imports SET status = 'undone', undone_at = ? WHERE id = ?").run(
383
+
now,
384
+
importId
385
+
);
386
+
db.exec('COMMIT');
387
+
return Number(result.changes);
388
+
} catch (err) {
389
+
db.exec('ROLLBACK');
390
+
throw err;
391
+
}
392
+
}
+86
src/lib/server/services/ledger.test.ts
+86
src/lib/server/services/ledger.test.ts
···
68
68
db.close();
69
69
});
70
70
71
+
/** Add a committed import plus one backfilled row. Kept out of testDb() so the
72
+
* existing count assertions stay meaningful. */
73
+
function withImport(db: DatabaseSync): number {
74
+
const now = new Date().toISOString();
75
+
const importId = Number(
76
+
db
77
+
.prepare(
78
+
`INSERT INTO imports (account_id, filename, uploaded_at, payload, status, committed_at)
79
+
VALUES ('chk', 'chase-2026.csv', ?, 'Date,Description,Amount', 'committed', ?)`
80
+
)
81
+
.run(now, now).lastInsertRowid
82
+
);
83
+
db.prepare(
84
+
`INSERT INTO transactions
85
+
(account_id, sfin_id, posted, amount_cents, description, pending, import_id, created_at)
86
+
VALUES ('chk', 'csv:abc123', ?, -1500, 'BACKFILLED LUNCH', 0, ?, ?)`
87
+
).run(JUNE_10, importId, now);
88
+
return importId;
89
+
}
90
+
91
+
Deno.test('source filter separates imported from synced', () => {
92
+
const db = testDb();
93
+
withImport(db);
94
+
95
+
const imported = listLedger(db, { source: 'imported' });
96
+
if (imported.length !== 1 || imported[0].description !== 'BACKFILLED LUNCH') {
97
+
throw new Error(`imported filter wrong: ${imported.map((r) => r.description).join(',')}`);
98
+
}
99
+
100
+
const synced = listLedger(db, { source: 'synced' });
101
+
if (synced.some((r) => r.description === 'BACKFILLED LUNCH')) {
102
+
throw new Error('imported row leaked into the synced filter');
103
+
}
104
+
if (synced.length !== 3) throw new Error(`expected the 3 synced rows, got ${synced.length}`);
105
+
106
+
// Unfiltered shows both — origin is an on-demand question, not a default lens.
107
+
if (listLedger(db).length !== 4) throw new Error('unfiltered should show every row');
108
+
db.close();
109
+
});
110
+
111
+
Deno.test('source filter composes with other filters', () => {
112
+
const db = testDb();
113
+
withImport(db);
114
+
115
+
if (listLedger(db, { source: 'imported', month: '2026-06' }).length !== 1) {
116
+
throw new Error('source + month wrong');
117
+
}
118
+
if (listLedger(db, { source: 'imported', month: '2026-07' }).length !== 0) {
119
+
throw new Error('source + month should exclude other months');
120
+
}
121
+
if (listLedger(db, { source: 'imported', accountId: 'chk' }).length !== 1) {
122
+
throw new Error('source + account wrong');
123
+
}
124
+
if (listLedger(db, { source: 'synced', month: '2026-06' }).length !== 1) {
125
+
throw new Error('source + month should still find the synced June row');
126
+
}
127
+
if (listLedger(db, { source: 'imported', q: 'lunch' }).length !== 1) {
128
+
throw new Error('source + search wrong');
129
+
}
130
+
if (listLedger(db, { source: 'synced', q: 'lunch' }).length !== 0) {
131
+
throw new Error('source + search should exclude the imported match');
132
+
}
133
+
if (listLedger(db, { source: 'imported', category: 'uncategorized' }).length !== 1) {
134
+
throw new Error('source + category wrong');
135
+
}
136
+
db.close();
137
+
});
138
+
139
+
Deno.test('ledger row carries its origin', () => {
140
+
const db = testDb();
141
+
const importId = withImport(db);
142
+
143
+
const [row] = listLedger(db, { source: 'imported' });
144
+
if (row.source !== 'imported') throw new Error('source not set');
145
+
if (row.importId !== importId) throw new Error('importId not joined');
146
+
if (row.importFilename !== 'chase-2026.csv') throw new Error('filename not joined');
147
+
if (!row.importedAt) throw new Error('importedAt not joined');
148
+
149
+
const synced = listLedger(db, { source: 'synced' })[0];
150
+
if (synced.source !== 'synced') throw new Error('synced row mislabeled');
151
+
if (synced.importId !== null || synced.importFilename !== null) {
152
+
throw new Error('synced row should carry no import origin');
153
+
}
154
+
db.close();
155
+
});
156
+
71
157
Deno.test('ledger filters: month, category, uncategorized, pending, account', () => {
72
158
const db = testDb();
73
159
const june = listLedger(db, { month: '2026-06' });
+24
-1
src/lib/server/services/ledger.ts
+24
-1
src/lib/server/services/ledger.ts
···
10
10
/** true = only pending, false = only posted, undefined = both */
11
11
pending?: boolean;
12
12
/**
13
+
* Where the row came from: synced from a connection, or backfilled from a CSV.
14
+
* A separate axis from `provenance`, which is about who chose the category.
15
+
*/
16
+
source?: 'synced' | 'imported';
17
+
/**
13
18
* Case-insensitive substring search over description, payee, memo, and the
14
19
* effective rule-applied display name — search matches what the ledger shows.
15
20
*/
···
40
45
provenanceRulePattern: string | null;
41
46
provenanceActorHandle: string | null;
42
47
provenanceActorDid: string | null;
48
+
/**
49
+
* Origin of the row itself. Shown in the history panel rather than on the row:
50
+
* within a backfilled date range every row is imported, so a per-row mark would
51
+
* carry no information exactly where it is densest.
52
+
*/
53
+
source: 'synced' | 'imported';
54
+
importId: number | null;
55
+
importFilename: string | null;
56
+
importedAt: string | null;
43
57
}
44
58
45
59
export function monthRange(month: string): { start: number; end: number } {
···
77
91
where.push('t.pending = ?');
78
92
params.push(filters.pending ? 1 : 0);
79
93
}
94
+
if (filters.source) {
95
+
where.push(filters.source === 'imported' ? 't.import_id IS NOT NULL' : 't.import_id IS NULL');
96
+
}
80
97
if (filters.q?.trim()) {
81
98
// r is the categorizing rule of the latest event (joined below), so the
82
99
// overlay display name is searchable — search finds what the user sees.
···
96
113
COALESCE(r.display_name, t.payee, t.description) AS display_label,
97
114
t.category_id, c.name AS category_name,
98
115
e.source AS prov_source, r.pattern AS prov_pattern, u.handle AS prov_handle,
99
-
e.actor_did AS prov_actor_did
116
+
e.actor_did AS prov_actor_did,
117
+
t.import_id, i.filename AS import_filename, i.committed_at AS imported_at
100
118
FROM transactions t
101
119
JOIN accounts a ON a.id = t.account_id
102
120
LEFT JOIN categories c ON c.id = t.category_id
···
105
123
)
106
124
LEFT JOIN rules r ON r.id = e.rule_id
107
125
LEFT JOIN users u ON u.did = e.actor_did
126
+
LEFT JOIN imports i ON i.id = t.import_id
108
127
WHERE ${where.join(' AND ')}
109
128
ORDER BY t.pending DESC, effective_at DESC, t.id DESC
110
129
LIMIT ?`
···
125
144
pending: r.pending === 1,
126
145
categoryId: r.category_id as number | null,
127
146
categoryName: r.category_name as string | null,
147
+
source: r.import_id == null ? ('synced' as const) : ('imported' as const),
148
+
importId: r.import_id as number | null,
149
+
importFilename: r.import_filename as string | null,
150
+
importedAt: r.imported_at as string | null,
128
151
provenance: (r.prov_source as EventSource | null) ?? null,
129
152
provenanceRulePattern: r.prov_pattern as string | null,
130
153
provenanceActorHandle: r.prov_handle as string | null,
+42
src/lib/server/services/sync.test.ts
+42
src/lib/server/services/sync.test.ts
···
2
2
import { openDatabase } from '../db.ts';
3
3
import { runSync } from './sync.ts';
4
4
import { categorizeManually } from './categorization.ts';
5
+
import { commitImport, createDraftImport, saveMapping } from './imports.ts';
5
6
import type { DatabaseSync } from 'node:sqlite';
6
7
7
8
const MIGRATIONS_DIR = new URL('../../../../migrations', import.meta.url).pathname.replace(
···
48
49
function count(db: DatabaseSync, sql: string): number {
49
50
return (db.prepare(sql).get() as { n: number }).n;
50
51
}
52
+
53
+
Deno.test('a sync after a backfill leaves the imported rows alone', async () => {
54
+
// The hazard this guards: ingestTransactions treats its feed as authoritative
55
+
// for the account and soft-removes anything pending that the feed omits. A
56
+
// backfill CSV appears in no feed, ever. Imported rows are posted (pending = 0),
57
+
// which is what keeps them outside every branch of sync's authority — a
58
+
// structural property worth pinning down rather than rediscovering.
59
+
const db = testDb();
60
+
const pendingTxn = { id: 'p1', posted: null, amount: '-5.00', description: 'PENDING COFFEE' };
61
+
await runSync(db, fakeFetch(payloadWith([pendingTxn])));
62
+
63
+
// Backfill straight into the discovered account, well before the feed's reach.
64
+
const importId = createDraftImport(db, 'act-1', 'old.csv', 'Date,Description,Amount\n2023-02-01,ANCIENT LUNCH,-9.99\n');
65
+
saveMapping(db, importId, {
66
+
date: 'Date',
67
+
dateFormat: 'iso',
68
+
amountMode: 'signed',
69
+
amount: 'Amount',
70
+
description: 'Description'
71
+
});
72
+
commitImport(db, importId);
73
+
74
+
// A later sync of the same connection: the feed still knows nothing of 2023.
75
+
await runSync(db, fakeFetch(payloadWith([pendingTxn, { id: 't9', posted: NOW_S, amount: '-1.00', description: 'NEW THING' }])));
76
+
77
+
const imported = db
78
+
.prepare("SELECT removed_at, pending FROM transactions WHERE description = 'ANCIENT LUNCH'")
79
+
.get() as { removed_at: string | null; pending: number };
80
+
if (imported.removed_at !== null) throw new Error('a sync must not remove backfilled rows');
81
+
if (imported.pending !== 0) throw new Error('imported rows must stay posted');
82
+
83
+
if (count(db, "SELECT COUNT(*) n FROM transactions WHERE description = 'ANCIENT LUNCH'") !== 1) {
84
+
throw new Error('a sync must not duplicate a backfilled row');
85
+
}
86
+
// And the account is still healthy: the backfill did not make it look absent.
87
+
const account = db.prepare("SELECT state FROM accounts WHERE id = 'act-1'").get() as {
88
+
state: string;
89
+
};
90
+
if (account.state === 'INACTIVE') throw new Error('the account should not have gone inactive');
91
+
db.close();
92
+
});
51
93
52
94
Deno.test('sync archives raw payload, discovers account as NEW, snapshots balance', async () => {
53
95
const db = testDb();
+20
-1
src/routes/(app)/ledger/+page.server.ts
+20
-1
src/routes/(app)/ledger/+page.server.ts
···
22
22
else if (pending === '0') filters.pending = false;
23
23
const q = url.searchParams.get('q')?.trim();
24
24
if (q) filters.q = q;
25
+
const source = url.searchParams.get('source');
26
+
if (source === 'synced' || source === 'imported') filters.source = source;
25
27
26
28
const historyId = url.searchParams.get('history');
27
29
let history: {
28
30
id: number;
29
31
events: ReturnType<typeof listEvents>;
32
+
/** Where the row itself came from — a different axis from who categorized it. */
33
+
origin: { source: 'synced' | 'imported'; filename: string | null; importedAt: string | null };
30
34
bankRecord: {
31
35
description: string;
32
36
payee: string | null;
···
38
42
if (historyId) {
39
43
const id = Number(historyId);
40
44
const txn = db
41
-
.prepare('SELECT description, payee, memo, sfin_id, extra FROM transactions WHERE id = ?')
45
+
.prepare(
46
+
`SELECT t.description, t.payee, t.memo, t.sfin_id, t.extra,
47
+
t.import_id, i.filename AS import_filename, i.committed_at AS imported_at
48
+
FROM transactions t
49
+
LEFT JOIN imports i ON i.id = t.import_id
50
+
WHERE t.id = ?`
51
+
)
42
52
.get(id) as
43
53
| {
44
54
description: string;
···
46
56
memo: string | null;
47
57
sfin_id: string;
48
58
extra: string | null;
59
+
import_id: number | null;
60
+
import_filename: string | null;
61
+
imported_at: string | null;
49
62
}
50
63
| undefined;
51
64
history = {
52
65
id,
53
66
events: listEvents(db, id),
67
+
origin: {
68
+
source: txn?.import_id == null ? 'synced' : 'imported',
69
+
filename: txn?.import_filename ?? null,
70
+
importedAt: txn?.imported_at ?? null
71
+
},
54
72
bankRecord: txn
55
73
? {
56
74
description: txn.description,
···
76
94
category: category ?? '',
77
95
month: month ?? '',
78
96
pending: pending ?? '',
97
+
source: source ?? '',
79
98
q: q ?? ''
80
99
}
81
100
};
+30
src/routes/(app)/ledger/+page.svelte
+30
src/routes/(app)/ledger/+page.svelte
···
56
56
}
57
57
58
58
const timeFmt = new Intl.DateTimeFormat(undefined, { dateStyle: 'medium', timeStyle: 'short' });
59
+
const dayFmt = new Intl.DateTimeFormat(undefined, { dateStyle: 'medium' });
59
60
</script>
60
61
61
62
<h1>Ledger</h1>
···
110
111
<option value="" selected={data.filters.pending === ''}>Posted + pending</option>
111
112
<option value="0" selected={data.filters.pending === '0'}>Posted only</option>
112
113
<option value="1" selected={data.filters.pending === '1'}>Pending only</option>
114
+
</select>
115
+
116
+
<select
117
+
name="source"
118
+
onchange={(e) => (location.href = filterUrl({ source: e.currentTarget.value }))}
119
+
>
120
+
<option value="" selected={data.filters.source === ''}>Any source</option>
121
+
<option value="synced" selected={data.filters.source === 'synced'}>Synced only</option>
122
+
<option value="imported" selected={data.filters.source === 'imported'}>Imported only</option>
113
123
</select>
114
124
115
125
{#if data.filters.q}
···
223
233
<strong>Details</strong>
224
234
<a href={filterUrl({ history: '' })} data-sveltekit-noscroll>close</a>
225
235
</div>
236
+
<p class="origin">
237
+
{#if data.history.origin.source === 'imported'}
238
+
Imported from <code>{data.history.origin.filename ?? 'a CSV file'}</code>
239
+
{#if data.history.origin.importedAt}
240
+
· {dayFmt.format(new Date(data.history.origin.importedAt))}
241
+
{/if}
242
+
{:else}
243
+
Synced from SimpleFIN
244
+
{/if}
245
+
</p>
226
246
{#if data.history.bankRecord}
227
247
<dl class="bank-record">
228
248
{#if data.history.bankRecord.payee}
···
466
486
467
487
.no-events {
468
488
color: var(--text-faint);
489
+
}
490
+
491
+
.origin {
492
+
margin: 0;
493
+
color: var(--text-muted);
494
+
}
495
+
496
+
.origin code {
497
+
font-family: var(--font-mono);
498
+
overflow-wrap: anywhere;
469
499
}
470
500
471
501
@media (max-width: 879px) {
+9
src/routes/(app)/settings/+page.svelte
+9
src/routes/(app)/settings/+page.svelte
···
72
72
</section>
73
73
74
74
<section>
75
+
<h2>CSV import</h2>
76
+
<p class="hint">
77
+
Some banks hand SimpleFIN only a few weeks of history. Fill the gap for an account from that
78
+
bank's own CSV export.
79
+
</p>
80
+
<p><a href="/settings/import">Import a CSV →</a></p>
81
+
</section>
82
+
83
+
<section>
75
84
<h2>Categories</h2>
76
85
{#if form?.categoryMessage}<p class="feedback" role="status">{form.categoryMessage}</p>{/if}
77
86
+78
src/routes/(app)/settings/import/+page.server.ts
+78
src/routes/(app)/settings/import/+page.server.ts
···
1
+
import { fail, redirect } from '@sveltejs/kit';
2
+
import { getDb } from '$lib/server/db';
3
+
import { listAccounts } from '$lib/server/services/accounts';
4
+
import { createDraftImport, listImports, undoImport } from '$lib/server/services/imports';
5
+
import { detectMapping, parseCsv } from '$lib/server/services/csv-import';
6
+
import { saveMapping } from '$lib/server/services/imports';
7
+
import type { Actions, PageServerLoad } from './$types';
8
+
9
+
/** Guards against a stray upload of something enormous or binary. */
10
+
const MAX_BYTES = 10 * 1024 * 1024;
11
+
12
+
export const load: PageServerLoad = () => {
13
+
const db = getDb();
14
+
return {
15
+
// Accounts come only from sync; a CSV attaches to one, it never creates one.
16
+
accounts: listAccounts(db)
17
+
.filter((a) => a.state !== 'HIDDEN')
18
+
.map(({ id, name, displayName }) => ({ id, label: displayName ?? name })),
19
+
imports: listImports(db).map((record) => ({
20
+
id: record.id,
21
+
accountId: record.accountId,
22
+
filename: record.filename,
23
+
status: record.status,
24
+
uploadedAt: record.uploadedAt,
25
+
committedAt: record.committedAt,
26
+
undoneAt: record.undoneAt,
27
+
rowCount: record.rowCount
28
+
}))
29
+
};
30
+
};
31
+
32
+
export const actions: Actions = {
33
+
upload: async ({ request }) => {
34
+
const form = await request.formData();
35
+
const accountId = String(form.get('accountId') ?? '');
36
+
const file = form.get('file');
37
+
38
+
if (!accountId) return fail(400, { message: 'Choose an account for this file first.' });
39
+
if (!(file instanceof File) || file.size === 0) {
40
+
return fail(400, { message: 'Choose a CSV file to import.' });
41
+
}
42
+
if (file.size > MAX_BYTES) {
43
+
return fail(400, { message: 'That file is larger than 10 MB.' });
44
+
}
45
+
46
+
const payload = await file.text();
47
+
48
+
const db = getDb();
49
+
let importId: number;
50
+
try {
51
+
// Archive first, parse second: a file that defeats the parser is still kept.
52
+
importId = createDraftImport(db, accountId, file.name || null, payload);
53
+
} catch (err) {
54
+
return fail(400, { message: err instanceof Error ? err.message : 'Upload failed.' });
55
+
}
56
+
57
+
// A best guess now saves a step; the mapping screen confirms it either way.
58
+
try {
59
+
const guess = detectMapping(parseCsv(payload));
60
+
if (guess) saveMapping(db, importId, guess);
61
+
} catch {
62
+
// An unparseable file still reaches the mapping screen, which states why.
63
+
}
64
+
65
+
redirect(303, `/settings/import/${importId}`);
66
+
},
67
+
68
+
undo: async ({ request }) => {
69
+
const form = await request.formData();
70
+
const importId = Number(form.get('importId'));
71
+
try {
72
+
const removed = undoImport(getDb(), importId);
73
+
return { message: `Undone. ${removed} transaction${removed === 1 ? '' : 's'} removed.` };
74
+
} catch (err) {
75
+
return fail(400, { message: err instanceof Error ? err.message : 'Undo failed.' });
76
+
}
77
+
}
78
+
};
+172
src/routes/(app)/settings/import/+page.svelte
+172
src/routes/(app)/settings/import/+page.svelte
···
1
+
<script lang="ts">
2
+
import { enhance } from '$app/forms';
3
+
4
+
let { data, form } = $props();
5
+
6
+
const dateFmt = new Intl.DateTimeFormat(undefined, { dateStyle: 'medium', timeStyle: 'short' });
7
+
8
+
function accountLabel(id: string): string {
9
+
return data.accounts.find((a) => a.id === id)?.label ?? id;
10
+
}
11
+
</script>
12
+
13
+
<h1>Import CSV</h1>
14
+
15
+
<section>
16
+
<p class="hint">
17
+
Fill a gap in an account's history from a bank's CSV export. An import only adds transactions to
18
+
an account you already have — it never changes anything a sync brought in, and it can be undone.
19
+
</p>
20
+
21
+
{#if data.accounts.length === 0}
22
+
<p class="hint">No accounts yet. Connect a bank and sync before importing a file.</p>
23
+
{:else}
24
+
<form method="POST" action="?/upload" enctype="multipart/form-data" use:enhance class="stack">
25
+
<label for="accountId">Account</label>
26
+
<select id="accountId" name="accountId">
27
+
{#each data.accounts as account (account.id)}
28
+
<option value={account.id}>{account.label}</option>
29
+
{/each}
30
+
</select>
31
+
32
+
<label for="file">CSV file</label>
33
+
<input id="file" name="file" type="file" accept=".csv,text/csv" />
34
+
35
+
<button class="primary" type="submit">Continue</button>
36
+
</form>
37
+
{/if}
38
+
39
+
{#if form?.message}<p class="feedback" role="status">{form.message}</p>{/if}
40
+
</section>
41
+
42
+
<section>
43
+
<h2>Past imports</h2>
44
+
45
+
{#if data.imports.length === 0}
46
+
<p class="hint">No imports yet.</p>
47
+
{:else}
48
+
<ul class="plain">
49
+
{#each data.imports as record (record.id)}
50
+
<li class="row">
51
+
<div class="detail">
52
+
<span class="mono">{record.filename ?? 'unnamed.csv'}</span>
53
+
<span class="note tnum">
54
+
{accountLabel(record.accountId)} ·
55
+
{#if record.status === 'committed'}
56
+
{record.rowCount} transaction{record.rowCount === 1 ? '' : 's'} ·
57
+
imported {record.committedAt ? dateFmt.format(new Date(record.committedAt)) : ''}
58
+
{:else if record.status === 'undone'}
59
+
undone {record.undoneAt ? dateFmt.format(new Date(record.undoneAt)) : ''}
60
+
{:else}
61
+
not finished · started {dateFmt.format(new Date(record.uploadedAt))}
62
+
{/if}
63
+
</span>
64
+
</div>
65
+
66
+
<div class="actions">
67
+
{#if record.status === 'draft'}
68
+
<a class="ghost" href="/settings/import/{record.id}">Resume</a>
69
+
{:else if record.status === 'committed'}
70
+
<form
71
+
method="POST"
72
+
action="?/undo"
73
+
use:enhance
74
+
class="inline"
75
+
onsubmit={(e) => {
76
+
const ok = confirm(
77
+
`Remove ${record.rowCount} transaction${record.rowCount === 1 ? '' : 's'} imported from ${record.filename ?? 'this file'}? Synced transactions are not affected.`
78
+
);
79
+
if (!ok) e.preventDefault();
80
+
}}
81
+
>
82
+
<input type="hidden" name="importId" value={record.id} />
83
+
<button class="ghost" type="submit">Undo</button>
84
+
</form>
85
+
{/if}
86
+
</div>
87
+
</li>
88
+
{/each}
89
+
</ul>
90
+
{/if}
91
+
</section>
92
+
93
+
<style>
94
+
section {
95
+
margin-top: var(--space-6);
96
+
max-width: 620px;
97
+
display: flex;
98
+
flex-direction: column;
99
+
gap: var(--space-4);
100
+
}
101
+
102
+
.hint {
103
+
color: var(--text-muted);
104
+
}
105
+
106
+
.stack {
107
+
display: flex;
108
+
flex-direction: column;
109
+
gap: var(--space-2);
110
+
align-items: flex-start;
111
+
}
112
+
113
+
.stack label {
114
+
font-size: var(--text-xs);
115
+
font-weight: 500;
116
+
color: var(--text-muted);
117
+
}
118
+
119
+
.mono {
120
+
font-family: var(--font-mono);
121
+
font-size: var(--text-xs);
122
+
}
123
+
124
+
.note,
125
+
.feedback {
126
+
color: var(--text-muted);
127
+
font-size: var(--text-xs);
128
+
}
129
+
130
+
.plain {
131
+
list-style: none;
132
+
margin: 0;
133
+
padding: 0;
134
+
}
135
+
136
+
.row {
137
+
display: flex;
138
+
justify-content: space-between;
139
+
align-items: center;
140
+
gap: var(--space-2);
141
+
padding: var(--space-2) 0;
142
+
border-bottom: 1px solid var(--border);
143
+
}
144
+
145
+
.detail {
146
+
display: flex;
147
+
flex-direction: column;
148
+
gap: 2px;
149
+
min-width: 0;
150
+
}
151
+
152
+
.actions {
153
+
display: flex;
154
+
gap: var(--space-1);
155
+
}
156
+
157
+
.inline {
158
+
display: inline;
159
+
}
160
+
161
+
.ghost {
162
+
background: none;
163
+
border: none;
164
+
color: var(--text-muted);
165
+
font-size: var(--text-xs);
166
+
padding: var(--space-1) var(--space-2);
167
+
}
168
+
169
+
.ghost:hover {
170
+
background: var(--bg-sunken);
171
+
}
172
+
</style>
+158
src/routes/(app)/settings/import/[id]/+page.server.ts
+158
src/routes/(app)/settings/import/[id]/+page.server.ts
···
1
+
import { error, fail } from '@sveltejs/kit';
2
+
import { getDb } from '$lib/server/db';
3
+
import { listAccounts } from '$lib/server/services/accounts';
4
+
import {
5
+
commitImport,
6
+
getImport,
7
+
previewImport,
8
+
saveDecisions,
9
+
saveMapping
10
+
} from '$lib/server/services/imports';
11
+
import { parseCsv, type CsvDateFormat, type CsvMapping } from '$lib/server/services/csv-import';
12
+
import type { Actions, PageServerLoad } from './$types';
13
+
14
+
/** Enough to see a wrong column at a glance without rendering the whole file. */
15
+
const SAMPLE_ROWS = 5;
16
+
17
+
function readMapping(form: FormData): CsvMapping {
18
+
const value = (name: string) => {
19
+
const raw = String(form.get(name) ?? '').trim();
20
+
return raw === '' ? null : raw;
21
+
};
22
+
const date = value('date');
23
+
const description = value('description');
24
+
if (!date) throw new Error('Choose the column holding the date.');
25
+
if (!description) throw new Error('Choose the column holding the description.');
26
+
27
+
const amountMode = String(form.get('amountMode') ?? 'signed') as CsvMapping['amountMode'];
28
+
if (amountMode === 'signed' && !value('amount')) {
29
+
throw new Error('Choose the column holding the amount.');
30
+
}
31
+
if (amountMode === 'debit-credit' && !value('debit') && !value('credit')) {
32
+
throw new Error('Choose a debit column, a credit column, or both.');
33
+
}
34
+
35
+
return {
36
+
date,
37
+
transactedDate: value('transactedDate'),
38
+
dateFormat: (String(form.get('dateFormat') ?? 'mdy') as CsvDateFormat) ?? 'mdy',
39
+
amountMode,
40
+
amount: value('amount'),
41
+
debit: value('debit'),
42
+
credit: value('credit'),
43
+
description,
44
+
payee: value('payee'),
45
+
memo: value('memo')
46
+
};
47
+
}
48
+
49
+
export const load: PageServerLoad = ({ params }) => {
50
+
const db = getDb();
51
+
const record = getImport(db, Number(params.id));
52
+
if (!record) error(404, 'No such import.');
53
+
54
+
const account = listAccounts(db).find((a) => a.id === record.accountId);
55
+
const accountLabel = account ? (account.displayName ?? account.name) : record.accountId;
56
+
57
+
const base = {
58
+
id: record.id,
59
+
filename: record.filename,
60
+
status: record.status,
61
+
accountId: record.accountId,
62
+
accountLabel,
63
+
committedAt: record.committedAt
64
+
};
65
+
66
+
// A committed or undone import is a record, not a wizard.
67
+
if (record.status !== 'draft') {
68
+
const counts = db
69
+
.prepare(
70
+
'SELECT COUNT(*) AS n FROM transactions WHERE import_id = ? AND removed_at IS NULL'
71
+
)
72
+
.get(record.id) as { n: number };
73
+
return { ...base, headers: [], mapping: null, sample: [], preview: null, parseError: null, rowCount: counts.n };
74
+
}
75
+
76
+
let headers: string[] = [];
77
+
let parseError: string | null = null;
78
+
try {
79
+
headers = parseCsv(record.payload).headers;
80
+
} catch (err) {
81
+
parseError = err instanceof Error ? err.message : 'This file could not be read as CSV.';
82
+
}
83
+
84
+
let preview = null;
85
+
if (record.mapping && !parseError) {
86
+
try {
87
+
const full = previewImport(db, record.id);
88
+
preview = {
89
+
errors: full.errors,
90
+
rangeStart: full.rangeStart,
91
+
rangeEnd: full.rangeEnd,
92
+
newCount: full.newCount,
93
+
alreadyPresentCount: full.alreadyPresentCount,
94
+
flaggedCount: full.flaggedCount,
95
+
sample: full.classified.slice(0, SAMPLE_ROWS).map((c) => ({
96
+
posted: c.candidate.posted,
97
+
amountCents: c.candidate.amountCents,
98
+
description: c.candidate.description
99
+
})),
100
+
flagged: full.classified
101
+
.filter((c) => c.status === 'flagged')
102
+
.map((c) => ({
103
+
syntheticId: c.candidate.syntheticId,
104
+
posted: c.candidate.posted,
105
+
amountCents: c.candidate.amountCents,
106
+
description: c.candidate.description,
107
+
match: c.match
108
+
}))
109
+
};
110
+
} catch (err) {
111
+
parseError = err instanceof Error ? err.message : 'This file could not be read as CSV.';
112
+
}
113
+
}
114
+
115
+
return {
116
+
...base,
117
+
headers,
118
+
mapping: record.mapping,
119
+
preview,
120
+
parseError,
121
+
rowCount: 0
122
+
};
123
+
};
124
+
125
+
export const actions: Actions = {
126
+
mapping: async ({ request, params }) => {
127
+
const form = await request.formData();
128
+
try {
129
+
saveMapping(getDb(), Number(params.id), readMapping(form));
130
+
} catch (err) {
131
+
return fail(400, { message: err instanceof Error ? err.message : 'Mapping failed.' });
132
+
}
133
+
return { message: 'Mapping updated.' };
134
+
},
135
+
136
+
commit: async ({ request, params }) => {
137
+
const form = await request.formData();
138
+
const db = getDb();
139
+
const importId = Number(params.id);
140
+
141
+
// Only checked rows are kept. An unchecked (or absent) flagged row is skipped:
142
+
// where both sources claim a transaction the synced row is the better record.
143
+
const decisions: Record<string, 'keep' | 'skip'> = {};
144
+
for (const id of form.getAll('keep')) decisions[String(id)] = 'keep';
145
+
146
+
try {
147
+
saveDecisions(db, importId, decisions);
148
+
const result = commitImport(db, importId);
149
+
const parts = [`Imported ${result.inserted + result.revived}`];
150
+
if (result.skipped) parts.push(`skipped ${result.skipped} possible duplicate${result.skipped === 1 ? '' : 's'}`);
151
+
if (result.alreadyPresent) parts.push(`${result.alreadyPresent} already present`);
152
+
if (result.ruleCategorized) parts.push(`${result.ruleCategorized} categorized by rules`);
153
+
return { message: `${parts.join(', ')}.` };
154
+
} catch (err) {
155
+
return fail(400, { message: err instanceof Error ? err.message : 'Import failed.' });
156
+
}
157
+
}
158
+
};
+346
src/routes/(app)/settings/import/[id]/+page.svelte
+346
src/routes/(app)/settings/import/[id]/+page.svelte
···
1
+
<script lang="ts">
2
+
import { untrack } from 'svelte';
3
+
import { enhance } from '$app/forms';
4
+
import { formatCents, formatDay } from '$lib/format';
5
+
6
+
let { data, form } = $props();
7
+
8
+
// Seeded from the stored mapping, then owned by the user: if a save is rejected,
9
+
// their choice must survive rather than snap back to what's on disk.
10
+
let amountMode = $state(untrack(() => data.mapping?.amountMode) ?? 'signed');
11
+
12
+
const dateFmt = new Intl.DateTimeFormat(undefined, { dateStyle: 'medium' });
13
+
14
+
function selected(value: string | null | undefined, header: string): boolean {
15
+
return value === header;
16
+
}
17
+
</script>
18
+
19
+
<h1>Import CSV</h1>
20
+
21
+
<p class="crumb"><a href="/settings/import">← All imports</a></p>
22
+
23
+
{#if data.status !== 'draft'}
24
+
<section>
25
+
<h2>{data.filename ?? 'unnamed.csv'}</h2>
26
+
{#if data.status === 'committed'}
27
+
<p class="hint tnum">
28
+
Imported into {data.accountLabel}. {data.rowCount} transaction{data.rowCount === 1
29
+
? ''
30
+
: 's'} are in the ledger from this file.
31
+
</p>
32
+
{#if form?.message}<p class="feedback" role="status">{form.message}</p>{/if}
33
+
<p>
34
+
<a href="/ledger?account={data.accountId}&source=imported">
35
+
See these transactions in the ledger
36
+
</a>
37
+
</p>
38
+
{:else}
39
+
<p class="hint">
40
+
This import was undone. Its transactions are no longer in the ledger.
41
+
</p>
42
+
{/if}
43
+
</section>
44
+
{:else if data.parseError}
45
+
<section>
46
+
<h2>{data.filename ?? 'unnamed.csv'}</h2>
47
+
<p class="hint">{data.parseError}</p>
48
+
<p class="hint">
49
+
The file is saved as uploaded. Export it again as CSV and start a new import.
50
+
</p>
51
+
</section>
52
+
{:else}
53
+
<section>
54
+
<h2>Columns</h2>
55
+
<p class="hint tnum">
56
+
<span class="mono">{data.filename ?? 'unnamed.csv'}</span> → {data.accountLabel}
57
+
</p>
58
+
59
+
<form method="POST" action="?/mapping" use:enhance class="grid">
60
+
<label for="date">Date</label>
61
+
<select id="date" name="date">
62
+
{#each data.headers as header (header)}
63
+
<option value={header} selected={selected(data.mapping?.date, header)}>{header}</option>
64
+
{/each}
65
+
</select>
66
+
67
+
<label for="dateFormat">Date order</label>
68
+
<select id="dateFormat" name="dateFormat">
69
+
<option value="iso" selected={data.mapping?.dateFormat === 'iso'}>Year-month-day</option>
70
+
<option value="mdy" selected={data.mapping?.dateFormat === 'mdy'}>Month/day/year</option>
71
+
<option value="dmy" selected={data.mapping?.dateFormat === 'dmy'}>Day/month/year</option>
72
+
</select>
73
+
74
+
<label for="amountMode">Amount</label>
75
+
<select id="amountMode" name="amountMode" bind:value={amountMode}>
76
+
<option value="signed">One signed column</option>
77
+
<option value="debit-credit">Separate debit and credit</option>
78
+
</select>
79
+
80
+
{#if amountMode === 'signed'}
81
+
<label for="amount">Amount column</label>
82
+
<select id="amount" name="amount">
83
+
{#each data.headers as header (header)}
84
+
<option value={header} selected={selected(data.mapping?.amount, header)}>{header}</option>
85
+
{/each}
86
+
</select>
87
+
{:else}
88
+
<label for="debit">Debit column</label>
89
+
<select id="debit" name="debit">
90
+
<option value="">—</option>
91
+
{#each data.headers as header (header)}
92
+
<option value={header} selected={selected(data.mapping?.debit, header)}>{header}</option>
93
+
{/each}
94
+
</select>
95
+
96
+
<label for="credit">Credit column</label>
97
+
<select id="credit" name="credit">
98
+
<option value="">—</option>
99
+
{#each data.headers as header (header)}
100
+
<option value={header} selected={selected(data.mapping?.credit, header)}>{header}</option>
101
+
{/each}
102
+
</select>
103
+
{/if}
104
+
105
+
<label for="description">Description</label>
106
+
<select id="description" name="description">
107
+
{#each data.headers as header (header)}
108
+
<option value={header} selected={selected(data.mapping?.description, header)}>
109
+
{header}
110
+
</option>
111
+
{/each}
112
+
</select>
113
+
114
+
<label for="transactedDate">Transaction date</label>
115
+
<select id="transactedDate" name="transactedDate">
116
+
<option value="">— none</option>
117
+
{#each data.headers as header (header)}
118
+
<option value={header} selected={selected(data.mapping?.transactedDate, header)}>
119
+
{header}
120
+
</option>
121
+
{/each}
122
+
</select>
123
+
124
+
<label for="payee">Payee</label>
125
+
<select id="payee" name="payee">
126
+
<option value="">— none</option>
127
+
{#each data.headers as header (header)}
128
+
<option value={header} selected={selected(data.mapping?.payee, header)}>{header}</option>
129
+
{/each}
130
+
</select>
131
+
132
+
<label for="memo">Memo</label>
133
+
<select id="memo" name="memo">
134
+
<option value="">— none</option>
135
+
{#each data.headers as header (header)}
136
+
<option value={header} selected={selected(data.mapping?.memo, header)}>{header}</option>
137
+
{/each}
138
+
</select>
139
+
140
+
<span></span>
141
+
<button type="submit">Apply columns</button>
142
+
</form>
143
+
144
+
{#if form?.message}<p class="feedback" role="status">{form.message}</p>{/if}
145
+
</section>
146
+
147
+
{#if !data.mapping}
148
+
<section>
149
+
<p class="hint">
150
+
Quantum could not guess this file's columns. Choose them above to continue.
151
+
</p>
152
+
</section>
153
+
{:else if data.preview}
154
+
<section>
155
+
<h2>Preview</h2>
156
+
157
+
{#if data.preview.rangeStart === null}
158
+
<p class="hint">No rows parsed. Check the columns above.</p>
159
+
{:else}
160
+
<p class="hint tnum">
161
+
{dateFmt.format(new Date(data.preview.rangeStart * 1000))} to
162
+
{dateFmt.format(new Date(data.preview.rangeEnd! * 1000))} ·
163
+
{data.preview.newCount} to import{#if data.preview.alreadyPresentCount}, {data.preview
164
+
.alreadyPresentCount} already here{/if}{#if data.preview.flaggedCount}, {data.preview
165
+
.flaggedCount} possibly already here{/if}.
166
+
</p>
167
+
168
+
<table class="tnum sample">
169
+
<tbody>
170
+
{#each data.preview.sample as row, i (i)}
171
+
<tr>
172
+
<td>{formatDay(row.posted)}</td>
173
+
<td class="desc">{row.description}</td>
174
+
<td class="right">{formatCents(row.amountCents)}</td>
175
+
</tr>
176
+
{/each}
177
+
</tbody>
178
+
</table>
179
+
<p class="note">First rows as Quantum reads them. If the dates or amounts look wrong, fix the columns above.</p>
180
+
{/if}
181
+
182
+
{#if data.preview.errors.length > 0}
183
+
<details>
184
+
<summary class="note">
185
+
{data.preview.errors.length} row{data.preview.errors.length === 1 ? '' : 's'} could not be
186
+
read and will be left out
187
+
</summary>
188
+
<ul class="note">
189
+
{#each data.preview.errors.slice(0, 20) as message (message)}
190
+
<li>{message}</li>
191
+
{/each}
192
+
{#if data.preview.errors.length > 20}
193
+
<li>…and {data.preview.errors.length - 20} more.</li>
194
+
{/if}
195
+
</ul>
196
+
</details>
197
+
{/if}
198
+
</section>
199
+
200
+
<form method="POST" action="?/commit" use:enhance>
201
+
{#if data.preview.flaggedCount > 0}
202
+
<section>
203
+
<h2>Possible duplicates</h2>
204
+
<p class="hint">
205
+
These already look like transactions in {data.accountLabel}. They are left out unless you
206
+
say otherwise — the synced record is the better copy. Tick one only if it is genuinely a
207
+
separate transaction.
208
+
</p>
209
+
210
+
<ul class="plain">
211
+
{#each data.preview.flagged as row (row.syntheticId)}
212
+
<li class="dupe">
213
+
<label>
214
+
<input type="checkbox" name="keep" value={row.syntheticId} />
215
+
<span class="dupe-body">
216
+
<span class="side tnum">
217
+
<span class="side-tag">This file</span>
218
+
<span>{formatDay(row.posted)} · {formatCents(row.amountCents)}</span>
219
+
<span class="desc">{row.description}</span>
220
+
</span>
221
+
<span class="side tnum">
222
+
<span class="side-tag">
223
+
Already here · {row.match?.source === 'imported' ? 'imported' : 'synced'}
224
+
</span>
225
+
<span>
226
+
{row.match?.effectiveAt ? formatDay(row.match.effectiveAt) : '—'} ·
227
+
{formatCents(row.match?.amountCents ?? 0)}
228
+
</span>
229
+
<span class="desc">{row.match?.description}</span>
230
+
</span>
231
+
</span>
232
+
</label>
233
+
</li>
234
+
{/each}
235
+
</ul>
236
+
</section>
237
+
{/if}
238
+
239
+
<section>
240
+
<button class="primary" type="submit" disabled={data.preview.rangeStart === null}>
241
+
Import {data.preview.newCount} transaction{data.preview.newCount === 1 ? '' : 's'}
242
+
</button>
243
+
</section>
244
+
</form>
245
+
{/if}
246
+
{/if}
247
+
248
+
<style>
249
+
section {
250
+
margin-top: var(--space-6);
251
+
max-width: 620px;
252
+
display: flex;
253
+
flex-direction: column;
254
+
gap: var(--space-4);
255
+
}
256
+
257
+
.crumb {
258
+
margin-top: var(--space-4);
259
+
font-size: var(--text-xs);
260
+
}
261
+
262
+
.hint {
263
+
color: var(--text-muted);
264
+
}
265
+
266
+
.mono {
267
+
font-family: var(--font-mono);
268
+
font-size: var(--text-xs);
269
+
}
270
+
271
+
.note,
272
+
.feedback {
273
+
color: var(--text-muted);
274
+
font-size: var(--text-xs);
275
+
}
276
+
277
+
.grid {
278
+
display: grid;
279
+
grid-template-columns: max-content 1fr;
280
+
gap: var(--space-2) var(--space-4);
281
+
align-items: center;
282
+
}
283
+
284
+
.grid label {
285
+
font-size: var(--text-xs);
286
+
font-weight: 500;
287
+
color: var(--text-muted);
288
+
}
289
+
290
+
.plain {
291
+
list-style: none;
292
+
margin: 0;
293
+
padding: 0;
294
+
}
295
+
296
+
.sample td {
297
+
padding: var(--space-1) var(--space-2) var(--space-1) 0;
298
+
border-bottom: 1px solid var(--border);
299
+
}
300
+
301
+
.right {
302
+
text-align: right;
303
+
}
304
+
305
+
.desc {
306
+
overflow-wrap: anywhere;
307
+
}
308
+
309
+
.dupe {
310
+
padding: var(--space-2) 0;
311
+
border-bottom: 1px solid var(--border);
312
+
}
313
+
314
+
.dupe label {
315
+
display: flex;
316
+
gap: var(--space-3);
317
+
align-items: flex-start;
318
+
}
319
+
320
+
.dupe-body {
321
+
display: grid;
322
+
grid-template-columns: 1fr 1fr;
323
+
gap: var(--space-4);
324
+
flex: 1;
325
+
min-width: 0;
326
+
}
327
+
328
+
.side {
329
+
display: flex;
330
+
flex-direction: column;
331
+
gap: 2px;
332
+
min-width: 0;
333
+
}
334
+
335
+
.side-tag {
336
+
font-size: var(--text-xs);
337
+
color: var(--text-faint);
338
+
}
339
+
340
+
@media (max-width: 879px) {
341
+
.dupe-body {
342
+
grid-template-columns: 1fr;
343
+
gap: var(--space-2);
344
+
}
345
+
}
346
+
</style>