Florian Schmidt
aacdc28d70
Two threads that ended up in the same files. One is navigation: a meter
swap that happened today had no click path at all, and most per-meter
tasks were reachable only by knowing which admin page owned them. The
other is attribution: readings on 1 August and 16 September showed six
weeks of water under September and nothing under August.
Meter events from the UI
Swap, counter reset, tank level, delivery and note are recorded through
MeterEventService rather than ad-hoc inserts, so the dialog's verdict and
the saved result come from the same Validate call, and every record or
delete recomputes the meter inside one transaction. MeterEventRules
decides which events a mode offers -- a tank has no register to swap, and
Correction is offered nowhere because nothing reads it.
A swap is stored as the event at T plus a manual reading of the new
register's start value at exactly T. That pairing is the whole trick: the
boundary window is (previousReading, reading], so the old register's tail
books at T and every later reading counts from the new start. Writing the
old final value as the reading at T instead -- the obvious thing -- double
counts the tail and then rejects every reading the new register produces.
Deleting a swap removes that start reading only while it is still the
untouched start value, and only Manual readings can be deleted at all.
Navigation
The meter page is now the hub: primary entry by mode, a "Record event"
menu, and Edit through a shared MeterEditor that also owns tank setup.
Other pages link into it with MeterLinks (/meters/{id}?tab=...&action=...),
whose action is consumed once after the interactive render and dropped
from the address -- the reverse order flashes the dialog and closes it,
because a circuit's first location change dismisses every open dialog.
The app bar gains a "Find a meter" dialog with the same quick entry.
A source that has no usable connector now links to creating (or enabling)
one and comes back to the same source dialog with the connector picked
and everything typed still there; the draft survives in a circuit-scoped
DraftStore, and the way back is a meter id rather than a URL, so the page
cannot be made to redirect anywhere else. The connector list shows which
meters use each connector, import batches list the meters and categories
they wrote to, the meter editor owns the meter's own cost categories, and
the dashboard's empty cost panel names the first missing step instead of
listing every admin page.
Months
A reading is an instant, and what it measures accrued over the time since
the previous one. Booking the whole delta at the closing reading misfiles
it whenever the interval crosses a month boundary, so a plain increase is
now divided at local month boundaries in proportion to elapsed time, each
share stamped inside its month and marked estimated: the meter recorded a
total, not a shape. The parts always sum to the original.
Imported monthly tables are the exception that keeps the golden fixtures
reconciling. "Mai 2026" carries the register at the end of May but is
stamped on the 1st, so the importer -- the only place that still knows
whether the date cell named a month or a day -- flags it MonthLabel, and
the engine reads it as the end of its month. Inferring that from the
stamp instead would catch day-dated rows: a sheet with "01.08.2026" in it
is not a monthly table, and reading it as one moves two thirds of July
into August.
ReadingTimeline is the single ordering built on that: effective time,
then stamp. The register normalizers walk it, and so do the decrease
guard and the event dialog, which is what stops them disagreeing about
which reading is "previous" -- a sheet imported after live readings of the
same month used to count that month twice, and a mid-month reading below
the month's end value was rejected as a drop. A swap detected in a
monthly table applies from the start of that local month, i.e. to the
first reading in it, and a recorded start value never counts above the
reading it lands on.
Every reader buckets in the configured timezone rather than a hardcoded
one, and turns a requested date into that zone's local midnight, so the
divided shares are read back under the months they were stamped in. The
zone id is normalised to its IANA form, because .NET accepts a Windows id
that PostgreSQL will not bucket by, and both are checked at startup.
Stored consumption is derived, so a rule change reaches a meter only at
its next reading -- weeks, for a meter read monthly. NormalizationUpgrade
records the revision and zone the stored series was built with and
rebuilds everything once at startup when either differs, each meter in
its own transaction. A meter that fails is logged, kept in
normalization_pending and retried at the next start: one bad series must
never keep the application down.
What an operator sees once
Existing charts change on the first start after the update: months that
carried a neighbour's use give it back. Rows of earlier imports from
monthly tables are marked as such before anything is recomputed, and if
that marking fails nothing is rebuilt or recorded, so the upgrade simply
runs again next time rather than shifting every imported month by one. A
wizard import whose date format was left on auto-detect is treated as a
monthly table when all of its rows sit on the 1st across at least two
months -- exactly how those rows were attributed before -- and each such
batch is named in the log, because a day-dated sheet always read on the
1st looks identical; revert and re-import it with the day format if that
is what it was.
Tests: 120 unit and 230 integration, including the reference fixtures,
which still reconcile month for month.
105 lines
20 KiB
Markdown
105 lines
20 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## What this repo is
|
||
|
||
MeterVault is a self-hosted, local-first energy & utility metering platform: it ingests meter data from Home Assistant, Tasmota and MQTT on a schedule, stores every reading timestamped and immutable, normalizes it into consumption, and turns it into cost dashboards. Energy types (electricity, water, heating oil, gas, …) and meters are **user-defined, never hardcoded**.
|
||
|
||
**Status: implemented (M0–M7) + SDD §8 panels.** The full solution is built and green — five projects, ~350 tests, working Docker deploy. `docs/SDD.md` remains the design reference; the milestone map (§12) matches the git history (M0…M7 commits). The dedicated **PV/Solar** (`/solar`), **Oil/consumable** (`/consumables`) and **meter-detail** (`/meters/{id}`) views (SDD §8.4–§8.6) are implemented as read models in `Infrastructure/Dashboard` (`SolarService`, `ConsumableService`, `MeterDetailService`) — PV meters are found by `Mode == GenerationCounter` and grid/load meters by a `role` tag in `Meter.Meta` (`MeterRoles`/`MeterMeta`), so nothing is hardcoded by name. **Admin write-CRUD** (SDD §8.7) is implemented as MudBlazor inline-dialog pages: energy types, meters (+ recompute on mode/baseline change), a meter's ingest sources (meter-detail Sources tab), tariffs, cost categories + members, and connectors (`ingestion_endpoint`, secrets as an env-var reference or typed in and encrypted at rest). **Manual readings** are entered from the meter-detail Readings tab ("Add reading"): a touch-first dialog prefilled with the meter's last register value and the current local time, with an on-screen keypad for phone entry at the meter, a live parsed-value + delta-since-last readout, and the decrease guard surfaced before saving. It goes through `IngestionService.IngestByMeterAsync(quality: Manual)`, so it is stamped `ReadingQuality.Manual` and renormalizes inline like any other ingest — the layout of that dialog deliberately reserves fixed space for its verdict line, because anything that reflows moves the keys out from under the user's thumb mid-entry. `/admin/settings` is a read-only effective-config view (settings are env-driven and reproducible, not DB-stored). **Home Assistant reading** is configured here: an HA connector (`BaseUrl` + `TokenEnv`) + an HA source (entity id) drives `HomeAssistantWorker`'s REST poll, or — with the connector's **WebSocket push** toggle (`HaEndpointConfig.UseWebSocket`) — `HomeAssistantWebSocketWorker` holds a persistent `state_changed` subscription and ingests in real time (the poll worker skips WS endpoints, so each is served once; `HaWebSocketProtocol` is the pure, unit-tested handshake/parse logic). `HaConnectionTester` powers the connector "Test connection" button. **Meter topology & flow**: `MeterLink` (a directed `from→to` edge; a downstream meter is a *subsection* of an upstream one, multi-parent allowed) drives a per-energy-type page `/energy/{id}` with a hand-rolled SVG **Sankey** (`SankeyChart.razor`, since ApexCharts has no Sankey type) computed by `FlowService` (link value = downstream consumption, split proportionally across multiple parents; unaccounted remainder → an "Other" node). Upstream meters are wired cycle-safely in the meter editor; the nav lists a link per energy type. **CSV mapping wizard** (`/import/wizard`): upload an arbitrary CSV, map columns → meters/roles, dry-run preview, then commit as a revertible `import_batch` (the `/import` page lists batches with one-click revert). The `instant_rate` mode is normalized (`InstantRateNormalizer`: rate integrated over time, trapezoidal). **Meter events from the UI** (swap, counter reset, tank level, delivery, note) go through `MeterEventService` (`Infrastructure/Ingestion`), never ad-hoc inserts: `MeterEventRules.RecordableFor(mode)` (Core) decides which events a mode offers (no `Correction` — nothing reads it), `Validate` gives the dialog the same verdict the save reaches, and every record/delete recomputes the meter in one transaction. A swap/reset is stored as the event at T **plus a manual reading of the new register's start value at exactly T** (flagged `MeterSwap`/`CounterReset`) — the boundary window is `(prevReading, reading]`, so this books the old tail at T and later readings count from the new start; never write the old final value as the reading at T (double-counts, then rejects every new-register reading). Deleting a swap removes its start reading only while it is still the untouched start value; only `Manual` readings are deletable in the UI. **Navigation conventions:** the meter page is the per-meter hub (header actions: primary entry by mode, "Record event" menu, Edit via the shared `Shared/MeterEditor.razor`, which also owns tank setup); link into it with `MeterLinks` (`/meters/{id}?tab=…&action=…`, action consumed once after the interactive render and dropped from the address); the app-bar "Find a meter" dialog offers the same quick entry; `NavState` tells the per-circuit nav to reload energy types after admin edits. A source that lacks a usable connector detours through `/admin/connectors?new=…|edit=…&meter=…` and comes back to that source dialog with the connector picked and everything typed restored (the page saves the open dialog to the circuit-scoped `DraftStore` on dispose) (`MeterLinks.Source`/`NewConnector`/`EditConnector`; the way back is a meter id, never a URL, so it cannot redirect off-site); the connector list shows which meters use each connector. The meter editor owns the meter's own cost-category memberships (type-level ones are only named), import batches list the meters/categories they wrote to, and the dashboard's empty cost panel names the first missing setup step (`DashboardService.GetCostSetupAsync` → `CostSetup.FirstGap`). **UI language** (M7's last item) is now English + German end to end — see *Localization* below. Set `MeterVault__SeedReferenceData=true` (compose: `METERVAULT_SEED=true`) for a one-command populated demo.
|
||
|
||
## Source of truth
|
||
|
||
`docs/SDD.md` is the authoritative spec and build brief — read it before implementing anything. Key protocol from §0 that governs all work here:
|
||
|
||
- **Build strictly in milestone order (§12, M0→M7).** Each milestone is independently runnable and testable; do not start Mn+1 until Mn's tests pass.
|
||
- **The four CSVs in `sampledata/` are golden fixtures.** Every parsing / consumption / cost rule must reconcile against them (§13). **If a computed number disagrees with the spreadsheet, the spreadsheet wins** unless the discrepancy is a deliberately documented correctness fix.
|
||
- When a design decision is ambiguous, check §14 (open questions): if listed, take the stated default and flag it; if not listed, ask before guessing.
|
||
- Keep the **domain layer free of infrastructure concerns** (the domain model and DB schema are UI-agnostic by design).
|
||
|
||
## Committed tech stack (do not re-litigate; see SDD §4.1)
|
||
|
||
.NET (current LTS — .NET 10, .NET 8 acceptable), C# · ASP.NET Core + **Blazor Server** · **MudBlazor** components · **ApexCharts** (Blazor-ApexCharts) · **MQTTnet** · **PostgreSQL + TimescaleDB** · **EF Core (Npgsql)** for schema/CRUD + **Dapper** for hot-path time-series reads · `BackgroundService` hosted services for ingestion/aggregation · **xUnit + Testcontainers** (Timescale image) · Docker Compose + GHCR.
|
||
|
||
## Project layout
|
||
|
||
```
|
||
/src/Core domain entities + enums; pure Normalization engine (mode strategies,
|
||
expression evaluator); Parsing (German dialect); Costing (TariffResolver)
|
||
/src/Infrastructure MeterVaultDbContext + migrations (relational + raw-SQL Timescale);
|
||
Import (CsvImporter, profiles, ImportService), Ingestion (MQTT/HA workers,
|
||
IngestionService), Normalization service, Costing/Dashboard/Backup services
|
||
/src/App ASP.NET Core host: Blazor Server UI (Components/), REST API (Api/), hosted
|
||
workers, Program.cs (Serilog, migrate+seed on startup, /healthz)
|
||
/tests/Core.Tests unit (no Docker): parsers, normalizers, swap→12, tariff resolver
|
||
/tests/Integration.Tests Testcontainers (Timescale): reconciliation vs the 4 fixtures,
|
||
import commit/revert, ingestion, cost, CAgg refresh, API, export, render
|
||
/deploy Dockerfile, docker-compose.yml (app + timescaledb), build-and-push.ps1, unraid-template.xml
|
||
```
|
||
|
||
Central package versions live in `Directory.Packages.props`; shared build/style in
|
||
`Directory.Build.props` + `.editorconfig`. Snake_case table/column mapping via
|
||
`UseSnakeCaseNamingConvention`. EF migrations are exempt from code-style enforcement (see `.editorconfig`).
|
||
|
||
## Commands
|
||
|
||
```powershell
|
||
dotnet build # build the solution
|
||
dotnet test # all tests (Integration.Tests needs Docker for Testcontainers)
|
||
dotnet test tests/Core.Tests # unit tests only (no Docker needed)
|
||
dotnet test tests/Integration.Tests --filter "FullyQualifiedName~Reconciliation" # one class/area
|
||
dotnet ef migrations add <Name> -p src/Infrastructure -s src/App -o Persistence/Migrations
|
||
dotnet run --project src/App # run app + workers locally (needs a Timescale DB)
|
||
docker compose -f deploy/docker-compose.yml up # app + TimescaleDB together
|
||
```
|
||
|
||
**Timescale-in-EF gotchas** (already handled — follow the pattern): hypertable/CAgg DDL lives in
|
||
raw-SQL migrations; continuous-aggregate creation + policies use `migrationBuilder.Sql(..., suppressTransaction: true)`, one statement each; CAgg policy `end_offset` must be ≥ one bucket. Tests
|
||
pause the compression job (historical fixture data would otherwise deadlock imports).
|
||
|
||
## Core architecture (the part that spans multiple files)
|
||
|
||
**Data pipeline — one direction, layered (SDD §4.2, §5, §7):**
|
||
|
||
```
|
||
sources (Tasmota/HA/MQTT/manual/CSV)
|
||
→ Ingestion workers write raw `reading` rows (immutable audit truth)
|
||
→ Normalization derives append-only `consumption` (deltas in base unit)
|
||
→ TimescaleDB continuous aggregates roll consumption to hourly/daily/monthly/yearly
|
||
→ Cost engine joins aggregates with time-ranged `tariff`
|
||
→ Blazor dashboard + REST API read aggregates + cost views
|
||
```
|
||
|
||
**Invariants that shape everything:**
|
||
|
||
- **Raw `reading` is immutable audit truth.** Everything derived (consumption, cost, balances, forecasts) is computed on top and must be reproducible. Never mutate readings to fix a derived number. Live ingestion recomputes the meter inline (`IngestionService.RenormalizeAsync`) — without it, polled readings never become consumption.
|
||
- **Consumption is attributed to the months it accrued in** (`GapAttribution`, SDD §7.1). A plain increase whose interval crosses a *local* month boundary (instance timezone) is divided at those boundaries by elapsed time, each share stamped inside its month (the closing reading keeps its own row when it lies in that month), marked `Estimated`. Imported monthly-table rows are month-end snapshots, marked by the **importer** with `ReadingFlags.MonthLabel` when the date cell named a month (`IsMonthLabel` = the flag; never infer it from a midnight-on-the-1st stamp — a day-dated "01.08.2026" is an instant). A manual correction keeps the flag; a live/API value or a swap start reading written onto that instant clears it (`IngestionService.UpsertAsync`). `EffectiveTime` reads a label as the end of its month; **`ReadingTimeline`** (Core) is the one ordering — effective time, then stamp — used by `CounterNormalizerBase` and `RuntimeCounterNormalizer`, and via `RegisterNeighbours` by the ingestion decrease guard and `MeterEventService.GetContextAsync`, so none of them disagrees about which reading is "previous". Consecutive rows span exactly one month and book unchanged (the golden fixtures reconcile), a live reading after the last imported row counts from that month's end, and a sheet imported after live readings does not double-count. A swap/reset stamped exactly at a label sits at the start of that label's local month (`ReadingTimeline.BoundaryTime`), and `RegisterBoundary.Advance` never counts a start value above the reading. `StampTime` keeps a label's row inside its own local month (zones behind UTC; also used by `DirectDeltaNormalizer`); `NormalizationEngine` coalesces any rows that still share a (time, kind) key; `GapAttribution.LocalMidnight` verifies its answer so contradictory zone data cannot stall the month walk. `GapSplittingIsInertOnFixturesTests` pins that no fixture interval is divided, in UTC or Berlin. Swaps/resets/decreases are never divided. **Readers bucket in the configured `MeterVault__TimeZone`** (`CostService`, `SolarService`, `ConsumableService`, `MeterPeriodService` take `@tz`) and turn requested dates into instants with `InstanceTimeZone.StartOf` (local midnight, not UTC midnight — `DashboardService`, `FlowService`, `EnergyView` too); a hard-coded zone or UTC-midnight range would re-file the divided shares. `Program` post-configures the zone id to its IANA form (`InstanceTimeZone.Canonical`) and logs an error when .NET or PostgreSQL does not know it. Stored consumption records `normalization_revision` and `normalization_zone` in `app_setting`; `NormalizationUpgrade` (startup migration step) first flags month rows of pre-revision-2 imports (reference profiles by name, wizard `MonthName` batches, and wizard `Auto` batches whose every row sits on the 1st across ≥2 months — logged per batch), lifting TimescaleDB's decompression cap for that one UPDATE; if flagging fails nothing is rebuilt or recorded. It then rebuilds every non-virtual meter (virtual ones are computed on read) when revision or zone differs, per meter in its own transaction; a failing meter is logged, kept in `normalization_pending` and retried next start — the upgrade must never crash startup. Bump `CurrentRevision` whenever the engine books existing readings differently.
|
||
- **Dashboards and charts read aggregates only — never scan `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). Raw is kept for a bounded window (default 3y); `consumption` + aggregates are the long-term source of truth.
|
||
- **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries − usage + forecast), `direct_delta`, `instant_rate`, `virtual` (expression over other meters). New ingestion/normalization logic dispatches on mode.
|
||
- **Nothing domain-specific is hardcoded.** Energy types are data. Cost **categories are decoupled from energy types** (Heizung may be oil today, heat-pump tomorrow). PV self-consumption/savings/net are **virtual meters** with user-defined expressions, not special-cased code. Tariffs are time-ranged (price history), scoped global / per-type / per-meter.
|
||
|
||
**Timescale vs EF split (SDD §5.3):** EF Core migrations own the relational tables. **Timescale-specific DDL — `create_hypertable`, compression policies, continuous aggregates, retention — is not expressible via EF's model builder and must live in raw-SQL migrations.** `reading` and `consumption` are hypertables.
|
||
|
||
**Time & DST (SDD §10):** store UTC everywhere; bucket and display in the instance timezone (default `Europe/Berlin`). "Daily cost" boundaries are local-midnight — use `time_bucket(..., 'Europe/Berlin')`.
|
||
|
||
**In-app update (`UpdateRunner`):** the dashboard shows a banner when a newer tag exists (`UpdateCheckService`, cached, never blocks a render). Triggering an update is **off by default**; `MeterVault__AllowInAppUpdate` is the *only* gate — no API key, by explicit owner decision. With it on, anything that can reach the app can trigger a rebuild+restart as root (realistically a DoS, since the build comes from the owner's own repo; RCE if that repo is compromised). The REST endpoint additionally requires an `X-MeterVault-Update` header — a CSRF guard, not auth, so a foreign page cannot drive it via a LAN browser. Launches detached via `systemd-run` because the update restarts the service. Treat any change here as security-critical; `UpdateRunnerTests` pins that the flag defaults off and that API keys alone don't enable it.
|
||
|
||
**Localization (SDD §12, M7 — en + de):** UI strings live in `src/App/Localization/Strings.resx` (neutral = English) and `Strings.de.resx`. MSBuild generates a **strongly-typed** `Strings` class from the neutral resx (see the `EmbeddedResource` block in `MeterVault.App.csproj`), aliased as `S` in `_Imports.razor` — so components write `@S.Common_Save`, never a string key, and a stale key is a build error. `Loc.F(S.Key, args)` formats the `{0}` ones. **Adding a string means editing both resx files**: `StringResourceTests` fails the build on a missing or blank translation, a placeholder mismatch, an orphan, or a key nothing references — resource fallback would otherwise hide a half-translated release. Domain **enums stay bare identifiers** (they are persisted as text and appear in the REST API); `DisplayNames.Display()` is the single place that decides how each value is *spoken*, and `EnumDisplayNameTests` fails if a value has no wording. Anything from the database (meter names, energy-type display names, category names) is user data and is **never** translated. Language is per-request: the cookie the `/culture/set` endpoint writes, else `Accept-Language`, else `MeterVault__Locale` (default `en`). Switching must be a **full reload** (`forceLoad`) — a Blazor Server circuit is fixed to the culture of the request that opened it. `Format.*` formats against `CurrentCulture`, so numbers and month labels follow the reader; the CSV importer's de-DE parsing is unrelated and unchanged, because that dialect belongs to the files, not the reader.
|
||
|
||
**Secrets (SDD §6.4):** broker/HA tokens are **never** stored in DB plaintext. Two forms, chosen per connector in the admin UI: a *reference* (`token_env`/`password_env` naming an env var or Docker secret path) resolved at runtime, or *encrypted at rest* (`token_enc`/`password_enc`) via `SecretProtector` over the ASP.NET Core data-protection key ring. Exactly one survives a save; `EndpointSecret.Resolve` is the single resolution path (encrypted wins). The key ring lives outside the app directory (`MeterVault__DataProtectionKeyPath`, default `/var/lib/metervault/keys`) because the LXC updater republishes `/opt/metervault`. `ExportService` drops `*_enc` values — they are bound to the originating key ring.
|
||
|
||
## Reference-data behaviours the code must reproduce (from `sampledata/`)
|
||
|
||
These CSVs are the German-dialect *Energiebilanz* spreadsheet export and define the minimum feature bar (SDD §2, Appendix A). When writing the importer or normalization, honour:
|
||
|
||
- **German number dialect:** decimal comma (`180,8244706`), thousands dot (`2.940,19`), trailing-`€` currency (`120,00 €`), **unit suffixes on values** (`411kWh`, `49` cm, `2287` L) — strip and validate.
|
||
- **Two date formats:** `Monat YYYY` (German month names, monthly tables) and `DD.MM.YYYY` (event rows).
|
||
- **Skip inline summary rows** (`Total`, `Heute`, `Seitbeginn Tage`, `Seit YYYY`) and **all-zero future placeholder rows** (e.g. Dec 2026) — do not ingest them. **Negatives are valid** (savings, grid balance).
|
||
- **Water register swaps mid-series** (…861 → 2 → 15): consumption must stay continuous across the boundary via a `meter_swap` event.
|
||
- **Electricity has 5 meters** (Haus, Netz, Auto, Solar 1, Solar 2) plus derived columns. Verified relations to reproduce: `Netz Einsparung = Haus − Netz`, `Ersparnis = Netz Einsparung × €/kWh`, `Kosten = Verbrauchskosten − Ersparnis` — but implement these as **user-definable virtual-meter expressions**, not hardcoded formulas.
|
||
- **Heating oil is the versatility stress test:** a consumable/tank model where consumption is derivable two ways — tank-level Δ, or burner runtime × rate (rate `fixed` from nozzle spec, or `empirical` = Δlevel ÷ Δhours). Early rows (1997–2004) carry deliveries only (no burner hours yet). Includes cm→litre dipstick calibration and forecast-to-empty.
|
||
|
||
## Git
|
||
|
||
Remote `origin` is `https://git.finalfactory.de/FinalFactory/MeterVault.git` (Gitea; default branch `master`). CI/release is **Gitea Actions** under `.gitea/workflows/` — edit `VERSION` on `master` to tag + publish the image to the Gitea container registry.
|