Florian Schmidt
bfa0b537ee
ci / build-test (push) Failing after 35s
The last open item on the M7 list. Number and currency formatting was already locale-aware, but every string in the UI was an English literal, so a German instance read half in each language -- German data, English chrome. This translates all of it and adds the machinery to keep it translated. Strings live in Localization/Strings.resx (English, neutral) and Strings.de.resx. The neutral file generates a strongly-typed accessor at build time, aliased as S in _Imports.razor, so components reference compiled properties -- @S.Common_Save, not a string key. That choice is the point: across 4,500 lines of markup, a key lookup that silently falls back to its own name is a defect you find in production, while a renamed property is a build error. Generation runs in MSBuild rather than the IDE designer, so dotnet build alone reproduces it anywhere. Resource fallback is the hazard here. Ask for a key the German satellite lacks and ResourceManager quietly serves the English one -- correct at runtime, disastrous at release time, because a half-translated build looks perfectly healthy. StringResourceTests reads each satellite with tryParents: false, which is the only way to see what one actually contains, and fails on a missing or blank translation, a placeholder that changed arity, an orphan, or a key nothing references. Three things needed more than substitution: - Domain enums reached the screen as bare identifiers. They stay bare in the model -- they are persisted as text and appear in the REST API, so their names are part of the data contract -- and DisplayNames is now the single place that decides how each value is spoken. Every arm ends in a fallback returning the identifier, so a value added later cannot throw mid-render; EnumDisplayNameTests is what stops that safety net quietly becoming the shipping behaviour. - Infrastructure was writing display text: FlowService's "Other (X)", MeterPeriodView's "Generation"/"Consumption", the HA connection-test verdicts, the updater's snackbar, the CSV importer's row warnings. Each now returns an outcome value and the UI supplies the words, which is where the reader's language is known. Diagnostics that are not ours -- an HTTP status, systemd's stderr, an exception message -- are passed through untranslated, and every English summary is kept alongside the outcome so log lines never move with the UI language. The UpdateRunner change is additive only; no gate was touched. - Importer warnings carry their arguments rather than a finished sentence, so the numbers inside them pick up the reader's grouping. A register that reads 2.940,19 everywhere else must not read 2940.19 only inside a warning. Switching language is a redirect through /culture/set followed by a full reload, not an interactive state change: a Blazor Server circuit is fixed to the culture of the request that opened it. That makes the endpoint a redirector taking its target from the query string, so anything but a local path is refused rather than followed. Preference order is the cookie, then Accept-Language, then MeterVault__Locale -- an instance can be pinned to one language and a reader can still switch. Locale keeps its documented default of "en". Format now follows CurrentCulture instead of a hardcoded de-DE, so an instance with nothing configured and a browser asking for English will show English number formatting where it previously showed German; set MeterVault__Locale=de to pin the old behaviour. The importer's de-DE parsing is untouched and stays that way -- that dialect is a property of the spreadsheets, not of whoever is looking at the dashboard. Anything that comes from the database -- meter names, energy-type display names, category names -- is user data and is never translated. Claude-Session: https://claude.ai/code/session_0112ezeWqaZ85kTj5bYu9JHx
105 lines
15 KiB
Markdown
105 lines
15 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## What this repo is
|
||
|
||
MeterVault is a self-hosted, local-first energy & utility metering platform: it ingests meter data from Home Assistant, Tasmota and MQTT on a schedule, stores every reading timestamped and immutable, normalizes it into consumption, and turns it into cost dashboards. Energy types (electricity, water, heating oil, gas, …) and meters are **user-defined, never hardcoded**.
|
||
|
||
**Status: implemented (M0–M7) + SDD §8 panels.** The full solution is built and green — five projects, ~108 tests, working Docker deploy. `docs/SDD.md` remains the design reference; the milestone map (§12) matches the git history (M0…M7 commits). The dedicated **PV/Solar** (`/solar`), **Oil/consumable** (`/consumables`) and **meter-detail** (`/meters/{id}`) views (SDD §8.4–§8.6) are implemented as read models in `Infrastructure/Dashboard` (`SolarService`, `ConsumableService`, `MeterDetailService`) — PV meters are found by `Mode == GenerationCounter` and grid/load meters by a `role` tag in `Meter.Meta` (`MeterRoles`/`MeterMeta`), so nothing is hardcoded by name. **Admin write-CRUD** (SDD §8.7) is implemented as MudBlazor inline-dialog pages: energy types, meters (+ recompute on mode/baseline change), a meter's ingest sources (meter-detail Sources tab), tariffs, cost categories + members, and connectors (`ingestion_endpoint`, secrets by env-var reference only). **Manual readings** are entered from the meter-detail Readings tab ("Add reading"): a touch-first dialog prefilled with the meter's last register value and the current local time, with an on-screen keypad for phone entry at the meter, a live parsed-value + delta-since-last readout, and the decrease guard surfaced before saving. It goes through `IngestionService.IngestByMeterAsync(quality: Manual)`, so it is stamped `ReadingQuality.Manual` and renormalizes inline like any other ingest — the layout of that dialog deliberately reserves fixed space for its verdict line, because anything that reflows moves the keys out from under the user's thumb mid-entry. `/admin/settings` is a read-only effective-config view (settings are env-driven and reproducible, not DB-stored). **Home Assistant reading** is configured here: an HA connector (`BaseUrl` + `TokenEnv`) + an HA source (entity id) drives `HomeAssistantWorker`'s REST poll, or — with the connector's **WebSocket push** toggle (`HaEndpointConfig.UseWebSocket`) — `HomeAssistantWebSocketWorker` holds a persistent `state_changed` subscription and ingests in real time (the poll worker skips WS endpoints, so each is served once; `HaWebSocketProtocol` is the pure, unit-tested handshake/parse logic). `HaConnectionTester` powers the connector "Test connection" button. **Meter topology & flow**: `MeterLink` (a directed `from→to` edge; a downstream meter is a *subsection* of an upstream one, multi-parent allowed) drives a per-energy-type page `/energy/{id}` with a hand-rolled SVG **Sankey** (`SankeyChart.razor`, since ApexCharts has no Sankey type) computed by `FlowService` (link value = downstream consumption, split proportionally across multiple parents; unaccounted remainder → an "Other" node). Upstream meters are wired cycle-safely in the meter editor; the nav lists a link per energy type. **CSV mapping wizard** (`/import/wizard`): upload an arbitrary CSV, map columns → meters/roles, dry-run preview, then commit as a revertible `import_batch` (the `/import` page lists batches with one-click revert). The `instant_rate` mode is normalized (`InstantRateNormalizer`: rate integrated over time, trapezoidal). **UI language** (M7's last item) is now English + German end to end — see *Localization* below. Set `MeterVault__SeedReferenceData=true` (compose: `METERVAULT_SEED=true`) for a one-command populated demo.
|
||
|
||
## Source of truth
|
||
|
||
`docs/SDD.md` is the authoritative spec and build brief — read it before implementing anything. Key protocol from §0 that governs all work here:
|
||
|
||
- **Build strictly in milestone order (§12, M0→M7).** Each milestone is independently runnable and testable; do not start Mn+1 until Mn's tests pass.
|
||
- **The four CSVs in `sampledata/` are golden fixtures.** Every parsing / consumption / cost rule must reconcile against them (§13). **If a computed number disagrees with the spreadsheet, the spreadsheet wins** unless the discrepancy is a deliberately documented correctness fix.
|
||
- When a design decision is ambiguous, check §14 (open questions): if listed, take the stated default and flag it; if not listed, ask before guessing.
|
||
- Keep the **domain layer free of infrastructure concerns** (the domain model and DB schema are UI-agnostic by design).
|
||
|
||
## Committed tech stack (do not re-litigate; see SDD §4.1)
|
||
|
||
.NET (current LTS — .NET 10, .NET 8 acceptable), C# · ASP.NET Core + **Blazor Server** · **MudBlazor** components · **ApexCharts** (Blazor-ApexCharts) · **MQTTnet** · **PostgreSQL + TimescaleDB** · **EF Core (Npgsql)** for schema/CRUD + **Dapper** for hot-path time-series reads · `BackgroundService` hosted services for ingestion/aggregation · **xUnit + Testcontainers** (Timescale image) · Docker Compose + GHCR.
|
||
|
||
## Project layout
|
||
|
||
```
|
||
/src/Core domain entities + enums; pure Normalization engine (mode strategies,
|
||
expression evaluator); Parsing (German dialect); Costing (TariffResolver)
|
||
/src/Infrastructure MeterVaultDbContext + migrations (relational + raw-SQL Timescale);
|
||
Import (CsvImporter, profiles, ImportService), Ingestion (MQTT/HA workers,
|
||
IngestionService), Normalization service, Costing/Dashboard/Backup services
|
||
/src/App ASP.NET Core host: Blazor Server UI (Components/), REST API (Api/), hosted
|
||
workers, Program.cs (Serilog, migrate+seed on startup, /healthz)
|
||
/tests/Core.Tests unit (no Docker): parsers, normalizers, swap→12, tariff resolver
|
||
/tests/Integration.Tests Testcontainers (Timescale): reconciliation vs the 4 fixtures,
|
||
import commit/revert, ingestion, cost, CAgg refresh, API, export, render
|
||
/deploy Dockerfile, docker-compose.yml (app + timescaledb), build-and-push.ps1, unraid-template.xml
|
||
```
|
||
|
||
Central package versions live in `Directory.Packages.props`; shared build/style in
|
||
`Directory.Build.props` + `.editorconfig`. Snake_case table/column mapping via
|
||
`UseSnakeCaseNamingConvention`. EF migrations are exempt from code-style enforcement (see `.editorconfig`).
|
||
|
||
## Commands
|
||
|
||
```powershell
|
||
dotnet build # build the solution
|
||
dotnet test # all tests (Integration.Tests needs Docker for Testcontainers)
|
||
dotnet test tests/Core.Tests # unit tests only (no Docker needed)
|
||
dotnet test tests/Integration.Tests --filter "FullyQualifiedName~Reconciliation" # one class/area
|
||
dotnet ef migrations add <Name> -p src/Infrastructure -s src/App -o Persistence/Migrations
|
||
dotnet run --project src/App # run app + workers locally (needs a Timescale DB)
|
||
docker compose -f deploy/docker-compose.yml up # app + TimescaleDB together
|
||
```
|
||
|
||
**Timescale-in-EF gotchas** (already handled — follow the pattern): hypertable/CAgg DDL lives in
|
||
raw-SQL migrations; continuous-aggregate creation + policies use `migrationBuilder.Sql(..., suppressTransaction: true)`, one statement each; CAgg policy `end_offset` must be ≥ one bucket. Tests
|
||
pause the compression job (historical fixture data would otherwise deadlock imports).
|
||
|
||
## Core architecture (the part that spans multiple files)
|
||
|
||
**Data pipeline — one direction, layered (SDD §4.2, §5, §7):**
|
||
|
||
```
|
||
sources (Tasmota/HA/MQTT/manual/CSV)
|
||
→ Ingestion workers write raw `reading` rows (immutable audit truth)
|
||
→ Normalization derives append-only `consumption` (deltas in base unit)
|
||
→ TimescaleDB continuous aggregates roll consumption to hourly/daily/monthly/yearly
|
||
→ Cost engine joins aggregates with time-ranged `tariff`
|
||
→ Blazor dashboard + REST API read aggregates + cost views
|
||
```
|
||
|
||
**Invariants that shape everything:**
|
||
|
||
- **Raw `reading` is immutable audit truth.** Everything derived (consumption, cost, balances, forecasts) is computed on top and must be reproducible. Never mutate readings to fix a derived number. Live ingestion recomputes the meter inline (`IngestionService.RenormalizeAsync`) — without it, polled readings never become consumption.
|
||
- **Long gaps are apportioned, short ones are not** (`GapAttribution`, SDD §7.1). An interval containing ≥2 whole calendar months is split across those months, proportional to elapsed time, marked `Estimated`. A monthly series contains exactly one and is untouched — that's what keeps the golden fixtures reconciling. `GapSplittingIsInertOnFixturesTests` asserts the rule declines to fire on the reference data, so this can't silently drift.
|
||
- **Dashboards and charts read aggregates only — never scan `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). Raw is kept for a bounded window (default 3y); `consumption` + aggregates are the long-term source of truth.
|
||
- **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries − usage + forecast), `direct_delta`, `instant_rate`, `virtual` (expression over other meters). New ingestion/normalization logic dispatches on mode.
|
||
- **Nothing domain-specific is hardcoded.** Energy types are data. Cost **categories are decoupled from energy types** (Heizung may be oil today, heat-pump tomorrow). PV self-consumption/savings/net are **virtual meters** with user-defined expressions, not special-cased code. Tariffs are time-ranged (price history), scoped global / per-type / per-meter.
|
||
|
||
**Timescale vs EF split (SDD §5.3):** EF Core migrations own the relational tables. **Timescale-specific DDL — `create_hypertable`, compression policies, continuous aggregates, retention — is not expressible via EF's model builder and must live in raw-SQL migrations.** `reading` and `consumption` are hypertables.
|
||
|
||
**Time & DST (SDD §10):** store UTC everywhere; bucket and display in the instance timezone (default `Europe/Berlin`). "Daily cost" boundaries are local-midnight — use `time_bucket(..., 'Europe/Berlin')`.
|
||
|
||
**In-app update (`UpdateRunner`):** the dashboard shows a banner when a newer tag exists (`UpdateCheckService`, cached, never blocks a render). Triggering an update is **off by default**; `MeterVault__AllowInAppUpdate` is the *only* gate — no API key, by explicit owner decision. With it on, anything that can reach the app can trigger a rebuild+restart as root (realistically a DoS, since the build comes from the owner's own repo; RCE if that repo is compromised). The REST endpoint additionally requires an `X-MeterVault-Update` header — a CSRF guard, not auth, so a foreign page cannot drive it via a LAN browser. Launches detached via `systemd-run` because the update restarts the service. Treat any change here as security-critical; `UpdateRunnerTests` pins that the flag defaults off and that API keys alone don't enable it.
|
||
|
||
**Localization (SDD §12, M7 — en + de):** UI strings live in `src/App/Localization/Strings.resx` (neutral = English) and `Strings.de.resx`. MSBuild generates a **strongly-typed** `Strings` class from the neutral resx (see the `EmbeddedResource` block in `MeterVault.App.csproj`), aliased as `S` in `_Imports.razor` — so components write `@S.Common_Save`, never a string key, and a stale key is a build error. `Loc.F(S.Key, args)` formats the `{0}` ones. **Adding a string means editing both resx files**: `StringResourceTests` fails the build on a missing or blank translation, a placeholder mismatch, an orphan, or a key nothing references — resource fallback would otherwise hide a half-translated release. Domain **enums stay bare identifiers** (they are persisted as text and appear in the REST API); `DisplayNames.Display()` is the single place that decides how each value is *spoken*, and `EnumDisplayNameTests` fails if a value has no wording. Anything from the database (meter names, energy-type display names, category names) is user data and is **never** translated. Language is per-request: the cookie the `/culture/set` endpoint writes, else `Accept-Language`, else `MeterVault__Locale` (default `en`). Switching must be a **full reload** (`forceLoad`) — a Blazor Server circuit is fixed to the culture of the request that opened it. `Format.*` formats against `CurrentCulture`, so numbers and month labels follow the reader; the CSV importer's de-DE parsing is unrelated and unchanged, because that dialect belongs to the files, not the reader.
|
||
|
||
**Secrets (SDD §6.4):** broker/HA tokens are **never** stored in DB plaintext. Two forms, chosen per connector in the admin UI: a *reference* (`token_env`/`password_env` naming an env var or Docker secret path) resolved at runtime, or *encrypted at rest* (`token_enc`/`password_enc`) via `SecretProtector` over the ASP.NET Core data-protection key ring. Exactly one survives a save; `EndpointSecret.Resolve` is the single resolution path (encrypted wins). The key ring lives outside the app directory (`MeterVault__DataProtectionKeyPath`, default `/var/lib/metervault/keys`) because the LXC updater republishes `/opt/metervault`. `ExportService` drops `*_enc` values — they are bound to the originating key ring.
|
||
|
||
## Reference-data behaviours the code must reproduce (from `sampledata/`)
|
||
|
||
These CSVs are the German-dialect *Energiebilanz* spreadsheet export and define the minimum feature bar (SDD §2, Appendix A). When writing the importer or normalization, honour:
|
||
|
||
- **German number dialect:** decimal comma (`180,8244706`), thousands dot (`2.940,19`), trailing-`€` currency (`120,00 €`), **unit suffixes on values** (`411kWh`, `49` cm, `2287` L) — strip and validate.
|
||
- **Two date formats:** `Monat YYYY` (German month names, monthly tables) and `DD.MM.YYYY` (event rows).
|
||
- **Skip inline summary rows** (`Total`, `Heute`, `Seitbeginn Tage`, `Seit YYYY`) and **all-zero future placeholder rows** (e.g. Dec 2026) — do not ingest them. **Negatives are valid** (savings, grid balance).
|
||
- **Water register swaps mid-series** (…861 → 2 → 15): consumption must stay continuous across the boundary via a `meter_swap` event.
|
||
- **Electricity has 5 meters** (Haus, Netz, Auto, Solar 1, Solar 2) plus derived columns. Verified relations to reproduce: `Netz Einsparung = Haus − Netz`, `Ersparnis = Netz Einsparung × €/kWh`, `Kosten = Verbrauchskosten − Ersparnis` — but implement these as **user-definable virtual-meter expressions**, not hardcoded formulas.
|
||
- **Heating oil is the versatility stress test:** a consumable/tank model where consumption is derivable two ways — tank-level Δ, or burner runtime × rate (rate `fixed` from nozzle spec, or `empirical` = Δlevel ÷ Δhours). Early rows (1997–2004) carry deliveries only (no burner hours yet). Includes cm→litre dipstick calibration and forecast-to-empty.
|
||
|
||
## Git
|
||
|
||
Remote `origin` is `https://git.finalfactory.de/FinalFactory/MeterVault.git` (Gitea; default branch `master`). CI/release is **Gitea Actions** under `.gitea/workflows/` — edit `VERSION` on `master` to tag + publish the image to the Gitea container registry.
|