a08e9f781fb7254b3df0859d551d79ad0b7c2555
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a08e9f781f |
Analysis: read a rarely-read meter as coarse, not absent; newest rows first
ci / build-test (push) Successful in 2m41s
Three things a reported Heizoel page got wrong at once. Its tank is dipped a few times a year and its burner read every few months, which is exactly the shape the coverage rules had not been walked through. "No data" for data that exists. A tank books nothing until the next dipstick closes the interval, so the stretch after the last dipstick is covered by no run at all, and a bucket no run covers was reported missing. The burner, whose run reaches into the window, said "only coarser data" -- the honest answer -- so one card claimed there was nothing while the coverage panel beside it listed years of data. A bucket that no run covers, no gap overlaps and no opening balance explains now reports the meter's resolution when its preceding coverage is within one interval of its own class: it is not silent, it is read rarely. A meter that does book its own buckets and stops -- a dead hourly source, a sheet asked about a later month -- still reads missing. Auto answering twelve months with one bar. Coarse only means "longer than a month", so a dipstick taken each autumn straddles a New Year as surely as a month start: coarsening the chart to years bought nothing and cost every point. The planning resolution now caps coarse at month when a run crosses a local year edge, and a series that cannot resolve the natural size no longer coarsens the whole chart -- it is drawn at that size with its buckets marked, which the chart and table already explain. A page contradicting itself. The comparison line above the ranking was fed the leading measure's matched coverage but worded as if it spoke for the page, directly above a burner row that did compare. It now names the figure it is about. Alongside: the "largest changes" ranking no longer drops a meter whose change is not comparable. It ranks what can be ranked, then lists the rest with their values and the reason -- the tank had been vanishing from its own energy type. And every dated table now reads newest first, as lists are read; charts stay chronological left to right, and the CSV export stays ascending for spreadsheets. A-41 to A-43 in the note record the three rules. |
||
|
|
8940ef25c3 |
Analysis: one selected period, one set of numbers, on every page
ci / build-test (push) Successful in 2m31s
The dashboards told several stories at once. Overview asked for full calendar years, meter detail for a fixed 12-month window that was really 13, Trends for 24 months with an Apply button, and the energy pages for 60. Each page derived "today" from UTC, so the first hours of a local day belonged to yesterday. A missing tariff, a month nobody measured and a genuine zero all rendered as 0. And a virtual meter -- the one thing the spreadsheet leans on hardest -- was excluded from analysis outright: MeterPeriodService returned null for it and the page offered a flow diagram instead. docs/DASHBOARD_ANALYSIS_CHANGE_BRIEF.md is the work order. Every choice it left open is settled in docs/ANALYSIS_IMPLEMENTATION_NOTE.md as D-01..D-58 plus amendments A-01..A-30; code, tests and release notes cite those ids. The analysis layer Core/Analysis holds the pure rules: period presets resolved once in the instance zone into a local date range and a half-open UTC range, bucket plans, calendar-unit comparisons, coverage runs with a resolution class, normalized quantities and units, the totals policy, the virtual formula parser/validator/evaluator, and the cost calculator. "Now" comes from TimeProvider; services never read the clock. Normalization now writes, in the same transaction as consumption and by diff, per-meter rollups by local day and month plus coverage runs and a rollup state (AnalysisDataWriter). AnalysisReader answers a request from those tables -- month rollups for month and year buckets, day rollups otherwise, at most two partial edge days from consumption -- and CostReader prices the result month by month. Pages, /api/v1 and the CSV export read nothing else. The unused continuous aggregates are dropped. The reader's statement count per request is constant whether it covers one meter or a thousand. On a synthetic 1,000-meter, ten-year instance the brief's target request (100 meters, ten years, monthly) takes 374 ms against a two-second target, and the Overview went from 48,244 SQL statements per load to 205. Missing is not zero Every bucket carries a status -- available, partial, missing, unresolved, invalid, pending -- derived from coverage, never from the amount, with provenance and a reason code beside it. A true zero is a number and a bar on the baseline; an unknown bucket is a gap that says why; a month whose data only exists monthly says so instead of inventing daily detail; a scope with no tariff says "not priced" instead of 0. Rows whose interval closes after now are reported separately rather than counted. Virtual meters are analysis subjects A virtual meter stores a canonical definition -- expression over m<id> references, result kind, unit and cost rule -- validated on save and on read for syntax, unknown or self references, loops and unit/kind rules. It is evaluated on read from its sources' rollups over their joint coverage: a missing source makes the bucket missing, an observed zero is a valid input, a non-finite result is invalid with its dependency path, and the page lists each source's contribution. Topology links are topology only and never rewrite a saved calculation; expression-less meters from older installs are converted once at startup. The editor has Sum, Difference and Advanced modes with a live preview. Totals and the bill Per energy type the totals policy separates use, grid import, export, generation and runtime, marks breakdown meters as breakdowns and virtual meters as views, and never adds across units. The bill follows it: grid import where there is one, separately priced subsections at their own price, feed-in only on export meters, standing charges once per scope per local day, manual costs once on their start day, categories as non-overlapping covers whose composition reconciles to the bill. The seeded demo's yearly totals now match the spreadsheet. Pages and navigation The period lives in the URL and every page reads the same contract, so a link, a reload and the browser's Back button keep it. Shared components carry it: page header with breadcrumbs, period toolbar, theme-aware chart with an accessible table beside it, metric cards, comparison and availability states, attention items that each link to the one action that fixes them. Meter detail leads with an Analysis tab and resolves its tabs by key; the energy page has Overview, History, Flow and Meters; the old cost-only Trends page is a general Analysis page over portfolio, type, category, meter or a meter comparison. Records tabs are paged server-side instead of showing the latest 200. Everything is English and German, light and dark, down to 360px. Some figures change on purpose; docs/RELEASE_NOTES.md lists each one and what the first start after the update does (it rebuilds all analysis data before the web server listens). docs/SDD.md and CLAUDE.md describe the system as it now is. Tests: 1,733 Core and 746 integration, all green, plus an opt-in performance suite with a synthetic 1,000-meter generator. |
||
|
|
aacdc28d70 |
Meters: record events from the UI, and book consumption in the months it accrued in
Two threads that ended up in the same files. One is navigation: a meter
swap that happened today had no click path at all, and most per-meter
tasks were reachable only by knowing which admin page owned them. The
other is attribution: readings on 1 August and 16 September showed six
weeks of water under September and nothing under August.
Meter events from the UI
Swap, counter reset, tank level, delivery and note are recorded through
MeterEventService rather than ad-hoc inserts, so the dialog's verdict and
the saved result come from the same Validate call, and every record or
delete recomputes the meter inside one transaction. MeterEventRules
decides which events a mode offers -- a tank has no register to swap, and
Correction is offered nowhere because nothing reads it.
A swap is stored as the event at T plus a manual reading of the new
register's start value at exactly T. That pairing is the whole trick: the
boundary window is (previousReading, reading], so the old register's tail
books at T and every later reading counts from the new start. Writing the
old final value as the reading at T instead -- the obvious thing -- double
counts the tail and then rejects every reading the new register produces.
Deleting a swap removes that start reading only while it is still the
untouched start value, and only Manual readings can be deleted at all.
Navigation
The meter page is now the hub: primary entry by mode, a "Record event"
menu, and Edit through a shared MeterEditor that also owns tank setup.
Other pages link into it with MeterLinks (/meters/{id}?tab=...&action=...),
whose action is consumed once after the interactive render and dropped
from the address -- the reverse order flashes the dialog and closes it,
because a circuit's first location change dismisses every open dialog.
The app bar gains a "Find a meter" dialog with the same quick entry.
A source that has no usable connector now links to creating (or enabling)
one and comes back to the same source dialog with the connector picked
and everything typed still there; the draft survives in a circuit-scoped
DraftStore, and the way back is a meter id rather than a URL, so the page
cannot be made to redirect anywhere else. The connector list shows which
meters use each connector, import batches list the meters and categories
they wrote to, the meter editor owns the meter's own cost categories, and
the dashboard's empty cost panel names the first missing step instead of
listing every admin page.
Months
A reading is an instant, and what it measures accrued over the time since
the previous one. Booking the whole delta at the closing reading misfiles
it whenever the interval crosses a month boundary, so a plain increase is
now divided at local month boundaries in proportion to elapsed time, each
share stamped inside its month and marked estimated: the meter recorded a
total, not a shape. The parts always sum to the original.
Imported monthly tables are the exception that keeps the golden fixtures
reconciling. "Mai 2026" carries the register at the end of May but is
stamped on the 1st, so the importer -- the only place that still knows
whether the date cell named a month or a day -- flags it MonthLabel, and
the engine reads it as the end of its month. Inferring that from the
stamp instead would catch day-dated rows: a sheet with "01.08.2026" in it
is not a monthly table, and reading it as one moves two thirds of July
into August.
ReadingTimeline is the single ordering built on that: effective time,
then stamp. The register normalizers walk it, and so do the decrease
guard and the event dialog, which is what stops them disagreeing about
which reading is "previous" -- a sheet imported after live readings of the
same month used to count that month twice, and a mid-month reading below
the month's end value was rejected as a drop. A swap detected in a
monthly table applies from the start of that local month, i.e. to the
first reading in it, and a recorded start value never counts above the
reading it lands on.
Every reader buckets in the configured timezone rather than a hardcoded
one, and turns a requested date into that zone's local midnight, so the
divided shares are read back under the months they were stamped in. The
zone id is normalised to its IANA form, because .NET accepts a Windows id
that PostgreSQL will not bucket by, and both are checked at startup.
Stored consumption is derived, so a rule change reaches a meter only at
its next reading -- weeks, for a meter read monthly. NormalizationUpgrade
records the revision and zone the stored series was built with and
rebuilds everything once at startup when either differs, each meter in
its own transaction. A meter that fails is logged, kept in
normalization_pending and retried at the next start: one bad series must
never keep the application down.
What an operator sees once
Existing charts change on the first start after the update: months that
carried a neighbour's use give it back. Rows of earlier imports from
monthly tables are marked as such before anything is recomputed, and if
that marking fails nothing is rebuilt or recorded, so the upgrade simply
runs again next time rather than shifting every imported month by one. A
wizard import whose date format was left on auto-detect is treated as a
monthly table when all of its rows sit on the 1st across at least two
months -- exactly how those rows were attributed before -- and each such
batch is named in the log, because a day-dated sheet always read on the
1st looks identical; revert and re-import it with the day format if that
is what it was.
Tests: 120 unit and 230 integration, including the reference fixtures,
which still reconcile month for month.
|
||
|
|
bfa0b537ee |
i18n: ship the UI in English and German
ci / build-test (push) Failing after 35s
The last open item on the M7 list. Number and currency formatting was already locale-aware, but every string in the UI was an English literal, so a German instance read half in each language -- German data, English chrome. This translates all of it and adds the machinery to keep it translated. Strings live in Localization/Strings.resx (English, neutral) and Strings.de.resx. The neutral file generates a strongly-typed accessor at build time, aliased as S in _Imports.razor, so components reference compiled properties -- @S.Common_Save, not a string key. That choice is the point: across 4,500 lines of markup, a key lookup that silently falls back to its own name is a defect you find in production, while a renamed property is a build error. Generation runs in MSBuild rather than the IDE designer, so dotnet build alone reproduces it anywhere. Resource fallback is the hazard here. Ask for a key the German satellite lacks and ResourceManager quietly serves the English one -- correct at runtime, disastrous at release time, because a half-translated build looks perfectly healthy. StringResourceTests reads each satellite with tryParents: false, which is the only way to see what one actually contains, and fails on a missing or blank translation, a placeholder that changed arity, an orphan, or a key nothing references. Three things needed more than substitution: - Domain enums reached the screen as bare identifiers. They stay bare in the model -- they are persisted as text and appear in the REST API, so their names are part of the data contract -- and DisplayNames is now the single place that decides how each value is spoken. Every arm ends in a fallback returning the identifier, so a value added later cannot throw mid-render; EnumDisplayNameTests is what stops that safety net quietly becoming the shipping behaviour. - Infrastructure was writing display text: FlowService's "Other (X)", MeterPeriodView's "Generation"/"Consumption", the HA connection-test verdicts, the updater's snackbar, the CSV importer's row warnings. Each now returns an outcome value and the UI supplies the words, which is where the reader's language is known. Diagnostics that are not ours -- an HTTP status, systemd's stderr, an exception message -- are passed through untranslated, and every English summary is kept alongside the outcome so log lines never move with the UI language. The UpdateRunner change is additive only; no gate was touched. - Importer warnings carry their arguments rather than a finished sentence, so the numbers inside them pick up the reader's grouping. A register that reads 2.940,19 everywhere else must not read 2940.19 only inside a warning. Switching language is a redirect through /culture/set followed by a full reload, not an interactive state change: a Blazor Server circuit is fixed to the culture of the request that opened it. That makes the endpoint a redirector taking its target from the query string, so anything but a local path is refused rather than followed. Preference order is the cookie, then Accept-Language, then MeterVault__Locale -- an instance can be pinned to one language and a reader can still switch. Locale keeps its documented default of "en". Format now follows CurrentCulture instead of a hardcoded de-DE, so an instance with nothing configured and a browser asking for English will show English number formatting where it previously showed German; set MeterVault__Locale=de to pin the old behaviour. The importer's de-DE parsing is untouched and stays that way -- that dialect is a property of the spreadsheets, not of whoever is looking at the dashboard. Anything that comes from the database -- meter names, energy-type display names, category names -- is user data and is never translated. Claude-Session: https://claude.ai/code/session_0112ezeWqaZ85kTj5bYu9JHx |
||
|
|
8fe5f4411b |
Fix defects found auditing the ingestion, import and connector changes
ci / build-test (push) Successful in 1m16s
An audit of this session's commits found several real problems, three of which
lose or expose data. Ordered by severity.
Live recompute was not atomic. RecomputeMeterAsync clears a meter's series with
ExecuteDelete, which commits by itself when no transaction is ambient, and only
then adds the rebuilt rows. Between the two the meter had *no* consumption:
a dashboard read reported zero, and a crash or cancelled request made the loss
permanent, for data the SDD treats as the long-term source of truth (§5.5).
Import and the events API already wrapped their recomputes; live ingestion,
which I added this session, did not. Now shares one transaction, joining an
ambient one rather than nesting.
The MQTT backfill migration counted brokers without regard to is_enabled. One
live broker plus a disabled leftover counted two, declined to backfill, and left
those sources unbound — which under endpoint-scoped routing means silently and
permanently dead. The "two or more is ambiguous" reasoning did not hold there:
the worker only ever connected to enabled endpoints. Corrected by a follow-up
migration rather than an edit, since the original may already have run; it
touches only rows still NULL, so hand-made bindings are safe.
A mapping edited after a dry run committed the *old* staged rows under the
*new* mapping. Readings went to the previous meter while the batch recorded the
current mapping — wrong data, provenance contradicting it, no exception. The
earlier fix re-validated but did not detect staleness. Commit now compares the
mapping against the one the preview was staged under and refuses.
"Test connection" sent a stored token to whatever Base URL was in the dialog.
Encrypting secrets at rest means the UI can decrypt what the operator can no
longer read, so this turned the button into an exfiltration primitive: point it
at any host, the token arrives as a Bearer header. A stored token now only goes
to the origin it was saved for; testing elsewhere requires typing it again.
A source that cannot ingest looked identical to a healthy one. Endpoint-scoped
routing made unbound and mis-bound sources silently dead, while the Sources tab
showed no connector at all and the delete dialog still promised sources would be
"unlinked". Added a Connector column that names the fault, stopped offering
disabled connectors (both workers filter on IsEnabled), and made the delete
warning say ingestion stops.
Virtual meters rendered four zero tiles: they evaluate on read and only
materialize when a cost category references them (§14.1), so summing
consumption is a confident lie about a working meter. They now report nothing
and the page explains why.
Re-importing an overlapping file failed at the database with EF's "An error
occurred while saving the entity changes", naming neither meter nor date — the
diagnosis problem
|
||
|
|
e23df37a3f |
Connectors: allow secrets to be entered in the UI, encrypted at rest
ci / build-test (push) Successful in 1m12s
Reference-only secrets (SDD §6.4) meant adding a connector required editing a file on the server and restarting the service. In practice that leads to the token being pasted into the env-var *name* field, which fails with "environment variable '<token>' is not set" and gives no hint what went wrong. Add a second storage form, chosen per connector: type the secret in and it is encrypted via ASP.NET Core data protection before it is stored. The env-var reference stays as an equal alternative — this widens the choice rather than replacing it. Exactly one form survives a save, so a stale secret cannot linger and silently win; EndpointSecret.Resolve is the single resolution path. The guarantee that matters is preserved: no plaintext in the database, so pg_dump and JSON exports carry nothing usable. The trust boundary is stated plainly in §6.4 — the key ring is on disk, so this protects against leaked database content, not an attacker who already has the host, which is the same boundary an env var has. Details worth noting: - Key ring defaults to /var/lib/metervault/keys, outside the app directory, because the LXC updater republishes /opt/metervault on every update. Docker gets a named volume. Overridable via MeterVault__DataProtectionKeyPath. - Undecryptable ciphertext (key ring lost) falls back rather than throwing: an ingestion worker on a timer should degrade, not crash. - The stored secret is never sent to the browser; a blank field means "unchanged", not "cleared". - MQTT usernames are stored as-is — §6.4 covers tokens and passwords, and encrypting a username would only blank the field on every edit. - ExportService drops *_enc values: bound to the originating key ring, so useless where an export would be restored. Expect to re-enter after a restore. - HaConnectionTester now takes a resolved token, so the admin UI can test a token that has been typed but not yet saved. SDD §6.4 and §9 updated to describe both forms rather than contradict the code. Claude-Session: https://claude.ai/code/session_01V6joyergfvVLFEizH1hJLd |
||
|
|
8550ed8d9e |
Ingestion: Home Assistant WebSocket push path
ci / build-test (push) Successful in 1m16s
HomeAssistantWebSocketWorker holds a persistent state_changed subscription per HA endpoint that opts in via the connector's WebSocket toggle (HaEndpointConfig.UseWebSocket): auth handshake, subscribe, ingest in real time, capped-backoff reconnect. The REST poll worker skips WS endpoints so each is served once. HaWebSocketProtocol holds the pure handshake/parse logic. Verified by 11 protocol unit tests + a live integration test against an in-process fake HA server. CLAUDE.md updated. Claude-Session: https://claude.ai/code/session_01Kib2MniVFbD95fkgLgBBnB |
||
|
|
85d650a8f5 |
Fix: enable global interactivity so dialogs/selects actually work
ci / build-test (push) Successful in 1m12s
Editing a meter (any admin dialog, select dropdown, snackbar, dark-mode toggle) did nothing: pages declared @rendermode InteractiveServer individually, but MainLayout — which hosts MudDialogProvider/MudPopoverProvider/MudSnackbarProvider — was rendered by <Routes>, which was static. Inline MudDialogs and MudSelect popovers render through those providers, so with the providers non-interactive no dialog could ever open. Set the render mode on <Routes> and <HeadOutlet> in App.razor (global interactivity) and removed the now-redundant per-page @rendermode declarations (they would otherwise throw "parent already has a render mode"). Why it slipped through: the integration render tests only issue a GET (static prerender), which never exercises the SignalR circuit. Verified this fix with a real headless-browser run (Playwright): the meter edit dialog opens and the "Sub-meter of (upstream meters)" multi-select opens with options — proving the layout providers are now interactive. 116 tests still green. Claude-Session: https://claude.ai/code/session_01Lz2RqAsnQhetqWNoCDfexK |
||
|
|
09cd435c2b |
Admin write-CRUD, Home Assistant connector config, wiring audit
ci / build-test (push) Successful in 1m19s
Three requested phases.
1) Admin section (SDD §8.7) — MudBlazor inline-dialog CRUD, consistent pattern,
delete guards, snackbar feedback, shared Confirm helper:
- Energy types: create/edit/delete (blocks delete when meters reference it).
- Meters: create/edit/delete; recomputes consumption when mode/baseline
changes (NormalizationService over a fresh factory context, in a tx);
delete cascades data (consumption+readings are Restrict → removed first).
- A meter's ingest sources: manage on the meter-detail Sources tab
(add/edit/delete MQTT/Tasmota/HA sources with typed config).
- Tariffs: full CRUD (scope/component/value/validity).
- Cost categories: CRUD + member management (meter or energy-type members).
- Connectors: ingestion_endpoint CRUD (MQTT broker + Home Assistant);
secrets referenced by env-var name only, never stored.
- Settings: read-only effective-config view (settings are env-driven and
reproducible, so an editable form would change nothing — kept honest).
PV role is now editable on meters (MeterMeta.SetRole can clear a role).
2) Read Home Assistant — extracted a shared public HaEndpointConfig (was a
private record in the worker), added HaConnectionTester (powers the connector
"Test connection": checks base URL + env-resolved token, optionally reads one
entity). Configuring an HA connector + an HA source on a meter drives the
existing REST-poll worker end to end. (WebSocket push stays a future
optimization; REST poll already reads HA.)
3) Wiring/placeholder audit — swept every OnClick/Href: all handlers are real,
all internal links resolve to real routes, no TODO/stub/placeholder code.
Fixed one genuine gap: MainLayout had no drawer toggle, so the nav was
unreachable on narrow screens — added a hamburger button.
Tests: +6 (MeterMeta.SetRole role-removal; HaConnectionTester fail-closed
guard branches with a throwing HttpClientFactory proving no network on bad
config); render test now covers all admin routes. 69 Core + 45 Integration =
114 green. Live-verified in Docker: all admin pages 200, drawer toggle present,
Settings shows real effective config.
Claude-Session: https://claude.ai/code/session_01Lz2RqAsnQhetqWNoCDfexK
|
||
|
|
a6edec2b12 |
Polish/audit: fix bugs found by 3 subsystem audits
ci / build-test (push) Successful in 2m45s
Correctness/data: - Fix demo cost double-count: reference importer no longer imports the Kosten Strom/Wasser columns for categories that are metered (only Heizung), so Wasser rollup is 70€ not 140€. - Spurious-decrease guard: only a reset/swap in the window (prevReading, thisReading] explains a decrease — an old historical reset no longer permanently disables the guard. - Gate swap auto-detection on MappingProfile.DetectCumulativeSwaps (flag was ignored). - Prorate basePrice by bucket length (day/month/year); guard virtual expressions against NaN/Inf. Concurrency/infra: - Blazor: register a DbContextFactory; CostService/DashboardService and the read pages now use short-lived per-operation contexts (no shared circuit DbContext); guard Trends re-entrancy. - /events: wrap event insert + consumption recompute in one transaction (atomic); 404 (not 500) on unknown meter. - MQTT worker: subscribe to newly-added topics on each tick; move client cleanup into finally. - Migrations: CREATE MATERIALIZED VIEW IF NOT EXISTS + if_not_exists on CAgg/compression/ hypertable calls (re-run-safe after a mid-migration crash). - HA worker: prune stale poll-schedule entries; export: null dangling ImportBatchIds on restore. API/security: - API fail-closed by default: with no keys and AllowAnonymousApi off, /api/v1 returns 401 (protects /export and /import). New MeterVault:AllowAnonymousApi opt-in. - Cap /readings batch at 5000; report ignored (unknown-meter) count; enums as strings in JSON. +4 regression tests (guard window, API closed, /events 404, no demo double-count). 98 tests green; Docker deploy re-verified healthy with the API fail-closed. Claude-Session: https://claude.ai/code/session_01WujdMtMJPbxDpDnMeK22rr |
||
|
|
d5419729e5 |
M5: Blazor dashboard (MudBlazor + ApexCharts)
- MudBlazor theme (dark default) + responsive drawer/appbar layout + nav. - DashboardService read model: KPIs with period-over-period deltas, category breakdown, "what cost more/less" difference view, monthly trends. - Pages: Overview (KPI cards + DeltaChip + donut + difference table), Trends (range-select bar chart), Meters (list + source status), Import (load reference dataset + CSV dry-run preview), Admin (energy types, tariffs). Charts isolated into components to avoid the ApexCharts/MudBlazor Color/Format name clashes. - ReferenceDataImporter: one-click load of all four sheets as a starter dataset (meters, tank, tariff history, category memberships) — bundled sample CSVs copied to app output. - End-to-end render test: import creates meters + consumption; overview/meters/trends/ import/admin pages all return 200 with KPI cards rendered. 92 tests green (56 Core + 36 integration). Deferred to polish: dedicated PV & oil/consumable panels, meter-detail page, full admin CRUD, prev-year trend overlay. Claude-Session: https://claude.ai/code/session_01WujdMtMJPbxDpDnMeK22rr |