c0f52dbb6f58a4e658246308164bbf844f742ac4
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Florian Schmidt
|
aacdc28d70 |
Meters: record events from the UI, and book consumption in the months it accrued in
Two threads that ended up in the same files. One is navigation: a meter
swap that happened today had no click path at all, and most per-meter
tasks were reachable only by knowing which admin page owned them. The
other is attribution: readings on 1 August and 16 September showed six
weeks of water under September and nothing under August.
Meter events from the UI
Swap, counter reset, tank level, delivery and note are recorded through
MeterEventService rather than ad-hoc inserts, so the dialog's verdict and
the saved result come from the same Validate call, and every record or
delete recomputes the meter inside one transaction. MeterEventRules
decides which events a mode offers -- a tank has no register to swap, and
Correction is offered nowhere because nothing reads it.
A swap is stored as the event at T plus a manual reading of the new
register's start value at exactly T. That pairing is the whole trick: the
boundary window is (previousReading, reading], so the old register's tail
books at T and every later reading counts from the new start. Writing the
old final value as the reading at T instead -- the obvious thing -- double
counts the tail and then rejects every reading the new register produces.
Deleting a swap removes that start reading only while it is still the
untouched start value, and only Manual readings can be deleted at all.
Navigation
The meter page is now the hub: primary entry by mode, a "Record event"
menu, and Edit through a shared MeterEditor that also owns tank setup.
Other pages link into it with MeterLinks (/meters/{id}?tab=...&action=...),
whose action is consumed once after the interactive render and dropped
from the address -- the reverse order flashes the dialog and closes it,
because a circuit's first location change dismisses every open dialog.
The app bar gains a "Find a meter" dialog with the same quick entry.
A source that has no usable connector now links to creating (or enabling)
one and comes back to the same source dialog with the connector picked
and everything typed still there; the draft survives in a circuit-scoped
DraftStore, and the way back is a meter id rather than a URL, so the page
cannot be made to redirect anywhere else. The connector list shows which
meters use each connector, import batches list the meters and categories
they wrote to, the meter editor owns the meter's own cost categories, and
the dashboard's empty cost panel names the first missing step instead of
listing every admin page.
Months
A reading is an instant, and what it measures accrued over the time since
the previous one. Booking the whole delta at the closing reading misfiles
it whenever the interval crosses a month boundary, so a plain increase is
now divided at local month boundaries in proportion to elapsed time, each
share stamped inside its month and marked estimated: the meter recorded a
total, not a shape. The parts always sum to the original.
Imported monthly tables are the exception that keeps the golden fixtures
reconciling. "Mai 2026" carries the register at the end of May but is
stamped on the 1st, so the importer -- the only place that still knows
whether the date cell named a month or a day -- flags it MonthLabel, and
the engine reads it as the end of its month. Inferring that from the
stamp instead would catch day-dated rows: a sheet with "01.08.2026" in it
is not a monthly table, and reading it as one moves two thirds of July
into August.
ReadingTimeline is the single ordering built on that: effective time,
then stamp. The register normalizers walk it, and so do the decrease
guard and the event dialog, which is what stops them disagreeing about
which reading is "previous" -- a sheet imported after live readings of the
same month used to count that month twice, and a mid-month reading below
the month's end value was rejected as a drop. A swap detected in a
monthly table applies from the start of that local month, i.e. to the
first reading in it, and a recorded start value never counts above the
reading it lands on.
Every reader buckets in the configured timezone rather than a hardcoded
one, and turns a requested date into that zone's local midnight, so the
divided shares are read back under the months they were stamped in. The
zone id is normalised to its IANA form, because .NET accepts a Windows id
that PostgreSQL will not bucket by, and both are checked at startup.
Stored consumption is derived, so a rule change reaches a meter only at
its next reading -- weeks, for a meter read monthly. NormalizationUpgrade
records the revision and zone the stored series was built with and
rebuilds everything once at startup when either differs, each meter in
its own transaction. A meter that fails is logged, kept in
normalization_pending and retried at the next start: one bad series must
never keep the application down.
What an operator sees once
Existing charts change on the first start after the update: months that
carried a neighbour's use give it back. Rows of earlier imports from
monthly tables are marked as such before anything is recomputed, and if
that marking fails nothing is rebuilt or recorded, so the upgrade simply
runs again next time rather than shifting every imported month by one. A
wizard import whose date format was left on auto-detect is treated as a
monthly table when all of its rows sit on the 1st across at least two
months -- exactly how those rows were attributed before -- and each such
batch is named in the log, because a day-dated sheet always read on the
1st looks identical; revert and re-import it with the day format if that
is what it was.
Tests: 120 unit and 230 integration, including the reference fixtures,
which still reconcile month for month.
|
||
 schmidt.florian
|
af786c7b28 |
Meters: add manual reading entry from the meter-detail Readings tab
ci / build-test (push) Successful in 1m37s
Entering a reading by hand previously meant POST /api/v1/readings with an API key, or a one-row CSV through the import wizard. SourceType.Manual existed in the enum but nothing was behind it. This adds the click path, built for the case it is actually used in: walking to each manual meter with a phone in hand. "Add reading" on the Readings tab opens a dialog prefilled with the meter's last register value and the current local time, both editable: - An on-screen keypad, because a register is read standing at the meter. It behaves like a calculator against the prefill - the first digit replaces it (a fresh register), while backspace edits it in place, which is the common case since only a register's last digits move. - Typed input accepts both separators (last one wins), so a German and an English phone keyboard both do the right thing. ReadingEntry owns that rule and is unit-tested; it deliberately differs from GermanNumber, where a lone dot really is a thousands separator. - A live parsed-value echo plus delta-since-last, which is the net that catches a mistyped digit before it is committed. - Decrease / replaces-existing / future / backdated surfaced before saving, and DST spring-forward gaps refused rather than shifted. The verdict line sits in a fixed-height, no-wrap slot above the keypad. That is load-bearing, not cosmetic: an alert that appears there when the value dips below the last reading moves the keys out from under the user's thumb mid-entry, which is a guaranteed mistype on a phone. The long-form explanation goes below the keypad, where reflow is harmless. Saving goes through IngestionService.IngestByMeterAsync, so the monotonic-decrease guard and inline renormalization apply exactly as for any other ingest. A new optional quality parameter stamps the row ReadingQuality.Manual; null preserves today's behaviour, so a source re-reporting the same timestamp updates the value without silently relabelling a hand-entered or imported reading. Also: the meter-detail tabs now render times in the instance timezone per SDD section 10, instead of raw UTC. Without it a reading entered at 18:00 reads back as 16:00. Side effect is that historic imported monthly rows show 01:00/02:00 rather than 00:00 - correct, if noisier. Claude-Session: https://claude.ai/code/session_01D4x3JbNKCSV4cBR9s7bJmX |
||
|
schmidt.florian
|
8fe5f4411b |
Fix defects found auditing the ingestion, import and connector changes
ci / build-test (push) Successful in 1m16s
An audit of this session's commits found several real problems, three of which
lose or expose data. Ordered by severity.
Live recompute was not atomic. RecomputeMeterAsync clears a meter's series with
ExecuteDelete, which commits by itself when no transaction is ambient, and only
then adds the rebuilt rows. Between the two the meter had *no* consumption:
a dashboard read reported zero, and a crash or cancelled request made the loss
permanent, for data the SDD treats as the long-term source of truth (§5.5).
Import and the events API already wrapped their recomputes; live ingestion,
which I added this session, did not. Now shares one transaction, joining an
ambient one rather than nesting.
The MQTT backfill migration counted brokers without regard to is_enabled. One
live broker plus a disabled leftover counted two, declined to backfill, and left
those sources unbound — which under endpoint-scoped routing means silently and
permanently dead. The "two or more is ambiguous" reasoning did not hold there:
the worker only ever connected to enabled endpoints. Corrected by a follow-up
migration rather than an edit, since the original may already have run; it
touches only rows still NULL, so hand-made bindings are safe.
A mapping edited after a dry run committed the *old* staged rows under the
*new* mapping. Readings went to the previous meter while the batch recorded the
current mapping — wrong data, provenance contradicting it, no exception. The
earlier fix re-validated but did not detect staleness. Commit now compares the
mapping against the one the preview was staged under and refuses.
"Test connection" sent a stored token to whatever Base URL was in the dialog.
Encrypting secrets at rest means the UI can decrypt what the operator can no
longer read, so this turned the button into an exfiltration primitive: point it
at any host, the token arrives as a Bearer header. A stored token now only goes
to the origin it was saved for; testing elsewhere requires typing it again.
A source that cannot ingest looked identical to a healthy one. Endpoint-scoped
routing made unbound and mis-bound sources silently dead, while the Sources tab
showed no connector at all and the delete dialog still promised sources would be
"unlinked". Added a Connector column that names the fault, stopped offering
disabled connectors (both workers filter on IsEnabled), and made the delete
warning say ingestion stops.
Virtual meters rendered four zero tiles: they evaluate on read and only
materialize when a cost category references them (§14.1), so summing
consumption is a confident lie about a working meter. They now report nothing
and the page explains why.
Re-importing an overlapping file failed at the database with EF's "An error
occurred while saving the entity changes", naming neither meter nor date — the
diagnosis problem
|
||
|
schmidt.florian
|
cedd60ab45 |
Audit fixes: batch recompute, negative-baseline percentages, key-ring persistence
ci / build-test (push) Successful in 1m17s
Three defects found reviewing the last few commits. Deriving consumption on ingest made the batch reading endpoint quadratic. A recompute rewrites a meter's entire consumption series, and POST /api/v1/readings ran one per reading -- 500 readings for one meter meant 500 full rewrites. IngestByMeterAsync takes renormalize:false and the endpoint normalizes each touched meter once after the batch. Percentage change divided by a possibly negative baseline. A net-export meter going from -100 to -150 exported half again as much and would have been reported as "+50%", reading as more consumption. A non-positive baseline now reports no basis rather than a confident lie. The data-protection key ring had no persistent home outside Docker Compose. The LXC installer now creates /var/lib/metervault/keys at 0700 -- the app would otherwise create it under the default umask, leaving a key ring world-readable -- and the Unraid template maps it, since without that every UI-entered secret was lost whenever the container was recreated. README documents the variable and the trust boundary: keys on disk protect against leaked database content, not against an attacker who already has the host. Claude-Session: https://claude.ai/code/session_01V6joyergfvVLFEizH1hJLd |
||
|
schmidt.florian
|
62d102c335 |
Ingestion: derive consumption on ingest, and poll HA in minutes not seconds
ci / build-test (push) Successful in 1m13s
Live ingestion wrote the raw reading and stopped there. Import, the REST push endpoint and the meter editor all recompute afterwards; the MQTT/Tasmota/HA path was the one that did not, so a polled reading landed in `reading` and every derived figure stayed frozen at the last import. Observed on a GenerationCounter: 45 readings, 44 consumption rows, generation pinned to the register value of the last imported reading. Recompute inline rather than behind a debounce. Normalizing a whole meter is cheap at metering cadence and a background dirty-set worker is machinery this does not yet need; the remark on RenormalizeAsync records when it would. Fixes a latent bug this surfaced in NormalizationService: ExecuteDelete drops the consumption rows in the database but leaves them in the change tracker, so a second recompute on the same context threw an identity conflict on (meter, time, kind). One worker scope ingesting two readings was enough to hit it. Detach the stale entries after the delete. Poll interval is now minutes, default 60, replacing seconds/60. A meter answers "how much this month, what will it cost" — an hourly sample answers that exactly as well as a per-second one, with far less raw volume (SDD §5.5). The `pollSeconds` key no longer binds, so existing sources fall back to the 60 default and move from every-60-seconds to hourly, which is the intent. A source that had deliberately set e.g. 300 seconds also lands on 60 minutes. Two test cleanups now delete consumption before the meter: live ingestion never produced any before, so the FK had nothing to trip on. Claude-Session: https://claude.ai/code/session_01V6joyergfvVLFEizH1hJLd |
||
|
schmidt.florian
|
a6edec2b12 |
Polish/audit: fix bugs found by 3 subsystem audits
ci / build-test (push) Successful in 2m45s
Correctness/data: - Fix demo cost double-count: reference importer no longer imports the Kosten Strom/Wasser columns for categories that are metered (only Heizung), so Wasser rollup is 70€ not 140€. - Spurious-decrease guard: only a reset/swap in the window (prevReading, thisReading] explains a decrease — an old historical reset no longer permanently disables the guard. - Gate swap auto-detection on MappingProfile.DetectCumulativeSwaps (flag was ignored). - Prorate basePrice by bucket length (day/month/year); guard virtual expressions against NaN/Inf. Concurrency/infra: - Blazor: register a DbContextFactory; CostService/DashboardService and the read pages now use short-lived per-operation contexts (no shared circuit DbContext); guard Trends re-entrancy. - /events: wrap event insert + consumption recompute in one transaction (atomic); 404 (not 500) on unknown meter. - MQTT worker: subscribe to newly-added topics on each tick; move client cleanup into finally. - Migrations: CREATE MATERIALIZED VIEW IF NOT EXISTS + if_not_exists on CAgg/compression/ hypertable calls (re-run-safe after a mid-migration crash). - HA worker: prune stale poll-schedule entries; export: null dangling ImportBatchIds on restore. API/security: - API fail-closed by default: with no keys and AllowAnonymousApi off, /api/v1 returns 401 (protects /export and /import). New MeterVault:AllowAnonymousApi opt-in. - Cap /readings batch at 5000; report ignored (unknown-meter) count; enums as strings in JSON. +4 regression tests (guard window, API closed, /events 404, no demo double-count). 98 tests green; Docker deploy re-verified healthy with the API fail-closed. Claude-Session: https://claude.ai/code/session_01WujdMtMJPbxDpDnMeK22rr |
||
|
schmidt.florian
|
9abc2937c2 |
M6: REST API + API-key auth + OpenAPI
- Minimal API under /api/v1 (SDD §9): POST /readings (idempotent HA push), GET /meters, /energy-types, /consumption, /cost, /dashboard/summary, POST /events (records + recomputes), GET+POST /tariffs, GET /sources/status. - IngestionService.IngestByMeterAsync for direct REST push (batch-safe upsert via Local cache). - ApiKeyFilter: X-Api-Key enforced against configured keys (open only when none set). - ReverseProxyTrust middleware: adopt X-Forwarded-User/Remote-User behind Authelia/Traefik. - Swagger/OpenAPI (Swashbuckle) at /swagger. - Tests: push rejected without key (401), accepted + persisted with key; meters + swagger live. 94 tests green (56 Core + 38 integration). Claude-Session: https://claude.ai/code/session_01WujdMtMJPbxDpDnMeK22rr |
||
|
schmidt.florian
|
4b0cad67df |
M3: live ingestion (MQTT/Tasmota + Home Assistant)
- PayloadExtractor: dot-path value/time extraction (Tasmota ENERGY.Total, bare scalars). - MqttTopicMatcher: standard +/# wildcard matching. - IngestionService: scale/offset, idempotent upsert on (meter_id, time), and a spurious- decrease guard for monotonic registers (allowed only with a reset/swap event) + source last-seen status. - MqttMessageRouter + MqttIngestionWorker (MQTTnet 5): per-endpoint persistent connections, topic subscription, graceful degradation; secrets resolved by env-var reference. - Home Assistant: HaStateClient (REST /api/states parse) + HomeAssistantWorker polling on each source's interval. HA-via-MQTT also works through the MQTT path. - Ingestion workers gated by MeterVault:EnableLiveIngestion (off in tests). 85 tests green (53 Core + 32 integration): Tasmota payload → reading verified end to end. Follow-up (polish): HA WebSocket push (state_changed) as an alternative to REST poll; source-topic index caching in the router. Claude-Session: https://claude.ai/code/session_01WujdMtMJPbxDpDnMeK22rr |