diff --git a/CLAUDE.md b/CLAUDE.md index b8f725c..8dc7bbe 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,7 +7,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co MeterVault is a self-hosted, local-first energy & utility metering platform: it ingests meter data from Home Assistant, Tasmota and MQTT on a schedule, stores every reading timestamped and immutable, normalizes it into consumption, and turns it into cost dashboards. Energy types (electricity, water, heating oil, gas, …) and meters are **user-defined, never hardcoded**. **Status: implemented (M0–M7) + the dashboard/analysis rework (next release 0.4.0).** -- **Size:** five projects, ~2,480 tests (Core 1,733, Integration 746), working Docker deploy. +- **Size:** five projects, ~2,490 tests (Core 1,733, Integration 753 incl. 2 opt-in performance facts), working Docker deploy. - **Docs:** - `docs/SDD.md` is the design reference. It marks in place every section the system now deviates from (list: note D-58). Its milestone map (§12) matches the git history (M0…M7). - The rework's work order is `docs/DASHBOARD_ANALYSIS_CHANGE_BRIEF.md`. Its decisions are D-01…D-58 and amendments A-01…A-39 in `docs/ANALYSIS_IMPLEMENTATION_NOTE.md`. **Read that note before touching analysis, costing, rollups, virtual meters or the analysis pages.** @@ -160,7 +160,7 @@ sources (Tasmota/HA/MQTT/manual/CSV) - Then it rebuilds consumption + rollups + coverage + state of **every** meter when the revision or zone differs. Otherwise it rebuilds only meters whose `meter_rollup_state` is missing or outdated, plus `normalization_pending`. Virtual meters are purged, since they store nothing. Each meter runs in its own transaction. - A meter whose oldest consumption predates its oldest reading or event is skipped and logged instead of truncating history. A failing meter is logged, kept pending and retried at the next start. - Until its rebuild runs, a meter reads as `Pending` ("analysis being prepared"), never "no data". **Bump `CurrentRevision` whenever the engine books existing readings differently.** The rebuild runs before the web server listens; roughly 0.1 s per monthly meter and ~1.4 s per meter with a year of hourly data (it grows with the reading count). -- **Dashboards and charts read rollups only — never `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). `consumption` + rollups are the analytical history. `reading` is read by the paged Readings tab, the manual-entry checks and the freshness query (latest 20 reading times per meter, D-18). **Raw retention is not enforced** (D-57, a documented blocker): every recompute rebuilds a meter from its readings, so dropping old readings would destroy history. `/admin/settings` and the Readings tab say so. +- **Dashboards and charts read rollups only — never `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). `consumption` + rollups are the analytical history. `reading` is read by the paged Readings tab, the manual-entry checks, and — only for a meter with a live source, only over the last 90 days — the rhythm query behind D-18 (A-40). The freshness *mark* is `meter_rollup_state.last_reading_at`, written by the recompute behind every write path, so an import-only meter's mark is exact without touching the raw chunks. **Raw retention is not enforced** (D-57, a documented blocker): every recompute rebuilds a meter from its readings, so dropping old readings would destroy history. `/admin/settings` and the Readings tab say so. - **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries − usage + forecast), `direct_delta`, `instant_rate`, `virtual` (a formula over other meters, **evaluated on read, never stored**: D-27, SDD §14.1). New ingestion/normalization logic dispatches on mode. `NormalizedQuantity` (D-20) gives each meter's analysis (kind, unit): - runtime: `h`, or the tank unit with a fixed rate; - instant rate: the rate unit without `/h` (W → Wh); diff --git a/docs/ANALYSIS_IMPLEMENTATION_NOTE.md b/docs/ANALYSIS_IMPLEMENTATION_NOTE.md index e7dcde9..5c97d68 100644 --- a/docs/ANALYSIS_IMPLEMENTATION_NOTE.md +++ b/docs/ANALYSIS_IMPLEMENTATION_NOTE.md @@ -8,6 +8,7 @@ The note was kept current through Phase 5: - §11: amendments from the Phase 1 module review. - §12: amendments from the acceptance review. - §13: decisions recorded with the final documentation. +- §14: amendments from the performance measurement. - §9 and §10: extended with what the implementation measured and changed. The outcome is in [ANALYSIS_REPORT.md](ANALYSIS_REPORT.md), the user-facing changes in @@ -708,3 +709,32 @@ figure. keep the existing detour and its draft. - The theme defaults to dark when no `mv-theme` cookie is set. - The Calculation tab words a calculation problem as the attention list does, one wording per `VirtualProblemKind`. + +## 14. Amendments from the performance measurement + +The measurement of brief §9.10 / D-56 on a synthetic 1,000-meter × 10-year instance named three costs that grow with +history rather than with what a page asks for. They are fixed here. No golden bill, reconciliation figure or displayed +value changes; only what a request reads does. + +- **A-40 What a request reads (D-15, D-18, D-56).** + - **The freshness mark is stored, not searched.** `meter_rollup_state` gains `last_reading_at`: the stamp of the + meter's latest raw reading, written by the recompute that every write path already runs. D-18 is unchanged — the + last reading or event time is still the mark — but a request no longer queries `reading` to find it, so an + import-only meter's mark stays exactly as old as its data without planning across a decade of raw chunks. The + migration backfills the column in one pass, so nothing waits for a rebuild. + - **Only a live source's rhythm is sampled, and only from the recent past.** The median of D-18 is consulted for a + meter with a live source, so the reading times are read for those meters alone, bounded by + `FreshnessRules.RecentWindow` (90 days) — the bound is what lets PostgreSQL exclude the older chunks at plan time. + A live meter that delivered nothing inside the window has no rhythm there; those few meters are read again over + their whole history, so a long-silent source is still called stale by its own rhythm. An instance without any live + source reads `reading` not at all. + - **The window-sum statement carries its overall bounds.** Its windows arrive through an `unnest` join, so their + bounds are columns and exclude no chunk. The minimum start and maximum end of the window set are repeated as + constants in the `WHERE` clause. No row outside them can match any window, so no tally changes; with enough + windows it is the difference between an indexed probe and a parallel scan of the whole hypertable with a sort + spilling to disk. + - **One Overview load holds one catalog.** The page's quantities, its bill and its comparison's bill are three + figures of one period. `DashboardService.GetOverviewAsync` loads the meters, tanks, links and rollup states once + and passes that snapshot to all three (`CostReader.ReadAsync(db, catalog, …)`), so every figure answers from the + same snapshot and the load is not repeated. The cards, the change table and the composition are derived from those + results and are never priced again. `OverviewReadBudgetTests` asserts the statement count of one load. diff --git a/docs/ANALYSIS_REPORT.md b/docs/ANALYSIS_REPORT.md index d9083d3..15ff922 100644 --- a/docs/ANALYSIS_REPORT.md +++ b/docs/ANALYSIS_REPORT.md @@ -10,8 +10,9 @@ decisions were applied to existing data, what was tested, and what is still miss behaviour is deliberate, the decision id says so. - **User-facing changes and the upgrade path:** [RELEASE_NOTES.md](RELEASE_NOTES.md). This report does not repeat them. -- **Build state at the end:** `dotnet build` 0 errors; Core.Tests 1,733 of 1,733 and Integration.Tests 746 of 746 - passing; the golden spreadsheet reconciliation and the seeded bill goldens unmoved by the last fix round. +- **Build state at the end:** `dotnet build` 0 errors; Core.Tests 1,733 of 1,733 and Integration.Tests 751 of 751 + passing (plus the two opt-in performance facts, skipped); the golden spreadsheet reconciliation and the seeded + bill goldens unmoved by the last fix round. ## 1. Findings A01 – A14 @@ -176,10 +177,11 @@ not repeated here. | Suite | Count | Needs | Covers | |---|---:|---|---| | `tests/Core.Tests` | 1,733 | nothing | parsers, normalizers, swap→12, and `Analysis/`: periods, DST, comparisons, buckets, coverage, rollups, totals, category cover, virtual formulas and evaluation, the cost calculator | -| `tests/Integration.Tests` | 746 | Docker (TimescaleDB) for the database facts | reconciliation against the four golden CSVs, import commit and revert, ingestion and events, rollups, the reader, the cost engine, the seeded bill, API contracts, the export, rendered pages, plus the pure UI-model tests under `Analysis/`, `MeterPage/`, `Overview/`, `Editor/` and `Specialized/` | +| `tests/Integration.Tests` | 751 | Docker (TimescaleDB) for the database facts | reconciliation against the four golden CSVs, import commit and revert, ingestion and events, rollups, the reader, the cost engine, the seeded bill, API contracts, the export, rendered pages, plus the pure UI-model tests under `Analysis/`, `MeterPage/`, `Overview/`, `Editor/` and `Specialized/` | | `tests/Integration.Tests/Performance` | 2 | `METERVAULT_PERF=1` | the synthetic 1,000-meter × 10-year dataset, reader and cost timings, statement counts, query plans; skipped by default | -All green at the end of the last fix round, where Integration rose from 727 to 746. +All green at the end of the last fix round, where Integration rose from 727 to 746, and again after the performance +fixes of §5.4, which added five more. ### 4.2 Kinds of tests @@ -281,31 +283,34 @@ Measured with the opt-in Performance trait (`METERVAULT_PERF=1`, types, 21 links, 109 tariff rows, 64 manual costs and 20 virtual meters nested up to three levels. Hardware: Ryzen 9 9950X3D (16 cores, 32 threads), 64 GB, Windows 11, Docker Desktop, PostgreSQL 16.6 with -TimescaleDB 2.17.2 (`jit=off`), .NET 10 Release. **These numbers are preliminary**: they were taken while other -agents were building and testing on the same machine (host 20–40 % CPU). They are re-runnable as above. +TimescaleDB 2.17.2 (`jit=off`), .NET 10 Release. The figures below are the final run, taken on an otherwise idle +machine (host 7–13 % CPU) after the fixes of §5.4. The "first run" column is the earlier measurement, taken before +those fixes and while other work was building and testing on the same machine — it is kept to show what the fixes +moved, not as a like-for-like comparison. ### 5.1 Reader and cost requests Median and p95 of 10 runs after 2 warm-ups; SQL is the number of commands one extra, untimed run sent. -| Request | Median ms | p95 ms | SQL | -|---|---:|---:|---:| -| 100 selected meters, 10 years by month, catalog cached — **the brief's 2 s target** | 374 | 509 | 8 | -| Portfolio, last 12 months by month (measures only, as the Overview) | 268 | 284 | 12 | -| … with one series per meter (1,000 series) | 379 | 403 | 12 | -| … with the previous-year comparison | 414 | 458 | 14 | -| One energy type (402 meters), 10 years by month | 313 | 347 | 12 | -| … with one series per meter (the type page's table) | 917 | 1,016 | 12 | -| One meter: 10 years by week, or 365 days by day | 29–36 | 32–58 | 11 | -| Portfolio bill, 12 months by month, with categories | 320 | 357 | 16 | -| Portfolio bill, 10 years by month | 838 | 875 | 15 | -| Virtual meter nested three levels, 10 years by month | 245 | 272 | 11 | -| Virtual difference of a 40-meter and an 8-meter sum, 10 years by month | 377 | 442 | 11 | -| Catalog load: 1,000 meters, tanks, links, states, validation, classification | 7 | 8 | 4 | -| 1,000-meter selection with the limit raised, 12 months by month | 353 | 394 | 12 | -| Refused: 1,000 meters against the 6-series limit; portfolio or bill by day over 10 years | 0 | 0 | **0** | +| Request | Median ms | p95 ms | SQL | First run, median | +|---|---:|---:|---:|---:| +| 100 selected meters, 10 years by month, catalog cached — **the brief's 2 s target** | 286 | 472 | 8 | 374 | +| … the same request through the public API | 274 | 317 | 12 | 361 | +| Portfolio, last 12 months by month (measures only, as the Overview) | 70 | 75 | 12 | 268 | +| … with one series per meter (1,000 series) | 170 | 186 | 12 | 379 | +| … with the previous-year comparison | 156 | 177 | 14 | 414 | +| One energy type (402 meters), 10 years by month | 156 | 182 | 12 | 313 | +| … with one series per meter (the type page's table) | 698 | 731 | 12 | 917 | +| One meter: 10 years by week, or 365 days by day | 12–16 | 13–27 | 10–11 | 29–36 | +| Portfolio bill, 12 months by month, with categories | 144 | 204 | 16 | 320 | +| Portfolio bill, 10 years by month | 577 | 627 | 15 | 838 | +| Virtual meter nested three levels, 10 years by month | 197 | 216 | 10 | 245 | +| Virtual difference of a 40-meter and an 8-meter sum, 10 years by month | 300 | 344 | 11 | 377 | +| Catalog load: 1,000 meters, tanks, links, states, validation, classification | 6 | 10 | 4 | 7 | +| 1,000-meter selection with the limit raised, 12 months by month | 151 | 171 | 12 | 353 | +| Refused: 1,000 meters against the 6-series limit; portfolio or bill by day over 10 years | 0 | 0 | **0** | 0 | -The brief's target is met with margin: 374 ms median and 509 ms p95 against 2,000 ms. The statement count per +The brief's target is met with margin: 286 ms median and 472 ms p95 against 2,000 ms. The statement count per request is constant at 8–16 whether the request covers 1, 100 or 1,000 meters — there is no per-meter query storm. Refused requests send no SQL at all, because the 400-point and 6-series limits are checked before any query is built. Rollup reads use the primary-key index: 4 ms for 125 meters over 10 years of month rollups, 7 ms for 980 @@ -313,10 +318,10 @@ meters over 11 months. ### 5.2 Rebuild and per-reading recompute -Rebuilding all 1,000 meters at startup through `NormalizationUpgrade` took 344–392 s: about 0.1 s per monthly -meter, ~0.6 s for a daily meter (3,670 readings) and ~1.4 s for a meter with a year of hourly data (9,314 -readings). That is also the cost of one ingested reading, because every write path recomputes its meter in full -(D-57). +Rebuilding all 1,000 meters at startup through `NormalizationUpgrade` took 279 s, about 3.6 meters per second. +One meter's full recompute is 51–56 ms with monthly readings, 0.50 s for a daily meter (3,670 readings) and 1.16 s +for a meter with a year of hourly data (9,314 readings). That is also the cost of one ingested reading, because +every write path recomputes its meter in full (D-57). ### 5.3 Page loads, old against new @@ -334,24 +339,31 @@ per GET taken from a separate logging pass. The old Overview ran a meter load, a tariff load and a `time_bucket` query 8,021 times per request. The individual meter pages and `/consumables` stayed in the same range (tens to low hundreds of milliseconds) and are not listed. -The new-side page numbers were taken on the foundation-era pages, before the page rework landed, so they are an -order of magnitude rather than a final figure. +The new-side numbers in that table came from the foundation-era pages on a busy machine, so they are an order of +magnitude rather than a final figure; the finished pages, measured on the idle machine against the same +1,000-meter instance, load in 455 ms (`/`), 383 ms (`/meters`), 469 ms (`/solar`), 641 ms (`/energy/1`), +684 ms (`/trends`), 104 ms (`/meters/1`) and 152 ms (a nested virtual meter). -### 5.4 Hotspots found and not fixed +### 5.4 Hotspots found and fixed -1. **The freshness query has no time bound.** `AnalysisQueries.RecentReadingsAsync` (the latest 20 reading times - per meter, D-18) plans across every raw chunk: 14–24 ms of planning over 123 `reading` chunks, 189 ms on a cold - connection, and 38–46 ms of execution for a portfolio walking back through compressed chunks. With a constant - 90-day lower bound the same single-meter statement plans in 0.9 ms. It grows with history length. -2. **Window sums get no plan-time chunk exclusion.** `AnalysisQueries.WindowSumsAsync` takes its bounds only from - the `unnest` join. Real requests send few windows and cost 4.6 ms, but a stress case of ~2,000 windows flips to - a parallel sequential scan of 1.39 M rows with a sort spilling to disk, 138–152 ms. Latent, and it grows with - total history rather than with the number of windows. -3. **The reworked Overview computes several portfolio bills per request** — nine in the measured build — which - shows as repeated catalog, tariff and manual-cost loads (526 ms of SQL inside a 2.4 s page). +The first measurement found three. All three are fixed (A-40), each proved by `EXPLAIN` or a statement count +before and after, with the tallies unchanged. -In-process work dominates the rest and scales with series × buckets (roughly 10–16 µs per series bucket). That is -the budget that matters for very large exports and tables, not for a normal page. +| Hotspot | Before | After | +|---|---|---| +| **The freshness query had no time bound.** `AnalysisQueries.RecentReadingsAsync` planned across all 123 raw chunks on every request, for every meter. | 13.8 ms planning + 36.1 ms execution for the portfolio (980 meters); 189 ms on a cold connection | 0.58 + 0.44 ms, over 4 chunks and only the 40 meters with a live source; an instance without a live source never reads `reading` at all | +| **Window sums could not exclude chunks.** `AnalysisQueries.WindowSumsAsync` took its bounds only from the `unnest` join. | a 1,960-window stress case: 42 chunks, parallel scan of 1.39 M rows, ~45 MB sort spill, 121.5 ms | 5 chunks, nested-loop index scans, 8.7 ms | +| **The Overview loaded the catalog three times**, once per read (quantities, bill, comparison bill). | 31 statements per load; 64 statements and 133 ms of SQL on the 1,000-meter instance | 23 statements per load; 39 statements and 81 ms of SQL | + +The freshness fix splits what D-18 had conflated: the *mark* (a meter's last reading time) is now stored on +`meter_rollup_state` and maintained by every recompute, with a one-pass backfill in the `FreshnessMark` migration, +so an import-only meter keeps its years-old "last activity" without touching `reading`; the *rhythm* (the sample +intervals that decide stale versus live) is sampled inside a 90-day window, and only for meters that have a live +source. A live source that has been silent longer than that window is re-read unbounded, so it is still reported +stale by its own rhythm rather than by a default. + +In-process work dominates what is left, and scales with series × buckets (roughly 10–16 µs per series bucket). +That is the budget that matters for very large exports and tables, not for a normal page. ## 6. Remaining limitations and known gaps @@ -368,7 +380,9 @@ the budget that matters for very large exports and tables, not for a normal page **Further limitations, measured or decided during the work:** -- The three performance hotspots in §5.4 are known and unfixed. +- `AnalysisQueries.OpeningBalancesAsync` has the same unbounded shape the freshness query had (a lateral + `ORDER BY time LIMIT 1` over `consumption`, 6.4 ms in the heaviest measured request). It was left alone: smaller, + and `consumption` has 42 chunks against `reading`'s 123. - The billing basis (grid meter or household use) is chosen per energy type for all time and cannot switch month by month, because the category composition would need the same per-month basis to stay reconciled with the bill (A-17). diff --git a/docs/RELEASE_NOTES.md b/docs/RELEASE_NOTES.md index adeabfc..be922f0 100644 --- a/docs/RELEASE_NOTES.md +++ b/docs/RELEASE_NOTES.md @@ -24,15 +24,18 @@ Back up the database (`pg_dump`) first. state of each meter. - This runs **before the web server listens**. The app is unreachable meanwhile, and Compose may report the container unhealthy. Let it finish. - - The time grows with the number of raw readings. One meter took about 0.1 s with monthly readings, ~0.6 s with ten - years of daily readings (3,700 readings), and ~1.4 s with a year of hourly readings (9,300 readings). - - A synthetic 1,000-meter × 10-year instance (1.34 M readings) took 5.6–6.5 minutes (338–392 s) on a Ryzen 9 - 9950X3D. That run was preliminary, on a busy machine. It is about 20 % slower per meter than the 0.3.0 rebuild, - because rollups and coverage are written too. + - The time grows with the number of raw readings. One meter took about 0.05 s with monthly readings, ~0.5 s with ten + years of daily readings (3,700 readings), and ~1.2 s with a year of hourly readings (9,300 readings). + - A synthetic 1,000-meter × 10-year instance (1.34 M readings) took 4.7 minutes (279 s) on an idle Ryzen 9 9950X3D — + about 3.6 meters a second. It writes rollups and coverage on top of consumption, so it does more per meter than + the 0.3.0 rebuild did. - Progress is logged every 100 meters. A meter that fails is logged, kept pending and retried at the next start. A meter whose stored consumption is older than its oldest remaining reading is skipped and logged, so its history is not truncated. - Until a meter is rebuilt, its pages say "analysis being prepared", never "no data". +- **The migration records each meter's last reading time** in the analysis state table and fills it in for the meters + you already have, in one pass over the readings. It is what the pages show as "last activity" and what decides + whether a live source is late, so no page has to search the raw readings for it any more. - **Virtual meters without a formula** (such as a seeded *Summe Solar* from an earlier version) are converted at startup. When their incoming links name meters of one unit and kind, the implied sum is stored as an explicit formula (D-28). The log lists the converted meters, and the meters that still need configuration because their @@ -214,7 +217,7 @@ engine, as listed above: actuals stop at now, virtual meters are evaluated, and and offers the monthly view. - **Bonus, Discount and Tax tariffs are stored but not applied** (D-57). - **Every live reading recomputes its meter in full** (D-57). That is fine for monthly and daily meters, but costs - about 1.4 s per reading for a meter with a year of hourly data, and grows with history. + about 1.2 s per reading for a meter with a year of hourly data, and grows with history. - **Months cannot switch billing basis:** the billing basis (grid meter or household use) is chosen per energy type for all time (A-17). - **Batteries are not modelled.** Without a grid-export meter, Solar's feed-in is calculated, and labelled as such. diff --git a/src/Core/Analysis/Freshness.cs b/src/Core/Analysis/Freshness.cs index 05dbaea..fb61930 100644 --- a/src/Core/Analysis/Freshness.cs +++ b/src/Core/Analysis/Freshness.cs @@ -57,6 +57,13 @@ public static class FreshnessRules /// How many recent reading times the median is taken over: 20 intervals need 21 readings. public const int RecentReadingCount = 21; + /// + /// How far back a reader looks for those reading times (A-40). A live source's rhythm is minutes to days, so a + /// quarter of a year holds every rhythm that can make a meter stale; a meter that delivered nothing in it is read + /// from its own history instead. The bound is what lets the query skip the raw chunks it cannot need. + /// + public static TimeSpan RecentWindow => TimeSpan.FromDays(90); + /// How many intervals (or poll intervals) of silence make a live source stale. public const double StaleFactor = 3; diff --git a/src/Infrastructure/Analysis/AnalysisQueries.cs b/src/Infrastructure/Analysis/AnalysisQueries.cs index ca97e4b..9d0dffd 100644 --- a/src/Infrastructure/Analysis/AnalysisQueries.cs +++ b/src/Infrastructure/Analysis/AnalysisQueries.cs @@ -228,6 +228,12 @@ internal static class AnalysisQueries /// The sums of the exact partial-day windows the leaves registered for summing (D-15): one statement for every /// window of every meter, returning a tally per window instead of its rows. /// + /// + /// The windows arrive through an unnest join, so their bounds are columns and say nothing about the rows + /// at plan time: PostgreSQL would plan — and, with enough windows, scan — every chunk of consumption + /// (A-40). The overall [min from, max to) of the set is therefore repeated as constants, which is exactly + /// the predicate chunk exclusion needs. It never changes a result: no row outside it can match any window. + /// public static async Task WindowSumsAsync( DbConnection connection, IReadOnlyDictionary leaves, CancellationToken cancellationToken) { @@ -245,13 +251,18 @@ internal static class AnalysisQueries coalesce(sum(c.amount) FILTER (WHERE c.quality NOT IN (@measured, @manual, @imported)), 0) FROM unnest(@ids, @froms, @tos) WITH ORDINALITY AS w(meter_id, from_time, to_time, idx) JOIN consumption c ON c.meter_id = w.meter_id AND c.time >= w.from_time AND c.time < w.to_time + WHERE c.time >= @min_from AND c.time < @max_to GROUP BY w.idx """; + var froms = windows.Select(w => w.From.ToUniversalTime()).ToArray(); + var tos = windows.Select(w => w.To.ToUniversalTime()).ToArray(); await using var command = Command(connection, sql); command.Parameters.AddWithValue("ids", windows.Select(w => w.Id).ToArray()); - command.Parameters.AddWithValue("froms", windows.Select(w => w.From.ToUniversalTime()).ToArray()); - command.Parameters.AddWithValue("tos", windows.Select(w => w.To.ToUniversalTime()).ToArray()); + command.Parameters.AddWithValue("froms", froms); + command.Parameters.AddWithValue("tos", tos); + command.Parameters.AddWithValue("min_from", froms.Min()); + command.Parameters.AddWithValue("max_to", tos.Max()); command.Parameters.AddWithValue("measured", (short)ReadingQuality.Measured); command.Parameters.AddWithValue("manual", (short)ReadingQuality.Manual); command.Parameters.AddWithValue("imported", (short)ReadingQuality.Imported); @@ -264,9 +275,17 @@ internal static class AnalysisQueries } } - /// The latest reading times per meter (D-18). + /// + /// The latest reading times per meter, for the rhythm a live + /// source is judged by (D-18). Only these times are read from reading; the freshness mark itself comes + /// from meter_rollup_state (A-40). + /// + /// + /// The lower time bound, or null for the whole history. A bound is what lets PostgreSQL leave the raw chunks + /// before it out of the plan — unbounded, a decade of chunks is planned on every request. + /// public static async Task>> RecentReadingsAsync( - DbConnection connection, IReadOnlyCollection meterIds, CancellationToken cancellationToken) + DbConnection connection, IReadOnlyCollection meterIds, DateTimeOffset? since, CancellationToken cancellationToken) { var result = new Dictionary>(); if (meterIds.Count == 0) @@ -274,15 +293,26 @@ internal static class AnalysisQueries return result; } - const string sql = """ + const string bounded = """ + SELECT m.id, r.time + FROM unnest(@ids) AS m(id) + CROSS JOIN LATERAL ( + SELECT time FROM reading WHERE meter_id = m.id AND time >= @since ORDER BY time DESC LIMIT @count) r + """; + const string all = """ SELECT m.id, r.time FROM unnest(@ids) AS m(id) CROSS JOIN LATERAL ( SELECT time FROM reading WHERE meter_id = m.id ORDER BY time DESC LIMIT @count) r """; - await using var command = Command(connection, sql); + await using var command = Command(connection, since is null ? all : bounded); command.Parameters.AddWithValue("ids", meterIds.ToArray()); + if (since is { } bound) + { + command.Parameters.AddWithValue("since", bound.ToUniversalTime()); + } + command.Parameters.AddWithValue("count", FreshnessRules.RecentReadingCount); await using var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) diff --git a/src/Infrastructure/Analysis/AnalysisReader.cs b/src/Infrastructure/Analysis/AnalysisReader.cs index 909a5e2..367454f 100644 --- a/src/Infrastructure/Analysis/AnalysisReader.cs +++ b/src/Infrastructure/Analysis/AnalysisReader.cs @@ -153,6 +153,13 @@ public sealed class AnalysisReader(IDbContextFactory contex return await AnalysisCatalog.LoadAsync(db, Zone, cancellationToken).ConfigureAwait(false); } + /// The same, on a context the caller holds — for a page that reads several figures from one snapshot (A-40). + internal Task LoadCatalogAsync(MeterVaultDbContext db, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(db); + return AnalysisCatalog.LoadAsync(db, Zone, cancellationToken); + } + private void CheckZone(TimeZoneInfo zone) { if (!string.Equals(zone.Id, Zone.Id, StringComparison.Ordinal)) diff --git a/src/Infrastructure/Analysis/AnalysisRun.cs b/src/Infrastructure/Analysis/AnalysisRun.cs index 6ab1110..45761ed 100644 --- a/src/Infrastructure/Analysis/AnalysisRun.cs +++ b/src/Infrastructure/Analysis/AnalysisRun.cs @@ -474,7 +474,16 @@ internal sealed class AnalysisRun } } - /// Freshness of every data leaf (D-18): recent reading times, the last event, and the live sources. + /// Freshness of every data leaf (D-18): the last reading and event, and the live sources' rhythm. + /// + /// The mark — the meter's last reading — is read from its stored rollup state, which the recompute behind every + /// write path records (A-40); an import-only meter's mark is years old and must stay exact, so it is never + /// guessed from a bounded sample. Only the *rhythm* of a live source needs raw reading times, and only from the + /// recent past: that query carries as its lower bound, so it plans over + /// the newest raw chunks instead of every chunk a decade of history has. A live meter that delivered nothing + /// inside the window has no rhythm there; those few meters are asked again over their whole history, so the + /// stale verdict of a long-silent source is unchanged. + /// private async Task LoadFreshnessAsync(CancellationToken cancellationToken) { var ids = _dataLeaves.ToList(); @@ -483,17 +492,30 @@ internal sealed class AnalysisRun return; } - var readings = await AnalysisQueries.RecentReadingsAsync(_connection, ids, cancellationToken).ConfigureAwait(false); - var events = await AnalysisQueries.LastEventsAsync(_connection, ids, cancellationToken).ConfigureAwait(false); var sources = await _db.MeterSources.AsNoTracking() .Include(s => s.Endpoint) .Where(s => ids.Contains(s.MeterId) && s.IsEnabled) .ToListAsync(cancellationToken).ConfigureAwait(false); + var liveIds = ids.Where(id => sources.Exists(s => s.MeterId == id && IsLive(s))).ToList(); + var readings = await AnalysisQueries + .RecentReadingsAsync(_connection, liveIds, Now - FreshnessRules.RecentWindow, cancellationToken).ConfigureAwait(false); + var silent = liveIds.Where(id => (readings.GetValueOrDefault(id)?.Count ?? 0) < 2).ToList(); + if (silent.Count > 0) + { + foreach (var (id, times) in await AnalysisQueries + .RecentReadingsAsync(_connection, silent, since: null, cancellationToken).ConfigureAwait(false)) + { + readings[id] = times; + } + } + + var events = await AnalysisQueries.LastEventsAsync(_connection, ids, cancellationToken).ConfigureAwait(false); + foreach (var id in ids) { var times = readings.GetValueOrDefault(id) ?? []; - var live = sources.Where(s => s.MeterId == id && s.SourceType is SourceType.Mqtt or SourceType.Tasmota or SourceType.HomeAssistant).ToList(); + var live = sources.Where(s => s.MeterId == id && IsLive(s)).ToList(); TimeSpan? poll = null; foreach (var source in live.Where(s => s.SourceType == SourceType.HomeAssistant)) { @@ -507,7 +529,7 @@ internal sealed class AnalysisRun } var input = new FreshnessInput( - times.Count > 0 ? times.Max() : null, + LastReadingOf(id, times), events.TryGetValue(id, out var lastEvent) ? lastEvent : null, times, live.Count > 0, @@ -516,6 +538,21 @@ internal sealed class AnalysisRun } } + /// A source that is expected to deliver on its own (D-18). + private static bool IsLive(MeterSource source) => + source.SourceType is SourceType.Mqtt or SourceType.Tasmota or SourceType.HomeAssistant; + + /// + /// The meter's last reading time (A-40): what its stored analysis data recorded, and — for a live meter whose + /// rhythm was read — whatever of the two is later, so a reading ingested since the last recompute still counts. + /// + private DateTimeOffset? LastReadingOf(int id, IReadOnlyList times) + { + var stored = _catalog.Meters.TryGetValue(id, out var meter) ? meter.State?.LastReadingAt : null; + var sampled = times.Count > 0 ? times.Max() : (DateTimeOffset?)null; + return stored is { } s && sampled is { } r ? (s >= r ? s : r) : stored ?? sampled; + } + // ---------------------------------------------------------------- physical values /// A physical meter's sums per bucket of one side, and their total. diff --git a/src/Infrastructure/Costing/BillRun.cs b/src/Infrastructure/Costing/BillRun.cs index 62953ec..5bc4e6a 100644 --- a/src/Infrastructure/Costing/BillRun.cs +++ b/src/Infrastructure/Costing/BillRun.cs @@ -42,6 +42,9 @@ internal sealed class BillRun private readonly Dictionary _firstData = []; private readonly List _quantityProblems = []; + /// A catalog the caller already loaded and wants this run to reuse (A-40); null to load one. + private readonly AnalysisCatalog? _sharedCatalog; + private AnalysisCatalog _catalog = null!; private TariffBook _book = null!; private TotalsClassification _full = null!; @@ -59,7 +62,8 @@ internal sealed class BillRun private Dictionary> _periodSpans = []; private Dictionary _series = []; - public BillRun(MeterVaultDbContext db, AnalysisReader reader, CostAnalysisRequest request, string currency) + public BillRun( + MeterVaultDbContext db, AnalysisReader reader, CostAnalysisRequest request, string currency, AnalysisCatalog? catalog = null) { _db = db; _reader = reader; @@ -68,6 +72,7 @@ internal sealed class BillRun _zone = reader.Zone; _currency = currency; _today = PeriodResolver.LocalDate(_period.Now, _zone); + _sharedCatalog = catalog; } private bool WantsCategories => @@ -120,7 +125,10 @@ internal sealed class BillRun private async Task LoadAsync(bool categories, CancellationToken cancellationToken) { - _catalog = await AnalysisCatalog.LoadAsync(_db, _zone, cancellationToken).ConfigureAwait(false); + // The catalog is a snapshot of meters, tanks, links and rollup states; a caller that reads several figures of + // one page from it (the Overview's bill and its comparison) passes the one it already holds, so they answer + // from the same snapshot and the load is not repeated (A-40). + _catalog = _sharedCatalog ?? await AnalysisCatalog.LoadAsync(_db, _zone, cancellationToken).ConfigureAwait(false); // One tariff load per request; ordered so that the book's deterministic tie-break never depends on the plan. var tariffs = await _db.Tariffs.AsNoTracking().OrderBy(t => t.Id).ToListAsync(cancellationToken).ConfigureAwait(false); diff --git a/src/Infrastructure/Costing/CostReader.cs b/src/Infrastructure/Costing/CostReader.cs index 825154a..5832e35 100644 --- a/src/Infrastructure/Costing/CostReader.cs +++ b/src/Infrastructure/Costing/CostReader.cs @@ -67,6 +67,30 @@ public sealed class CostReader( return await new BillRun(db, _reader, request, Currency).ExecuteAsync(cancellationToken).ConfigureAwait(false); } + /// + /// Prices a scope on a context and catalog the caller already holds — a page that reads several figures of one + /// period (the Overview: its quantities, its bill and the comparison's bill) loads the catalog once and passes it + /// here, so every figure answers from the same snapshot (A-40). + /// + /// The period was resolved in another zone than the reader's. + internal async Task ReadAsync( + MeterVaultDbContext db, AnalysisCatalog catalog, CostAnalysisRequest request, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(db); + ArgumentNullException.ThrowIfNull(catalog); + ArgumentNullException.ThrowIfNull(request); + CheckZone(request.Period.Zone); + + var plan = request.Plan + ?? (request.Bucket != BucketSize.Auto ? BucketPlanner.Plan(request.Period, request.Bucket, maxPoints: request.MaxPoints) : null); + if (plan is { Refused: true }) + { + return Refused(request, plan); + } + + return await new BillRun(db, _reader, request, Currency, catalog).ExecuteAsync(cancellationToken).ConfigureAwait(false); + } + /// /// What a cost scope has data for as of (D-19): its priced meters' coverage plus its manual /// costs, and the latest month holding either — for the all preset on a cost view and "go to latest data". diff --git a/src/Infrastructure/Dashboard/DashboardService.cs b/src/Infrastructure/Dashboard/DashboardService.cs index 6e37098..6f5085b 100644 --- a/src/Infrastructure/Dashboard/DashboardService.cs +++ b/src/Infrastructure/Dashboard/DashboardService.cs @@ -61,21 +61,29 @@ public sealed class DashboardService( ArgumentNullException.ThrowIfNull(period); ArgumentNullException.ThrowIfNull(comparison); + // One context and one catalog for the whole page (A-40): the quantities, the bill and the comparison's bill are + // three figures of one period, so they are read from one snapshot of the meters instead of loading it three + // times. Everything else the page shows is derived from these results — never priced again. + await using var db = await _contextFactory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + var catalog = await _reader.LoadCatalogAsync(db, cancellationToken).ConfigureAwait(false); + var quantities = await _reader.ReadAsync( + db, + catalog, new AnalysisRequest(AnalysisScope.Portfolio, period) { Bucket = bucket, Comparison = comparison, IncludeMeterSeries = true }, cancellationToken).ConfigureAwait(false); - var types = await EnergyTypesAsync(cancellationToken).ConfigureAwait(false); + var types = await EnergyTypesAsync(db, cancellationToken).ConfigureAwait(false); var resolution = ComparisonResolver.Resolve(period, comparison); var request = new CostAnalysisRequest(CostScope.Portfolio, period) { Bucket = bucket, IncludeCategories = true }; if (quantities.Refusal != AnalysisRefusal.None) { // Refused before anything was read: the toolbar offers a coarser bucket; nothing is priced either. - var refused = await _costs.ReadAsync(request with { Plan = quantities.Plan }, cancellationToken).ConfigureAwait(false); + var refused = await _costs.ReadAsync(db, catalog, request with { Plan = quantities.Plan }, cancellationToken).ConfigureAwait(false); return new DashboardOverview(period, quantities, refused, resolution, null, [], types); } - var cost = await _costs.ReadAsync(request with { Plan = quantities.Plan }, cancellationToken).ConfigureAwait(false); + var cost = await _costs.ReadAsync(db, catalog, request with { Plan = quantities.Plan }, cancellationToken).ConfigureAwait(false); CostAnalysis? previous = null; IReadOnlyList pairs = []; @@ -84,6 +92,8 @@ public sealed class DashboardService( pairs = ComparisonResolver.PairBuckets(period, resolution.Period, cost.Plan.Buckets); var plan = new BucketPlan(cost.Plan.Size, cost.Plan.Size, [.. pairs.Select(p => p.Comparison)], pairs.Count, Refused: false, Suggested: null); previous = await _costs.ReadAsync( + db, + catalog, new CostAnalysisRequest(CostScope.Portfolio, resolution.Period.ToResolvedPeriod(period)) { Plan = plan, IncludeCategories = true }, cancellationToken).ConfigureAwait(false); } @@ -368,9 +378,9 @@ public sealed class DashboardService( } /// Every energy type with its name and meter count, ordered by id (the navigation's order). - private async Task> EnergyTypesAsync(CancellationToken cancellationToken) + private static async Task> EnergyTypesAsync( + MeterVaultDbContext db, CancellationToken cancellationToken) { - await using var db = await _contextFactory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); var types = await db.EnergyTypes.AsNoTracking() .OrderBy(t => t.Id) .Select(t => new { t.Id, t.DisplayName, Meters = t.Meters.Count }) diff --git a/src/Infrastructure/Normalization/AnalysisDataWriter.cs b/src/Infrastructure/Normalization/AnalysisDataWriter.cs index b12b047..46b7ca9 100644 --- a/src/Infrastructure/Normalization/AnalysisDataWriter.cs +++ b/src/Infrastructure/Normalization/AnalysisDataWriter.cs @@ -101,6 +101,7 @@ internal sealed class AnalysisDataWriter(MeterVaultDbContext db) Zone = state.Zone, NormalizedUnit = state.NormalizedUnit, Kind = state.Kind, + LastReadingAt = state.LastReadingAt, BuiltAt = now.ToUniversalTime(), }); return true; @@ -117,13 +118,15 @@ internal sealed class AnalysisDataWriter(MeterVaultDbContext db) || stored.Revision != state.Revision || !string.Equals(stored.Zone, state.Zone, StringComparison.Ordinal) || !string.Equals(stored.NormalizedUnit, state.NormalizedUnit, StringComparison.Ordinal) - || stored.Kind != state.Kind; + || stored.Kind != state.Kind + || !Nullable.Equals(stored.LastReadingAt, state.LastReadingAt); if (changed || stateChanged) { stored.Revision = state.Revision; stored.Zone = state.Zone; stored.NormalizedUnit = state.NormalizedUnit; stored.Kind = state.Kind; + stored.LastReadingAt = state.LastReadingAt; stored.BuiltAt = now.ToUniversalTime(); } diff --git a/src/Infrastructure/Normalization/NormalizationService.cs b/src/Infrastructure/Normalization/NormalizationService.cs index 46577e4..ed7a287 100644 --- a/src/Infrastructure/Normalization/NormalizationService.cs +++ b/src/Infrastructure/Normalization/NormalizationService.cs @@ -54,6 +54,10 @@ public sealed class NormalizationService( Tank? tank = null; IReadOnlyList consumption = []; + + // The freshness mark of D-18, recorded with the derived data instead of read back from the raw + // hypertable on every analysis request (A-40). A virtual meter has no readings of its own. + DateTimeOffset? lastReading = null; if (meter.Mode != MeterMode.Virtual) { tank = await _db.Tanks.FirstOrDefaultAsync(t => t.MeterId == meterId, cancellationToken).ConfigureAwait(false); @@ -69,6 +73,8 @@ public sealed class NormalizationService( .OrderBy(e => e.Time) .ToListAsync(cancellationToken).ConfigureAwait(false); + lastReading = readings.Count > 0 ? readings[^1].Time.ToUniversalTime() : null; + var context = new NormalizationContext { Meter = config, Readings = readings, Events = events, TimeZone = _zone }; consumption = _engine.Normalize(context); } @@ -107,6 +113,7 @@ public sealed class NormalizationService( Zone = _zone.Id, NormalizedUnit = quantity.Unit, Kind = quantity.Kind, + LastReadingAt = lastReading, }; await new AnalysisDataWriter(_db) diff --git a/src/Infrastructure/Persistence/Analysis/AnalysisEntities.cs b/src/Infrastructure/Persistence/Analysis/AnalysisEntities.cs index 44ad6be..15e8848 100644 --- a/src/Infrastructure/Persistence/Analysis/AnalysisEntities.cs +++ b/src/Infrastructure/Persistence/Analysis/AnalysisEntities.cs @@ -178,6 +178,13 @@ public sealed class MeterRollupState /// What the meter's amounts measure (D-20). public QuantityKind Kind { get; set; } + /// + /// The stamp of the meter's latest raw reading when its analysis data was built, or null when it has none + /// (A-40). Every write path recomputes the meter inline, so this is the freshness mark of D-18 without a + /// query over reading — whose chunks a reader would otherwise have to plan across on every request. + /// + public DateTimeOffset? LastReadingAt { get; set; } + /// /// When the meter's analysis data last changed: a rollup or coverage row was added, changed or removed, or /// one of the columns above changed. A recompute that reproduces the stored data leaves it alone, so it diff --git a/src/Infrastructure/Persistence/Migrations/20260920083742_FreshnessMark.Designer.cs b/src/Infrastructure/Persistence/Migrations/20260920083742_FreshnessMark.Designer.cs new file mode 100644 index 0000000..73d341f --- /dev/null +++ b/src/Infrastructure/Persistence/Migrations/20260920083742_FreshnessMark.Designer.cs @@ -0,0 +1,1149 @@ +// +using System; +using MeterVault.Infrastructure.Persistence; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; + +#nullable disable + +namespace MeterVault.Infrastructure.Persistence.Migrations +{ + [DbContext(typeof(MeterVaultDbContext))] + [Migration("20260920083742_FreshnessMark")] + partial class FreshnessMark + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.9") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.HasPostgresExtension(modelBuilder, "timescaledb"); + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); + + modelBuilder.Entity("MeterVault.Core.Domain.AppSetting", b => + { + b.Property("Key") + .HasMaxLength(128) + .HasColumnType("character varying(128)") + .HasColumnName("key"); + + b.Property("Value") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("value"); + + b.HasKey("Key") + .HasName("pk_app_setting"); + + b.ToTable("app_setting", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Consumption", b => + { + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("Time") + .HasColumnType("timestamp with time zone") + .HasColumnName("time"); + + b.Property("Kind") + .HasColumnType("smallint") + .HasColumnName("kind"); + + b.Property("Amount") + .HasColumnType("double precision") + .HasColumnName("amount"); + + b.Property("ImportBatchId") + .HasColumnType("integer") + .HasColumnName("import_batch_id"); + + b.Property("Quality") + .HasColumnType("smallint") + .HasColumnName("quality"); + + b.HasKey("MeterId", "Time", "Kind") + .HasName("pk_consumption"); + + b.HasIndex("ImportBatchId") + .HasDatabaseName("ix_consumption_import_batch_id"); + + b.ToTable("consumption", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.CostCategory", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("ColorHex") + .HasColumnType("text") + .HasColumnName("color_hex"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(128) + .HasColumnType("character varying(128)") + .HasColumnName("name"); + + b.Property("Sort") + .HasColumnType("integer") + .HasColumnName("sort"); + + b.HasKey("Id") + .HasName("pk_cost_category"); + + b.ToTable("cost_category", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.CostCategoryMember", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CategoryId") + .HasColumnType("integer") + .HasColumnName("category_id"); + + b.Property("EnergyTypeId") + .HasColumnType("smallint") + .HasColumnName("energy_type_id"); + + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.HasKey("Id") + .HasName("pk_cost_category_member"); + + b.HasIndex("CategoryId") + .HasDatabaseName("ix_cost_category_member_category_id"); + + b.HasIndex("EnergyTypeId") + .HasDatabaseName("ix_cost_category_member_energy_type_id"); + + b.HasIndex("MeterId") + .HasDatabaseName("ix_cost_category_member_meter_id"); + + b.ToTable("cost_category_member", null, t => + { + t.HasCheckConstraint("ck_cost_category_member_target", "meter_id IS NOT NULL OR energy_type_id IS NOT NULL"); + }); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.EnergyType", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("smallint") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("BaseUnit") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("base_unit"); + + b.Property("ColorHex") + .HasColumnType("text") + .HasColumnName("color_hex"); + + b.Property("CreatedAt") + .ValueGeneratedOnAdd() + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at") + .HasDefaultValueSql("now()"); + + b.Property("DefaultMode") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("character varying(32)") + .HasColumnName("default_mode"); + + b.Property("DisplayName") + .IsRequired() + .HasMaxLength(128) + .HasColumnType("character varying(128)") + .HasColumnName("display_name"); + + b.Property("Icon") + .HasColumnType("text") + .HasColumnName("icon"); + + b.Property("Key") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("key"); + + b.HasKey("Id") + .HasName("pk_energy_type"); + + b.HasIndex("Key") + .IsUnique() + .HasDatabaseName("ix_energy_type_key"); + + b.ToTable("energy_type", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.ImportBatch", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CreatedAt") + .ValueGeneratedOnAdd() + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at") + .HasDefaultValueSql("now()"); + + b.Property("Mapping") + .HasColumnType("jsonb") + .HasColumnName("mapping"); + + b.Property("RevertedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("reverted_at"); + + b.Property("RowCount") + .HasColumnType("integer") + .HasColumnName("row_count"); + + b.Property("SourceName") + .HasColumnType("text") + .HasColumnName("source_name"); + + b.HasKey("Id") + .HasName("pk_import_batch"); + + b.ToTable("import_batch", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.IngestionEndpoint", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Config") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("jsonb") + .HasColumnName("config") + .HasDefaultValueSql("'{}'::jsonb"); + + b.Property("IsEnabled") + .HasColumnType("boolean") + .HasColumnName("is_enabled"); + + b.Property("LastSeenAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_seen_at"); + + b.Property("LastStatus") + .HasColumnType("text") + .HasColumnName("last_status"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(128) + .HasColumnType("character varying(128)") + .HasColumnName("name"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("character varying(32)") + .HasColumnName("type"); + + b.HasKey("Id") + .HasName("pk_ingestion_endpoint"); + + b.ToTable("ingestion_endpoint", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.ManualCost", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Amount") + .HasColumnType("double precision") + .HasColumnName("amount"); + + b.Property("CategoryId") + .HasColumnType("integer") + .HasColumnName("category_id"); + + b.Property("Currency") + .IsRequired() + .HasMaxLength(8) + .HasColumnType("character varying(8)") + .HasColumnName("currency"); + + b.Property("ImportBatchId") + .HasColumnType("integer") + .HasColumnName("import_batch_id"); + + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("Notes") + .HasColumnType("text") + .HasColumnName("notes"); + + b.Property("PeriodEnd") + .HasColumnType("date") + .HasColumnName("period_end"); + + b.Property("PeriodStart") + .HasColumnType("date") + .HasColumnName("period_start"); + + b.HasKey("Id") + .HasName("pk_manual_cost"); + + b.HasIndex("CategoryId") + .HasDatabaseName("ix_manual_cost_category_id"); + + b.HasIndex("ImportBatchId") + .HasDatabaseName("ix_manual_cost_import_batch_id"); + + b.HasIndex("MeterId") + .HasDatabaseName("ix_manual_cost_meter_id"); + + b.ToTable("manual_cost", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Meter", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CreatedAt") + .ValueGeneratedOnAdd() + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at") + .HasDefaultValueSql("now()"); + + b.Property("EnergyTypeId") + .HasColumnType("smallint") + .HasColumnName("energy_type_id"); + + b.Property("InitialBaseline") + .HasColumnType("double precision") + .HasColumnName("initial_baseline"); + + b.Property("InstalledAt") + .HasColumnType("date") + .HasColumnName("installed_at"); + + b.Property("IsActive") + .ValueGeneratedOnAdd() + .HasColumnType("boolean") + .HasDefaultValue(true) + .HasColumnName("is_active"); + + b.Property("Location") + .HasColumnType("text") + .HasColumnName("location"); + + b.Property("Manufacturer") + .HasColumnType("text") + .HasColumnName("manufacturer"); + + b.Property("Meta") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("jsonb") + .HasColumnName("meta") + .HasDefaultValueSql("'{}'::jsonb"); + + b.Property("Mode") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("character varying(32)") + .HasColumnName("mode"); + + b.Property("Model") + .HasColumnType("text") + .HasColumnName("model"); + + b.Property("Name") + .IsRequired() + .HasColumnType("text") + .HasColumnName("name"); + + b.Property("RetiredAt") + .HasColumnType("date") + .HasColumnName("retired_at"); + + b.Property("SerialNumber") + .HasColumnType("text") + .HasColumnName("serial_number"); + + b.Property("Unit") + .IsRequired() + .HasColumnType("text") + .HasColumnName("unit"); + + b.Property("UpdatedAt") + .ValueGeneratedOnAdd() + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at") + .HasDefaultValueSql("now()"); + + b.HasKey("Id") + .HasName("pk_meter"); + + b.HasIndex("EnergyTypeId", "IsActive") + .HasDatabaseName("ix_meter_energy_type_id_is_active"); + + b.ToTable("meter", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.MeterEvent", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Amount") + .HasColumnType("double precision") + .HasColumnName("amount"); + + b.Property("EventType") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("character varying(32)") + .HasColumnName("event_type"); + + b.Property("ImportBatchId") + .HasColumnType("integer") + .HasColumnName("import_batch_id"); + + b.Property("Meta") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("jsonb") + .HasColumnName("meta") + .HasDefaultValueSql("'{}'::jsonb"); + + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("NewValue") + .HasColumnType("double precision") + .HasColumnName("new_value"); + + b.Property("Notes") + .HasColumnType("text") + .HasColumnName("notes"); + + b.Property("PrevValue") + .HasColumnType("double precision") + .HasColumnName("prev_value"); + + b.Property("Time") + .HasColumnType("timestamp with time zone") + .HasColumnName("time"); + + b.Property("Unit") + .HasColumnType("text") + .HasColumnName("unit"); + + b.HasKey("Id") + .HasName("pk_meter_event"); + + b.HasIndex("ImportBatchId") + .HasDatabaseName("ix_meter_event_import_batch_id"); + + b.HasIndex("MeterId", "Time") + .HasDatabaseName("ix_meter_event_meter_id_time"); + + b.ToTable("meter_event", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.MeterLink", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("FromMeterId") + .HasColumnType("integer") + .HasColumnName("from_meter_id"); + + b.Property("ToMeterId") + .HasColumnType("integer") + .HasColumnName("to_meter_id"); + + b.HasKey("Id") + .HasName("pk_meter_link"); + + b.HasIndex("ToMeterId") + .HasDatabaseName("ix_meter_link_to_meter_id"); + + b.HasIndex("FromMeterId", "ToMeterId") + .IsUnique() + .HasDatabaseName("ix_meter_link_from_meter_id_to_meter_id"); + + b.ToTable("meter_link", null, t => + { + t.HasCheckConstraint("ck_meter_link_distinct", "from_meter_id <> to_meter_id"); + }); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.MeterSource", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Config") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("jsonb") + .HasColumnName("config") + .HasDefaultValueSql("'{}'::jsonb"); + + b.Property("EndpointId") + .HasColumnType("integer") + .HasColumnName("endpoint_id"); + + b.Property("IsEnabled") + .HasColumnType("boolean") + .HasColumnName("is_enabled"); + + b.Property("LastSeenAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_seen_at"); + + b.Property("LastStatus") + .HasColumnType("text") + .HasColumnName("last_status"); + + b.Property("LastValue") + .HasColumnType("double precision") + .HasColumnName("last_value"); + + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("Offset") + .HasColumnType("double precision") + .HasColumnName("offset"); + + b.Property("Priority") + .HasColumnType("integer") + .HasColumnName("priority"); + + b.Property("Scale") + .ValueGeneratedOnAdd() + .HasColumnType("double precision") + .HasDefaultValue(1.0) + .HasColumnName("scale"); + + b.Property("SourceType") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("character varying(32)") + .HasColumnName("source_type"); + + b.Property("ValueKind") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("value_kind"); + + b.HasKey("Id") + .HasName("pk_meter_source"); + + b.HasIndex("EndpointId") + .HasDatabaseName("ix_meter_source_endpoint_id"); + + b.HasIndex("MeterId") + .HasDatabaseName("ix_meter_source_meter_id"); + + b.ToTable("meter_source", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Reading", b => + { + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("Time") + .HasColumnType("timestamp with time zone") + .HasColumnName("time"); + + b.Property("Flags") + .HasColumnType("integer") + .HasColumnName("flags"); + + b.Property("ImportBatchId") + .HasColumnType("integer") + .HasColumnName("import_batch_id"); + + b.Property("Quality") + .HasColumnType("smallint") + .HasColumnName("quality"); + + b.Property("SourceId") + .HasColumnType("integer") + .HasColumnName("source_id"); + + b.Property("Value") + .HasColumnType("double precision") + .HasColumnName("value"); + + b.HasKey("MeterId", "Time") + .HasName("pk_reading"); + + b.HasIndex("ImportBatchId") + .HasDatabaseName("ix_reading_import_batch_id"); + + b.ToTable("reading", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Tank", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("CachedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("cached_at"); + + b.Property("CachedBalance") + .HasColumnType("double precision") + .HasColumnName("cached_balance"); + + b.Property("Calibration") + .HasColumnType("jsonb") + .HasColumnName("calibration"); + + b.Property("Capacity") + .HasColumnType("double precision") + .HasColumnName("capacity"); + + b.Property("FixedRate") + .HasColumnType("double precision") + .HasColumnName("fixed_rate"); + + b.Property("LowThreshold") + .HasColumnType("double precision") + .HasColumnName("low_threshold"); + + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("RateMode") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("rate_mode"); + + b.Property("ReorderThreshold") + .HasColumnType("double precision") + .HasColumnName("reorder_threshold"); + + b.Property("Unit") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("unit"); + + b.HasKey("Id") + .HasName("pk_tank"); + + b.HasIndex("MeterId") + .IsUnique() + .HasDatabaseName("ix_tank_meter_id"); + + b.ToTable("tank", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Tariff", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("integer") + .HasColumnName("id"); + + NpgsqlPropertyBuilderExtensions.UseIdentityByDefaultColumn(b.Property("Id")); + + b.Property("Component") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("component"); + + b.Property("Currency") + .IsRequired() + .HasMaxLength(8) + .HasColumnType("character varying(8)") + .HasColumnName("currency"); + + b.Property("Notes") + .HasColumnType("text") + .HasColumnName("notes"); + + b.Property("ScopeId") + .HasColumnType("integer") + .HasColumnName("scope_id"); + + b.Property("ScopeType") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("scope_type"); + + b.Property("Unit") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("unit"); + + b.Property("ValidFrom") + .HasColumnType("date") + .HasColumnName("valid_from"); + + b.Property("ValidTo") + .HasColumnType("date") + .HasColumnName("valid_to"); + + b.Property("Value") + .HasColumnType("double precision") + .HasColumnName("value"); + + b.HasKey("Id") + .HasName("pk_tariff"); + + b.HasIndex("ScopeType", "ScopeId", "Component", "ValidFrom") + .HasDatabaseName("ix_tariff_scope_type_scope_id_component_valid_from"); + + b.ToTable("tariff", (string)null); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.ConsumptionRollup", b => + { + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("Day") + .HasColumnType("date") + .HasColumnName("day"); + + b.Property("Kind") + .HasColumnType("smallint") + .HasColumnName("kind"); + + b.Property("Amount") + .HasColumnType("double precision") + .HasColumnName("amount"); + + b.Property("Estimated") + .HasColumnType("double precision") + .HasColumnName("estimated"); + + b.Property("Flags") + .HasColumnType("integer") + .HasColumnName("flags"); + + b.Property("Imported") + .HasColumnType("double precision") + .HasColumnName("imported"); + + b.Property("Manual") + .HasColumnType("double precision") + .HasColumnName("manual"); + + b.Property("MaxIntervalEnd") + .HasColumnType("timestamp with time zone") + .HasColumnName("max_interval_end"); + + b.Property("Measured") + .HasColumnType("double precision") + .HasColumnName("measured"); + + b.Property("Rows") + .HasColumnType("integer") + .HasColumnName("rows"); + + b.HasKey("MeterId", "Day", "Kind") + .HasName("pk_consumption_rollup"); + + b.ToTable("consumption_rollup", (string)null); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.ConsumptionRollupMonth", b => + { + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("Month") + .HasColumnType("date") + .HasColumnName("month"); + + b.Property("Kind") + .HasColumnType("smallint") + .HasColumnName("kind"); + + b.Property("Amount") + .HasColumnType("double precision") + .HasColumnName("amount"); + + b.Property("Estimated") + .HasColumnType("double precision") + .HasColumnName("estimated"); + + b.Property("Flags") + .HasColumnType("integer") + .HasColumnName("flags"); + + b.Property("Imported") + .HasColumnType("double precision") + .HasColumnName("imported"); + + b.Property("Manual") + .HasColumnType("double precision") + .HasColumnName("manual"); + + b.Property("MaxIntervalEnd") + .HasColumnType("timestamp with time zone") + .HasColumnName("max_interval_end"); + + b.Property("Measured") + .HasColumnType("double precision") + .HasColumnName("measured"); + + b.Property("Rows") + .HasColumnType("integer") + .HasColumnName("rows"); + + b.HasKey("MeterId", "Month", "Kind") + .HasName("pk_consumption_rollup_month"); + + b.ToTable("consumption_rollup_month", (string)null); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.MeterCoverageRun", b => + { + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("SpanFrom") + .HasColumnType("timestamp with time zone") + .HasColumnName("span_from"); + + b.Property("DividedAtMonths") + .HasColumnType("boolean") + .HasColumnName("divided_at_months"); + + b.Property("GapReason") + .HasColumnType("smallint") + .HasColumnName("gap_reason"); + + b.Property("LastIntervalStart") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_interval_start"); + + b.Property("ResolutionClass") + .HasColumnType("smallint") + .HasColumnName("resolution_class"); + + b.Property("SpanTo") + .HasColumnType("timestamp with time zone") + .HasColumnName("span_to"); + + b.HasKey("MeterId", "SpanFrom") + .HasName("pk_meter_coverage"); + + b.ToTable("meter_coverage", (string)null); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.MeterRollupState", b => + { + b.Property("MeterId") + .HasColumnType("integer") + .HasColumnName("meter_id"); + + b.Property("BuiltAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("built_at"); + + b.Property("Kind") + .HasColumnType("smallint") + .HasColumnName("kind"); + + b.Property("LastReadingAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_reading_at"); + + b.Property("NormalizedUnit") + .IsRequired() + .HasColumnType("text") + .HasColumnName("normalized_unit"); + + b.Property("Revision") + .HasColumnType("integer") + .HasColumnName("revision"); + + b.Property("Zone") + .IsRequired() + .HasColumnType("text") + .HasColumnName("zone"); + + b.HasKey("MeterId") + .HasName("pk_meter_rollup_state"); + + b.ToTable("meter_rollup_state", (string)null); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Consumption", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_consumption_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.CostCategoryMember", b => + { + b.HasOne("MeterVault.Core.Domain.CostCategory", "Category") + .WithMany("Members") + .HasForeignKey("CategoryId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_cost_category_member_cost_category_category_id"); + + b.HasOne("MeterVault.Core.Domain.EnergyType", null) + .WithMany() + .HasForeignKey("EnergyTypeId") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_cost_category_member_energy_type_energy_type_id"); + + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_cost_category_member_meter_meter_id"); + + b.Navigation("Category"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.ManualCost", b => + { + b.HasOne("MeterVault.Core.Domain.CostCategory", null) + .WithMany() + .HasForeignKey("CategoryId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_manual_cost_cost_category_category_id"); + + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_manual_cost_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Meter", b => + { + b.HasOne("MeterVault.Core.Domain.EnergyType", "EnergyType") + .WithMany("Meters") + .HasForeignKey("EnergyTypeId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_meter_energy_type_energy_type_id"); + + b.Navigation("EnergyType"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.MeterEvent", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_meter_event_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.MeterLink", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", "FromMeter") + .WithMany() + .HasForeignKey("FromMeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_meter_link_meter_from_meter_id"); + + b.HasOne("MeterVault.Core.Domain.Meter", "ToMeter") + .WithMany() + .HasForeignKey("ToMeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_meter_link_meter_to_meter_id"); + + b.Navigation("FromMeter"); + + b.Navigation("ToMeter"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.MeterSource", b => + { + b.HasOne("MeterVault.Core.Domain.IngestionEndpoint", "Endpoint") + .WithMany() + .HasForeignKey("EndpointId") + .OnDelete(DeleteBehavior.SetNull) + .HasConstraintName("fk_meter_source_ingestion_endpoints_endpoint_id"); + + b.HasOne("MeterVault.Core.Domain.Meter", "Meter") + .WithMany("Sources") + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_meter_source_meter_meter_id"); + + b.Navigation("Endpoint"); + + b.Navigation("Meter"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Reading", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_reading_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Tank", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", "Meter") + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_tank_meter_meter_id"); + + b.Navigation("Meter"); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.ConsumptionRollup", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_consumption_rollup_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.ConsumptionRollupMonth", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_consumption_rollup_month_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.MeterCoverageRun", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithMany() + .HasForeignKey("MeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_meter_coverage_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Infrastructure.Persistence.Analysis.MeterRollupState", b => + { + b.HasOne("MeterVault.Core.Domain.Meter", null) + .WithOne() + .HasForeignKey("MeterVault.Infrastructure.Persistence.Analysis.MeterRollupState", "MeterId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_meter_rollup_state_meter_meter_id"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.CostCategory", b => + { + b.Navigation("Members"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.EnergyType", b => + { + b.Navigation("Meters"); + }); + + modelBuilder.Entity("MeterVault.Core.Domain.Meter", b => + { + b.Navigation("Sources"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/Infrastructure/Persistence/Migrations/20260920083742_FreshnessMark.cs b/src/Infrastructure/Persistence/Migrations/20260920083742_FreshnessMark.cs new file mode 100644 index 0000000..9f65f18 --- /dev/null +++ b/src/Infrastructure/Persistence/Migrations/20260920083742_FreshnessMark.cs @@ -0,0 +1,38 @@ +using System; +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace MeterVault.Infrastructure.Persistence.Migrations +{ + /// + public partial class FreshnessMark : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.AddColumn( + name: "last_reading_at", + table: "meter_rollup_state", + type: "timestamp with time zone", + nullable: true); + + // Backfill once, so an existing instance keeps every meter's freshness mark (D-18) without waiting for a + // rebuild. From here on every recompute records it, and no analysis request reads `reading` for it (A-40). + migrationBuilder.Sql(""" + UPDATE meter_rollup_state s + SET last_reading_at = r.last_time + FROM (SELECT meter_id, max(time) AS last_time FROM reading GROUP BY meter_id) r + WHERE r.meter_id = s.meter_id + """); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.DropColumn( + name: "last_reading_at", + table: "meter_rollup_state"); + } + } +} diff --git a/src/Infrastructure/Persistence/Migrations/MeterVaultDbContextModelSnapshot.cs b/src/Infrastructure/Persistence/Migrations/MeterVaultDbContextModelSnapshot.cs index 7fa97bc..83876f8 100644 --- a/src/Infrastructure/Persistence/Migrations/MeterVaultDbContextModelSnapshot.cs +++ b/src/Infrastructure/Persistence/Migrations/MeterVaultDbContextModelSnapshot.cs @@ -928,6 +928,10 @@ namespace MeterVault.Infrastructure.Persistence.Migrations .HasColumnType("smallint") .HasColumnName("kind"); + b.Property("LastReadingAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_reading_at"); + b.Property("NormalizedUnit") .IsRequired() .HasColumnType("text") diff --git a/tests/Integration.Tests/Analysis/AnalysisReaderTests.cs b/tests/Integration.Tests/Analysis/AnalysisReaderTests.cs index 0324e0d..9c8b1eb 100644 --- a/tests/Integration.Tests/Analysis/AnalysisReaderTests.cs +++ b/tests/Integration.Tests/Analysis/AnalysisReaderTests.cs @@ -10,6 +10,7 @@ using MeterVault.Infrastructure.Ingestion; using MeterVault.Infrastructure.Normalization; using MeterVault.Infrastructure.Options; using MeterVault.Infrastructure.Persistence; +using MeterVault.Integration.Tests.Performance; using Microsoft.EntityFrameworkCore; using static MeterVault.Integration.Tests.Reconciliation.ReconciliationSupport; @@ -694,6 +695,49 @@ public sealed class AnalysisReaderTests(TimescaleFixture fx) : IAsyncLifetime Assert.DoesNotContain(result.Problems, p => p.Kind == AnalysisProblemKind.StaleSource && p.MeterId != stale); } + [Fact] + public async Task An_import_only_meter_keeps_its_years_old_mark_without_reading_the_raw_series() + { + // D-18, A-40: the mark is the meter's own last reading — for an import-only meter as old as its data — and it + // is read from the stored rollup state, so the request never plans across the raw chunks to find it. + var type = await TypeAsync(); + var historical = await MeterAsync(type, MeterMode.CumulativeCounter, "kWh", installedAt: new DateOnly(2018, 1, 1)); + await ReadingsAsync(historical, (Local(Berlin, 2018, 3, 5, 9), 100), (Local(Berlin, 2018, 4, 5, 9), 140)); + + using var counter = CommandCounter.Start(); + var result = await Reader().ReadAsync(new AnalysisRequest(AnalysisScope.ForMeter(historical), Resolve(PeriodPreset.MonthToDate))); + + var freshness = result.SeriesFor(historical)!.Freshness; + Assert.Equal(FreshnessState.Historical, freshness.State); + Assert.Equal(Local(Berlin, 2018, 4, 5, 9), freshness.LastActivity); + Assert.DoesNotContain(counter.Statements, s => s.Contains("SELECT m.id, r.time", StringComparison.Ordinal)); + } + + [Fact] + public async Task A_live_source_silent_for_longer_than_the_recent_window_is_still_stale_by_its_own_rhythm() + { + // A-40: the bounded sample finds nothing for a meter that has been quiet for months, so its whole history is + // read for those few meters — the verdict and the mark are the same as before the bound existed. + var type = await TypeAsync(); + var meter = await MeterAsync(type, MeterMode.CumulativeCounter, "kWh", installedAt: new DateOnly(2026, 1, 1)); + await using (var db = fx.CreateContext()) + { + db.MeterSources.Add(new MeterSource { MeterId = meter, SourceType = SourceType.Mqtt, Config = """{"topic":"tele/d/SENSOR"}""" }); + await db.SaveChangesAsync(); + } + + // Hourly for a day in February, then nothing: seven months of silence, far outside FreshnessRules.RecentWindow. + var start = Local(Berlin, 2026, 2, 1); + await ReadingsAsync(meter, [.. Enumerable.Range(0, 25).Select(h => (start.AddHours(h), (double)h))]); + + var result = await Reader().ReadAsync(new AnalysisRequest(AnalysisScope.ForMeter(meter), Resolve(PeriodPreset.YearToDate))); + + var freshness = result.SeriesFor(meter)!.Freshness; + Assert.Equal(FreshnessState.Stale, freshness.State); + Assert.Equal(TimeSpan.FromHours(3), freshness.StaleAfter); + Assert.Equal(start.AddHours(24), freshness.LastActivity); + } + // ------------------------------------------------------------------------------------------------ limits, availability [Fact] diff --git a/tests/Integration.Tests/Analysis/WindowSumPlanTests.cs b/tests/Integration.Tests/Analysis/WindowSumPlanTests.cs new file mode 100644 index 0000000..f9d9613 --- /dev/null +++ b/tests/Integration.Tests/Analysis/WindowSumPlanTests.cs @@ -0,0 +1,184 @@ +using MeterVault.Core.Domain; +using MeterVault.Core.Normalization; +using MeterVault.Infrastructure.Normalization; +using MeterVault.Infrastructure.Options; +using MeterVault.Infrastructure.Persistence; +using Microsoft.EntityFrameworkCore; +using Npgsql; + +namespace MeterVault.Integration.Tests.Analysis; + +/// +/// The window-sum statement of the reader (D-15, A-40). Its windows arrive through an unnest join, so their +/// bounds are columns: without the overall [min from, max to) repeated as constants, PostgreSQL has nothing to +/// exclude chunks by and plans — and, with enough windows, scans — the whole consumption hypertable. The bounds +/// are pure arithmetic over the window set, so they may never change a tally; this pins both halves of that. +/// +[Collection("Timescale")] +public sealed class WindowSumPlanTests(TimescaleFixture fx) : IAsyncLifetime +{ + private const string BerlinId = "Europe/Berlin"; + + private static readonly TimeZoneInfo Berlin = TimeZoneInfo.FindSystemTimeZoneById(BerlinId); + + /// The reader's statement (AnalysisQueries.WindowSumsAsync). + private const string Bounded = """ + SELECT w.idx, count(*)::int, sum(c.amount) + FROM unnest(@ids, @froms, @tos) WITH ORDINALITY AS w(meter_id, from_time, to_time, idx) + JOIN consumption c ON c.meter_id = w.meter_id AND c.time >= w.from_time AND c.time < w.to_time + WHERE c.time >= @min_from AND c.time < @max_to + GROUP BY w.idx + """; + + /// The same without the overall bounds, as it was sent before A-40. + private const string Unbounded = """ + SELECT w.idx, count(*)::int, sum(c.amount) + FROM unnest(@ids, @froms, @tos) WITH ORDINALITY AS w(meter_id, from_time, to_time, idx) + JOIN consumption c ON c.meter_id = w.meter_id AND c.time >= w.from_time AND c.time < w.to_time + GROUP BY w.idx + """; + + private int _meter; + private short _type; + + public async Task InitializeAsync() + { + await using var db = fx.CreateContext(); + var type = new EnergyType + { + Key = $"windows-{Guid.NewGuid():N}", + DisplayName = "Window sums", + BaseUnit = "kWh", + DefaultMode = MeterMode.CumulativeCounter, + }; + db.EnergyTypes.Add(type); + await db.SaveChangesAsync(); + _type = type.Id; + + var meter = new Meter + { + Name = $"windows-{Guid.NewGuid():N}", + EnergyTypeId = _type, + Mode = MeterMode.CumulativeCounter, + Unit = "kWh", + InstalledAt = new DateOnly(2020, 1, 1), + }; + db.Meters.Add(meter); + await db.SaveChangesAsync(); + _meter = meter.Id; + + // Three years of daily readings: `consumption` is chunked by 90 days, so the meter's own rows alone spread + // over a dozen chunks — enough for plan-time exclusion to be visible. + var register = 0d; + for (var day = new DateOnly(2020, 1, 2); day <= new DateOnly(2022, 12, 31); day = day.AddDays(1)) + { + register += 1.5; + db.Readings.Add(new Reading + { + MeterId = _meter, + Time = GapAttribution.LocalMidnight(day, Berlin).AddHours(6), + Value = register, + Quality = ReadingQuality.Measured, + }); + } + + await db.SaveChangesAsync(); + await Normalization(db).RecomputeMeterAsync(_meter, null); + await db.SaveChangesAsync(); + } + + public async Task DisposeAsync() + { + await using var db = fx.CreateContext(); + await db.Consumption.Where(c => c.MeterId == _meter).ExecuteDeleteAsync(); + await db.Readings.Where(r => r.MeterId == _meter).ExecuteDeleteAsync(); + await db.Meters.Where(m => m.Id == _meter).ExecuteDeleteAsync(); + await db.EnergyTypes.Where(t => t.Id == _type).ExecuteDeleteAsync(); + } + + [Fact] + public async Task The_overall_bounds_exclude_chunks_and_change_no_tally() + { + await using var connection = new NpgsqlConnection(fx.ConnectionString); + await connection.OpenAsync(); + + // Two windows, both inside the last three months of the meter's history. + var from = GapAttribution.LocalMidnight(new DateOnly(2022, 12, 20), Berlin); + var ids = new[] { _meter, _meter }; + var froms = new[] { from, from.AddDays(3) }; + var tos = new[] { from.AddDays(1), from.AddDays(4) }; + + var bounded = await TalliesAsync(connection, Bounded, ids, froms, tos, bounds: true); + var unbounded = await TalliesAsync(connection, Unbounded, ids, froms, tos, bounds: false); + Assert.Equal(2, bounded.Count); + Assert.Equal(unbounded, bounded); + + var boundedChunks = await ChunksAsync(connection, Bounded, ids, froms, tos, bounds: true); + var unboundedChunks = await ChunksAsync(connection, Unbounded, ids, froms, tos, bounds: false); + + // The old statement has to keep every chunk of the table in its plan; the bounded one keeps the few the + // windows can fall into. A table with a single chunk would make this vacuous, so the shape is asserted too. + Assert.True(unboundedChunks > 4, $"expected a chunked consumption table, saw {unboundedChunks} chunks in the plan"); + Assert.True( + boundedChunks < unboundedChunks, + $"the bounded statement planned {boundedChunks} chunks, the unbounded one {unboundedChunks}"); + Assert.True(boundedChunks <= 2, $"two windows three days apart should reach at most two chunks, not {boundedChunks}"); + } + + private static async Task> TalliesAsync( + NpgsqlConnection connection, string sql, int[] ids, DateTimeOffset[] froms, DateTimeOffset[] tos, bool bounds) + { + await using var command = Prepare(connection, sql, ids, froms, tos, bounds); + var rows = new List<(long, int, double)>(); + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + { + rows.Add((reader.GetInt64(0), reader.GetInt32(1), reader.GetDouble(2))); + } + + rows.Sort(); + return rows; + } + + /// How many hypertable chunks the plan of mentions. + private static async Task ChunksAsync( + NpgsqlConnection connection, string sql, int[] ids, DateTimeOffset[] froms, DateTimeOffset[] tos, bool bounds) + { + await using var command = Prepare(connection, "EXPLAIN " + sql, ids, froms, tos, bounds); + var chunks = new HashSet(StringComparer.Ordinal); + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + { + foreach (var word in reader.GetString(0).Split([' ', '(', ')', ','], StringSplitOptions.RemoveEmptyEntries)) + { + if (word.StartsWith("_hyper_", StringComparison.Ordinal)) + { + chunks.Add(word); + } + } + } + + return chunks.Count; + } + + private static NpgsqlCommand Prepare( + NpgsqlConnection connection, string sql, int[] ids, DateTimeOffset[] froms, DateTimeOffset[] tos, bool bounds) + { + var command = new NpgsqlCommand(sql, connection); + command.Parameters.AddWithValue("ids", ids); + command.Parameters.AddWithValue("froms", froms); + command.Parameters.AddWithValue("tos", tos); + if (bounds) + { + command.Parameters.AddWithValue("min_from", froms.Min()); + command.Parameters.AddWithValue("max_to", tos.Max()); + } + + return command; + } + + private static NormalizationService Normalization(MeterVaultDbContext db) => + new(db, NormalizationEngine.CreateDefault(), + Microsoft.Extensions.Options.Options.Create(new MeterVaultOptions { TimeZone = BerlinId }), + TimeProvider.System); +} diff --git a/tests/Integration.Tests/Overview/OverviewReadBudgetTests.cs b/tests/Integration.Tests/Overview/OverviewReadBudgetTests.cs new file mode 100644 index 0000000..9f6af4c --- /dev/null +++ b/tests/Integration.Tests/Overview/OverviewReadBudgetTests.cs @@ -0,0 +1,134 @@ +using MeterVault.App; +using MeterVault.Core.Analysis; +using MeterVault.Infrastructure.Costing; +using MeterVault.Infrastructure.Dashboard; +using MeterVault.Infrastructure.Import; +using MeterVault.Infrastructure.Options; +using MeterVault.Infrastructure.Persistence; +using MeterVault.Integration.Tests.Performance; +using Microsoft.EntityFrameworkCore; +using static MeterVault.Integration.Tests.Costing.CostSandbox; + +namespace MeterVault.Integration.Tests.Overview; + +/// +/// What one Overview load costs in SQL (D-15, D-56, A-40). The Overview is the widest read in the app — every energy +/// type, every meter's series, the bill with its composition and the comparison's bill — and the page builds all of it +/// from one . That call must stay a constant handful of statements +/// whatever the instance holds: one quantity read, one bill for the period and one for the comparison, each reusing the +/// one result for the cards, the change table and the composition. A figure computed twice shows up here as a repeated +/// catalog, tariff or manual-cost load long before it shows up as a slow page. +/// +/// The Overview reads the whole instance, so every test starts from — and leaves — an instance without meters. +[Collection("Timescale")] +public sealed class OverviewReadBudgetTests(TimescaleFixture fx) : IAsyncLifetime +{ + /// + /// What one load with a comparison sends: four for the shared catalog, seven for the quantities, two for the + /// manual-cost dates and the energy types, and five for each of the two bills. It is asserted exactly, so any new + /// per-type, per-category or per-meter read fails here instead of being measured later. + /// + private const int Budget = 23; + + private static readonly ComparisonRequest PreviousYear = new(ComparisonKind.PreviousYear); + + public async Task InitializeAsync() + { + await using var db = fx.CreateContext(); + await ClearAsync(db); + } + + public async Task DisposeAsync() + { + await using var db = fx.CreateContext(); + await ClearAsync(db); + } + + [Fact] + public async Task One_overview_load_prices_the_period_once_and_its_comparison_once() + { + await LoadReferenceDataAsync(); + var dashboard = Dashboard(); + + // Warm up: the first call of the process compiles EF queries and opens the pool, which sends its own statements. + await dashboard.GetOverviewAsync(Preset(PeriodPreset.PreviousYear), BucketSize.Auto, PreviousYear); + + using var counter = CommandCounter.Start(); + var overview = await dashboard.GetOverviewAsync(Preset(PeriodPreset.PreviousYear), BucketSize.Auto, PreviousYear); + var statements = counter.Statements; + + // The figures are the ones OverviewDataTests pins; this test only counts what it took to get them. + Assert.Equal(12, overview.Plan.Buckets.Count); + Assert.NotNull(overview.PreviousCost); + Assert.NotEmpty(overview.CategoryChanges); + Assert.NotEmpty(overview.LineChanges); + + Assert.True( + statements.Count == Budget, + $"One Overview load sent {statements.Count} statements, not {Budget}:\n{string.Join('\n', statements)}"); + + // The catalog — meters, tanks, links, rollup states — is loaded once and shared by all three reads (A-40). + Assert.Equal(1, Count(statements, "m.initial_baseline")); + Assert.Equal(1, Count(statements, "m.meter_id, m.built_at")); + + // One bill for the period and one for the comparison, each loading tariffs, manual costs and categories once. + Assert.Equal(2, Count(statements, "t.valid_from")); + Assert.Equal(2, Count(statements, "m.amount, m.category_id")); + Assert.Equal(2, Count(statements, "c.color_hex")); + + // One quantity read for the page, plus the one each bill makes for the meters it prices. + Assert.Equal(3, Count(statements, "span_from")); + Assert.Equal(3, Count(statements, "r.meter_id, r.month")); + + // The freshness mark comes from the stored rollup state, so nothing reads the raw hypertable: the seeded + // instance has no live source, and an import-only meter's mark is not sampled from recent readings (A-40). + Assert.Equal(0, Count(statements, "SELECT m.id, r.time")); + } + + [Fact] + public async Task An_overview_without_a_comparison_prices_the_period_once() + { + await LoadReferenceDataAsync(); + var dashboard = Dashboard(); + await dashboard.GetOverviewAsync(Preset(PeriodPreset.PreviousYear), BucketSize.Auto, new ComparisonRequest(ComparisonKind.None)); + + using var counter = CommandCounter.Start(); + await dashboard.GetOverviewAsync(Preset(PeriodPreset.PreviousYear), BucketSize.Auto, new ComparisonRequest(ComparisonKind.None)); + + Assert.Equal(1, Count(counter.Statements, "t.valid_from")); + Assert.Equal(1, Count(counter.Statements, "m.amount, m.category_id")); + Assert.Equal(1, Count(counter.Statements, "m.initial_baseline")); + } + + private static int Count(IReadOnlyList statements, string fragment) => + statements.Count(s => s.Contains(fragment, StringComparison.OrdinalIgnoreCase)); + + private DashboardService Dashboard() + { + var clock = new FixedTimeProvider(Now); + var options = Microsoft.Extensions.Options.Options.Create(new MeterVaultOptions { TimeZone = BerlinId }); + return new DashboardService(fx, new CostService(fx, options, clock), clock); + } + + private async Task LoadReferenceDataAsync() + { + await using var db = fx.CreateContext(); + var importer = new ReferenceDataImporter(db, new ImportService(db, Normalization(db)), new CsvImporter()); + await importer.LoadAsync(Path.Combine(AppContext.BaseDirectory, "fixtures")); + } + + private static async Task ClearAsync(MeterVaultDbContext db) + { + await db.MeterLinks.ExecuteDeleteAsync(); + await db.Consumption.ExecuteDeleteAsync(); + await db.Readings.ExecuteDeleteAsync(); + await db.MeterEvents.ExecuteDeleteAsync(); + await db.ManualCosts.ExecuteDeleteAsync(); + await db.CostCategoryMembers.ExecuteDeleteAsync(); + await db.Tariffs.ExecuteDeleteAsync(); + await db.Tanks.ExecuteDeleteAsync(); + await db.MeterSources.ExecuteDeleteAsync(); + await db.Meters.ExecuteDeleteAsync(); + await db.ImportBatches.ExecuteDeleteAsync(); + } +} diff --git a/tests/Integration.Tests/Performance/ReaderTimingTests.cs b/tests/Integration.Tests/Performance/ReaderTimingTests.cs index 5dc7ef0..8b84db5 100644 --- a/tests/Integration.Tests/Performance/ReaderTimingTests.cs +++ b/tests/Integration.Tests/Performance/ReaderTimingTests.cs @@ -304,6 +304,20 @@ public sealed class ReaderTimingTests(ITestOutputHelper output) """; const string raw = "SELECT meter_id, time, amount, quality FROM consumption WHERE meter_id = ANY(@i0) AND time >= @f0 AND time < @t0"; const string windows = """ + SELECT w.idx, count(*)::int, sum(c.amount), + coalesce(sum(c.amount) FILTER (WHERE c.quality = 0), 0), + coalesce(sum(c.amount) FILTER (WHERE c.quality = 2), 0), + coalesce(sum(c.amount) FILTER (WHERE c.quality = 3), 0), + coalesce(sum(c.amount) FILTER (WHERE c.quality NOT IN (0, 2, 3)), 0) + FROM unnest(@ids, @froms, @tos) WITH ORDINALITY AS w(meter_id, from_time, to_time, idx) + JOIN consumption c ON c.meter_id = w.meter_id AND c.time >= w.from_time AND c.time < w.to_time + WHERE c.time >= @min_from AND c.time < @max_to + GROUP BY w.idx + """; + + // Not the reader's SQL any more (A-40): the same statement without the overall bounds, to show what plan-time + // chunk exclusion saves. + const string windowsUnbounded = """ SELECT w.idx, count(*)::int, sum(c.amount), coalesce(sum(c.amount) FILTER (WHERE c.quality = 0), 0), coalesce(sum(c.amount) FILTER (WHERE c.quality = 2), 0), @@ -329,6 +343,15 @@ public sealed class ReaderTimingTests(ITestOutputHelper output) WHERE (f.flags & @flag) <> 0 """; const string recent = """ + SELECT m.id, r.time + FROM unnest(@ids) AS m(id) + CROSS JOIN LATERAL ( + SELECT time FROM reading WHERE meter_id = m.id AND time >= @since ORDER BY time DESC LIMIT @count) r + """; + + // Not the reader's SQL any more (A-40): the same statement without the window bound, which is what made every + // non-quantities-only request plan across every raw chunk. + const string recentUnbounded = """ SELECT m.id, r.time FROM unnest(@ids) AS m(id) CROSS JOIN LATERAL ( @@ -374,23 +397,23 @@ public sealed class ReaderTimingTests(ITestOutputHelper output) ("i0", leaves), ("f0", Midnight(today)), ("t0", Midnight(today.AddDays(1)))); await PlanAsync("(a) coverage runs", coverage, ("ids", leaves)); await PlanAsync("(a) opening balances", opening, ("ids", leaves), ("flag", (int)RollupFlags.OpeningBalance)); - await PlanAsync("(a) freshness — the latest readings per meter (reading is compressed after 30 days)", recent, ("ids", leaves), ("count", FreshnessRules.RecentReadingCount)); + var since = now.AddDays(-FreshnessRules.RecentWindow.TotalDays); + var liveMeters = manifest.Groups["live"]; + await PlanAsync( + $"(a) freshness — the rhythm of the live meters among the leaves, {FreshnessRules.RecentWindow.TotalDays:F0} days back (reading is compressed after 30 days)", + recent, ("ids", leaves.Where(liveMeters.Contains).ToArray()), ("since", since), ("count", FreshnessRules.RecentReadingCount)); await PlanAsync("(a) freshness — the latest event per meter", events, ("ids", leaves)); - await PlanAsync($"(b) freshness — the latest readings of every physical meter ({portfolio.Length}), as a portfolio request loads them", recent, - ("ids", portfolio), ("count", FreshnessRules.RecentReadingCount)); + await PlanAsync($"(b) freshness — the rhythm of every live meter ({liveMeters.Length}), as a portfolio request loads it", recent, + ("ids", liveMeters), ("since", since), ("count", FreshnessRules.RecentReadingCount)); + await PlanAsync( + $"(comparison, not the reader's SQL) the same statement unbounded for every physical meter ({portfolio.Length}), as it was sent before A-40", + recentUnbounded, ("ids", portfolio), ("count", FreshnessRules.RecentReadingCount)); - // Not the reader's SQL: the same statement with a constant lower time bound, to show what chunk exclusion saves. var daily = manifest.Samples["dailyMeter"]; - const string recentBounded = """ - SELECT m.id, r.time - FROM unnest(@ids) AS m(id) - CROSS JOIN LATERAL ( - SELECT time FROM reading WHERE meter_id = m.id AND time >= @since ORDER BY time DESC LIMIT @count) r - """; - await PlanAsync("(d) freshness — one daily meter, as every single-meter request sends it", recent, + await PlanAsync("(d) freshness — one live meter, as a single-meter request sends it", recent, + ("ids", new[] { manifest.Samples["liveMeter"] }), ("since", since), ("count", FreshnessRules.RecentReadingCount)); + await PlanAsync("(comparison, not the reader's SQL) the same for one daily meter, unbounded, as it was sent before A-40", recentUnbounded, ("ids", new[] { daily }), ("count", FreshnessRules.RecentReadingCount)); - await PlanAsync("(comparison, not the reader's SQL) the same for one daily meter with a constant 90-day lower bound", recentBounded, - ("ids", new[] { daily }), ("since", now.AddDays(-90)), ("count", FreshnessRules.RecentReadingCount)); await PlanAsync($"(b) month rollups — portfolio, {portfolio.Length} physical meters × the 11 complete months of the last 12", month, ("ids", portfolio), ("froms", portfolio.Select(_ => currentMonth.AddMonths(-11)).ToArray()), ("tos", portfolio.Select(_ => currentMonth).ToArray())); @@ -401,13 +424,20 @@ public sealed class ReaderTimingTests(ITestOutputHelper output) await PlanAsync("(b'') window sums — one meter's two partial days (today, and its image a year ago): the small case", windows, ("ids", new[] { single, single }), ("froms", new[] { Midnight(today), Midnight(yearAgo) }), - ("tos", new[] { now, yearAgoCut })); + ("tos", new[] { now, yearAgoCut }), + ("min_from", Midnight(yearAgo)), ("max_to", now)); + + var stressIds = portfolio.Concat(portfolio).ToArray(); + var stressFroms = portfolio.Select(_ => Midnight(today)).Concat(portfolio.Select(_ => Midnight(yearAgo))).ToArray(); + var stressTos = portfolio.Select(_ => now).Concat(portfolio.Select(_ => yearAgoCut)).ToArray(); await PlanAsync( $"(b'') window sums — stress case, not what (b'') sends (see its SQL list): today's partial day and its image a year ago for all {portfolio.Length} meters", windows, - ("ids", portfolio.Concat(portfolio).ToArray()), - ("froms", portfolio.Select(_ => Midnight(today)).Concat(portfolio.Select(_ => Midnight(yearAgo))).ToArray()), - ("tos", portfolio.Select(_ => now).Concat(portfolio.Select(_ => yearAgoCut)).ToArray())); + ("ids", stressIds), ("froms", stressFroms), ("tos", stressTos), + ("min_from", stressFroms.Min()), ("max_to", stressTos.Max())); + await PlanAsync( + "(comparison, not the reader's SQL) the same stress case without the overall bounds, as it was sent before A-40", + windowsUnbounded, ("ids", stressIds), ("froms", stressFroms), ("tos", stressTos)); await PlanAsync("(d'') day rollups — one daily meter, the last 365 days", day, ("ids", new[] { daily }), ("froms", new[] { today.AddDays(-364) }), ("tos", new[] { today }),