Analysis: bound the freshness query, let window sums skip chunks, share the Overview's catalog

Three things the 1,000-meter x 10-year measurement found, each proved by
EXPLAIN or a statement count before and after. No tally moves.

Freshness had two jobs in one unbounded query. The mark -- when a meter
last delivered -- is now stored on meter_rollup_state and maintained by
every recompute, with a one-pass backfill in the migration, so an
import-only meter keeps its years-old last activity without reading a
single raw row. The rhythm that decides stale versus live is sampled
inside a 90-day window and only for meters that actually have a live
source; a source silent for longer than that is re-read unbounded, so it
is still called stale by its own rhythm rather than by a default. The
portfolio query went from 13.8 ms planning plus 36.1 ms execution across
all 123 reading chunks to 0.58 plus 0.44 ms across four.

Window sums took their time bounds only from the unnest join, so the
planner could not exclude chunks: a 1,960-window case scanned 1.39 M rows
in parallel and spilled a 45 MB sort. Repeating the overall min and max as
constants makes it five chunks and nested-loop index scans, 121.5 ms to
8.7 ms.

The Overview read the catalog three times, once for the quantities and
once for each of its two bills. One context and one catalog snapshot now
feed all three: 31 statements per load to 23.

The final timings on an idle machine are in docs/ANALYSIS_REPORT.md: the
brief's target request (100 meters, ten years, monthly) is 286 ms against
two seconds, and a startup rebuild of 1,000 meters is 279 s.

docs/ANALYSIS_IMPLEMENTATION_NOTE.md

@@ -8,6 +8,7 @@ The note was kept current through Phase 5:
 - §11: amendments from the Phase 1 module review.
 - §12: amendments from the acceptance review.
 - §13: decisions recorded with the final documentation.
+- §14: amendments from the performance measurement.
 - §9 and §10: extended with what the implementation measured and changed.
 
 The outcome is in [ANALYSIS_REPORT.md](ANALYSIS_REPORT.md), the user-facing changes in
@@ -708,3 +709,32 @@ figure.
     keep the existing detour and its draft.
   - The theme defaults to dark when no `mv-theme` cookie is set.
   - The Calculation tab words a calculation problem as the attention list does, one wording per `VirtualProblemKind`.
+
+## 14. Amendments from the performance measurement
+
+The measurement of brief §9.10 / D-56 on a synthetic 1,000-meter × 10-year instance named three costs that grow with
+history rather than with what a page asks for. They are fixed here. No golden bill, reconciliation figure or displayed
+value changes; only what a request reads does.
+
+- **A-40 What a request reads (D-15, D-18, D-56).**
+  - **The freshness mark is stored, not searched.** `meter_rollup_state` gains `last_reading_at`: the stamp of the
+    meter's latest raw reading, written by the recompute that every write path already runs. D-18 is unchanged — the
+    last reading or event time is still the mark — but a request no longer queries `reading` to find it, so an
+    import-only meter's mark stays exactly as old as its data without planning across a decade of raw chunks. The
+    migration backfills the column in one pass, so nothing waits for a rebuild.
+  - **Only a live source's rhythm is sampled, and only from the recent past.** The median of D-18 is consulted for a
+    meter with a live source, so the reading times are read for those meters alone, bounded by
+    `FreshnessRules.RecentWindow` (90 days) — the bound is what lets PostgreSQL exclude the older chunks at plan time.
+    A live meter that delivered nothing inside the window has no rhythm there; those few meters are read again over
+    their whole history, so a long-silent source is still called stale by its own rhythm. An instance without any live
+    source reads `reading` not at all.
+  - **The window-sum statement carries its overall bounds.** Its windows arrive through an `unnest` join, so their
+    bounds are columns and exclude no chunk. The minimum start and maximum end of the window set are repeated as
+    constants in the `WHERE` clause. No row outside them can match any window, so no tally changes; with enough
+    windows it is the difference between an indexed probe and a parallel scan of the whole hypertable with a sort
+    spilling to disk.
+  - **One Overview load holds one catalog.** The page's quantities, its bill and its comparison's bill are three
+    figures of one period. `DashboardService.GetOverviewAsync` loads the meters, tanks, links and rollup states once
+    and passes that snapshot to all three (`CostReader.ReadAsync(db, catalog, …)`), so every figure answers from the
+    same snapshot and the load is not repeated. The cards, the change table and the composition are derived from those
+    results and are never priced again. `OverviewReadBudgetTests` asserts the statement count of one load.

docs/ANALYSIS_REPORT.md

@@ -10,8 +10,9 @@ decisions were applied to existing data, what was tested, and what is still miss
   behaviour is deliberate, the decision id says so.
 - **User-facing changes and the upgrade path:** [RELEASE_NOTES.md](RELEASE_NOTES.md). This report does not repeat
   them.
-- **Build state at the end:** `dotnet build` 0 errors; Core.Tests 1,733 of 1,733 and Integration.Tests 746 of 746
-  passing; the golden spreadsheet reconciliation and the seeded bill goldens unmoved by the last fix round.
+- **Build state at the end:** `dotnet build` 0 errors; Core.Tests 1,733 of 1,733 and Integration.Tests 751 of 751
+  passing (plus the two opt-in performance facts, skipped); the golden spreadsheet reconciliation and the seeded
+  bill goldens unmoved by the last fix round.
 
 ## 1. Findings A01 – A14
 
@@ -176,10 +177,11 @@ not repeated here.
 | Suite | Count | Needs | Covers |
 |---|---:|---|---|
 | `tests/Core.Tests` | 1,733 | nothing | parsers, normalizers, swap→12, and `Analysis/`: periods, DST, comparisons, buckets, coverage, rollups, totals, category cover, virtual formulas and evaluation, the cost calculator |
-| `tests/Integration.Tests` | 746 | Docker (TimescaleDB) for the database facts | reconciliation against the four golden CSVs, import commit and revert, ingestion and events, rollups, the reader, the cost engine, the seeded bill, API contracts, the export, rendered pages, plus the pure UI-model tests under `Analysis/`, `MeterPage/`, `Overview/`, `Editor/` and `Specialized/` |
+| `tests/Integration.Tests` | 751 | Docker (TimescaleDB) for the database facts | reconciliation against the four golden CSVs, import commit and revert, ingestion and events, rollups, the reader, the cost engine, the seeded bill, API contracts, the export, rendered pages, plus the pure UI-model tests under `Analysis/`, `MeterPage/`, `Overview/`, `Editor/` and `Specialized/` |
 | `tests/Integration.Tests/Performance` | 2 | `METERVAULT_PERF=1` | the synthetic 1,000-meter × 10-year dataset, reader and cost timings, statement counts, query plans; skipped by default |
 
-All green at the end of the last fix round, where Integration rose from 727 to 746.
+All green at the end of the last fix round, where Integration rose from 727 to 746, and again after the performance
+fixes of §5.4, which added five more.
 
 ### 4.2 Kinds of tests
 
@@ -281,31 +283,34 @@ Measured with the opt-in Performance trait (`METERVAULT_PERF=1`,
 types, 21 links, 109 tariff rows, 64 manual costs and 20 virtual meters nested up to three levels.
 
 Hardware: Ryzen 9 9950X3D (16 cores, 32 threads), 64 GB, Windows 11, Docker Desktop, PostgreSQL 16.6 with
-TimescaleDB 2.17.2 (`jit=off`), .NET 10 Release. **These numbers are preliminary**: they were taken while other
-agents were building and testing on the same machine (host 20–40 % CPU). They are re-runnable as above.
+TimescaleDB 2.17.2 (`jit=off`), .NET 10 Release. The figures below are the final run, taken on an otherwise idle
+machine (host 7–13 % CPU) after the fixes of §5.4. The "first run" column is the earlier measurement, taken before
+those fixes and while other work was building and testing on the same machine — it is kept to show what the fixes
+moved, not as a like-for-like comparison.
 
 ### 5.1 Reader and cost requests
 
 Median and p95 of 10 runs after 2 warm-ups; SQL is the number of commands one extra, untimed run sent.
 
-| Request | Median ms | p95 ms | SQL |
-|---|---:|---:|---:|
-| 100 selected meters, 10 years by month, catalog cached — **the brief's 2 s target** | 374 | 509 | 8 |
-| Portfolio, last 12 months by month (measures only, as the Overview) | 268 | 284 | 12 |
-| … with one series per meter (1,000 series) | 379 | 403 | 12 |
-| … with the previous-year comparison | 414 | 458 | 14 |
-| One energy type (402 meters), 10 years by month | 313 | 347 | 12 |
-| … with one series per meter (the type page's table) | 917 | 1,016 | 12 |
-| One meter: 10 years by week, or 365 days by day | 29–36 | 32–58 | 11 |
-| Portfolio bill, 12 months by month, with categories | 320 | 357 | 16 |
-| Portfolio bill, 10 years by month | 838 | 875 | 15 |
-| Virtual meter nested three levels, 10 years by month | 245 | 272 | 11 |
-| Virtual difference of a 40-meter and an 8-meter sum, 10 years by month | 377 | 442 | 11 |
-| Catalog load: 1,000 meters, tanks, links, states, validation, classification | 7 | 8 | 4 |
-| 1,000-meter selection with the limit raised, 12 months by month | 353 | 394 | 12 |
-| Refused: 1,000 meters against the 6-series limit; portfolio or bill by day over 10 years | 0 | 0 | **0** |
+| Request | Median ms | p95 ms | SQL | First run, median |
+|---|---:|---:|---:|---:|
+| 100 selected meters, 10 years by month, catalog cached — **the brief's 2 s target** | 286 | 472 | 8 | 374 |
+| … the same request through the public API | 274 | 317 | 12 | 361 |
+| Portfolio, last 12 months by month (measures only, as the Overview) | 70 | 75 | 12 | 268 |
+| … with one series per meter (1,000 series) | 170 | 186 | 12 | 379 |
+| … with the previous-year comparison | 156 | 177 | 14 | 414 |
+| One energy type (402 meters), 10 years by month | 156 | 182 | 12 | 313 |
+| … with one series per meter (the type page's table) | 698 | 731 | 12 | 917 |
+| One meter: 10 years by week, or 365 days by day | 12–16 | 13–27 | 10–11 | 29–36 |
+| Portfolio bill, 12 months by month, with categories | 144 | 204 | 16 | 320 |
+| Portfolio bill, 10 years by month | 577 | 627 | 15 | 838 |
+| Virtual meter nested three levels, 10 years by month | 197 | 216 | 10 | 245 |
+| Virtual difference of a 40-meter and an 8-meter sum, 10 years by month | 300 | 344 | 11 | 377 |
+| Catalog load: 1,000 meters, tanks, links, states, validation, classification | 6 | 10 | 4 | 7 |
+| 1,000-meter selection with the limit raised, 12 months by month | 151 | 171 | 12 | 353 |
+| Refused: 1,000 meters against the 6-series limit; portfolio or bill by day over 10 years | 0 | 0 | **0** | 0 |
 
-The brief's target is met with margin: 374 ms median and 509 ms p95 against 2,000 ms. The statement count per
+The brief's target is met with margin: 286 ms median and 472 ms p95 against 2,000 ms. The statement count per
 request is constant at 8–16 whether the request covers 1, 100 or 1,000 meters — there is no per-meter query storm.
 Refused requests send no SQL at all, because the 400-point and 6-series limits are checked before any query is
 built. Rollup reads use the primary-key index: 4 ms for 125 meters over 10 years of month rollups, 7 ms for 980
@@ -313,10 +318,10 @@ meters over 11 months.
 
 ### 5.2 Rebuild and per-reading recompute
 
-Rebuilding all 1,000 meters at startup through `NormalizationUpgrade` took 344–392 s: about 0.1 s per monthly
-meter, ~0.6 s for a daily meter (3,670 readings) and ~1.4 s for a meter with a year of hourly data (9,314
-readings). That is also the cost of one ingested reading, because every write path recomputes its meter in full
-(D-57).
+Rebuilding all 1,000 meters at startup through `NormalizationUpgrade` took 279 s, about 3.6 meters per second.
+One meter's full recompute is 51–56 ms with monthly readings, 0.50 s for a daily meter (3,670 readings) and 1.16 s
+for a meter with a year of hourly data (9,314 readings). That is also the cost of one ingested reading, because
+every write path recomputes its meter in full (D-57).
 
 ### 5.3 Page loads, old against new
 
@@ -334,24 +339,31 @@ per GET taken from a separate logging pass.
 
 The old Overview ran a meter load, a tariff load and a `time_bucket` query 8,021 times per request. The individual
 meter pages and `/consumables` stayed in the same range (tens to low hundreds of milliseconds) and are not listed.
-The new-side page numbers were taken on the foundation-era pages, before the page rework landed, so they are an
-order of magnitude rather than a final figure.
+The new-side numbers in that table came from the foundation-era pages on a busy machine, so they are an order of
+magnitude rather than a final figure; the finished pages, measured on the idle machine against the same
+1,000-meter instance, load in 455 ms (`/`), 383 ms (`/meters`), 469 ms (`/solar`), 641 ms (`/energy/1`),
+684 ms (`/trends`), 104 ms (`/meters/1`) and 152 ms (a nested virtual meter).
 
-### 5.4 Hotspots found and not fixed
+### 5.4 Hotspots found and fixed
 
-1. **The freshness query has no time bound.** `AnalysisQueries.RecentReadingsAsync` (the latest 20 reading times
-   per meter, D-18) plans across every raw chunk: 14–24 ms of planning over 123 `reading` chunks, 189 ms on a cold
-   connection, and 38–46 ms of execution for a portfolio walking back through compressed chunks. With a constant
-   90-day lower bound the same single-meter statement plans in 0.9 ms. It grows with history length.
-2. **Window sums get no plan-time chunk exclusion.** `AnalysisQueries.WindowSumsAsync` takes its bounds only from
-   the `unnest` join. Real requests send few windows and cost 4.6 ms, but a stress case of ~2,000 windows flips to
-   a parallel sequential scan of 1.39 M rows with a sort spilling to disk, 138–152 ms. Latent, and it grows with
-   total history rather than with the number of windows.
-3. **The reworked Overview computes several portfolio bills per request** — nine in the measured build — which
-   shows as repeated catalog, tariff and manual-cost loads (526 ms of SQL inside a 2.4 s page).
+The first measurement found three. All three are fixed (A-40), each proved by `EXPLAIN` or a statement count
+before and after, with the tallies unchanged.
 
-In-process work dominates the rest and scales with series × buckets (roughly 10–16 µs per series bucket). That is
-the budget that matters for very large exports and tables, not for a normal page.
+| Hotspot | Before | After |
+|---|---|---|
+| **The freshness query had no time bound.** `AnalysisQueries.RecentReadingsAsync` planned across all 123 raw chunks on every request, for every meter. | 13.8 ms planning + 36.1 ms execution for the portfolio (980 meters); 189 ms on a cold connection | 0.58 + 0.44 ms, over 4 chunks and only the 40 meters with a live source; an instance without a live source never reads `reading` at all |
+| **Window sums could not exclude chunks.** `AnalysisQueries.WindowSumsAsync` took its bounds only from the `unnest` join. | a 1,960-window stress case: 42 chunks, parallel scan of 1.39 M rows, ~45 MB sort spill, 121.5 ms | 5 chunks, nested-loop index scans, 8.7 ms |
+| **The Overview loaded the catalog three times**, once per read (quantities, bill, comparison bill). | 31 statements per load; 64 statements and 133 ms of SQL on the 1,000-meter instance | 23 statements per load; 39 statements and 81 ms of SQL |
+
+The freshness fix splits what D-18 had conflated: the *mark* (a meter's last reading time) is now stored on
+`meter_rollup_state` and maintained by every recompute, with a one-pass backfill in the `FreshnessMark` migration,
+so an import-only meter keeps its years-old "last activity" without touching `reading`; the *rhythm* (the sample
+intervals that decide stale versus live) is sampled inside a 90-day window, and only for meters that have a live
+source. A live source that has been silent longer than that window is re-read unbounded, so it is still reported
+stale by its own rhythm rather than by a default.
+
+In-process work dominates what is left, and scales with series × buckets (roughly 10–16 µs per series bucket).
+That is the budget that matters for very large exports and tables, not for a normal page.
 
 ## 6. Remaining limitations and known gaps
 
@@ -368,7 +380,9 @@ the budget that matters for very large exports and tables, not for a normal page
 
 **Further limitations, measured or decided during the work:**
 
-- The three performance hotspots in §5.4 are known and unfixed.
+- `AnalysisQueries.OpeningBalancesAsync` has the same unbounded shape the freshness query had (a lateral
+  `ORDER BY time LIMIT 1` over `consumption`, 6.4 ms in the heaviest measured request). It was left alone: smaller,
+  and `consumption` has 42 chunks against `reading`'s 123.
 - The billing basis (grid meter or household use) is chosen per energy type for all time and cannot switch month by
   month, because the category composition would need the same per-month basis to stay reconciled with the bill
   (A-17).

docs/RELEASE_NOTES.md

@@ -24,15 +24,18 @@ Back up the database (`pg_dump`) first.
     state of each meter.
   - This runs **before the web server listens**. The app is unreachable meanwhile, and Compose may report the
     container unhealthy. Let it finish.
-  - The time grows with the number of raw readings. One meter took about 0.1 s with monthly readings, ~0.6 s with ten
-    years of daily readings (3,700 readings), and ~1.4 s with a year of hourly readings (9,300 readings).
-  - A synthetic 1,000-meter × 10-year instance (1.34 M readings) took 5.6–6.5 minutes (338–392 s) on a Ryzen 9
-    9950X3D. That run was preliminary, on a busy machine. It is about 20 % slower per meter than the 0.3.0 rebuild,
-    because rollups and coverage are written too.
+  - The time grows with the number of raw readings. One meter took about 0.05 s with monthly readings, ~0.5 s with ten
+    years of daily readings (3,700 readings), and ~1.2 s with a year of hourly readings (9,300 readings).
+  - A synthetic 1,000-meter × 10-year instance (1.34 M readings) took 4.7 minutes (279 s) on an idle Ryzen 9 9950X3D —
+    about 3.6 meters a second. It writes rollups and coverage on top of consumption, so it does more per meter than
+    the 0.3.0 rebuild did.
   - Progress is logged every 100 meters. A meter that fails is logged, kept pending and retried at the next start.
     A meter whose stored consumption is older than its oldest remaining reading is skipped and logged, so its
     history is not truncated.
   - Until a meter is rebuilt, its pages say "analysis being prepared", never "no data".
+- **The migration records each meter's last reading time** in the analysis state table and fills it in for the meters
+  you already have, in one pass over the readings. It is what the pages show as "last activity" and what decides
+  whether a live source is late, so no page has to search the raw readings for it any more.
 - **Virtual meters without a formula** (such as a seeded *Summe Solar* from an earlier version) are converted at
   startup. When their incoming links name meters of one unit and kind, the implied sum is stored as an explicit
   formula (D-28). The log lists the converted meters, and the meters that still need configuration because their
@@ -214,7 +217,7 @@ engine, as listed above: actuals stop at now, virtual meters are evaluated, and
   and offers the monthly view.
 - **Bonus, Discount and Tax tariffs are stored but not applied** (D-57).
 - **Every live reading recomputes its meter in full** (D-57). That is fine for monthly and daily meters, but costs
-  about 1.4 s per reading for a meter with a year of hourly data, and grows with history.
+  about 1.2 s per reading for a meter with a year of hourly data, and grows with history.
 - **Months cannot switch billing basis:** the billing basis (grid meter or household use) is chosen per energy type
   for all time (A-17).
 - **Batteries are not modelled.** Without a grid-export meter, Solar's feed-in is calculated, and labelled as such.