Analysis: bound the freshness query, let window sums skip chunks, share the Overview's catalog

Three things the 1,000-meter x 10-year measurement found, each proved by
EXPLAIN or a statement count before and after. No tally moves.

Freshness had two jobs in one unbounded query. The mark -- when a meter
last delivered -- is now stored on meter_rollup_state and maintained by
every recompute, with a one-pass backfill in the migration, so an
import-only meter keeps its years-old last activity without reading a
single raw row. The rhythm that decides stale versus live is sampled
inside a 90-day window and only for meters that actually have a live
source; a source silent for longer than that is re-read unbounded, so it
is still called stale by its own rhythm rather than by a default. The
portfolio query went from 13.8 ms planning plus 36.1 ms execution across
all 123 reading chunks to 0.58 plus 0.44 ms across four.

Window sums took their time bounds only from the unnest join, so the
planner could not exclude chunks: a 1,960-window case scanned 1.39 M rows
in parallel and spilled a 45 MB sort. Repeating the overall min and max as
constants makes it five chunks and nested-loop index scans, 121.5 ms to
8.7 ms.

The Overview read the catalog three times, once for the quantities and
once for each of its two bills. One context and one catalog snapshot now
feed all three: 31 statements per load to 23.

The final timings on an idle machine are in docs/ANALYSIS_REPORT.md: the
brief's target request (100 meters, ten years, monthly) is 286 ms against
two seconds, and a startup rebuild of 1,000 meters is 279 s.

docs/ANALYSIS_REPORT.md

@@ -10,8 +10,9 @@ decisions were applied to existing data, what was tested, and what is still miss
   behaviour is deliberate, the decision id says so.
 - **User-facing changes and the upgrade path:** [RELEASE_NOTES.md](RELEASE_NOTES.md). This report does not repeat
   them.
-- **Build state at the end:** `dotnet build` 0 errors; Core.Tests 1,733 of 1,733 and Integration.Tests 746 of 746
-  passing; the golden spreadsheet reconciliation and the seeded bill goldens unmoved by the last fix round.
+- **Build state at the end:** `dotnet build` 0 errors; Core.Tests 1,733 of 1,733 and Integration.Tests 751 of 751
+  passing (plus the two opt-in performance facts, skipped); the golden spreadsheet reconciliation and the seeded
+  bill goldens unmoved by the last fix round.
 
 ## 1. Findings A01 – A14
 
@@ -176,10 +177,11 @@ not repeated here.
 | Suite | Count | Needs | Covers |
 |---|---:|---|---|
 | `tests/Core.Tests` | 1,733 | nothing | parsers, normalizers, swap→12, and `Analysis/`: periods, DST, comparisons, buckets, coverage, rollups, totals, category cover, virtual formulas and evaluation, the cost calculator |
-| `tests/Integration.Tests` | 746 | Docker (TimescaleDB) for the database facts | reconciliation against the four golden CSVs, import commit and revert, ingestion and events, rollups, the reader, the cost engine, the seeded bill, API contracts, the export, rendered pages, plus the pure UI-model tests under `Analysis/`, `MeterPage/`, `Overview/`, `Editor/` and `Specialized/` |
+| `tests/Integration.Tests` | 751 | Docker (TimescaleDB) for the database facts | reconciliation against the four golden CSVs, import commit and revert, ingestion and events, rollups, the reader, the cost engine, the seeded bill, API contracts, the export, rendered pages, plus the pure UI-model tests under `Analysis/`, `MeterPage/`, `Overview/`, `Editor/` and `Specialized/` |
 | `tests/Integration.Tests/Performance` | 2 | `METERVAULT_PERF=1` | the synthetic 1,000-meter × 10-year dataset, reader and cost timings, statement counts, query plans; skipped by default |
 
-All green at the end of the last fix round, where Integration rose from 727 to 746.
+All green at the end of the last fix round, where Integration rose from 727 to 746, and again after the performance
+fixes of §5.4, which added five more.
 
 ### 4.2 Kinds of tests
 
@@ -281,31 +283,34 @@ Measured with the opt-in Performance trait (`METERVAULT_PERF=1`,
 types, 21 links, 109 tariff rows, 64 manual costs and 20 virtual meters nested up to three levels.
 
 Hardware: Ryzen 9 9950X3D (16 cores, 32 threads), 64 GB, Windows 11, Docker Desktop, PostgreSQL 16.6 with
-TimescaleDB 2.17.2 (`jit=off`), .NET 10 Release. **These numbers are preliminary**: they were taken while other
-agents were building and testing on the same machine (host 20–40 % CPU). They are re-runnable as above.
+TimescaleDB 2.17.2 (`jit=off`), .NET 10 Release. The figures below are the final run, taken on an otherwise idle
+machine (host 7–13 % CPU) after the fixes of §5.4. The "first run" column is the earlier measurement, taken before
+those fixes and while other work was building and testing on the same machine — it is kept to show what the fixes
+moved, not as a like-for-like comparison.
 
 ### 5.1 Reader and cost requests
 
 Median and p95 of 10 runs after 2 warm-ups; SQL is the number of commands one extra, untimed run sent.
 
-| Request | Median ms | p95 ms | SQL |
-|---|---:|---:|---:|
-| 100 selected meters, 10 years by month, catalog cached — **the brief's 2 s target** | 374 | 509 | 8 |
-| Portfolio, last 12 months by month (measures only, as the Overview) | 268 | 284 | 12 |
-| … with one series per meter (1,000 series) | 379 | 403 | 12 |
-| … with the previous-year comparison | 414 | 458 | 14 |
-| One energy type (402 meters), 10 years by month | 313 | 347 | 12 |
-| … with one series per meter (the type page's table) | 917 | 1,016 | 12 |
-| One meter: 10 years by week, or 365 days by day | 29–36 | 32–58 | 11 |
-| Portfolio bill, 12 months by month, with categories | 320 | 357 | 16 |
-| Portfolio bill, 10 years by month | 838 | 875 | 15 |
-| Virtual meter nested three levels, 10 years by month | 245 | 272 | 11 |
-| Virtual difference of a 40-meter and an 8-meter sum, 10 years by month | 377 | 442 | 11 |
-| Catalog load: 1,000 meters, tanks, links, states, validation, classification | 7 | 8 | 4 |
-| 1,000-meter selection with the limit raised, 12 months by month | 353 | 394 | 12 |
-| Refused: 1,000 meters against the 6-series limit; portfolio or bill by day over 10 years | 0 | 0 | **0** |
+| Request | Median ms | p95 ms | SQL | First run, median |
+|---|---:|---:|---:|---:|
+| 100 selected meters, 10 years by month, catalog cached — **the brief's 2 s target** | 286 | 472 | 8 | 374 |
+| … the same request through the public API | 274 | 317 | 12 | 361 |
+| Portfolio, last 12 months by month (measures only, as the Overview) | 70 | 75 | 12 | 268 |
+| … with one series per meter (1,000 series) | 170 | 186 | 12 | 379 |
+| … with the previous-year comparison | 156 | 177 | 14 | 414 |
+| One energy type (402 meters), 10 years by month | 156 | 182 | 12 | 313 |
+| … with one series per meter (the type page's table) | 698 | 731 | 12 | 917 |
+| One meter: 10 years by week, or 365 days by day | 12–16 | 13–27 | 10–11 | 29–36 |
+| Portfolio bill, 12 months by month, with categories | 144 | 204 | 16 | 320 |
+| Portfolio bill, 10 years by month | 577 | 627 | 15 | 838 |
+| Virtual meter nested three levels, 10 years by month | 197 | 216 | 10 | 245 |
+| Virtual difference of a 40-meter and an 8-meter sum, 10 years by month | 300 | 344 | 11 | 377 |
+| Catalog load: 1,000 meters, tanks, links, states, validation, classification | 6 | 10 | 4 | 7 |
+| 1,000-meter selection with the limit raised, 12 months by month | 151 | 171 | 12 | 353 |
+| Refused: 1,000 meters against the 6-series limit; portfolio or bill by day over 10 years | 0 | 0 | **0** | 0 |
 
-The brief's target is met with margin: 374 ms median and 509 ms p95 against 2,000 ms. The statement count per
+The brief's target is met with margin: 286 ms median and 472 ms p95 against 2,000 ms. The statement count per
 request is constant at 8–16 whether the request covers 1, 100 or 1,000 meters — there is no per-meter query storm.
 Refused requests send no SQL at all, because the 400-point and 6-series limits are checked before any query is
 built. Rollup reads use the primary-key index: 4 ms for 125 meters over 10 years of month rollups, 7 ms for 980
@@ -313,10 +318,10 @@ meters over 11 months.
 
 ### 5.2 Rebuild and per-reading recompute
 
-Rebuilding all 1,000 meters at startup through `NormalizationUpgrade` took 344–392 s: about 0.1 s per monthly
-meter, ~0.6 s for a daily meter (3,670 readings) and ~1.4 s for a meter with a year of hourly data (9,314
-readings). That is also the cost of one ingested reading, because every write path recomputes its meter in full
-(D-57).
+Rebuilding all 1,000 meters at startup through `NormalizationUpgrade` took 279 s, about 3.6 meters per second.
+One meter's full recompute is 51–56 ms with monthly readings, 0.50 s for a daily meter (3,670 readings) and 1.16 s
+for a meter with a year of hourly data (9,314 readings). That is also the cost of one ingested reading, because
+every write path recomputes its meter in full (D-57).
 
 ### 5.3 Page loads, old against new
 
@@ -334,24 +339,31 @@ per GET taken from a separate logging pass.
 
 The old Overview ran a meter load, a tariff load and a `time_bucket` query 8,021 times per request. The individual
 meter pages and `/consumables` stayed in the same range (tens to low hundreds of milliseconds) and are not listed.
-The new-side page numbers were taken on the foundation-era pages, before the page rework landed, so they are an
-order of magnitude rather than a final figure.
+The new-side numbers in that table came from the foundation-era pages on a busy machine, so they are an order of
+magnitude rather than a final figure; the finished pages, measured on the idle machine against the same
+1,000-meter instance, load in 455 ms (`/`), 383 ms (`/meters`), 469 ms (`/solar`), 641 ms (`/energy/1`),
+684 ms (`/trends`), 104 ms (`/meters/1`) and 152 ms (a nested virtual meter).
 
-### 5.4 Hotspots found and not fixed
+### 5.4 Hotspots found and fixed
 
-1. **The freshness query has no time bound.** `AnalysisQueries.RecentReadingsAsync` (the latest 20 reading times
-   per meter, D-18) plans across every raw chunk: 14–24 ms of planning over 123 `reading` chunks, 189 ms on a cold
-   connection, and 38–46 ms of execution for a portfolio walking back through compressed chunks. With a constant
-   90-day lower bound the same single-meter statement plans in 0.9 ms. It grows with history length.
-2. **Window sums get no plan-time chunk exclusion.** `AnalysisQueries.WindowSumsAsync` takes its bounds only from
-   the `unnest` join. Real requests send few windows and cost 4.6 ms, but a stress case of ~2,000 windows flips to
-   a parallel sequential scan of 1.39 M rows with a sort spilling to disk, 138–152 ms. Latent, and it grows with
-   total history rather than with the number of windows.
-3. **The reworked Overview computes several portfolio bills per request** — nine in the measured build — which
-   shows as repeated catalog, tariff and manual-cost loads (526 ms of SQL inside a 2.4 s page).
+The first measurement found three. All three are fixed (A-40), each proved by `EXPLAIN` or a statement count
+before and after, with the tallies unchanged.
 
-In-process work dominates the rest and scales with series × buckets (roughly 10–16 µs per series bucket). That is
-the budget that matters for very large exports and tables, not for a normal page.
+| Hotspot | Before | After |
+|---|---|---|
+| **The freshness query had no time bound.** `AnalysisQueries.RecentReadingsAsync` planned across all 123 raw chunks on every request, for every meter. | 13.8 ms planning + 36.1 ms execution for the portfolio (980 meters); 189 ms on a cold connection | 0.58 + 0.44 ms, over 4 chunks and only the 40 meters with a live source; an instance without a live source never reads `reading` at all |
+| **Window sums could not exclude chunks.** `AnalysisQueries.WindowSumsAsync` took its bounds only from the `unnest` join. | a 1,960-window stress case: 42 chunks, parallel scan of 1.39 M rows, ~45 MB sort spill, 121.5 ms | 5 chunks, nested-loop index scans, 8.7 ms |
+| **The Overview loaded the catalog three times**, once per read (quantities, bill, comparison bill). | 31 statements per load; 64 statements and 133 ms of SQL on the 1,000-meter instance | 23 statements per load; 39 statements and 81 ms of SQL |
+
+The freshness fix splits what D-18 had conflated: the *mark* (a meter's last reading time) is now stored on
+`meter_rollup_state` and maintained by every recompute, with a one-pass backfill in the `FreshnessMark` migration,
+so an import-only meter keeps its years-old "last activity" without touching `reading`; the *rhythm* (the sample
+intervals that decide stale versus live) is sampled inside a 90-day window, and only for meters that have a live
+source. A live source that has been silent longer than that window is re-read unbounded, so it is still reported
+stale by its own rhythm rather than by a default.
+
+In-process work dominates what is left, and scales with series × buckets (roughly 10–16 µs per series bucket).
+That is the budget that matters for very large exports and tables, not for a normal page.
 
 ## 6. Remaining limitations and known gaps
 
@@ -368,7 +380,9 @@ the budget that matters for very large exports and tables, not for a normal page
 
 **Further limitations, measured or decided during the work:**
 
-- The three performance hotspots in §5.4 are known and unfixed.
+- `AnalysisQueries.OpeningBalancesAsync` has the same unbounded shape the freshness query had (a lateral
+  `ORDER BY time LIMIT 1` over `consumption`, 6.4 ms in the heaviest measured request). It was left alone: smaller,
+  and `consumption` has 42 chunks against `reading`'s 123.
 - The billing basis (grid meter or household use) is chosen per energy type for all time and cannot switch month by
   month, because the category composition would need the same per-month basis to stay reconciled with the bill
   (A-17).