Analysis: bound the freshness query, let window sums skip chunks, share the Overview's catalog
ci / build-test (push) Successful in 2m34s
ci / build-test (push) Successful in 2m34s
Three things the 1,000-meter x 10-year measurement found, each proved by EXPLAIN or a statement count before and after. No tally moves. Freshness had two jobs in one unbounded query. The mark -- when a meter last delivered -- is now stored on meter_rollup_state and maintained by every recompute, with a one-pass backfill in the migration, so an import-only meter keeps its years-old last activity without reading a single raw row. The rhythm that decides stale versus live is sampled inside a 90-day window and only for meters that actually have a live source; a source silent for longer than that is re-read unbounded, so it is still called stale by its own rhythm rather than by a default. The portfolio query went from 13.8 ms planning plus 36.1 ms execution across all 123 reading chunks to 0.58 plus 0.44 ms across four. Window sums took their time bounds only from the unnest join, so the planner could not exclude chunks: a 1,960-window case scanned 1.39 M rows in parallel and spilled a 45 MB sort. Repeating the overall min and max as constants makes it five chunks and nested-loop index scans, 121.5 ms to 8.7 ms. The Overview read the catalog three times, once for the quantities and once for each of its two bills. One context and one catalog snapshot now feed all three: 31 statements per load to 23. The final timings on an idle machine are in docs/ANALYSIS_REPORT.md: the brief's target request (100 meters, ten years, monthly) is 286 ms against two seconds, and a startup rebuild of 1,000 meters is 279 s.
This commit is contained in:
@@ -7,7 +7,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
MeterVault is a self-hosted, local-first energy & utility metering platform: it ingests meter data from Home Assistant, Tasmota and MQTT on a schedule, stores every reading timestamped and immutable, normalizes it into consumption, and turns it into cost dashboards. Energy types (electricity, water, heating oil, gas, …) and meters are **user-defined, never hardcoded**.
|
||||
|
||||
**Status: implemented (M0–M7) + the dashboard/analysis rework (next release 0.4.0).**
|
||||
- **Size:** five projects, ~2,480 tests (Core 1,733, Integration 746), working Docker deploy.
|
||||
- **Size:** five projects, ~2,490 tests (Core 1,733, Integration 753 incl. 2 opt-in performance facts), working Docker deploy.
|
||||
- **Docs:**
|
||||
- `docs/SDD.md` is the design reference. It marks in place every section the system now deviates from (list: note D-58). Its milestone map (§12) matches the git history (M0…M7).
|
||||
- The rework's work order is `docs/DASHBOARD_ANALYSIS_CHANGE_BRIEF.md`. Its decisions are D-01…D-58 and amendments A-01…A-39 in `docs/ANALYSIS_IMPLEMENTATION_NOTE.md`. **Read that note before touching analysis, costing, rollups, virtual meters or the analysis pages.**
|
||||
@@ -160,7 +160,7 @@ sources (Tasmota/HA/MQTT/manual/CSV)
|
||||
- Then it rebuilds consumption + rollups + coverage + state of **every** meter when the revision or zone differs. Otherwise it rebuilds only meters whose `meter_rollup_state` is missing or outdated, plus `normalization_pending`. Virtual meters are purged, since they store nothing. Each meter runs in its own transaction.
|
||||
- A meter whose oldest consumption predates its oldest reading or event is skipped and logged instead of truncating history. A failing meter is logged, kept pending and retried at the next start.
|
||||
- Until its rebuild runs, a meter reads as `Pending` ("analysis being prepared"), never "no data". **Bump `CurrentRevision` whenever the engine books existing readings differently.** The rebuild runs before the web server listens; roughly 0.1 s per monthly meter and ~1.4 s per meter with a year of hourly data (it grows with the reading count).
|
||||
- **Dashboards and charts read rollups only — never `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). `consumption` + rollups are the analytical history. `reading` is read by the paged Readings tab, the manual-entry checks and the freshness query (latest 20 reading times per meter, D-18). **Raw retention is not enforced** (D-57, a documented blocker): every recompute rebuilds a meter from its readings, so dropping old readings would destroy history. `/admin/settings` and the Readings tab say so.
|
||||
- **Dashboards and charts read rollups only — never `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). `consumption` + rollups are the analytical history. `reading` is read by the paged Readings tab, the manual-entry checks, and — only for a meter with a live source, only over the last 90 days — the rhythm query behind D-18 (A-40). The freshness *mark* is `meter_rollup_state.last_reading_at`, written by the recompute behind every write path, so an import-only meter's mark is exact without touching the raw chunks. **Raw retention is not enforced** (D-57, a documented blocker): every recompute rebuilds a meter from its readings, so dropping old readings would destroy history. `/admin/settings` and the Readings tab say so.
|
||||
- **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries − usage + forecast), `direct_delta`, `instant_rate`, `virtual` (a formula over other meters, **evaluated on read, never stored**: D-27, SDD §14.1). New ingestion/normalization logic dispatches on mode. `NormalizedQuantity` (D-20) gives each meter's analysis (kind, unit):
|
||||
- runtime: `h`, or the tank unit with a fixed rate;
|
||||
- instant rate: the rate unit without `/h` (W → Wh);
|
||||
|
||||
Reference in New Issue
Block a user