Florian Schmidt
8940ef25c3
ci / build-test (push) Successful in 2m31s
The dashboards told several stories at once. Overview asked for full calendar years, meter detail for a fixed 12-month window that was really 13, Trends for 24 months with an Apply button, and the energy pages for 60. Each page derived "today" from UTC, so the first hours of a local day belonged to yesterday. A missing tariff, a month nobody measured and a genuine zero all rendered as 0. And a virtual meter -- the one thing the spreadsheet leans on hardest -- was excluded from analysis outright: MeterPeriodService returned null for it and the page offered a flow diagram instead. docs/DASHBOARD_ANALYSIS_CHANGE_BRIEF.md is the work order. Every choice it left open is settled in docs/ANALYSIS_IMPLEMENTATION_NOTE.md as D-01..D-58 plus amendments A-01..A-30; code, tests and release notes cite those ids. The analysis layer Core/Analysis holds the pure rules: period presets resolved once in the instance zone into a local date range and a half-open UTC range, bucket plans, calendar-unit comparisons, coverage runs with a resolution class, normalized quantities and units, the totals policy, the virtual formula parser/validator/evaluator, and the cost calculator. "Now" comes from TimeProvider; services never read the clock. Normalization now writes, in the same transaction as consumption and by diff, per-meter rollups by local day and month plus coverage runs and a rollup state (AnalysisDataWriter). AnalysisReader answers a request from those tables -- month rollups for month and year buckets, day rollups otherwise, at most two partial edge days from consumption -- and CostReader prices the result month by month. Pages, /api/v1 and the CSV export read nothing else. The unused continuous aggregates are dropped. The reader's statement count per request is constant whether it covers one meter or a thousand. On a synthetic 1,000-meter, ten-year instance the brief's target request (100 meters, ten years, monthly) takes 374 ms against a two-second target, and the Overview went from 48,244 SQL statements per load to 205. Missing is not zero Every bucket carries a status -- available, partial, missing, unresolved, invalid, pending -- derived from coverage, never from the amount, with provenance and a reason code beside it. A true zero is a number and a bar on the baseline; an unknown bucket is a gap that says why; a month whose data only exists monthly says so instead of inventing daily detail; a scope with no tariff says "not priced" instead of 0. Rows whose interval closes after now are reported separately rather than counted. Virtual meters are analysis subjects A virtual meter stores a canonical definition -- expression over m<id> references, result kind, unit and cost rule -- validated on save and on read for syntax, unknown or self references, loops and unit/kind rules. It is evaluated on read from its sources' rollups over their joint coverage: a missing source makes the bucket missing, an observed zero is a valid input, a non-finite result is invalid with its dependency path, and the page lists each source's contribution. Topology links are topology only and never rewrite a saved calculation; expression-less meters from older installs are converted once at startup. The editor has Sum, Difference and Advanced modes with a live preview. Totals and the bill Per energy type the totals policy separates use, grid import, export, generation and runtime, marks breakdown meters as breakdowns and virtual meters as views, and never adds across units. The bill follows it: grid import where there is one, separately priced subsections at their own price, feed-in only on export meters, standing charges once per scope per local day, manual costs once on their start day, categories as non-overlapping covers whose composition reconciles to the bill. The seeded demo's yearly totals now match the spreadsheet. Pages and navigation The period lives in the URL and every page reads the same contract, so a link, a reload and the browser's Back button keep it. Shared components carry it: page header with breadcrumbs, period toolbar, theme-aware chart with an accessible table beside it, metric cards, comparison and availability states, attention items that each link to the one action that fixes them. Meter detail leads with an Analysis tab and resolves its tabs by key; the energy page has Overview, History, Flow and Meters; the old cost-only Trends page is a general Analysis page over portfolio, type, category, meter or a meter comparison. Records tabs are paged server-side instead of showing the latest 200. Everything is English and German, light and dark, down to 360px. Some figures change on purpose; docs/RELEASE_NOTES.md lists each one and what the first start after the update does (it rebuilds all analysis data before the web server listens). docs/SDD.md and CLAUDE.md describe the system as it now is. Tests: 1,733 Core and 746 integration, all green, plus an opt-in performance suite with a synthetic 1,000-meter generator.
222 lines
18 KiB
Markdown
222 lines
18 KiB
Markdown
# Release notes
|
||
|
||
User-visible changes per release. Earlier releases are described in the git history and in the README's upgrade
|
||
sections. Decision ids (D-nn, A-nn) refer to [`ANALYSIS_IMPLEMENTATION_NOTE.md`](ANALYSIS_IMPLEMENTATION_NOTE.md).
|
||
|
||
## 0.4.0 — Dashboard, navigation and historical analysis (unreleased)
|
||
|
||
This release rebuilds how MeterVault analyses and prices data, and the pages that show it. Every page now works on
|
||
the same selected period and the same numbers. A virtual (calculated) meter can be analysed like a physical one. A
|
||
true zero, missing data, data that exists only per month, and a missing price are always told apart.
|
||
|
||
**Some figures change on purpose.** Most visibly, an energy type's bill now counts its grid meter, not every meter
|
||
that exists. [Changed figures](#changed-figures) lists every deliberate change. The seeded demo's yearly costs now
|
||
match the spreadsheet: 2025 comes to 7,907.65 € against the sheet's 7,907.64 €. The seeded "this year" cost for 2026
|
||
was 4,402.19 € in 0.3.0, which priced Haus + Netz + Auto and water at 5 €/m³. It is now 2,940.19 €, the sheet's
|
||
figure.
|
||
|
||
### Upgrading
|
||
|
||
Back up the database (`pg_dump`) first.
|
||
|
||
- **The first start rebuilds all analysis data** (normalization revision 3):
|
||
- For every meter it rebuilds consumption and the new day and month rollups plus their coverage. It records the
|
||
state of each meter.
|
||
- This runs **before the web server listens**. The app is unreachable meanwhile, and Compose may report the
|
||
container unhealthy. Let it finish.
|
||
- The time grows with the number of raw readings. One meter took about 0.1 s with monthly readings, ~0.6 s with ten
|
||
years of daily readings (3,700 readings), and ~1.4 s with a year of hourly readings (9,300 readings).
|
||
- A synthetic 1,000-meter × 10-year instance (1.34 M readings) took 5.6–6.5 minutes (338–392 s) on a Ryzen 9
|
||
9950X3D. That run was preliminary, on a busy machine. It is about 20 % slower per meter than the 0.3.0 rebuild,
|
||
because rollups and coverage are written too.
|
||
- Progress is logged every 100 meters. A meter that fails is logged, kept pending and retried at the next start.
|
||
A meter whose stored consumption is older than its oldest remaining reading is skipped and logged, so its
|
||
history is not truncated.
|
||
- Until a meter is rebuilt, its pages say "analysis being prepared", never "no data".
|
||
- **Virtual meters without a formula** (such as a seeded *Summe Solar* from an earlier version) are converted at
|
||
startup. When their incoming links name meters of one unit and kind, the implied sum is stored as an explicit
|
||
formula (D-28). The log lists the converted meters, and the meters that still need configuration because their
|
||
links are ambiguous, in mixed units, or loop. The conversion runs once; a rerun changes nothing. From then on, links
|
||
are topology only and never change a calculation.
|
||
- **The migration drops the unused continuous aggregates** and their hourly refresh jobs. It also deletes consumption
|
||
rows stored for virtual meters, which nothing read (D-17).
|
||
- **Check the attention items** on the Overview after the first start. They name missing prices with a direct "Add
|
||
tariff" link, invalid calculations, stale live sources, rows dated after now, and meters that may be counted twice.
|
||
- **Rolling back** to 0.3.0 works. The old version ignores the new tables and never read the dropped aggregates.
|
||
Consumption stays as 0.4.0 booked it until each meter next ingests a reading. Stored virtual formulas remain, and
|
||
0.3.0 goes back to summing links.
|
||
|
||
### What's new
|
||
|
||
**Navigation.**
|
||
- The sidebar has a fixed structure: Overview, Analysis, Meters, an **Energy types** group, **Specialized views**
|
||
(Solar, Tanks & consumables; always listed, with a setup hint when not configured), Data import, and
|
||
**Configuration**.
|
||
- "Configuration → Energy types" edits the definitions; the Energy types group is for analysis.
|
||
- Expanded groups are remembered, and the current page's group always opens. If the energy types cannot be loaded,
|
||
the menu shows an error with Retry instead of silently dropping them.
|
||
- Breadcrumbs (Overview → energy type → meter) keep the selected period, and so does Back.
|
||
- "Find a meter" opens the meter's analysis with the current period and still offers quick entry. It is a text
|
||
button on desktop and an icon on phones.
|
||
|
||
**One period everywhere.**
|
||
- Every analysis page has the same toolbar: this month to date, last month, year to date, previous year, last 12 or
|
||
24 months, all history, or custom dates with one Apply.
|
||
- It also offers a bucket size (automatic, day, week, month, year) and a comparison: previous period, same period
|
||
last year (the default), or any calendar year.
|
||
- The effective dates are shown next to the choice, with the time zone. The selection lives in the address, so
|
||
reload, Back and shared links reproduce the page.
|
||
- Rapid clicks can no longer leave one page showing another selection's data.
|
||
|
||
**Overview.**
|
||
- It shows one selected period (default month to date):
|
||
- the cost, split into metered use, standing charges, manual costs and feed-in credit;
|
||
- a card per energy type with its quantities in their own units, its cost and billing basis, the change and
|
||
freshness;
|
||
- a history chart with a table view;
|
||
- "What changed", by category or by meter;
|
||
- the cost composition;
|
||
- attention items, each with one targeted action.
|
||
- Changes are compared only over the part both periods cover, and the page states both date ranges.
|
||
- When the period has no data, the page names the dates that do have data and offers "Go to latest data". It never
|
||
silently switches to an older month.
|
||
|
||
**Analysis page** (`/trends`, formerly the cost trend).
|
||
- Explore everything, one energy type, a cost category, one meter, or up to six meters side by side, by quantity or
|
||
by cost.
|
||
- Compare calendar years with an overlay and a comparison table. Click a bar or a row to drill into a finer period.
|
||
- Export exactly what is shown as CSV.
|
||
|
||
**Energy type pages** are titled with the type's own name and have four tabs:
|
||
- **Overview:** measures such as total use, grid import, generation and runtime, each in its own unit and never added
|
||
across units; the cost with its billing basis; coverage; the largest changes.
|
||
- **History:** the total, or up to six individual meters with an explanation of how each one counts.
|
||
- **Flow:** the Sankey, now using the same values as every other page. Calculated and estimated connections are
|
||
marked. It comes with a table version and **Manage connections**.
|
||
- **Meters:** each meter's value for the period and its data quality.
|
||
|
||
**Meter page.**
|
||
- The tabs sit directly under the header: Analysis, Readings, Normalized data, Events, Tariffs, Sources. A calculated
|
||
meter has Calculation instead of Sources.
|
||
- The Analysis tab shows:
|
||
- the period total with its unit and status, and the cost with its rule, or the reason it has none;
|
||
- the change against the comparison period;
|
||
- a labelled projection, where there is enough data for one;
|
||
- a full chart with the previous-year overlay, and a table;
|
||
- a "Data quality and coverage" section;
|
||
- events and tariff changes in the range.
|
||
- The record tabs page through the **whole history** (100 rows at a time, filtered by the selected dates), no longer
|
||
just the latest 200. Rows dated after now are marked.
|
||
- Existing links such as `?tab=consumption`, `?tab=readings&action=reading` and `?tab=events&action=swap` still open
|
||
the intended tab and dialog once.
|
||
|
||
**Virtual (calculated) meters.**
|
||
- Create them with **Sum**, **Difference** or a **Formula** over other meters, picked by name. A live preview of the
|
||
selected period shows every source's values and flags incomplete months.
|
||
- The result kind (consumption, generation, net or indicator), unit and cost rule are stored with the formula.
|
||
- A virtual meter gets the same analysis as a physical one, plus "Source meters" showing each source's contribution.
|
||
- **The rules:**
|
||
- A missing source month makes the result "no data" for that month, never a silent zero.
|
||
- An observed zero is a real zero.
|
||
- A division by zero or a loop is reported, and names the meters involved.
|
||
- Differences stay negative.
|
||
- New calculated meters are *analysis only*: they never add to a type's totals or the bill. The "Always count" option
|
||
lets one replace its sources instead.
|
||
|
||
**Solar and Tanks & consumables** use the same toolbar, cards and charts.
|
||
- **Solar** works out self-consumption, feed-in, site use, savings and autarky from the meters' roles. For a missing
|
||
role it shows a setup card with candidate meters instead of raw role tags.
|
||
- **Tanks** keep "Last dipstick (date)" apart from "Estimated now". A past period shows the contents at its end, not
|
||
today's. The forecast is a labelled projection, hidden when the dipstick is older than 60 days.
|
||
|
||
**Tariffs and configuration.**
|
||
- An "Add tariff" link from a missing price opens the tariff editor once, prefilled with the scope, component and
|
||
first uncovered month.
|
||
- The unit is checked against what it prices: a wrong unit or currency blocks the save.
|
||
- A new tariff needs a value; a typed 0 is a deliberate free period.
|
||
- The editor notes that Bonus, Discount and Tax are stored but **not applied** yet.
|
||
- Meter roles have friendly names and one-line meanings. A role is unique per energy type, and saving it names the
|
||
meter it moves from.
|
||
- The Settings page shows the analysis data state and that raw retention is not enforced.
|
||
|
||
**Also:**
|
||
- CSV export of any analysis view (`/export/analysis.csv`): statuses, provenance, costs and comparison values; unknown
|
||
values are empty cells, never 0.
|
||
- The light/dark choice persists across reloads and language switches, and charts follow it immediately.
|
||
- Visible keyboard focus, and tables for every chart.
|
||
- MudBlazor's own labels are in German too.
|
||
- Pages work at phone width.
|
||
- Amounts use the configured currency (`MeterVault__Currency`) instead of a hard-coded €.
|
||
- Deleting a meter or an energy type also deletes the tariffs scoped to it.
|
||
- JSON export/import now carries meter connections, re-links virtual formulas to the new meter ids, and no longer
|
||
restores the tariffs of deleted meters onto other meters.
|
||
|
||
### Changed figures
|
||
|
||
Every change below is deliberate. The golden spreadsheet reconciliation (consumption of all four sheets, Netz
|
||
Einsparung) is unchanged. The seeded yearly bill matches the sheet's `Jahreskosten` within ±0.02 € for 2022, 2025 and
|
||
2026. It differs by 3.78 € (2023) and 0.46 € (2024) only because the sheet multiplies by unrounded prices.
|
||
|
||
| Area | 0.3.0 | 0.4.0 |
|
||
|---|---|---|
|
||
| What an energy type's bill counts | Every meter's cost was summed: Haus + Netz + Auto for the seeded Strom | The type's grid import meter when it has one, otherwise its household use. Submeters are breakdowns; generation is never billed. Seeded Strom is Zähler Netz × price, like the sheet (2025: 4,742.64 €) (D-22, D-34) |
|
||
| Feed-in credit | Credited on all generation | Only on a meter with the grid-export role, at the feed-in price (D-34) |
|
||
| A subsection with its own meter price | Added on top | Billed at its own price and taken out of the meter above it; quantities unchanged (D-35, A-19) |
|
||
| Missing tariff | Cost 0 | "Not priced (no tariff)" with an Add tariff action; a hole in a price history is a price gap (unavailable). The quantities stay visible. An explicit 0 tariff is still a valid zero (D-38) |
|
||
| Standing charges | Per meter, per month that had readings, and copied onto every meter of the type | Per local day over the scope's service period (including reading gaps), **once per scope**. Type and global charges are their own rows; meter fees stay on their meter (D-40, A-18) |
|
||
| Price of a longer bucket | Month buckets used the price of the 15th, year buckets the price of 1 July | Every bucket is priced month by month at the price of the 15th, so a year equals the sum of its months and changing the bucket never changes a total (D-36) |
|
||
| Months in which the billed grid meter was not yet (or no longer) in service | — (0.3.0 summed every meter) | Unavailable rather than free while use was measured, with an attention item naming the grid meter and the months (A-17) |
|
||
| Tank, runtime, direct-delta or instant-rate readings more than a month apart (a tank dipped every few months, quarterly burner hours) | Booked whole in the month of the later reading, with zeros in between | The months in between read "only coarser data" (not zero). A year or longer bucket is priced when all its months share one price; otherwise it is unavailable with an attention item (A-16) |
|
||
| Manual costs | Counted in the Overview but not in the trend; a cost dated later this month counted at once | Counted once, on their start day, when that day has come, everywhere: Overview, Analysis, categories, export (D-41) |
|
||
| Cost categories | Sum of their member meters' costs | The priced non-overlapping cover of their members plus their manual costs. Seeded Strom = Netz × price. A category overlapping another is shown as a view, apart from the composition. A category whose members price nothing says so (D-42, A-22) |
|
||
| Virtual meters | No analysis; the flow summed incoming links, ignoring any formula | Full analysis from the stored formula; the Sankey uses the same values (D-27, D-30) |
|
||
| Summe Solar and other generation sums | — | Analysed as generation (seeded 2025: 4,750 kWh = Solar 1 3,123 + Solar 2 1,627) and **not costed**, because generation is never billed (A-15) |
|
||
| Readings at exactly midnight | Booked in the following day | Booked in the day they close (D-11) |
|
||
| "Last 12 months" | 13–14 buckets, including a partial future month | 12 calendar buckets ending with the current month; actuals stop at now (D-02) |
|
||
| Rows dated after now | Counted in "to date" totals | Excluded and shown as "recorded after now", e.g. a sheet row labelled the current month or a future-stamped reading. A day that holds such a row reads partial (D-04, A-14, A-20) |
|
||
| Overview "this month / this year" | Current month and year against the complete previous ones | The selected period against the same elapsed part of the comparison period, measured over what both cover (D-07) |
|
||
| "Latest month with data" | An amount without its month; consumption only | The month and its basis (meter data, manual costs or both) (D-19) |
|
||
| Percentages | A negative baseline was divided by its absolute value | "Not applicable" for a zero or negative baseline; the absolute difference is always shown (D-08) |
|
||
| Meter dates | — | Outside its install and retire dates a meter counts as a known zero; retired meters keep their history (D-24) |
|
||
| Currency | € hard-coded in most views | `MeterVault__Currency` everywhere; a tariff in another currency is reported as not fitting, never converted (D-43) |
|
||
| Continuous aggregates | Refreshed hourly, read by nothing | Dropped; rollup tables are written with each recompute (D-12, D-17) |
|
||
| Seed | — | Adds the water price of 7.00 €/m³ from 2026-01-01, and stores Summe Solar's formula (`m4 + m5`, generation, not costed). This affects new seeds; existing seeded instances get the formula through the startup conversion (D-44) |
|
||
|
||
### REST API
|
||
|
||
Every existing field keeps its name and type. What changed is only added as new fields. The numbers follow the new
|
||
engine, as listed above: actuals stop at now, virtual meters are evaluated, and costs are the bill's.
|
||
|
||
- **`GET /api/v1/consumption`:**
|
||
- New fields: `status`, `issue`, `kind`, `unit`.
|
||
- A month without data is left out instead of reported as 0.
|
||
- `from`/`to` with a UTC offset are accepted; they used to fail with a server error.
|
||
- Virtual meters return evaluated values.
|
||
- **`GET /api/v1/cost`:**
|
||
- New fields: `costStatus` (`Priced`, `Partial`, `NotPriced`, `PriceGap`, `UnitMismatch`), `costAvailability` (the
|
||
state of the quantities behind the cost, e.g. `Unresolved`, `Invalid`), `costRule`, `notCosted`, `missingPrices[]`
|
||
(component, reason, scope, first and last month, tariff, unit issue, credit), `status`, `issue`, `kind`, `unit`.
|
||
- `cost` stays numeric and is 0 when nothing could be priced; check `costStatus` and `costAvailability` before
|
||
trusting a 0.
|
||
- **Behaviour change:** generation and runtime meters, indicators and calculations that cannot be evaluated now
|
||
report `costStatus: NotPriced`, with `notCosted` giving the reason. They used to report `Priced`, and a generation
|
||
meter could carry a negative feed-in cost (A-21).
|
||
- **`GET /api/v1/dashboard/summary`:**
|
||
- New fields: `deltaPercentApplicable` per KPI, and `latestMonth` `{period, basis}`.
|
||
- The month and year windows are unchanged (the calendar month and year to now, against the whole previous ones),
|
||
but the values are the new bill. `deltaPercent` is 0 when not applicable.
|
||
|
||
### Known limitations
|
||
|
||
- **Raw retention is not enforced** (D-57). `MeterVault__RawRetentionDays` is shown but nothing deletes readings,
|
||
because every recompute rebuilds a meter from its readings.
|
||
- **Monthly data is never interpolated to days** (D-57). A day or week view of monthly data says "only coarser data"
|
||
and offers the monthly view.
|
||
- **Bonus, Discount and Tax tariffs are stored but not applied** (D-57).
|
||
- **Every live reading recomputes its meter in full** (D-57). That is fine for monthly and daily meters, but costs
|
||
about 1.4 s per reading for a meter with a year of hourly data, and grows with history.
|
||
- **Months cannot switch billing basis:** the billing basis (grid meter or household use) is chosen per energy type
|
||
for all time (A-17).
|
||
- **Batteries are not modelled.** Without a grid-export meter, Solar's feed-in is calculated, and labelled as such.
|
||
- **No CSV export on the Solar page**, because the export has no derived measures.
|