MeterVault/docs/RELEASE_NOTES.md

# Release notes

User-visible changes per release. Earlier releases are described in the git history and in the README's upgrade
sections. Decision ids (D-nn, A-nn) refer to [`ANALYSIS_IMPLEMENTATION_NOTE.md`](ANALYSIS_IMPLEMENTATION_NOTE.md).

## 0.4.0 — Dashboard, navigation and historical analysis (unreleased)

This release rebuilds how MeterVault analyses and prices data, and the pages that show it. Every page now works on
the same selected period and the same numbers. A virtual (calculated) meter can be analysed like a physical one. A
true zero, missing data, data that exists only per month, and a missing price are always told apart.

**Some figures change on purpose.** Most visibly, an energy type's bill now counts its grid meter, not every meter
that exists. [Changed figures](#changed-figures) lists every deliberate change. The seeded demo's yearly costs now
match the spreadsheet: 2025 comes to 7,907.65 € against the sheet's 7,907.64 €. The seeded "this year" cost for 2026
was 4,402.19 € in 0.3.0, which priced Haus + Netz + Auto and water at 5 €/m³. It is now 2,940.19 €, the sheet's
figure.

### Upgrading

Back up the database (`pg_dump`) first.

- **The first start rebuilds all analysis data** (normalization revision 3):
  - For every meter it rebuilds consumption and the new day and month rollups plus their coverage. It records the
    state of each meter.
  - This runs **before the web server listens**. The app is unreachable meanwhile, and Compose may report the
    container unhealthy. Let it finish.
  - The time grows with the number of raw readings. One meter took about 0.05 s with monthly readings, ~0.5 s with ten
    years of daily readings (3,700 readings), and ~1.2 s with a year of hourly readings (9,300 readings).
  - A synthetic 1,000-meter × 10-year instance (1.34 M readings) took 4.7 minutes (279 s) on an idle Ryzen 9 9950X3D —
    about 3.6 meters a second. It writes rollups and coverage on top of consumption, so it does more per meter than
    the 0.3.0 rebuild did.
  - Progress is logged every 100 meters. A meter that fails is logged, kept pending and retried at the next start.
    A meter whose stored consumption is older than its oldest remaining reading is skipped and logged, so its
    history is not truncated.
  - Until a meter is rebuilt, its pages say "analysis being prepared", never "no data".
- **The migration records each meter's last reading time** in the analysis state table and fills it in for the meters
  you already have, in one pass over the readings. It is what the pages show as "last activity" and what decides
  whether a live source is late, so no page has to search the raw readings for it any more.
- **Virtual meters without a formula** (such as a seeded *Summe Solar* from an earlier version) are converted at
  startup. When their incoming links name meters of one unit and kind, the implied sum is stored as an explicit
  formula (D-28). The log lists the converted meters, and the meters that still need configuration because their
  links are ambiguous, in mixed units, or loop. The conversion runs once; a rerun changes nothing. From then on, links
  are topology only and never change a calculation.
- **The migration drops the unused continuous aggregates** and their hourly refresh jobs. It also deletes consumption
  rows stored for virtual meters, which nothing read (D-17).
- **Check the attention items** on the Overview after the first start. They name missing prices with a direct "Add
  tariff" link, invalid calculations, stale live sources, rows dated after now, and meters that may be counted twice.
- **Rolling back** to 0.3.0 works. The old version ignores the new tables and never read the dropped aggregates.
  Consumption stays as 0.4.0 booked it until each meter next ingests a reading. Stored virtual formulas remain, and
  0.3.0 goes back to summing links.

### What's new

**Navigation.**
- The sidebar has a fixed structure: Overview, Analysis, Meters, an **Energy types** group, **Specialized views**
  (Solar, Tanks & consumables; always listed, with a setup hint when not configured), Data import, and
  **Configuration**.
- "Configuration → Energy types" edits the definitions; the Energy types group is for analysis.
- Expanded groups are remembered, and the current page's group always opens. If the energy types cannot be loaded,
  the menu shows an error with Retry instead of silently dropping them.
- Breadcrumbs (Overview → energy type → meter) keep the selected period, and so does Back.
- "Find a meter" opens the meter's analysis with the current period and still offers quick entry. It is a text
  button on desktop and an icon on phones.

**One period everywhere.**
- Every analysis page has the same toolbar: this month to date, last month, year to date, previous year, last 12 or
  24 months, all history, or custom dates with one Apply.
- It also offers a bucket size (automatic, day, week, month, year) and a comparison: previous period, same period
  last year (the default), or any calendar year.
- The effective dates are shown next to the choice, with the time zone. The selection lives in the address, so
  reload, Back and shared links reproduce the page.
- Rapid clicks can no longer leave one page showing another selection's data.

**Overview.**
- It shows one selected period (default month to date):
  - the cost, split into metered use, standing charges, manual costs and feed-in credit;
  - a card per energy type with its quantities in their own units, its cost and billing basis, the change and
    freshness;
  - a history chart with a table view;
  - "What changed", by category or by meter;
  - the cost composition;
  - attention items, each with one targeted action.
- Changes are compared only over the part both periods cover, and the page states both date ranges.
- When the period has no data, the page names the dates that do have data and offers "Go to latest data". It never
  silently switches to an older month.

**Analysis page** (`/trends`, formerly the cost trend).
- Explore everything, one energy type, a cost category, one meter, or up to six meters side by side, by quantity or
  by cost.
- Compare calendar years with an overlay and a comparison table. Click a bar or a row to drill into a finer period.
- Export exactly what is shown as CSV.

**Energy type pages** are titled with the type's own name and have four tabs:
- **Overview:** measures such as total use, grid import, generation and runtime, each in its own unit and never added
  across units; the cost with its billing basis; coverage; the largest changes.
- **History:** the total, or up to six individual meters with an explanation of how each one counts.
- **Flow:** the Sankey, now using the same values as every other page. Calculated and estimated connections are
  marked. It comes with a table version and **Manage connections**.
- **Meters:** each meter's value for the period and its data quality.

**Meter page.**
- The tabs sit directly under the header: Analysis, Readings, Normalized data, Events, Tariffs, Sources. A calculated
  meter has Calculation instead of Sources.
- The Analysis tab shows:
  - the period total with its unit and status, and the cost with its rule, or the reason it has none;
  - the change against the comparison period;
  - a labelled projection, where there is enough data for one;
  - a full chart with the previous-year overlay, and a table;
  - a "Data quality and coverage" section;
  - events and tariff changes in the range.
- The record tabs page through the **whole history** (100 rows at a time, filtered by the selected dates), no longer
  just the latest 200. Rows dated after now are marked.
- Existing links such as `?tab=consumption`, `?tab=readings&action=reading` and `?tab=events&action=swap` still open
  the intended tab and dialog once.

**Virtual (calculated) meters.**
- Create them with **Sum**, **Difference** or a **Formula** over other meters, picked by name. A live preview of the
  selected period shows every source's values and flags incomplete months.
- The result kind (consumption, generation, net or indicator), unit and cost rule are stored with the formula.
- A virtual meter gets the same analysis as a physical one, plus "Source meters" showing each source's contribution.
- **The rules:**
  - A missing source month makes the result "no data" for that month, never a silent zero.
  - An observed zero is a real zero.
  - A division by zero or a loop is reported, and names the meters involved.
  - Differences stay negative.
- New calculated meters are *analysis only*: they never add to a type's totals or the bill. The "Always count" option
  lets one replace its sources instead.

**Solar and Tanks & consumables** use the same toolbar, cards and charts.
- **Solar** works out self-consumption, feed-in, site use, savings and autarky from the meters' roles. For a missing
  role it shows a setup card with candidate meters instead of raw role tags.
- **Tanks** keep "Last dipstick (date)" apart from "Estimated now". A past period shows the contents at its end, not
  today's. The forecast is a labelled projection, hidden when the dipstick is older than 60 days.

**Tariffs and configuration.**
- An "Add tariff" link from a missing price opens the tariff editor once, prefilled with the scope, component and
  first uncovered month.
- The unit is checked against what it prices: a wrong unit or currency blocks the save.
- A new tariff needs a value; a typed 0 is a deliberate free period.
- The editor notes that Bonus, Discount and Tax are stored but **not applied** yet.
- Meter roles have friendly names and one-line meanings. A role is unique per energy type, and saving it names the
  meter it moves from.
- The Settings page shows the analysis data state and that raw retention is not enforced.

**Also:**
- CSV export of any analysis view (`/export/analysis.csv`): statuses, provenance, costs and comparison values; unknown
  values are empty cells, never 0.
- The light/dark choice persists across reloads and language switches, and charts follow it immediately.
- Visible keyboard focus, and tables for every chart.
- MudBlazor's own labels are in German too.
- Pages work at phone width.
- Amounts use the configured currency (`MeterVault__Currency`) instead of a hard-coded €.
- Deleting a meter or an energy type also deletes the tariffs scoped to it.
- JSON export/import now carries meter connections, re-links virtual formulas to the new meter ids, and no longer
  restores the tariffs of deleted meters onto other meters.

### Changed figures

Every change below is deliberate. The golden spreadsheet reconciliation (consumption of all four sheets, Netz
Einsparung) is unchanged. The seeded yearly bill matches the sheet's `Jahreskosten` within ±0.02 € for 2022, 2025 and
2026. It differs by 3.78 € (2023) and 0.46 € (2024) only because the sheet multiplies by unrounded prices.

| Area | 0.3.0 | 0.4.0 |
|---|---|---|
| What an energy type's bill counts | Every meter's cost was summed: Haus + Netz + Auto for the seeded Strom | The type's grid import meter when it has one, otherwise its household use. Submeters are breakdowns; generation is never billed. Seeded Strom is Zähler Netz × price, like the sheet (2025: 4,742.64 €) (D-22, D-34) |
| Feed-in credit | Credited on all generation | Only on a meter with the grid-export role, at the feed-in price (D-34) |
| A subsection with its own meter price | Added on top | Billed at its own price and taken out of the meter above it; quantities unchanged (D-35, A-19) |
| Missing tariff | Cost 0 | "Not priced (no tariff)" with an Add tariff action; a hole in a price history is a price gap (unavailable). The quantities stay visible. An explicit 0 tariff is still a valid zero (D-38) |
| Standing charges | Per meter, per month that had readings, and copied onto every meter of the type | Per local day over the scope's service period (including reading gaps), **once per scope**. Type and global charges are their own rows; meter fees stay on their meter (D-40, A-18) |
| Price of a longer bucket | Month buckets used the price of the 15th, year buckets the price of 1 July | Every bucket is priced month by month at the price of the 15th, so a year equals the sum of its months and changing the bucket never changes a total (D-36) |
| Months in which the billed grid meter was not yet (or no longer) in service | — (0.3.0 summed every meter) | Unavailable rather than free while use was measured, with an attention item naming the grid meter and the months (A-17) |
| Tank, runtime, direct-delta or instant-rate readings more than a month apart (a tank dipped every few months, quarterly burner hours) | Booked whole in the month of the later reading, with zeros in between | The months in between read "only coarser data" (not zero). A year or longer bucket is priced when all its months share one price; otherwise it is unavailable with an attention item (A-16) |
| Manual costs | Counted in the Overview but not in the trend; a cost dated later this month counted at once | Counted once, on their start day, when that day has come, everywhere: Overview, Analysis, categories, export (D-41) |
| Cost categories | Sum of their member meters' costs | The priced non-overlapping cover of their members plus their manual costs. Seeded Strom = Netz × price. A category overlapping another is shown as a view, apart from the composition. A category whose members price nothing says so (D-42, A-22) |
| Virtual meters | No analysis; the flow summed incoming links, ignoring any formula | Full analysis from the stored formula; the Sankey uses the same values (D-27, D-30) |
| Summe Solar and other generation sums | — | Analysed as generation (seeded 2025: 4,750 kWh = Solar 1 3,123 + Solar 2 1,627) and **not costed**, because generation is never billed (A-15) |
| Readings at exactly midnight | Booked in the following day | Booked in the day they close (D-11) |
| "Last 12 months" | 13–14 buckets, including a partial future month | 12 calendar buckets ending with the current month; actuals stop at now (D-02) |
| Rows dated after now | Counted in "to date" totals | Excluded and shown as "recorded after now", e.g. a sheet row labelled the current month or a future-stamped reading. A day that holds such a row reads partial (D-04, A-14, A-20) |
| Overview "this month / this year" | Current month and year against the complete previous ones | The selected period against the same elapsed part of the comparison period, measured over what both cover (D-07) |
| "Latest month with data" | An amount without its month; consumption only | The month and its basis (meter data, manual costs or both) (D-19) |
| Percentages | A negative baseline was divided by its absolute value | "Not applicable" for a zero or negative baseline; the absolute difference is always shown (D-08) |
| Meter dates | — | Outside its install and retire dates a meter counts as a known zero; retired meters keep their history (D-24) |
| Currency | € hard-coded in most views | `MeterVault__Currency` everywhere; a tariff in another currency is reported as not fitting, never converted (D-43) |
| Continuous aggregates | Refreshed hourly, read by nothing | Dropped; rollup tables are written with each recompute (D-12, D-17) |
| Seed | — | Adds the water price of 7.00 €/m³ from 2026-01-01, and stores Summe Solar's formula (`m4 + m5`, generation, not costed). This affects new seeds; existing seeded instances get the formula through the startup conversion (D-44) |

### REST API

Every existing field keeps its name and type. What changed is only added as new fields. The numbers follow the new
engine, as listed above: actuals stop at now, virtual meters are evaluated, and costs are the bill's.

- **`GET /api/v1/consumption`:**
  - New fields: `status`, `issue`, `kind`, `unit`.
  - A month without data is left out instead of reported as 0.
  - `from`/`to` with a UTC offset are accepted; they used to fail with a server error.
  - Virtual meters return evaluated values.
- **`GET /api/v1/cost`:**
  - New fields: `costStatus` (`Priced`, `Partial`, `NotPriced`, `PriceGap`, `UnitMismatch`), `costAvailability` (the
    state of the quantities behind the cost, e.g. `Unresolved`, `Invalid`), `costRule`, `notCosted`, `missingPrices[]`
    (component, reason, scope, first and last month, tariff, unit issue, credit), `status`, `issue`, `kind`, `unit`.
  - `cost` stays numeric and is 0 when nothing could be priced; check `costStatus` and `costAvailability` before
    trusting a 0.
  - **Behaviour change:** generation and runtime meters, indicators and calculations that cannot be evaluated now
    report `costStatus: NotPriced`, with `notCosted` giving the reason. They used to report `Priced`, and a generation
    meter could carry a negative feed-in cost (A-21).
- **`GET /api/v1/dashboard/summary`:**
  - New fields: `deltaPercentApplicable` per KPI, and `latestMonth` `{period, basis}`.
  - The month and year windows are unchanged (the calendar month and year to now, against the whole previous ones),
    but the values are the new bill. `deltaPercent` is 0 when not applicable.

### Known limitations

- **Raw retention is not enforced** (D-57). `MeterVault__RawRetentionDays` is shown but nothing deletes readings,
  because every recompute rebuilds a meter from its readings.
- **Monthly data is never interpolated to days** (D-57). A day or week view of monthly data says "only coarser data"
  and offers the monthly view.
- **Bonus, Discount and Tax tariffs are stored but not applied** (D-57).
- **Every live reading recomputes its meter in full** (D-57). That is fine for monthly and daily meters, but costs
  about 1.2 s per reading for a meter with a year of hourly data, and grows with history.
- **Months cannot switch billing basis:** the billing basis (grid meter or household use) is chosen per energy type
  for all time (A-17).
- **Batteries are not modelled.** Without a grid-export meter, Solar's feed-in is calculated, and labelled as such.
- **No CSV export on the Solar page**, because the export has no derived measures.