Compare commits

...
2 Commits
Author SHA1 Message Date
schmidt.florian cedd60ab45 Audit fixes: batch recompute, negative-baseline percentages, key-ring persistence
ci / build-test (push) Successful in 1m17s
Three defects found reviewing the last few commits.

Deriving consumption on ingest made the batch reading endpoint quadratic. A
recompute rewrites a meter's entire consumption series, and POST
/api/v1/readings ran one per reading -- 500 readings for one meter meant 500
full rewrites. IngestByMeterAsync takes renormalize:false and the endpoint
normalizes each touched meter once after the batch.

Percentage change divided by a possibly negative baseline. A net-export meter
going from -100 to -150 exported half again as much and would have been
reported as "+50%", reading as more consumption. A non-positive baseline now
reports no basis rather than a confident lie.

The data-protection key ring had no persistent home outside Docker Compose. The
LXC installer now creates /var/lib/metervault/keys at 0700 -- the app would
otherwise create it under the default umask, leaving a key ring world-readable
-- and the Unraid template maps it, since without that every UI-entered secret
was lost whenever the container was recreated. README documents the variable
and the trust boundary: keys on disk protect against leaked database content,
not against an attacker who already has the host.

Claude-Session: https://claude.ai/code/session_01V6joyergfvVLFEizH1hJLd
2026-07-18 19:39:56 +02:00
schmidt.florian ad896db051 Normalization: apportion a long unread gap across the months it covers
A counter delta is booked at the reading that closes it. That is right at the
reporting cadence -- a monthly series books December against the 1 January
reading, which is what the reference sheet does -- and wrong after an outage.
Observed on Solar 1: 78 days of generation arrived as one July row, leaving
June looking like the array was switched off.

An interval containing two or more complete calendar months is now divided
across them in proportion to elapsed time. The meter recorded a total, not a
shape, so every row a split produces is marked Estimated. The sum is exact: the
final segment absorbs the rounding remainder, so a split never creates or
destroys energy.

Counting whole months *contained* rather than boundaries *crossed* is what
makes the rule safe. A monthly series contains exactly one whole month per
interval and is untouched, so the golden fixtures keep measuring the normalizer
rather than the splitter; and a reading landing hours late cannot tip the rule
and hand the new month a sliver. GapSplittingIsInertOnFixturesTests asserts the
rule declines to fire on every reference interval, so this cannot drift into
the oracle unnoticed.

Not apportioned: swap and reset amounts (explicit corrections booked at their
event -- apportioning one would rewrite a number the operator supplied), a
rejected decrease, and a zero delta, which would otherwise fan out into rows
that say nothing.

Segments are stamped at their end, keeping the existing convention that a row
records the period ending at its timestamp -- so nothing shifts relative to how
unsplit intervals are already labelled.

Claude-Session: https://claude.ai/code/session_01V6joyergfvVLFEizH1hJLd
2026-07-18 19:39:42 +02:00
14 changed files with 524 additions and 18 deletions
+2 -1
View File
@@ -72,7 +72,8 @@ sources (Tasmota/HA/MQTT/manual/CSV)
**Invariants that shape everything:** **Invariants that shape everything:**
- **Raw `reading` is immutable audit truth.** Everything derived (consumption, cost, balances, forecasts) is computed on top and must be reproducible. Never mutate readings to fix a derived number. - **Raw `reading` is immutable audit truth.** Everything derived (consumption, cost, balances, forecasts) is computed on top and must be reproducible. Never mutate readings to fix a derived number. Live ingestion recomputes the meter inline (`IngestionService.RenormalizeAsync`) — without it, polled readings never become consumption.
- **Long gaps are apportioned, short ones are not** (`GapAttribution`, SDD §7.1). An interval containing ≥2 whole calendar months is split across those months, proportional to elapsed time, marked `Estimated`. A monthly series contains exactly one and is untouched — that's what keeps the golden fixtures reconciling. `GapSplittingIsInertOnFixturesTests` asserts the rule declines to fire on the reference data, so this can't silently drift.
- **Dashboards and charts read aggregates only — never scan `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). Raw is kept for a bounded window (default 3y); `consumption` + aggregates are the long-term source of truth. - **Dashboards and charts read aggregates only — never scan `reading`.** This is what makes 1000 meters × 50 years feasible (§5.5). Raw is kept for a bounded window (default 3y); `consumption` + aggregates are the long-term source of truth.
- **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries usage + forecast), `direct_delta`, `instant_rate`, `virtual` (expression over other meters). New ingestion/normalization logic dispatches on mode. - **`meter.mode` (measurement mode) is the central abstraction** for how raw readings become consumption (SDD §5.2): `cumulative_counter`, `generation_counter`, `runtime_counter` (Δhours × rate), `consumable_balance` (tank: deliveries usage + forecast), `direct_delta`, `instant_rate`, `virtual` (expression over other meters). New ingestion/normalization logic dispatches on mode.
- **Nothing domain-specific is hardcoded.** Energy types are data. Cost **categories are decoupled from energy types** (Heizung may be oil today, heat-pump tomorrow). PV self-consumption/savings/net are **virtual meters** with user-defined expressions, not special-cased code. Tariffs are time-ranged (price history), scoped global / per-type / per-meter. - **Nothing domain-specific is hardcoded.** Energy types are data. Cost **categories are decoupled from energy types** (Heizung may be oil today, heat-pump tomorrow). PV self-consumption/savings/net are **virtual meters** with user-defined expressions, not special-cased code. Tariffs are time-ranged (price history), scoped global / per-type / per-meter.
+11 -2
View File
@@ -72,12 +72,21 @@ Configuration is via environment variables (`Section__Key` double-underscore map
| `MeterVault__ReverseProxyTrust` | `true` to honour `X-Forwarded-User` behind an auth proxy | | `MeterVault__ReverseProxyTrust` | `true` to honour `X-Forwarded-User` behind an auth proxy |
| `MeterVault__EnableLiveIngestion` | `false` to disable the MQTT/HA workers | | `MeterVault__EnableLiveIngestion` | `false` to disable the MQTT/HA workers |
| `MeterVault__SeedReferenceData` | `true` to load the bundled demo dataset on first start (idempotent) | | `MeterVault__SeedReferenceData` | `true` to load the bundled demo dataset on first start (idempotent) |
| `MeterVault__DataProtectionKeyPath` | Where the key ring for UI-entered connector secrets lives (default `/var/lib/metervault/keys`) |
The REST API is **closed by default**: with no `ApiKeys` configured and `AllowAnonymousApi` off, it The REST API is **closed by default**: with no `ApiKeys` configured and `AllowAnonymousApi` off, it
returns 401. Set at least one API key (or open it explicitly for a trusted network). returns 401. Set at least one API key (or open it explicitly for a trusted network).
Secrets (broker/HA tokens) are **never** stored in the database — endpoint configs hold the *name* Secrets (broker/HA tokens) are **never** stored in the database as plaintext. Each connector picks
of an environment variable, resolved at runtime. one of two forms: the *name* of an environment variable, resolved at runtime, or the secret typed
into the admin UI and encrypted at rest under the data-protection key ring. Either way a `pg_dump`
or JSON export carries nothing usable.
Keep the key ring on persistent storage outside the app directory — the default
`/var/lib/metervault/keys` survives an LXC update, and the Compose file mounts a named volume for it.
Lose it and every UI-entered secret must be re-entered. The key ring is on disk, so this protects
against leaked database content, not against an attacker who already has the host; that is the same
trust boundary an environment variable has.
## Pushing readings (Home Assistant) ## Pushing readings (Home Assistant)
+9
View File
@@ -23,6 +23,7 @@
: "${INSTALL_DIR:=/opt/metervault}" : "${INSTALL_DIR:=/opt/metervault}"
: "${SOURCE_DIR:=/opt/metervault-src}" : "${SOURCE_DIR:=/opt/metervault-src}"
: "${ENV_FILE:=/etc/metervault/environment}" : "${ENV_FILE:=/etc/metervault/environment}"
: "${KEYRING_DIR:=/var/lib/metervault/keys}"
: "${DB_NAME:=metervault}" : "${DB_NAME:=metervault}"
: "${DB_USER:=metervault}" : "${DB_USER:=metervault}"
@@ -203,6 +204,13 @@ EOF
chmod 600 "${ENV_FILE}" chmod 600 "${ENV_FILE}"
} }
# Key ring for connector secrets typed into the admin UI (SDD §6.4). The app creates this itself if
# missing, but with the default umask — created here instead so it is 0700 from the start, and so it
# is visibly outside /opt/metervault, which the updater republishes on every run.
write_keyring_dir() {
install -d -m 0700 "${KEYRING_DIR}"
}
write_systemd() { write_systemd() {
cat <<'EOF' >/etc/systemd/system/metervault.service cat <<'EOF' >/etc/systemd/system/metervault.service
[Unit] [Unit]
@@ -246,6 +254,7 @@ main() {
install_dotnet_sdk install_dotnet_sdk
build_metervault build_metervault
write_env write_env
write_keyring_dir
write_systemd write_systemd
systemctl daemon-reload 2>/dev/null || true systemctl daemon-reload 2>/dev/null || true
+2
View File
@@ -23,4 +23,6 @@
<Config Name="API key" Target="MeterVault__ApiKeys__0" Default="" Mode="" Description="API key for the REST API (X-Api-Key header). Leave blank to leave the API open." Type="Variable" Display="always" Required="false" Mask="true"/> <Config Name="API key" Target="MeterVault__ApiKeys__0" Default="" Mode="" Description="API key for the REST API (X-Api-Key header). Leave blank to leave the API open." Type="Variable" Display="always" Required="false" Mask="true"/>
<Config Name="Reverse-proxy trust" Target="MeterVault__ReverseProxyTrust" Default="false" Mode="" Description="Honour X-Forwarded-User from a trusted auth proxy" Type="Variable" Display="advanced" Required="false">false</Config> <Config Name="Reverse-proxy trust" Target="MeterVault__ReverseProxyTrust" Default="false" Mode="" Description="Honour X-Forwarded-User from a trusted auth proxy" Type="Variable" Display="advanced" Required="false">false</Config>
<Config Name="Secret key ring" Target="/var/lib/metervault/keys" Default="/mnt/user/appdata/metervault/keys" Mode="rw" Description="Encryption keys for connector secrets entered in the web UI. Must persist: without this mapping every stored token is lost when the container is recreated." Type="Path" Display="always" Required="true">/mnt/user/appdata/metervault/keys</Config>
</Container> </Container>
+4
View File
@@ -429,6 +429,10 @@ The key ring must be persisted outside the app directory (`MeterVault__DataProte
### 7.1 Register → consumption ### 7.1 Register → consumption
For `cumulative_counter`/`generation_counter`: for each new reading, `amount = value previous_value`. Persist to `consumption`. Cross a `meter_swap` as `(old_final prev) + (curr new_initial)`; a `counter_reset` starts a fresh baseline. Ignore/annotate negative deltas that lack an explaining event (flag as anomaly). For `cumulative_counter`/`generation_counter`: for each new reading, `amount = value previous_value`. Persist to `consumption`. Cross a `meter_swap` as `(old_final prev) + (curr new_initial)`; a `counter_reset` starts a fresh baseline. Ignore/annotate negative deltas that lack an explaining event (flag as anomaly).
**Gap attribution.** A delta is booked at the reading that closes it — correct at the reporting cadence, and what the reference sheets do. After a long unread stretch it misleads: 78 days of PV output arriving as one July row makes June look idle. So an interval containing **two or more complete calendar months** is apportioned across the months it covers, in proportion to elapsed time, and every row it yields is marked `quality = estimated` — the meter recorded a total, not a shape.
The threshold is deliberately conservative. A monthly series contains exactly one whole month per interval and is never touched, which is what keeps the golden-fixture reconciliation (§13) measuring the normalizer rather than the splitter. Counting whole months *contained* rather than boundaries *crossed* keeps the rule stable when a reading lands hours late. Swap and reset amounts are never apportioned: they are explicit corrections booked at the event. Split points are UTC, so one can sit an hour or two from a displayed month edge (§10) — immaterial when dividing a multi-month gap, and the alternative is threading a timezone through an otherwise timezone-free engine.
### 7.2 Runtime → consumption (burner) ### 7.2 Runtime → consumption (burner)
For `runtime_counter`: `amount = Δhours × rate`. `rate` comes from the linked `tank`: `fixed` (nozzle spec, L/h) or `empirical` (`Δlevel ÷ Δhours` measured between deliveries/level reads — reproduce the spreadsheet's 1.87/1.94/2.92 … behaviour). Expose both; default empirical when level data exists, else fixed. For `runtime_counter`: `amount = Δhours × rate`. `rate` comes from the linked `tank`: `fixed` (nozzle spec, L/h) or `empirical` (`Δlevel ÷ Δhours` measured between deliveries/level reads — reproduce the spreadsheet's 1.87/1.94/2.92 … behaviour). Expose both; default empirical when level data exists, else fixed.
+11 -3
View File
@@ -36,17 +36,25 @@ public static class ApiEndpoints
} }
int written = 0, updated = 0, rejected = 0, ignored = 0; int written = 0, updated = 0, rejected = 0, ignored = 0;
var touched = new HashSet<int>();
foreach (var r in readings) foreach (var r in readings)
{ {
switch (await ingestion.IngestByMeterAsync(r.MeterId, r.Time, r.Value, ct)) // Normalize once per meter after the batch, not per reading: a recompute rewrites the
// meter's whole consumption series, so doing it inside the loop is quadratic.
switch (await ingestion.IngestByMeterAsync(r.MeterId, r.Time, r.Value, renormalize: false, ct))
{ {
case IngestionOutcome.Written: written++; break; case IngestionOutcome.Written: written++; touched.Add(r.MeterId); break;
case IngestionOutcome.Updated: updated++; break; case IngestionOutcome.Updated: updated++; touched.Add(r.MeterId); break;
case IngestionOutcome.RejectedDecrease: rejected++; break; case IngestionOutcome.RejectedDecrease: rejected++; break;
default: ignored++; break; // unknown meter default: ignored++; break; // unknown meter
} }
} }
foreach (var meterId in touched)
{
await ingestion.RenormalizeMeterAsync(meterId, ct);
}
return Results.Ok(new IngestResult(written, updated, rejected, ignored)); return Results.Ok(new IngestResult(written, updated, rejected, ignored));
}).WithSummary("Ingest one or more readings (idempotent). Lets Home Assistant push."); }).WithSummary("Ingest one or more readings (idempotent). Lets Home Assistant push.");
+110
View File
@@ -0,0 +1,110 @@
namespace MeterVault.Core.Normalization;
/// <summary>
/// Spreads a register delta that spans several calendar months across the months it actually covers.
/// </summary>
/// <remarks>
/// A counter delta is booked at the reading that closes it, which is right when readings arrive at
/// the reporting cadence: a monthly series books December's usage against the 1 January reading, and
/// that is what the reference spreadsheet does. It stops being right when a meter goes unread for a
/// long stretch — 78 days of PV generation arriving as a single July row makes June look idle and
/// July look extraordinary, when nothing unusual happened.
///
/// Splitting is therefore deliberately conservative: an interval is divided only when it contains
/// <em>two or more complete calendar months</em>. A normal monthly series contains exactly one, so it
/// is left completely untouched and the golden-fixture reconciliation stands (SDD §13); a series that
/// skipped a month or more contains two or more, which is precisely where lumping misleads.
///
/// Counting whole months contained, rather than boundaries crossed, is what makes this stable against
/// readings that do not land on midnight: a monthly reading arriving at 06:00 on the 1st still
/// contains one whole month, where a boundary count would tip over and hand the new month a sliver.
///
/// The division is by elapsed time, so it assumes a flat rate across the gap. That is a guess — the
/// meter recorded a total, not a shape — so every row it produces is marked
/// <see cref="Domain.ReadingQuality.Estimated"/>. The sum is exact: the final segment absorbs any
/// rounding remainder, so a split never creates or destroys energy.
///
/// Boundaries are UTC. The dashboard buckets in the instance timezone (SDD §10), so a split point
/// can sit an hour or two from the displayed month edge — immaterial for apportioning a multi-month
/// gap, and the alternative would be threading a timezone through the otherwise timezone-free engine.
/// </remarks>
public static class GapAttribution
{
/// <summary>
/// True when an interval contains two or more complete calendar months, and so would misattribute
/// a long gap to its closing month.
/// </summary>
public static bool ShouldSplit(DateTimeOffset start, DateTimeOffset end) =>
end > start && WholeMonthsInside(start, end) >= 2;
/// <summary>
/// Divides <paramref name="amount"/> across the calendar months between the two instants,
/// proportionally to the time spent in each.
/// </summary>
/// <remarks>
/// Each segment is stamped at its <em>end</em>, which keeps the existing convention that a
/// consumption row records the period ending at its timestamp — the same reason an unsplit delta
/// sits on its closing reading, and the reason the reference sheet's January row carries
/// December's usage. So the share covering May is stamped 1 June and buckets as June, exactly as
/// a May-to-June monthly reading pair already would. The last segment therefore keeps the closing
/// reading's own timestamp, and nothing shifts relative to how unsplit intervals are labelled.
/// </remarks>
public static IReadOnlyList<GapSegment> Split(DateTimeOffset start, DateTimeOffset end, double amount)
{
if (end <= start)
{
return [new GapSegment(end, amount)];
}
var total = end - start;
var segments = new List<GapSegment>();
var cursor = start;
var assigned = 0d;
while (cursor < end)
{
var nextBoundary = NextMonthStart(cursor);
var segmentEnd = nextBoundary < end ? nextBoundary : end;
if (segmentEnd >= end)
{
// Final segment takes the remainder, so the parts always sum to the original.
segments.Add(new GapSegment(end, amount - assigned));
break;
}
var share = amount * ((segmentEnd - cursor) / total);
segments.Add(new GapSegment(segmentEnd, share));
assigned += share;
cursor = segmentEnd;
}
return segments;
}
private static int WholeMonthsInside(DateTimeOffset start, DateTimeOffset end)
{
// A month counts only if it lies entirely within the interval, so a partial month at either
// edge never tips the decision.
var cursor = MonthStart(start) == start.ToUniversalTime() ? MonthStart(start) : NextMonthStart(start);
var whole = 0;
while (cursor.AddMonths(1) <= end)
{
whole++;
cursor = cursor.AddMonths(1);
}
return whole;
}
private static DateTimeOffset MonthStart(DateTimeOffset instant)
{
var utc = instant.ToUniversalTime();
return new DateTimeOffset(utc.Year, utc.Month, 1, 0, 0, 0, TimeSpan.Zero);
}
private static DateTimeOffset NextMonthStart(DateTimeOffset instant) => MonthStart(instant).AddMonths(1);
}
/// <summary>One month's share of a spread gap: the instant it closes and the amount attributed.</summary>
public sealed record GapSegment(DateTimeOffset Time, double Amount);
@@ -12,7 +12,10 @@ namespace MeterVault.Core.Normalization.Normalizers;
/// reconciles to 12), otherwise <c>(oldFinal prev) + (curr newInitial)</c>;</item> /// reconciles to 12), otherwise <c>(oldFinal prev) + (curr newInitial)</c>;</item>
/// <item>counter reset → baseline restarts at <c>NewValue</c> (default 0);</item> /// <item>counter reset → baseline restarts at <c>NewValue</c> (default 0);</item>
/// <item>unexplained decrease → 0 with an anomaly flagged (never a silent negative), rebaselined /// <item>unexplained decrease → 0 with an anomaly flagged (never a silent negative), rebaselined
/// to the current value.</item> /// to the current value;</item>
/// <item>a plain increase spanning two or more whole calendar months → apportioned across them
/// and marked estimated (<see cref="GapAttribution"/>), so an unread stretch does not land wholly
/// in its closing month. A monthly cadence never triggers this.</item>
/// </list> /// </list>
/// </summary> /// </summary>
public abstract class CounterNormalizerBase : IMeterNormalizer public abstract class CounterNormalizerBase : IMeterNormalizer
@@ -45,6 +48,7 @@ public abstract class CounterNormalizerBase : IMeterNormalizer
var swap = FindEvent(swaps, previousTime, reading.Time); var swap = FindEvent(swaps, previousTime, reading.Time);
double amount; double amount;
var plainIncrease = false;
if (swap is { EventType: MeterEventType.MeterSwap }) if (swap is { EventType: MeterEventType.MeterSwap })
{ {
amount = swap.Amount amount = swap.Amount
@@ -57,6 +61,7 @@ public abstract class CounterNormalizerBase : IMeterNormalizer
else if (reading.Value >= previous) else if (reading.Value >= previous)
{ {
amount = reading.Value - previous; amount = reading.Value - previous;
plainIncrease = true;
} }
else else
{ {
@@ -65,15 +70,40 @@ public abstract class CounterNormalizerBase : IMeterNormalizer
quality = ReadingQuality.Estimated; quality = ReadingQuality.Estimated;
} }
yield return new Consumption // Only a plain increase over an unread stretch is worth apportioning (SDD §7.1). A swap
// or reset amount is an explicit correction booked at its event; a rejected decrease
// contributes nothing; the first reading has no interval behind it; and fanning a zero
// out across three months just adds rows that say nothing.
var gapStart = plainIncrease && Math.Abs(amount) > 1e-9 ? previousTime : null;
if (gapStart is { } start && GapAttribution.ShouldSplit(start, reading.Time))
{ {
MeterId = context.Meter.MeterId, foreach (var segment in GapAttribution.Split(start, reading.Time, amount))
Time = reading.Time, {
Amount = amount, yield return new Consumption
Kind = Kind, {
Quality = quality, MeterId = context.Meter.MeterId,
ImportBatchId = reading.ImportBatchId, Time = segment.Time,
}; Amount = segment.Amount,
Kind = Kind,
// The total is measured; only its distribution across the gap is inferred.
Quality = ReadingQuality.Estimated,
ImportBatchId = reading.ImportBatchId,
};
}
}
else
{
yield return new Consumption
{
MeterId = context.Meter.MeterId,
Time = reading.Time,
Amount = amount,
Kind = Kind,
Quality = quality,
ImportBatchId = reading.ImportBatchId,
};
}
previous = reading.Value; previous = reading.Value;
previousTime = reading.Time; previousTime = reading.Time;
@@ -45,8 +45,13 @@ public sealed record MeterPeriodView(
public bool HasHistory => Last12Months.Count > 0; public bool HasHistory => Last12Months.Count > 0;
/// <summary>
/// Percentage change is only meaningful against a positive baseline. Dividing by a negative one
/// inverts the sign — a net-export meter going from 100 to 150 would report "+50% more used"
/// when it exported half as much again — so those report no basis rather than a confident lie.
/// </summary>
private static double? Ratio(double current, double previous) => private static double? Ratio(double current, double previous) =>
Math.Abs(previous) < 1e-9 ? null : (current - previous) / previous; previous <= 1e-9 ? null : (current - previous) / previous;
} }
/// <summary>A meter lifecycle/correction event row.</summary> /// <summary>A meter lifecycle/correction event row.</summary>
@@ -63,8 +63,14 @@ public sealed class IngestionService(
} }
/// <summary>Ingests directly against a meter (REST push, e.g. Home Assistant POST /api/v1/readings).</summary> /// <summary>Ingests directly against a meter (REST push, e.g. Home Assistant POST /api/v1/readings).</summary>
/// <param name="renormalize">
/// False to skip deriving consumption, for callers ingesting a batch into one meter: recomputing
/// rewrites the meter's entire series, so doing it per reading is quadratic in batch size. Such a
/// caller must recompute the affected meters itself once the batch is in.
/// </param>
public async Task<IngestionOutcome> IngestByMeterAsync( public async Task<IngestionOutcome> IngestByMeterAsync(
int meterId, DateTimeOffset time, double value, CancellationToken cancellationToken = default) int meterId, DateTimeOffset time, double value, bool renormalize = true,
CancellationToken cancellationToken = default)
{ {
var meter = await _db.Meters var meter = await _db.Meters
.FirstOrDefaultAsync(m => m.Id == meterId, cancellationToken).ConfigureAwait(false); .FirstOrDefaultAsync(m => m.Id == meterId, cancellationToken).ConfigureAwait(false);
@@ -82,10 +88,24 @@ public sealed class IngestionService(
var outcome = await UpsertAsync(meter, utc, value, sourceId: null, cancellationToken).ConfigureAwait(false); var outcome = await UpsertAsync(meter, utc, value, sourceId: null, cancellationToken).ConfigureAwait(false);
await _db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); await _db.SaveChangesAsync(cancellationToken).ConfigureAwait(false);
await RenormalizeAsync(meter.Id, outcome, cancellationToken).ConfigureAwait(false); if (renormalize)
{
await RenormalizeAsync(meter.Id, outcome, cancellationToken).ConfigureAwait(false);
}
return outcome; return outcome;
} }
/// <summary>
/// Derives consumption for one meter after a batch of readings has been written. The public
/// counterpart to skipping <c>renormalize</c> on each individual ingest.
/// </summary>
public async Task RenormalizeMeterAsync(int meterId, CancellationToken cancellationToken = default)
{
await _normalization.RecomputeMeterAsync(meterId, batchId: null, cancellationToken).ConfigureAwait(false);
await _db.SaveChangesAsync(cancellationToken).ConfigureAwait(false);
}
/// <summary> /// <summary>
/// Derives consumption from the reading just written. Without this a live-ingested reading sits /// Derives consumption from the reading just written. Without this a live-ingested reading sits
/// in <c>reading</c> forever and every derived figure — consumption, generation, cost — stays /// in <c>reading</c> forever and every derived figure — consumption, generation, cost — stays
+208
View File
@@ -0,0 +1,208 @@
using MeterVault.Core.Domain;
using MeterVault.Core.Normalization;
using static MeterVault.Core.Tests.TestData;
namespace MeterVault.Core.Tests;
/// <summary>
/// A counter delta is booked at the reading that closes it. That is correct at the reporting cadence
/// and wrong after a long outage, so a gap containing two or more whole months is apportioned.
/// The boundary between those two behaviours is what these pin down: a normal monthly series must
/// come out byte-for-byte unchanged, because it is what reconciles against the reference spreadsheet.
/// </summary>
public sealed class GapAttributionTests
{
private readonly INormalizationEngine _engine = NormalizationEngine.CreateDefault();
[Fact]
public void A_monthly_cadence_is_never_split()
{
// One whole month per interval — the reference-data shape. Splitting here would move energy
// between months and break reconciliation (SDD §13).
Assert.False(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 2)));
Assert.False(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 2).AddDays(-1)));
// A reading that lands hours late must not tip the rule and hand January a sliver.
Assert.False(GapAttribution.ShouldSplit(Month(2023, 12), Month(2024, 1).AddHours(6)));
// Nor should a six-week interval, which still contains only one whole month.
Assert.False(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 2).AddDays(14)));
}
[Fact]
public void Sub_month_intervals_are_never_split()
{
Assert.False(GapAttribution.ShouldSplit(Month(2023, 5), Month(2023, 5).AddHours(1)));
Assert.False(GapAttribution.ShouldSplit(Month(2023, 5).AddDays(10), Month(2023, 5).AddDays(20)));
}
[Fact]
public void A_skipped_month_is_split()
{
Assert.True(GapAttribution.ShouldSplit(Month(2023, 1), Month(2023, 3)));
Assert.True(GapAttribution.ShouldSplit(Month(2026, 5), new DateTimeOffset(2026, 7, 18, 15, 33, 0, TimeSpan.Zero)));
}
[Fact]
public void Splitting_preserves_the_total_and_keeps_the_closing_timestamp()
{
var start = Month(2026, 5);
var end = new DateTimeOffset(2026, 7, 18, 15, 33, 0, TimeSpan.Zero);
var segments = GapAttribution.Split(start, end, 714.5);
// May, June, July.
Assert.Equal(3, segments.Count);
Assert.Equal(714.5, segments.Sum(s => s.Amount), 6);
Assert.Equal(end, segments[^1].Time);
Assert.Equal(Month(2026, 6), segments[0].Time);
Assert.Equal(Month(2026, 7), segments[1].Time);
}
[Fact]
public void Each_month_gets_a_share_proportional_to_the_time_it_covers()
{
// Exactly two whole months: an even split, to the cent.
var segments = GapAttribution.Split(Month(2023, 1), Month(2023, 3), 620);
Assert.Equal(2, segments.Count);
var januaryShare = 31d / 59d; // 2023 is not a leap year: Jan 31 + Feb 28.
Assert.Equal(620 * januaryShare, segments[0].Amount, 6);
Assert.Equal(620, segments.Sum(s => s.Amount), 6);
}
[Fact]
public void A_gap_in_a_counter_series_is_spread_and_marked_estimated()
{
var ctx = new NormalizationContext
{
Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" },
Readings =
[
Reading(1, Month(2023, 1), 1000),
Reading(1, Month(2023, 4), 1900), // three months in one reading
],
};
var result = _engine.Normalize(ctx).ToList();
// Baseline row for the first reading, then Jan/Feb/Mar shares of the 900 gap.
Assert.Equal(4, result.Count);
Assert.Equal(1000 + 900, result.Sum(c => c.Amount), 6);
var spread = result.Skip(1).ToList();
Assert.All(spread, c => Assert.Equal(ReadingQuality.Estimated, c.Quality));
Assert.Equal(900, spread.Sum(c => c.Amount), 6);
}
[Fact]
public void An_ordinary_monthly_series_produces_one_measured_row_per_reading()
{
// The regression that matters: this is the reference-data shape, and it must not gain rows
// or lose its quality markers.
var ctx = new NormalizationContext
{
Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" },
Readings =
[
Reading(1, Month(2022, 9), 0),
Reading(1, Month(2022, 10), 411),
Reading(1, Month(2022, 11), 1153),
Reading(1, Month(2022, 12), 1968),
],
};
var result = _engine.Normalize(ctx).ToList();
Assert.Equal(4, result.Count);
Assert.DoesNotContain(result, c => c.Quality == ReadingQuality.Estimated);
Assert.Equal([0, 411, 742, 815], result.Select(c => c.Amount).ToArray());
}
[Fact]
public void The_observed_solar_gap_is_apportioned_across_the_months_it_covers()
{
// The case this exists for: Solar 1 read monthly to 1 May 2026, then a single live reading on
// 18 July. 714.5 kWh of generation arriving as one July row made June look like an outage.
var ctx = new NormalizationContext
{
Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.GenerationCounter, Unit = "kWh" },
Readings =
[
Reading(1, Month(2026, 4), 10308),
Reading(1, Month(2026, 5), 10731),
Reading(1, new DateTimeOffset(2026, 7, 18, 15, 33, 0, TimeSpan.Zero), 11445.5),
],
};
var result = _engine.Normalize(ctx).ToList();
var gap = result.Where(c => c.Time > Month(2026, 5)).ToList();
Assert.Equal(3, gap.Count);
Assert.Equal(714.5, gap.Sum(c => c.Amount), 6);
// No single month swallows the whole gap any more.
Assert.All(gap, c => Assert.True(c.Amount < 714.5 * 0.75, $"{c.Time:yyyy-MM-dd} took {c.Amount:0.#}"));
// Generation is preserved end to end: baseline 0 → 11445.5.
Assert.Equal(11445.5, result.Sum(c => c.Amount), 6);
}
[Fact]
public void An_unchanged_register_across_a_long_gap_does_not_fan_out_into_empty_rows()
{
// Nothing was used. Three rows of zero say no more than one, and would dilute the
// measured/estimated ratio on the detail page.
var ctx = new NormalizationContext
{
Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" },
Readings = [Reading(1, Month(2023, 1), 500), Reading(1, Month(2023, 5), 500)],
};
var result = _engine.Normalize(ctx).ToList();
Assert.Equal(2, result.Count);
Assert.Equal(0, result[^1].Amount, 6);
}
[Fact]
public void A_rejected_decrease_across_a_long_gap_stays_a_single_row()
{
// The decrease branch already yields 0 and rebaselines; spreading that zero would invent
// rows for months the meter never reported.
var ctx = new NormalizationContext
{
Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "kWh" },
Readings = [Reading(1, Month(2023, 1), 900), Reading(1, Month(2023, 5), 100)],
};
var result = _engine.Normalize(ctx).ToList();
Assert.Equal(2, result.Count);
Assert.Equal(0, result[^1].Amount, 6);
Assert.Equal(Month(2023, 5), result[^1].Time);
}
[Fact]
public void A_swap_across_a_long_gap_keeps_its_explicit_amount_in_one_row()
{
// Swap amounts are corrections booked at the event (the water …861 → 2 case reconciles to
// 12). Apportioning one across the gap would silently rewrite a number the operator supplied.
var ctx = new NormalizationContext
{
Meter = new MeterConfig { MeterId = 1, Mode = MeterMode.CumulativeCounter, Unit = "m³" },
Readings =
[
Reading(1, Month(2023, 1), 861),
Reading(1, Month(2023, 5), 15),
],
Events = [Swap(1, Month(2023, 3), prevValue: 861, newValue: 2, amount: 12)],
};
var result = _engine.Normalize(ctx).ToList();
Assert.Equal(2, result.Count);
Assert.Equal(12, result[^1].Amount, 6);
Assert.NotEqual(ReadingQuality.Estimated, result[^1].Quality);
}
}
@@ -174,6 +174,31 @@ public sealed class IngestionServiceTests(TimescaleFixture fx)
await CleanupAsync(db, meterId); await CleanupAsync(db, meterId);
} }
[Fact]
public async Task A_batch_can_defer_normalization_and_derive_the_same_series_once_at_the_end()
{
// Recomputing rewrites a meter's whole consumption series, so the batch endpoint skips it
// per reading and does it once. The result must be identical to normalizing as it goes.
await using var db = fx.CreateContext();
var (meterId, _) = await SetupAsync(db, MeterMode.CumulativeCounter);
var service = NewIngestion(db);
for (var hour = 0; hour < 5; hour++)
{
await service.IngestByMeterAsync(meterId, T0.AddHours(hour), 1000 + (hour * 10), renormalize: false);
}
Assert.False(await db.Consumption.AnyAsync(c => c.MeterId == meterId));
await service.RenormalizeMeterAsync(meterId);
var consumption = await db.Consumption.AsNoTracking().Where(c => c.MeterId == meterId).ToListAsync();
Assert.Equal(5, consumption.Count);
Assert.Equal(1040d, consumption.Sum(c => c.Amount), 3); // baseline 0 → 1000, then 4 × 10
await CleanupAsync(db, meterId);
}
private static IngestionService NewIngestion(MeterVaultDbContext db) => private static IngestionService NewIngestion(MeterVaultDbContext db) =>
new(db, new MeterVault.Infrastructure.Normalization.NormalizationService( new(db, new MeterVault.Infrastructure.Normalization.NormalizationService(
db, MeterVault.Core.Normalization.NormalizationEngine.CreateDefault())); db, MeterVault.Core.Normalization.NormalizationEngine.CreateDefault()));
@@ -77,6 +77,27 @@ public sealed class MeterPeriodServiceTests(TimescaleFixture fx)
await CleanupAsync(db, meterId); await CleanupAsync(db, meterId);
} }
[Fact]
public async Task A_negative_previous_period_reports_no_basis_rather_than_an_inverted_percentage()
{
// Net export: -100 -> -150 is half again as much exported, but dividing by a negative
// baseline would render it "+50%", which reads as more consumption.
await using var db = fx.CreateContext();
var meterId = await SetupAsync(db, MeterMode.CumulativeCounter);
var today = DateOnly.FromDateTime(DateTime.UtcNow);
var thisMonth = new DateOnly(today.Year, today.Month, 1);
await AddConsumptionAsync(db, meterId, ConsumptionKind.Consumption, thisMonth.AddDays(1), -150);
await AddConsumptionAsync(db, meterId, ConsumptionKind.Consumption, thisMonth.AddMonths(-1).AddDays(3), -100);
var view = await NewService().GetAsync(meterId);
Assert.NotNull(view);
Assert.Null(view!.MonthChange);
await CleanupAsync(db, meterId);
}
private MeterPeriodService NewService() private MeterPeriodService NewService()
{ {
var options = Microsoft.Extensions.Options.Options.Create( var options = Microsoft.Extensions.Options.Options.Create(
@@ -0,0 +1,54 @@
using MeterVault.Core.Domain;
using MeterVault.Core.Normalization;
using MeterVault.Infrastructure.Import;
using static MeterVault.Integration.Tests.Reconciliation.ReconciliationSupport;
namespace MeterVault.Integration.Tests.Reconciliation;
/// <summary>
/// Gap splitting apportions a long unread stretch across the months it covers. The reference sheets
/// are read monthly and must never trigger it, or their months would silently shift and the whole
/// golden-fixture oracle (SDD §13) would be measuring the splitter instead of the normalizer.
/// </summary>
/// <remarks>
/// The reconciliation suites already compare month by month, so a spurious split would surface there
/// as a numeric failure. This asserts the mechanism directly instead of relying on that side effect:
/// it proves the rule was evaluated against real fixture cadence and declined to fire, rather than
/// the fixtures simply having no gaps to find.
/// </remarks>
public sealed class GapSplittingIsInertOnFixturesTests
{
[Theory]
[InlineData(ReferenceProfiles.Haus, MeterMode.CumulativeCounter)]
[InlineData(ReferenceProfiles.Netz, MeterMode.CumulativeCounter)]
[InlineData(ReferenceProfiles.Auto, MeterMode.CumulativeCounter)]
[InlineData(ReferenceProfiles.Solar1, MeterMode.GenerationCounter)]
[InlineData(ReferenceProfiles.Solar2, MeterMode.GenerationCounter)]
public void Electricity_meters_produce_exactly_one_row_per_reading(int meterId, MeterMode mode)
{
var staged = Stage(ReferenceProfiles.Electricity(), Electricity);
var readings = staged.Readings.Count(r => r.MeterId == meterId);
var computed = Normalize(staged, new MeterConfig { MeterId = meterId, Mode = mode, Unit = "kWh" });
Assert.True(readings > 20, $"meter {meterId}: expected a real series, got {readings} readings.");
Assert.Equal(readings, computed.Count);
}
[Fact]
public void No_fixture_interval_is_long_enough_to_split()
{
var staged = Stage(ReferenceProfiles.Electricity(), Electricity);
foreach (var group in staged.Readings.GroupBy(r => r.MeterId))
{
var times = group.Select(r => r.Time).OrderBy(t => t).ToList();
for (var i = 1; i < times.Count; i++)
{
Assert.False(
GapAttribution.ShouldSplit(times[i - 1], times[i]),
$"meter {group.Key}: {times[i - 1]:yyyy-MM-dd} → {times[i]:yyyy-MM-dd} would be split.");
}
}
}
}