Analysis: bound the freshness query, let window sums skip chunks, share the Overview's catalog
ci / build-test (push) Successful in 2m34s

Three things the 1,000-meter x 10-year measurement found, each proved by
EXPLAIN or a statement count before and after. No tally moves.

Freshness had two jobs in one unbounded query. The mark -- when a meter
last delivered -- is now stored on meter_rollup_state and maintained by
every recompute, with a one-pass backfill in the migration, so an
import-only meter keeps its years-old last activity without reading a
single raw row. The rhythm that decides stale versus live is sampled
inside a 90-day window and only for meters that actually have a live
source; a source silent for longer than that is re-read unbounded, so it
is still called stale by its own rhythm rather than by a default. The
portfolio query went from 13.8 ms planning plus 36.1 ms execution across
all 123 reading chunks to 0.58 plus 0.44 ms across four.

Window sums took their time bounds only from the unnest join, so the
planner could not exclude chunks: a 1,960-window case scanned 1.39 M rows
in parallel and spilled a 45 MB sort. Repeating the overall min and max as
constants makes it five chunks and nested-loop index scans, 121.5 ms to
8.7 ms.

The Overview read the catalog three times, once for the quantities and
once for each of its two bills. One context and one catalog snapshot now
feed all three: 31 statements per load to 23.

The final timings on an idle machine are in docs/ANALYSIS_REPORT.md: the
brief's target request (100 meters, ten years, monthly) is 286 ms against
two seconds, and a startup rebuild of 1,000 meters is 279 s.
This commit is contained in:
Florian Schmidt
2026-09-20 11:16:48 +02:00
parent ec51419f64
commit 5a6f34a467
21 changed files with 1857 additions and 87 deletions
+42 -5
View File
@@ -474,7 +474,16 @@ internal sealed class AnalysisRun
}
}
/// <summary>Freshness of every data leaf (D-18): recent reading times, the last event, and the live sources.</summary>
/// <summary>Freshness of every data leaf (D-18): the last reading and event, and the live sources' rhythm.</summary>
/// <remarks>
/// The mark — the meter's last reading — is read from its stored rollup state, which the recompute behind every
/// write path records (A-40); an import-only meter's mark is years old and must stay exact, so it is never
/// guessed from a bounded sample. Only the *rhythm* of a live source needs raw reading times, and only from the
/// recent past: that query carries <see cref="FreshnessRules.RecentWindow"/> as its lower bound, so it plans over
/// the newest raw chunks instead of every chunk a decade of history has. A live meter that delivered nothing
/// inside the window has no rhythm there; those few meters are asked again over their whole history, so the
/// stale verdict of a long-silent source is unchanged.
/// </remarks>
private async Task LoadFreshnessAsync(CancellationToken cancellationToken)
{
var ids = _dataLeaves.ToList();
@@ -483,17 +492,30 @@ internal sealed class AnalysisRun
return;
}
var readings = await AnalysisQueries.RecentReadingsAsync(_connection, ids, cancellationToken).ConfigureAwait(false);
var events = await AnalysisQueries.LastEventsAsync(_connection, ids, cancellationToken).ConfigureAwait(false);
var sources = await _db.MeterSources.AsNoTracking()
.Include(s => s.Endpoint)
.Where(s => ids.Contains(s.MeterId) && s.IsEnabled)
.ToListAsync(cancellationToken).ConfigureAwait(false);
var liveIds = ids.Where(id => sources.Exists(s => s.MeterId == id && IsLive(s))).ToList();
var readings = await AnalysisQueries
.RecentReadingsAsync(_connection, liveIds, Now - FreshnessRules.RecentWindow, cancellationToken).ConfigureAwait(false);
var silent = liveIds.Where(id => (readings.GetValueOrDefault(id)?.Count ?? 0) < 2).ToList();
if (silent.Count > 0)
{
foreach (var (id, times) in await AnalysisQueries
.RecentReadingsAsync(_connection, silent, since: null, cancellationToken).ConfigureAwait(false))
{
readings[id] = times;
}
}
var events = await AnalysisQueries.LastEventsAsync(_connection, ids, cancellationToken).ConfigureAwait(false);
foreach (var id in ids)
{
var times = readings.GetValueOrDefault(id) ?? [];
var live = sources.Where(s => s.MeterId == id && s.SourceType is SourceType.Mqtt or SourceType.Tasmota or SourceType.HomeAssistant).ToList();
var live = sources.Where(s => s.MeterId == id && IsLive(s)).ToList();
TimeSpan? poll = null;
foreach (var source in live.Where(s => s.SourceType == SourceType.HomeAssistant))
{
@@ -507,7 +529,7 @@ internal sealed class AnalysisRun
}
var input = new FreshnessInput(
times.Count > 0 ? times.Max() : null,
LastReadingOf(id, times),
events.TryGetValue(id, out var lastEvent) ? lastEvent : null,
times,
live.Count > 0,
@@ -516,6 +538,21 @@ internal sealed class AnalysisRun
}
}
/// <summary>A source that is expected to deliver on its own (D-18).</summary>
private static bool IsLive(MeterSource source) =>
source.SourceType is SourceType.Mqtt or SourceType.Tasmota or SourceType.HomeAssistant;
/// <summary>
/// The meter's last reading time (A-40): what its stored analysis data recorded, and — for a live meter whose
/// rhythm was read — whatever of the two is later, so a reading ingested since the last recompute still counts.
/// </summary>
private DateTimeOffset? LastReadingOf(int id, IReadOnlyList<DateTimeOffset> times)
{
var stored = _catalog.Meters.TryGetValue(id, out var meter) ? meter.State?.LastReadingAt : null;
var sampled = times.Count > 0 ? times.Max() : (DateTimeOffset?)null;
return stored is { } s && sampled is { } r ? (s >= r ? s : r) : stored ?? sampled;
}
// ---------------------------------------------------------------- physical values
/// <summary>A physical meter's sums per bucket of one side, and their total.</summary>