1. The architecture on one page
Alarming is continuous and lives in a dedicated worker process
(worker/alarm_worker.py), spawned and supervised by the
platform (restart with backoff, log in data/alarm-worker.log).
The worker opens the database strictly read-only, keeps live values in
memory from the MQTT broker, and evaluates every project every 10 seconds
by default. It never writes the database: every state change is published
over MQTT and the main application, the single database writer for the
whole platform, persists it. This division is what lets the alarm engine
work as hard as it likes without ever endangering the historian.
The worker restores its lifecycle state across restarts from its own
retained MQTT snapshots, then reconciles against the database's shelves
and acknowledgements, so a restart neither loses an acknowledgement nor
re-annunciates a latched alarm. If the worker is offline or disabled, the
Alarms tab falls back to on-demand evaluation and says so; the rule
engines are the same code either way
(app/alarms.py, app/equip_logic.py), so
continuous and manual evaluation can never disagree about what an alarm
is.
2. The ISA-18.2 lifecycle
Every alarm, whatever produced it, carries the same lifecycle:
| State | Meaning | Leaves by |
|---|---|---|
unack | Active Unacknowledged: the condition is present, nobody has seen it. Blinking chip. | Acknowledge (to acked), or condition clears (to rtn_unack). |
acked | Active Acknowledged: still present, an operator has it. Steady chip with the operator's name. | Condition clears (to normal). |
rtn_unack | Returned to Normal, Unacknowledged: the condition cleared before anyone saw it. Latched on purpose: a 3 am transient is still visible at 8 am. | Acknowledge (to normal), or the condition returns (a fresh unack with a new raise time). |
shelved | Operator suppression with a hard expiry (30 minutes to 7 days). Evaluated silently the whole time. | Unshelve, or expiry: re-raises unacknowledged if still active, otherwise normal. |
Raise and clear are debounced: a condition must persist before it raises and be absent before it clears, both measured on the evaluation clock, so cadence changes never change semantics. Debounce profiles by alarm kind:
| Alarm kind (key prefix) | Raise after | Clear after | Why |
|---|---|---|---|
| Point, device and equipment alarms | 30 s | 60 s | Tunable: min_active_s, clear_s. |
comm: | 60 s | 30 s | One bad poll cycle must never page anyone. |
flood:, chatter: | immediate | 60 s | Derived from already debounced facts. |
anom: | 10 min | 15 min | An anomaly that cannot hold itself for ten minutes was noise. |
Acknowledge and shelve are operator actions, audited, persisted
(alarm_acks, alarm_shelved) and honoured across
worker restarts. Every transition is appended to
alarm_events, the history that feeds the KPIs, the
chattering detector, coincidence mining and Explain.
3. The detection layers
All layers run together in the worker, every cycle, and feed the same lifecycle. A layer that lacks evidence stays silent rather than guessing; that principle repeats everywhere below.
| Layer | Rules | Severity, category | Key shape |
|---|---|---|---|
| Basic rules | Per unit-family thresholds (temperature roles, humidity, CO2, percent bounds), flagged alarm points reading 1, stale data | warning or critical; various | {rule}:{point_id} |
| Sensor plausibility | Physically impossible readings by role (a 95 degC chilled water supply); faults suppress the comfort alarms they would otherwise cause | fault, sensor | sensor_fault:{point_id} |
| Advanced rules | User-defined thresholds and comparisons from the New rule popup; picked up within a minute of saving, or instantly via the reload hint | as configured | rule:{rule_id} |
| Device alarms | Alarms the BACnet devices themselves report, bridge-polled; priority under 100 maps to critical | critical or warning, device | dev:{device}:{type}:{instance} |
| Equipment logic | The five contradiction checks, section 5 | warning, equipment | eq:{check}:{equipment} |
| Anomalies and drift | Baseline deviation, flatline, delta-T decline, amps creep, section 6 | info (anomaly) or fault (flatline) | anom:{kind}:{id} |
| Meta | Communications loss, flood, chattering, section 4 | critical, warning, info | comm:site,
flood:{equipment}, chatter:{key} |
4. Suppression and alarm quality
Communications loss and the hold
When every bridge reporting for a project is failing or silent for
three minutes, the worker raises exactly one critical
comm:site alarm and holds everything else for that project:
point evidence over a dead link is neither fresh enough to raise on nor
fresh enough to clear on. Held alarms keep their lifecycle state through
the outage, nothing false-clears, nothing new raises, and every debounce
window restarts when the link returns, so the platform re-establishes the
truth instead of guessing it. Projects fed by plain MQTT with no bridge
are never comm-alarmed. This is Guideline 36 suppression in miniature; the
full source-load version arrives with the topology graph, whose first
edges the relationship mining (section 7) is already collecting.
Flood collapse and first-out
Five or more active alarms on one equipment
(flood_n) become one parent warning naming the count and the
first out, the earliest raiser, which is the best cheap cause signal a
flat point list can give. The table folds the grouped rows beneath the
parent; nothing is hidden, the screen just stops shouting.
Chattering
An alarm that raised ten or more times in 24 hours
(chatter_n) gets a calm info alarm naming it a bad actor,
computed from the event history on the reload cadence, with hysteresis
(clears below six) so the detector itself cannot chatter. Meta alarms
never breed meta alarms.
The scoreboard
The Alarm quality panel scores the system the ISA-18.2 way: annunciations per day against the roughly 150 one operator can actually work, peak hour, percentage of hours in flood (ten plus raises in an hour), median acknowledgement latency, standing alarms older than 24 hours, and the top ten bad actors of the window. These metrics are how an alarm system stops degrading: they make the noise visible before operators tune it out.
5. Equipment roles and the contradiction checks
The equipment logic needs to know which point plays which part. Roles
(heat_cmd, cool_cmd, run_cmd,
run_status, amps, supply_temp,
return_temp, setpoint) are guessed from names,
conservatively: one point per role, first strong match wins, ambiguity
stays unassigned, because a false contradiction burns trust faster than a
missed one. Every slot is admin-overridable under Admin, Alarms,
Equipment logic roles; a slot set to none means this equipment has no
such point, stop guessing. Manual decisions are never overridden and the
engine reloads instantly on change.
| Check | Fires when | Reads |
|---|---|---|
eq_heat_cool | Heating and cooling both commanded above 10% (binary commands understood) | heat_cmd, cool_cmd |
eq_no_status | Commanded on, run status present and off | run_cmd, run_status |
eq_zero_amps | Commanded or proven running, current below 0.5 A | run_cmd or run_status, amps |
eq_valve_starved | A valve saturated above 95% with the supply more than 2 K on the wrong side of setpoint; direction-aware on both the cooling and heating side | the valve, supply_temp, setpoint |
eq_low_dt | Under 2 K across a chilled or hot water circuit at 80% plus load; water-only by design, because air-side delta-T has economizer physics a flat check must not pretend to understand | supply_temp, return_temp, a valve as the load signal |
Every check demands fresh values, under 15 minutes old, for every point it reads; missing or stale evidence means no finding.
6. Anomalies, drift and forecasts
The statistical layer makes claims against the point's OWN history and keeps them humble: "unusual for a Tuesday at 14:00" is a different statement from "wrong".
Baseline deviation. Every numeric sensor gets a robust
profile per workday-or-weekend and hour-of-day bucket: the median and
MAD-derived spread of its per-hour means over
baseline_days (28) of its project's own history file,
rebuilt nightly. A fresh value beyond anomaly_sigma (6)
robust spreads raises an info alarm in the anomaly category. Guard rails:
twelve samples minimum per bucket, per-family spread floors so naturally
flat points cannot alarm on noise, stale values never score, and
setpoints, binary states and alarm-flag points are never baselined.
Flatline. Fresh timestamps with a value frozen for
flat_hours (6), on a point whose own baseline proves it
normally varies, is a frozen sensor, raised as a sensor fault. The three
failure modes are cleanly separated: frozen sensor here, comms loss owned
by the hold, stale data owned by the stale check.
Drift and the forecasts. Theil-Sen slopes over daily aggregates (the median of pairwise slopes: an outlier day cannot bend the line) watch for declining water delta-T across circuits the roles already know, and creeping running current on motors. Both need fourteen days and report in plain units per week. When a trend would cross its practical limit, 1.5 K of delta-T or 20% over the running median of amps, within 26 weeks, the finding says so in weeks: plain regression on the KPI, silent beyond that horizon because extrapolation stops deserving belief. Installs on the InfluxDB historian skip the statistical layer gracefully for now.
7. Cause and memory
The maintenance and observation log. A Log button on every alarm row and an Add observation button open a twenty second form: a cause from a fixed vocabulary, a note, the action taken. Each entry can bind to the exact alarm activation it closes (key plus raise timestamp), which is the join a supervised failure model will need in two to three years; until then it is the equipment's memory, and Explain already reads it.
Proposed relationships. Mining over the raise history finds equipment pairs whose alarms co-occur far beyond chance, with direction (who raises first) and typical lead time, under honest thresholds (five co-occurrences, 0.3 of the rarer equipment's raises). Admins confirm real plumbing or dismiss accidents; decided pairs stop being proposed; confirmed links seed the topology graph.
Explain. One button on any live alarm assembles a deterministic evidence bundle: the alarm, the equipment's role snapshot with fresh values marked fresh and stale marked stale, the surrounding event timeline, confirmed relationships, the maintenance history, active drift. The configured Anthropic model returns at most three ranked hypotheses, each tied to specific evidence with a concrete check, and the bundle is shown beneath the answer so nothing arrives unverifiable. Deterministic evidence, narrative on top, nothing invented.
8. Configuration reference
Everything lives in config.json under
alarm_worker; the worker restarts with the platform to pick
changes up.
| Key | Default | Meaning |
|---|---|---|
enabled | true | The continuous worker; off falls back to on-demand evaluation. |
eval_interval | 10 | Seconds between evaluation cycles. |
min_active_s | 30 | Raise debounce for point, device and equipment alarms. |
clear_s | 60 | Clear debounce for the same. |
flood_n | 5 | Active alarms on one equipment before flood collapse. |
chatter_n | 10 | Raises in 24 hours before an alarm is a bad actor. |
anomaly_enabled | true | The statistical layer. |
anomaly_sigma | 6 | Robust spreads before a baseline deviation raises. |
flat_hours | 6 | Hours frozen before a varying sensor is a flatline. |
baseline_days | 28 | Trailing window for the nightly baseline rebuild. |
9. API reference
| Endpoint | Role | Purpose |
|---|---|---|
GET /api/alarms/live?project_id= | viewer | The live snapshot plus worker health. |
POST /api/alarms/ack, /shelve, /unshelve | operator | Lifecycle actions; audited; survive restarts. |
GET /api/alarms/events?project_id= | viewer | The append-only transition history. |
GET /api/alarms/kpi?project_id=&days= | viewer | The ISA-18.2 scoreboard. |
POST /api/alarms/run-basic, /run-advanced | viewer | The on-demand fallback evaluations. |
GET/POST/DELETE /api/alarm-rules | admin | Advanced rule management; publishes a reload hint. |
GET/PUT/DELETE /api/equip-roles | admin | The role editor behind the contradiction checks. |
POST/GET /api/maintenance | operator, viewer | The label factory. |
GET /api/alarms/coincidence?project_id= | admin | Mined relationship proposals. |
GET/POST /api/equip-links | viewer, admin | Confirmed and dismissed relationships. |
POST /api/alarms/explain | operator | Ranked hypotheses over the evidence bundle; needs the API key under Admin, AI. |
10. MQTT topic reference
| Topic | Direction | Content |
|---|---|---|
$beez/alarm-events/{project_id} | worker to platform | One JSON message per lifecycle transition; the platform persists it. |
$beez/alarm-state/{project_id} | worker, retained | The snapshot of non-normal alarms with counts; the live view cache, and the worker's own restart memory. |
$beez/alarm-cmd/{project_id} | platform to worker | ack, shelve, unshelve, reload. |
$beez/worker-status/alarms | worker, retained | Heartbeat: evaluation age, machine counts, debounce settings. |
$beez/bridge-status/{bridge}/{project} | bridge, retained | Field link health; feeds the communications hold. |
11. Operational notes and honest limits
The worker follows the canonical five-segment topic convention (slug/site/equipment/point/value); projects fed through an explicit topic-prefix map keep on-demand evaluation for now. Cleared alarms are garbage-collected an hour after they last mattered. Freshly onboarded sites stay silent on baselines for a couple of weeks while evidence accrues, exactly as they should. And the fallback is always there: stop the worker and the Alarms tab still evaluates on demand with the same engines, which is also the honest way to compare the two if you ever doubt one.