Menu

Alarm management

The Warning the Operator Actually Notices

Notes · FirmwareNovember 19, 202627 min readSenova Systems Research

A plant with 400 configured alarms, 60 active at any moment, and operators who acknowledge the horn without reading the screen. EEMUA 191 says an operator can handle about one alarm every ten minutes, and at Milford Haven, two operators had to recognise, acknowledge, and act on 275 alarms in 11 minutes. The warning is a design problem, not a discipline problem.

Picture the control room at 3 a.m. The plant has 400 configured alarms, 60 of them active at any moment, and the horn is going. The operator does not read the screen.

The operator hits the acknowledge button to silence the horn, much like we do with our car alarms, and goes back to watching the process. (Control Engineering) The alarm list is a wall of red, and the red has been there for hours. Somewhere in that wall is the one alarm that matters: a vessel filling, and its outlet closed.

The operator will not see it, because the operator has stopped looking. The alarm system has trained the operator to ignore it.

This is the usual failure, and it is not a bug in a single line of code. "Most alarm-handling failures are not bugs in a single line. The usual failure is a plant with 400 configured alarms, 60 of them active at any moment, and operators who acknowledge the horn without reading the screen."

(TryPLC) The alarm system grew by accident: every vendor package arrived with its own alarms, every commissioning engineer added a few "just in case", and nobody removed any. The result is familiar, hundreds of alarms a day, the same handful repeating, and an operator who acknowledges without reading. When a real problem comes, it arrives inside a flood of alarms that started at the same moment.

(Zumatic)

The discipline that fixes it is called alarm management, and it has a founding document. EEMUA 191, Alarm Systems: A Guide to Design, Management and Procurement, was first published in the UK in 1999 and is now in its 4th edition, November 2024. Since it was first published, it has become the globally accepted and leading guide to good practice for all aspects of alarm systems, developed by users of alarm systems with input from the UK Health and Safety Executive.

(EEMUA) The two international standards for the management of alarm systems for the process industries, ISA 18.2 from the International Society of Automation and IEC 62682:2023, are aligned with it. ANSI/ISA-18.2, Management of Alarm Systems for the Process Industries, was first published in 2009, and IEC 62682 is its international counterpart. (TryPLC)

The incident usually cited behind EEMUA 191 is the 1994 explosion at the Milford Haven refinery, where the operators in the last minutes before it were facing far more alarms than anyone could read. (TryPLC) On 24 July 1994, the explosion and fires at the Texaco Milford Haven refinery injured twenty-six people and caused damage of around £48 million and significant production loss. The HSE's investigation found three key factors: there were too many alarms and they were poorly prioritised; the control room displays did not help the operators to understand what was happening; and there had been inadequate training for dealing with a stressful and sustained plant upset.

In the last 11 minutes before the explosion, the two operators had to recognise, acknowledge, and act on 275 alarms. (HSE, Chemical Sheet No 6) One alarm every 2.4 seconds, for eleven minutes, and per operator, split evenly, one every 4.8 seconds, while the plant tried to say the one thing that mattered: a vessel was full and its outlet was closed.

What an alarm is, and what it is not

An alarm demands an operator response within a defined time. It stays on the alarm list until someone acknowledges it, even if the condition has already gone away. ISA-18.2 and EEMUA 191, the two documents most alarm philosophies are written from, are blunt about this: **if nobody needs to act, it is not an alarm.

It is an event, and it belongs in a log.** (TryPLC) A trip, or fault, is different again: it stops equipment. The program drives outputs to a safe state on its own, without waiting for anyone, and refuses to restart until a person has confirmed it is safe.

The distinction is the whole discipline. An alarm is the annunciation of a condition that requires an operator response, not a condition worth knowing about, a condition requiring a response. (evasmartai) The HSE's guidance is blunt about the entry criterion: process status indicators should not be designated as alarms, and the first quick win it lists for an overloaded system is to eliminate or review alarms with no defined operator response. (HSE, via ashita)

Most alarm systems grow by accident, and the addiction is specific. "Once alarms effectively became free we used them for everything. Our biggest addiction is to information alarms, those that tell the operator something is going on that someone else thinks he should know about but typically isn't required to do anything or can't do anything about."

(Control Engineering) In the days of light-box annunciators, the cost per alarm and the real estate involved made engineers think carefully about things to be alarmed. Once alarms effectively became free, they were used for everything, and the result is still with us: "because-I-did-something" alarms, and alarm floods when a trip occurs. The fix, as Control Engineering puts it, is to sell the concept that if there is no operator action required, it is not an alarm, and to keep referring back to the criteria that were set, to make sure the rankings stay consistent.

An alarm has two independent pieces of state, and mixing them up is the root of most broken alarm code. One is whether the condition is still present, which detection tells you scan by scan. The other is whether the operator has seen it, which is memory, and only the operator's acknowledge clears it.

Together they give four states, and the alarm stays on the list in three of them: unacknowledged and present, shown, flashing, horn on; acknowledged and present, shown, steady, horn off; returned to normal, unacknowledged, shown, flashing, horn on; normal, removed, off, off. (TryPLC) The third row is the one that surprises people: the condition is gone, and the alarm is still on the list, still flashing, still sounding, because nobody has acknowledged it.

That is the design. An alarm is not a record of what happened. It is a claim on the operator's attention, and the claim is only released when the operator takes it.

The numbers: one alarm every ten minutes

The performance figures that most alarm philosophies quote from ISA-18.2 and EEMUA 191 are worth knowing, because a customer's specification may hold you to them. They are guidance, not hard limits, but they are useful in a design review. (TryPLC) All benchmarks are based on at least 30 days of data per operating position, state that, because a week of data does not establish a rate.

The average alarm rate. In steady operation, the very-likely-acceptable rate is about 1 alarm per 10 minutes, about 6 per hour, about 150 per day. The maximum manageable is about 2 per 10 minutes, about 12 per hour, about 300 per day. Above that, operators start missing alarms. (TryPLC; Zumatic; Talsoft; Sarom) One alarm every ten minutes over 24 hours is exactly 144 a day, the commonly quoted figure is "about 150", and Control Engineering puts it as "1 every 10 minutes, or not more than 144 a day." The difference between 144 and 150 is rounding; the point is the same: the operator has ten minutes per alarm to recognise it, acknowledge it, and act on it. At 2 per 10 minutes, five minutes. At 12.5 per 10 minutes, the demo plant's rate, 48 seconds.

The flood. An alarm flood is more than 10 alarms in 10 minutes per operator. EEMUA 191 treats the flood as continuing until a 10-minute period has fewer than five new alarms, and the time spent in flood should be under about 1%. (TryPLC; Zumatic; Sarom) After a major upset, the aim is no more than 10 alarms in the first 10 minutes. (Zumatic; itotconvergentie) Milford Haven ran at 25 times the flood line overall, 25 alarms a minute against a flood threshold of 1 a minute, and 12.5 times even split evenly between its two operators. The flood is defined relative to the operator, not the plant: ten or more alarms in ten minutes, per operator, is the rate beyond which a human stops managing what arrives.

The priority split. Roughly 80% low, 15% medium, 5% high. "When a third of the alarms are high priority, none of them are." (TryPLC) Priority is a claim on the operator's preemption, and preemption only works if it is rare. If a third of the alarms are high priority, the operator cannot preempt, there is nothing to preempt to. The priority is derived from the consequence and the response time, not guessed: the master alarm database records, for every alarm, the cause, the consequence, the operator action, the response time, and the priority, and the priority is read off a matrix of consequence severity crossed with required response time. (evasmartai)

The top ten. The ten most frequent alarms should cause only a small part of the load, typically 1–5%. If they produce more than about 20% of the total, a few nuisance alarms are doing most of the damage. (PharmaChemE; Zumatic; Talsoft) A single tag above 5% is the investigation trigger. The top-10 share is the fastest diagnostic in the whole discipline: it tells you, in one chart, how much rationalisation the system needs.

Chattering and stale. Chattering alarms, alarms that go on and off repeatedly in a short time, target zero. Stale or standing alarms, active more than 24 hours, target fewer than 5. (Zumatic; Talsoft; Sarom) High steady-state rates are usually caused by chattering alarms: the measured variable sits close to the alarm setting, either because the setting is inappropriate, or often because the alarm belongs to a piece of equipment that is currently shut down. De-bounce timers, filtering, and dead-band changes can often cure this type of alarm. (HazardEx) High upset rates happen for the opposite reason: most alarm settings are appropriate for normal running and become inappropriate when there is a plant trip, low flow, for example, is what you would expect on a pump that is stopped, and logic is required to remove these expected alarms.

The five levels. EEMUA 191 describes five levels of alarm system performance: Level 1 Overloaded, Level 2 Reactive, Level 3 Stable, Level 4 Robust, Level 5 Predictive. The recommended goal, based on industry publications, is to set alarm system performance at Robust rather than Predictive, the investment in additional effort, software, and cost may not be worth the return. (powermag) The steady-state chart plots the percentage of time above 1 alarm per 10 minutes against the average alarm rate, and the upset chart plots the percentage of time above 10 alarms per 10 minutes against the maximum alarm rate, and the two situations are measured separately, because the mechanisms for improving performance in them differ, and the abnormal situation is the harder to address. (HazardEx)

The scale of the problem, in the field: today it is very common to receive more than 2,000 alarms a day per operator, and most of those are nuisance alarms that provide no value. (ABB) Tens of thousands of alarms, most of which signal changes in the plant's efficiency, are often monitored by just three or four operators, and even when things are running smoothly, 30 to 60 alarms per hour is common.

(New Scientist) Thirty to sixty an hour is 5 to 10 times the very-likely-acceptable rate of 6 an hour. And the good news, from ABB: alarm system quality improvement projects can rapidly result in impressive alarm rate reductions, by 50% or more, and actually lead operators to perform their jobs more effectively. Many organisations working towards the EEMUA 191 recommendations find that, in practice, it may mean reducing the number of alarms by an order of magnitude.

(Energy Institute)

The screen: grey, quiet, and one colour at a time

The alarm list is only half the system. The other half is the screen the operator reads it on, and the screen is where the alarm that matters gets buried. High-performance HMI is a design approach where operator displays are built to make abnormal conditions immediately obvious.

Backgrounds are low-contrast grey, process elements are drawn in muted outline, and saturated colour is reserved almost entirely for alarms and abnormal states. It is codified in ISA-101, and it stands in contrast to the photorealistic, heavily coloured graphics that dominated early SCADA. (Curry Controls)

The principle is one sentence: colour carries meaning, and if everything is coloured, colour means nothing. In high-performance design, colour carries meaning: a neutral grey process mimic means all parameters are normal; a yellow element means a parameter is deviating or an alarm is active; red means a critical alarm or shutdown condition. Operators immediately see the abnormal condition without scanning the entire screen, the fundamental goal of HMI design for abnormal situation management.

(Precisol) Normal operation appears in grayscale, pipes are grey, vessels are grey, values are black. Colour appears only when attention is needed: yellow for approaching limits, orange for warnings, red for alarms. (Flux Controls) Running equipment is a subtle outline change, not a green fill.

A normal stopped pump is the same shape, lighter fill or hatched, distinguishable at a glance without colour. A critical alarm is saturated red, usually with a shape indicator, reserved exclusively for that. (Curry Controls)

The history is the reason. Back in 1995, the Abnormal Situation Management Consortium identified that the industry had evolved its graphics based on early computer technology that was very limited in capability, and had adopted many bad practices based on those limitations. The graphics were based on black backgrounds, which conflicted with the room lighting and promoted glare, and very bright colours were overused, so operators turned the lights off and sat in the dark day and night, impacting their ability to be alert and to adapt to 24/7 shifts.

No colour coding led to many errors and loss of situation awareness: simple coding of important information such as alarms was lost in the plethora of bright colours, and colours like red and yellow were not reserved only for alarm indication. One customer's graphic used the colour red to code 13 different items, so the graphic was covered in red during normal operations and did not allow the differentiation of a major abnormal event.

Flashing was used as an eye-catcher, but it was not always reserved for abnormal events: one example was rotating fan blades, which flashed regardless of the operation, defeating the intent of flashing representing running. (ASM Consortium, via Control Global)

The rules that came out of it are simple, and they are the same shape as the alarm rules. Depiction over decoration: flat, two-dimensional shapes, no bevels, gradients, or photorealistic vessels, which add nothing an operator uses and cost visual bandwidth. Analog context: a number alone tells you the value; a number with a bar showing the normal operating range and the alarm limits tells you whether to care.

Consistent layout: the same information in the same place on every display, so an operator does not have to search for where a pump status lives on this particular screen. Deliberate hierarchy: Level 1 is the plant overview (all major areas visible at once), Level 2 is the area or unit display, Level 3 is the detailed equipment screen, Level 4 is diagnostic and maintenance. Operators navigate down to detail and back to overview without losing process context.

(Curry Controls; Precisol; Zumatic) ANSI/ISA-101.01-2015 covers the philosophy, design, implementation, operation, and maintenance of human–machine interfaces across their lifecycle, and IEC 63303:2024, derived from ISA-101, sets requirements for consistency of design and a four-level display hierarchy. (Zumatic)

The test is the five-second test. Show an operator display to someone who does not run the plant, put one item into an abnormal state, and ask them to point at it. If it takes longer than about five seconds, the display is doing too much decoration and not enough signalling.

(Curry Controls) And the operator the display is designed for is the 3 a.m. operator, tired, distracted, possibly responding to an upset. (Flux Controls) Article #18 built this: the device will be used by tired, distracted, or untrained people, and the first field report will be a use error, not a hardware failure. The user is solely responsible in just 1 out of 100 instances (Wiklund), so the failure is a use error, not a user error, and the design is the cause in the vast majority of those cases.

The usability standard, IEC 62366-1:2015+AMD1:2020, says the same thing in its own language: specify the users and the use environment, identify the use-related hazards with severity ahead of probability, design the user interface to mitigate them with inherent safety first, and validate with representative users, at least 15 participants per distinct user group for the summative test, 5–8 per group for the formative rounds. Five users find about 85% of problems (Nielsen–Landauer), and 15 users find a minimum of 90%, an average of 97% (Faulkner 2003).

The HMI is a user interface, and the five-second test is its summative test: if a stranger cannot find the abnormal item in five seconds, the operator at 3 a.m. will not find it either.

The alarm system and the HMI are one system, read from two ends. The alarm list decides what claims the operator's attention. The HMI decides whether the operator can see it. A plant with 400 configured alarms and a screen covered in red during normal operations has the same failure twice: the list is too long, and the screen has no signal left in it. The fix is the same in both: reserve the signal for the abnormal.

The fix

The fix is not to remove alarms at random. It is to measure the alarm system the way any other system is measured, decide what each alarm is for, and keep measuring. (Zumatic) The discipline has a name, rationalisation, and a sequence, and the sequence is the same for a new system and for cleaning up a ten-year-old SCADA. (evasmartai)

Step 1, apply the definition. Both standards start from one sentence: an alarm is the annunciation of a condition that requires an operator response. For every alarm that survives, record five fields, the master alarm database, the alarm list's single source of truth: the cause (the physical condition that triggers the alarm), the consequence (what happens if nobody responds), the operator action (a concrete, one-line intervention), the response time (order of seconds or minutes), and the priority (consequence severity × response urgency, a matrix output, not a personal preference). Any alarm whose "operator action" line you cannot fill in leaves the list. If deleting feels too aggressive, demote it to event level: the record survives, the screen gets cleaner. (evasmartai)

Step 2, derive priority from a matrix, then measure the distribution. Priority is not whatever the engineer who wrote the alarm felt that day. Consequence severity, personnel safety, environment, equipment damage, production loss, is crossed with required response time in a matrix, and priority is read off from there. Then look at the distribution: high/critical ~5%, medium ~15%, low ~80%. A bulge in the medium band means the matrix was skipped. A small low share keeps operators permanently running. Producing the distribution is half a day of querying, and it shows in one chart how much rationalisation the system needs. (evasmartai)

Step 3, suppress noise at the source. Some alarms that survive rationalisation still arrive far too often. The fix is not switching them off, but separating measurement noise from the alarm logic. Deadband: without a hysteresis band around the threshold, a signal sitting near it will come and go hundreds of times. Commonly published starting values are a percentage of measurement span, around 5% for flow, 5% for level, 2% for pressure, 1% for temperature. Treat those as starting points; the real value comes from the signal's own noise amplitude, put the band above the peak-to-peak noise. On/off delay: short excursions do not need to reach the operator. An on-delay raises the alarm only when the condition persists; an off-delay stops it from vanishing the instant the condition clears. (evasmartai)

Step 4, suppress by design, by plant state. Qualifying an alarm by plant state is suppression by design: the alarm is deliberately not shown when the plant is in a state where it means nothing. (TryPLC) The worked example: switch PumpRunning on and leave FlowSwitch off, FlowDelay.ET climbs, and LowFlow comes on after 3 seconds. Switch FlowSwitch on at 2 seconds instead, and the timer resets: a normal start, no alarm. The LowFlow alarm is qualified by the pump running, and the same shape appears on a machine: the gripper did not raise is not an alarm condition until the machine is turned on and air pressure is present. Machine power off, air pressure low, and the emergency stop pressed are valid alarms; "gripper failed to open" and "gripper failed to raise" are preconditions that should be included in the logic, not alarms on their own. (Control Design) EEMUA 191 describes five levels of suppression: redundant alarm suppression (several I/O points for a single status), eclipsing logic (several alarm points from one process measurement), out-of-service conditions (equipment, system, or plant level), operating mode alarm suppression, and major event alarm suppression (to reduce alarm floods when alarm traffic may be drastically increased). (powermag) Alarm suppression should be evaluated as an addendum to prudent alarm system design, rather than a means of keeping alarms that may not meet the original alarm design principles.

Step 5, run the lifecycle, and keep measuring. ANSI/ISA-18.2 and IEC 62682 define an alarm management lifecycle: philosophy, identification, rationalisation, detailed design, implementation, operation, maintenance, monitoring and assessment, management of change, and audit. (Zumatic) The monitoring and assessment step is the one that keeps the system from rotting: the alarm system degrades over time after it has been improved, because the plant is dynamic, new alarms are added, settings drift, equipment is shut down and its alarms are left live. The fix is to keep measuring the same five numbers, rate, flood, top contributors, nuisance alarms (chattering and stale), and priority distribution, all available from the alarm log. (Zumatic) And the human-factors companion to EEMUA 191, from the Energy Institute, adds the question the numbers alone do not answer: do the high-priority alarms maximise the probability of successful control-room-operator response? The factors are not only the number of alarms, but whether the alarms help, rather than hinder, the operator's situation awareness.

The other two articles this one builds on are the ones that put the alarm in the firmware. Article #13: the firmware has outgrown the super-loop, three concurrent jobs, a radio stack, and a control loop fighting for the same CPU, and the alarms interrupt the tasks, so the task that raises the alarm and the task that services it have to be scheduled, not hoped for. Article #37: the watchdog is the alarm that fires when the firmware stops, the external supervisor IC, $0.30–$1.50, never kicked from a timer interrupt, and the STM32's independent watchdog, clocked by a 32 kHz LSI with 17–47 kHz tolerance, so a nominal 1,000 ms timeout is actually 681–1,882 ms. The alarm that fires when the firmware stops has to be designed the same way as the alarm that fires when the vessel fills: a defined condition, a defined response, a defined time, and a screen the operator can read at 3 a.m.

DEMO

The demo plant is the pipeline's own failure, measured: 400 configured alarms, 60 of them active at any moment, and operators who acknowledge the horn without reading the screen. The log is the article's own worked example, labelled, 30 days of alarm log, the minimum period the benchmarks are based on, with the arithmetic run in Python before writing.

Step 1, the rate. Thirty days of log: 54,000 annunciated alarms, 1,800 a day, 75 an hour, 12.5 per 10 minutes. Against the benchmarks: 12 times the very-likely-acceptable rate of 1 per 10 minutes (150 a day), and 6 times the maximum manageable of 2 per 10 minutes (300 a day). The operator's time per alarm is the number that makes it concrete: at 1,800 a day, the operator has 48 seconds per alarm, to recognise it, acknowledge it, and act on it. At 300 a day, 4.8 minutes. At 150 a day, 9.6 minutes. The plant's operator has 48 seconds, and the plant has 60 alarms active at any moment, 15% of the configured alarms, active at once. The operator is not ignoring the alarms out of laziness. The operator has 48 seconds per alarm, and the system has given the operator 1,800 a day.

Step 2, the flood. The log shows the operator in flood, more than 10 alarms in 10 minutes, for 4.2% of the time, against the target of under 1%. In a 24-hour day, 1% is 14.4 minutes; 4.2% is 60.5 minutes, an hour a day in flood. And the flood ends only when a 10-minute period has fewer than five new alarms, so a flood that starts at a trip can run for a long time before the rate falls below five. Milford Haven's last 11 minutes were a flood at 25 times the flood line overall, 25 alarms a minute against a threshold of 1 a minute, and 12.5 times even split evenly between the two operators, one alarm every 4.8 seconds per operator. The demo plant's operator, at 12.5 per 10 minutes, is in a permanent low-grade flood, and the flood is the state in which the operator stops managing what arrives.

Step 3, the top ten. The ten most frequent alarms cause 45% of the total, 24,300 of the 54,000 over 30 days, 810 a day, 33.8 an hour. The target is 1–5%; more than 20% means a few nuisance alarms are doing most of the damage. The single worst tag, a chattering level alarm on a noisy signal, is 12% of the total: 6,480 over 30 days, 216 a day, 9 an hour, 1.5 per 10 minutes. One tag alone exceeds the very-likely-acceptable rate for the whole operator. The top-10 share is the fastest diagnostic in the discipline, and here it says the same thing three ways: the system is not 400 alarms. It is 10 tags, doing 45% of the damage.

Step 4, the priority and the stale alarms. The priority split on the 400 configured alarms is 38% high, 27% medium, 35% low, 152 high-priority alarms, against a target of 5%, which is 20. When a third of the alarms are high priority, none of them are, and here more than a third are. The stale alarms are 47 active more than 24 hours, against a target of fewer than 5. The chattering tags are 14, against a target of zero. The plant's alarm list is not a list of warnings. It is a list of 47 alarms that have been active for more than a day, 14 tags that go on and off repeatedly, and 152 alarms that claim to be high priority. The operator who acknowledges the horn without reading the screen is reading the system correctly: most of it does not need to be read.

Step 5, the two passes. Pass 1: remove the top 3 chattering tags (30% of the total, 540 a day) + demote the no-action alarms to events (25%, 450 a day) + suppress the expected alarms during shutdown (15%, 270 a day), and the rate falls from 1,800 to 540 a day, 22.5 an hour, 3.75 per 10 minutes, a 70% reduction. Pass 2: the remaining chattering tags + the priority re-derivation, removing 240 a day, and the rate falls to 300 a day, 12.5 an hour, 2.08 per 10 minutes, just over the maximum manageable line. The overall reduction is 1,800 to 300, 83.3%, and the operator's time per alarm goes from 48 seconds to 4.8 minutes. Two passes get the plant to the maximum manageable line, and the third pass, the remaining top-10 tags and the stale alarms, is the one that gets it to the very-likely-acceptable 150 a day. Rationalisation is a programme, not a single pass, and the Energy Institute's point is the honest one: working towards the EEMUA 191 recommendations may mean reducing the number of alarms by an order of magnitude. The demo plant reduced them by 83% in two passes and still has one more to go, and that is the normal shape of the work.

Step 6, the screen. The demo plant's HMI is the old one: black background, bright colours, red used to code 13 different items, flashing fan blades. The five-second test: show the display to someone who does not run the plant, put one item into an abnormal state, and ask them to point at it. On the old screen, the stranger cannot find it, the screen is covered in red during normal operations, and the abnormal item does not stand out. On the high-performance screen, low-contrast grey background, muted outlines, saturated colour reserved for the abnormal, a bar with the normal range and the alarm limits beside the number, the same information in the same place on every display, the stranger finds it in under five seconds. The test is not a design exercise. It is the operator at 3 a.m., tired, distracted, possibly responding to an upset, finding the one alarm that matters in a wall of red.

The decision test

Five questions. If you can answer all five with a number from the log, the alarm system exists. If you cannot, the alarm system is an inbox with a horn.

  1. What is the average alarm rate, and how much time does the operator have per alarm? Very likely acceptable: about 1 per 10 minutes, about 6 an hour, about 150 a day. Maximum manageable: about 2 per 10 minutes, about 12 an hour, about 300 a day. At 150 a day, the operator has 9.6 minutes per alarm. At 1,800 a day, 48 seconds. If the answer is one blended number for the whole plant, the per-operator rate is hiding, and the benchmarks are per operator, based on at least 30 days of data per operating position.
  2. How much time is spent in flood, and what ends a flood? A flood is more than 10 alarms in 10 minutes, and it ends when a 10-minute period has fewer than five new alarms. The time in flood should be under about 1%, 14.4 minutes a day. After a major upset, no more than 10 alarms in the first 10 minutes. If the flood rate is not measured, the flood is the normal state.
  3. What do the top ten alarms cause, and what is the single worst tag? The ten most frequent alarms should cause 1–5% of the total. More than 20% means a few nuisance alarms are doing most of the damage. If one tag alone exceeds the 1-per-10-minutes rate for the whole operator, that tag is the alarm system.
  4. How many alarms are chattering, how many are stale, and what is the priority split? Chattering: target zero. Stale (active more than 24 hours): fewer than 5. Priority: roughly 80% low, 15% medium, 5% high. If a third of the alarms are high priority, none of them are, and the priority was guessed, not derived from a matrix of consequence and response time.
  5. Can a stranger find the abnormal item on the screen in five seconds? The five-second test: show the display to someone who does not run the plant, put one item into an abnormal state, and ask them to point at it. If it takes longer than about five seconds, the display is doing too much decoration and not enough signalling. And the colour test: if red codes 13 different items, red means nothing.

Objections

"Our operators are trained, they know which alarms matter." The operators at Milford Haven were trained, and they had to recognise, acknowledge, and act on 275 alarms in 11 minutes, one every 4.8 seconds per operator. The HSE's finding was not that the operators were untrained; it was that there were too many alarms, poorly prioritised, and the control room displays did not help the operators to understand what was happening, and there had been inadequate training for dealing with a stressful and sustained plant upset. High operator reliability requires a very obvious display of the specific alarm, few false alarms, a low operator workload, a simple well-defined operator response, well-trained operators, and testing of the effectiveness of operators' responses. Training is the fifth item on that list, not the first. The operator's capacity is an engineering constraint with a number, about one alarm every ten minutes, not a personal productivity failure.

"We can't remove alarms, what if we need them?" The alarm does not disappear when it is removed from the alarm list. If deleting feels too aggressive, demote it to event level: the record survives, the screen gets cleaner. The event log is where the information lives; the alarm list is where the operator's attention lives. The question is not whether the information exists. It is whether it claims the operator's attention, and an alarm that nobody needs to act on is an event, not an alarm.

"The vendor package came with these alarms, we can't change them." Every vendor package arrives with its own alarms, and nobody removes any, that is how most alarm systems grow by accident. The vendor's alarms are a starting point, not a philosophy. The alarm philosophy is the site's document, and the rationalisation is the site's work: every candidate alarm must justify its existence before it is allowed to exist, with a documented operator action, the consequence of no response, and the time available to respond. The vendor does not own the operator's attention. The site does.

"A grey screen looks unfinished, the customer wants it to look impressive." The customer wants the operator to find the abnormal item in five seconds at 3 a.m. A photorealistic vessel with bevels and gradients and a green running pump does not help the operator find it, it costs visual bandwidth, and the operator who cannot find it will not find it because the screen was impressive. The high-performance HMI inverts the design: the normal state is deliberately quiet, and anything that draws the eye is doing so because it needs attention. The screen that looks unfinished in a demo is the screen that works in a control room.

"Rationalisation is too expensive, a thorough review takes more than one shift per alarm." A thorough review is expensive, and the alternative is the one the plant already has: an operator with 48 seconds per alarm, an hour a day in flood, and a screen the operator has stopped reading. ABB's finding is that alarm system quality improvement projects can rapidly result in impressive alarm rate reductions, by 50% or more, and actually lead operators to perform their jobs more effectively. The demo plant reduced its rate by 70% in one pass, by removing three chattering tags, demoting the no-action alarms, and suppressing the expected shutdown alarms. The review pays for itself in the first pass, and the plant that does not do it pays in the flood.

Sources

  1. EEMUA, "EEMUA Publication 191"
    What we used

    "Since it was first published in 1999, EEMUA 191 has become the globally accepted and leading guide to good practice for all aspects of alarm systems." "The publication, developed by users of alarm systems with input from the UK Health and Safety Executive, gives comprehensive guidance on designing, managing and procuring an effective alarm system." "Both of the international standards for the management of alarm systems for the process industries, ISA 18.2 from the International Society of Automation and IEC 62682: 2023, are aligned with EEMUA 191." "The new Fourth Edition has been comprehensively updated and restructured to improve ease of use." "EEMUA 191 is primarily concerned with alarm systems provided for people operating industrial processes."

  2. Engineering Maintenance Info, "Good practice for all aspects of industrial alarm systems, new edition of EEMUA 191 released" (3 December 2024)
    What we used

    "EEMUA has released Edition 4 of EEMUA 191, *Alarm systems, A guide to design, management and procurement*." "Since it was first published in 1999, EEMUA 191 has become the globally accepted and leading guide to good practice for all aspects of alarm systems."

  3. The NBS, "Publication 191 Alarm systems: a guide to design, management and procurement. 4th edition"
    What we used

    "4th edition, November 2024. Supersedes the 3rd edition, 2013 edition." ISBN 9780859312431. The 1st edition (1999) is listed as withdrawn

  4. TryPLC, "Alarms and fault handling"
    What we used

    "An alarm demands an operator response within a defined time. It stays on the alarm list until someone acknowledges it, even if the condition has already gone away. ISA-18.2 and EEMUA 191, the two documents most alarm philosophies are written from, are blunt about this: if nobody needs to act, it is not an alarm. It is an event, and it belongs in a log." "A trip (or fault) stops equipment. The program drives outputs to a safe state on its own, without waiting for anyone, and refuses to restart until a person has confirmed it is safe." "Most alarm-handling failures are not bugs in a single line. The usual failure is a plant with 400 configured alarms, 60 of them active at any moment, and operators who acknowledge the horn without reading the screen. EEMUA 191 suggests that an operator can handle about one alarm every ten minutes in steady operation. Every alarm you add should be one somebody would be grateful for at 3 a.m." "ANSI/ISA-18.2, *Management of Alarm Systems for the Process Industries*, was first published in 2009, and IEC 62682 is its international counterpart. EEMUA 191 is older: the Engineering Equipment and Materials Users' Association first published it in the UK in 1999. The incident usually cited behind it is the 1994 explosion at the Milford Haven refinery, where the operators in the last minutes before it were facing far more alarms than anyone could read." The four-state table: unacknowledged/present, shown, flashing, horn on; acknowledged/present, shown, steady, horn off; returned to normal, unacknowledged, shown, flashing, horn on; normal, removed, off, off. "ISA-18.2 has names for both halves. Qualifying an alarm by plant state, as `PumpRunning` does here, is *suppression by design*: the alarm is deliberately not shown when the plant is in a state where it means nothing." The LowFlow worked example: switch PumpRunning on, leave FlowSwitch off, FlowDelay.ET climbs, LowFlow comes on after 3 s; switch FlowSwitch on at 2 s instead, and the timer resets: a normal start, no alarm. "The performance figures that most alarm philosophies quote from ISA-18.2 and EEMUA 191 are worth knowing, because a customer's specification may hold you to them." The benchmarks table: average alarm rate, very likely acceptable about 1 per 10 minutes (about 6 per hour, 150 per day), maximum manageable about 2 per 10 minutes (about 12 per hour, 300 per day); alarm flood, more than 10 alarms in 10 minutes, time spent in flood under about 1%; priority split, roughly 80% low, 15% medium, 5% high, "when a third of the alarms are high priority, none of them are." "The numbers are guidance, not hard limits, but they are useful in a design review."

  5. Zumatic, "Alarm Management KPIs: ISA-18.2, IEC 62682, EEMUA 191"
    What we used

    "An alarm should mean one thing: the operator must act. Alarms that need no action train people to ignore the ones that do." "Commonly cited targets are about 150 alarms per day per operator as very likely acceptable and 300 as the maximum manageable, time in flood below 1 %, and a priority mix near 80 / 15 / 5 % (low / medium / high)." "Most alarm systems grow by accident. Every vendor package arrives with its own alarms, every commissioning engineer adds a few 'just in case', and nobody removes any. The result is familiar: hundreds of alarms a day, the same handful repeating, and an operator who acknowledges without reading. When a real problem comes, it arrives inside a flood of alarms that started at the same moment." "The fix is not to remove alarms at random. It is to measure the alarm system the way any other system is measured, decide what each alarm is for, and keep measuring." "The same discipline applies to a pump station with forty alarms as to a refinery with forty thousand, only the numbers are smaller." "ANSI/ISA-18.2 and its international counterpart IEC 62682 define an alarm management lifecycle: philosophy, identification, rationalisation, detailed design, implementation, operation, maintenance, monitoring and assessment, management of change, and audit." "EEMUA 191 (3rd edition, 2013) is the long-established industry guide; it defines an alarm as an indication of a problem *requiring operator action*." "The performance metrics in these documents are commonly reproduced as follows, based on at least 30 days of data per operating position." The metrics table: annunciated alarms per day ~150 / ~300; per hour ~6 / ~12; per 10 minutes ~1 / ~2; time in flood < 1%; share of the 10 most frequent alarms ~1–5%; chattering and fleeting alarms 0; stale alarms (active > 24 h) < 5 per day; priority distribution ~80 / 15 / 5%. "A flood is commonly defined as more than 10 alarms in 10 minutes, ending when the rate falls below 5 in 10 minutes. A chattering alarm repeatedly changes between active and normal in a short time; a stale alarm stays active for a long period, for example 24 hours." "EEMUA 191 gives a long-term target of less than one alarm per 10 minutes in steady operation, and fewer than 10 alarms in the first 10 minutes after a major upset."

  6. Sarom, "Alarm Flood: Causes, Benchmarks & How to Stop It"
    What we used

    "An alarm flood occurs when an operator receives more alarms than they can effectively assess and respond to, commonly defined as more than 10 new alarms in 10 minutes on a single console." "EEMUA 191 adds an end point: the flood is over once a 10-minute period has fewer than five new alarms." "During steady operation, about 1 alarm per 10 minutes per console is the target. This equates to roughly six manageable alarms per operator each hour, or around 150 over a 24-hour period. ISA-18.2's metrics treat 2 alarms per 10 minutes (12 an hour, 300 a day) as the most an operator can manage." "These figures apply to steady running and assume at least 30 days of data." EEMUA 191 edition 4, November 2024; "EEMUA 191 gives practical targets, while ISA-18.2 and IEC 62682 set out the lifecycle that a well-run alarm system should follow." Peak alarm rate 10 or fewer, investigate if more than 20; stale or standing alarms (active more than 24 hours) fewer than 5 per day

  7. PharmaChemE, "Alarm Management Basics for Chemical and Pharma Plants"
    What we used

    "Alarm management is the discipline of keeping the alarm list small, meaningful and manageable. The two widely used references are EEMUA 191 (a guide to design, management and procurement of alarm systems) and ISA-18.2, also published as IEC 62682 (management of alarm systems for the process industries). Both are paid documents, so this article gives original explanations of the ideas and the commonly quoted benchmarks, not the text of the standards." "EEMUA 191 treats fewer than one new alarm per 10 minutes in steady running as very likely acceptable, and fewer than ten in 10 minutes after an upset as manageable but difficult. A long-term average of about 12 alarms per hour (roughly two per 10 minutes) is often quoted as an upper limit for one operator." "An alarm flood is commonly defined as a 10-minute period with more than 10 new alarms. ISA-18.2 (also quoted for EEMUA 191) treats the flood as continuing until a 10-minute period with fewer than five new alarms." The worked example is labelled made-up data: a seven-day alarm log of 2,104 alarms from 40 tags, including two upsets and one tag that chatters, 301 alarms per day, 12.5 per hour, 2.1 per 10 minutes, 6 flood periods of 1,008 (0.6%), peak 32 in 10 minutes. "The benchmarks here are commonly quoted screening values, and the worked example uses made-up data. Your alarm philosophy must be set from your own hazards, response times and the current editions of EEMUA 191, ISA-18.2 and IEC 62682."

  8. Talsoft, "Alarm management KPIs: EEMUA 191 and ISA-18.2 explained"
    What we used

    "About 6 alarms per hour (150 per day) or fewer: very likely acceptable. About 12 alarms per hour (300 per day): the maximum that is still manageable. Above that, operators start missing alarms." "A flood starts when one operator receives more than 10 alarms in 10 minutes. The target is to spend less than 1% of the time in flood." "Top-10 share: the ten most frequent alarms should cause only a small part of the load, typically 1 to 5%." "Chattering alarms: alarms that go on and off repeatedly in a short time. The target is zero." "Stale (standing) alarms: alarms that stay active for more than 24 hours. The target is fewer than 5."

  9. itotconvergentie, "What is Alarm Management? Approach and KPIs"
    What we used

    "The key standards are ISA-18.2, its international counterpart IEC 62682 and the British guideline EEMUA 191." "Average alarm rate per operator ≤ 1 per 10 minutes (about 6 per hour); 2 per 10 minutes is considered the maximum manageable." "Alarms after a major upset ≤ 10 in the first 10 minutes (EEMUA 191)." "Standing (stale) alarms fewer than 5 present on any day (ISA-18.2)." "According to EEMUA 191 and ISA-18.2, an average of one alarm per ten minutes per operator, about six per hour, is very likely to be acceptable. Two alarms per ten minutes is regarded as the maximum that is still manageable. During a major upset, an operator should receive no more than ten alarms in the first ten minutes."

  10. HSE, "Better alarm handling" (Chemical Sheet No 6)
    What we used

    "The 1994 explosion and fires at the Texaco Milford Haven refinery injured twenty-six people and caused damage of around £48 million and significant production loss. Key factors that emerged from the Health and Safety Executive's investigation were: there were too many alarms and they were poorly prioritised; the control room displays did not help the operators to understand what was happening; there had been inadequate training for dealing with a stressful and sustained plant upset." "In the last 11 minutes before the explosion the two operators had to recognise, acknowledge and act on 275 alarms." "Alarm rate targets: the long-term average alarm rate during normal operation should be no more than one every ten minutes; and no more than ten displayed in the first ten minutes following a major plant upset." "High operator reliability requires: very obvious display of the specific alarm; few false alarms; a low operator workload; a simple well-defined operator response; well trained operators; testing of the effectiveness of operators' responses." "Are there nuisance alarms, eg are large numbers of alarms acknowledged in quick succession, or are audible alarms regularly turned off?" "How many standing alarms are there?" "(Note: process status indicators should not be designated as alarms.)"

  11. New Scientist, "Cause for alarm"
    What we used

    "Tens of thousands of alarms, most of which signal changes in the plant's efficiency, are often monitored by just three or four operators. Even when things are running smoothly, 30 to 60 alarms per hour is common." "The 1994 explosion at Texaco's Milford Haven refinery in south Wales is a case in point. During the preceding 11 minutes, two operators had the impossible task of dealing with 275 alarms. The explosion injured 26 people and caused damage worth around £48 million." "'It's not the principle of alarm systems that's wrong,' says Andow, 'It's the number of alarms, and their configuration that needs improving.'"

  12. Control Engineering, "Managing chaos"
    What we used

    "In many of our facilities we have many more things that fall into the warning or even event log categories that flash and sound the horn than we have real alarms, so after a while the operator doesn't even look at the screen to see if it's important. They just hit the acknowledge button to silence the horn much like we do with our car alarms." "Once alarms effectively became free we used them for everything. Our biggest addiction is to information alarms, those that tell the operator something is going on that someone else thinks he should know about but typically isn't required to do anything or can't do anything about." "Sell the concept that if there is no operator action required, it isn't an alarm. This will be an ongoing battle with those addicted to informational alarms." "Identify the bad actors and target them before anything else. Alarms that chatter because of too little deadband or from being set too close to a normal operating condition on a noisy process or even on a badly tuned loop." "Develop an alarm management strategy and make it someone's job to ensure that every control system gets reviewed to comply with it."

  13. Control Engineering, "Managing Alarms"
    What we used

    "The average, recommends EEMUA's Publication 191, should be 1 every 10 minutes, or not more than 144 a day." "The EEMUA figures are recommended benchmarks from Publication 191 (1999), 'Alarm Systems: A Guide to Design, Management and Procurement.'"

  14. ABB, "Alarm Management for power generation" (white paper)
    What we used

    "Today it is very common to receive more than 2000 alarms a day per operator. Most of those alarms are nuisance alarms that provide no value for the operators." "EEMUA 191 focuses on the properties of the operator's information processing capabilities. It emphasizes the usability of an alarm system from the operator's perspective. If the operator is overloaded with alarms, the whole alarm system becomes useless." "Long term average alarm rate in steady operation: less than 1 alarm in 10 minutes. Number of alarms during first 10 minutes after a major plant upset: under 10. Alarm priority distribution: high (5%) - medium (15%) - low (80%). Average fewer than 10 standing alarms." "Alarm system quality improvement projects, often with the help of external consultants, can rapidly result in impressive alarm rate reductions, eg, by 50% or more and actually lead operators to perform their jobs more effectively."

  15. powermag, "How to Avoid Alarm Overload with Centralized Alarm Management"
    What we used

    "EEMUA 191 describes five levels of alarm system performance. They range from Level 1 to Level 5 and are described as Overloaded, Reactive, Stable, Robust, and Predictive, respectively. The recommended goal based on industry publications is to set alarm system performance at Robust in lieu of Predictive. This is due to the fact that the investment in additional effort, additional software, and additional cost may not be worth the return." The five levels of suppression: redundant alarm suppression (several I/O points for a single status), eclipsing logic (several alarm points from one process measurement), out-of-service conditions (equipment, system, or plant level), operating mode alarm suppression, major event alarm suppression (to reduce alarm floods when alarm traffic may be drastically increased). "Alarm suppression should be evaluated as an addendum to prudent alarm system design rather than a means of keeping alarms that may not meet the original alarm design principles." "Alarm prioritization should be designed using a three- to five-level system. Three levels of priority (1-2-3 or high-medium-low) are the most popular, according to *The Alarm Management Handbook* by Bill Hollifield and Eddie Habibi."

  16. HazardEx, "EEMUA 191: Implications of Revision 3 on KPIs"
    What we used

    "High alarm rates during Steady State are usually caused by 'chattering' alarms; this is when alarms repeatedly annunciate because the measured variable is close to the alarm setting, either because the setting is inappropriate, or often, when it belongs to a piece of equipment that is currently shut down. De-bounce timers, filtering and dead-band changes can often cure this type of alarm." "High Upset rates happen because most alarms settings are appropriate for normal running, and become inappropriate when there is a plant trip. Low flow, for example, is what we would expect on a pump that is stopped. Generally, logic is required to remove these expected alarms." "The Steady State chart plots % time > 1 alarm per 10 minutes against the average alarm rate. 1 alarm per 10 minutes is considered manageable and this should be the target we are attempting to achieve. The upset chart plots % time > 10 alarms per 10 minutes against the maximum alarm rate. 10 alarms per minutes is considered manageable for a short time and again, this should be our target."

  17. evasmartai, "How to build an alarm list operators actually read"
    What we used

    "Both standards start from one sentence: an alarm is the annunciation of a condition that requires an operator response." The master alarm database five fields: cause, consequence, operator action, response time, priority, "Priority is derived from this, not guessed." "Any alarm whose 'operator action' line you cannot fill in leaves the list. If deleting feels too aggressive, demote it to event level: the record survives, the screen gets cleaner." "Priority is not whatever the engineer who wrote the alarm felt that day. Consequence severity (personnel safety, environment, equipment damage, production loss) is crossed with required response time in a matrix, and priority is read off from there." Deadband starting values: around 5% for flow, 5% for level, 2% for pressure, 1% for temperature, as a percentage of measurement span, "Treat those as starting points; the real value comes from the signal's own noise amplitude, put the band above the peak-to-peak noise." "On/off delay: short excursions do not need to reach the operator. An on-delay raises the alarm only when the condition persists; an off-delay stops it from vanishing the instant the condition clears."

  18. Energy Institute, "Guidance for optimising operator plant situational awareness by rationalising control room alarms"
    What we used

    "Many organisations are working towards the recommendations set in EEMUA 191, which in practice may mean reducing the number of alarms by an order of magnitude. However, organisations undergoing such 'alarm rationalisation' struggle to determine how to do this and still maintain (or even optimise) operator situation awareness." "This publication helps operating companies answer three questions: what factors should be considered when trying to improve control room operator situation awareness? Are there an acceptable number of alarms? Do the high-priority alarms maximise the probability of successful CRO response?"

  19. Curry Controls, "High Performance HMI Design"
    What we used

    "High performance HMI is a design approach where operator displays are built to make abnormal conditions immediately obvious. Backgrounds are low-contrast grey, process elements are drawn in muted outline, and saturated color is reserved almost entirely for alarms and abnormal states. It is codified in ISA-101 and stands in contrast to the photorealistic, heavily colored graphics that dominated early SCADA." "Color carries meaning. If everything is colored, color means nothing." "Grey backgrounds with muted process elements make an abnormal condition jump out." "Analog values benefit from context: a bar with limits beats a bare number." "Hierarchical displays let an operator move from plant overview to detail predictably." "The test is whether a new operator can identify the abnormal item in a few seconds." "Low-contrast grey background: a mid-grey field rather than black or white. It reduces eye strain over a twelve-hour shift and gives both light and dark elements room to stand out." "Reserved color: saturated color is used almost exclusively for abnormal conditions and alarm priority. Running equipment is a subtle outline change, not a green fill." "Depiction over decoration: flat, two-dimensional shapes. No bevels, gradients, or photorealistic vessels. They add nothing an operator uses and cost visual bandwidth." "Analog context: a number alone tells you the value. A number with a bar showing the normal operating range and the alarm limits tells you whether to care." "Consistent layout: the same information in the same place on every display." "The five-second test: show an operator display to someone who does not run the plant, put one item into an abnormal state, and ask them to point at it. If it takes longer than about five seconds, the display is doing too much decoration and not enough signaling." Meaning table: normal running, dark outline, light fill, no saturated color; normal stopped, same shape, lighter fill or hatched; critical alarm, saturated red, usually with a shape indicator; high priority alarm, saturated amber or orange

  20. Precisol, "HMI Design Best Practices for Industrial Automation and Industry 4.0"
    What we used

    "Colour for State, Not Decoration: in high-performance human machine interface design, colour carries meaning. A neutral grey process mimic means all parameters are normal. A yellow element means a parameter is deviating or an alarm is active. Red means a critical alarm or shutdown condition. Operators immediately see the abnormal condition without scanning the entire screen, the fundamental goal of HMI design best practices for abnormal situation management." "Screen Hierarchy and Navigation: a four-level hierarchy: Level 1 is the plant overview (all major areas visible at once); Level 2 is area or unit displays; Level 3 is detailed equipment screens; Level 4 is diagnostic and maintenance screens." Touch target size: minimum 15×15 mm for gloved-hand operation

  21. Flux Controls, "HMI Design Best Practices"
    What we used

    "Reserve color for meaning. In high-performance design, normal operation appears in grayscale, pipes are gray, vessels are gray, values are black. Color appears only when attention is needed: yellow for approaching limits, orange for warnings, red for alarms." "Design for the 3AM operator, tired, distracted, possibly responding to an upset." "Alarm floods are the enemy of effective operations. Every alarm should require operator action, if no action is needed, it's not an alarm, it's a notification. Suppress alarms during startup and known conditions. Use alarm shelving appropriately. Aim for fewer than one alarm per operator per 10 minutes during normal operation."

  22. ASM Consortium (via Control Global, "ASM HMI Paper"
    What we used

    ), "Back in 1995, the Consortium identified that the industry had evolved their graphics based on early computer technology that was very limited in capability and that the industry had adopted many bad practices based on these limitations." "One of the outcomes of this technology was that graphics were based on black backgrounds which conflicted with the room lighting promoting glare issues and that very bright colors were over used in the design of the graphics. Resulting in operators turning the lights off and sitting in the dark day and night impacting their ability to be alert and to adapt to 24/7 shifts." "No color coding led to many errors and loss of situation awareness. Simple coding of important information such as alarms was lost in the plethora of bright colors and not reserving colors like Red and Yellow only for alarm indication. Some graphics had used the color Red for multiple codes such as closed valves, stopped the pump, heating or hot and much more. One customers graphic I discovered the color Red was used to code 13 different items. Hence, the graphic was covered in red during normal operations and did not allow the differentiation of a major abnormal event." "Flashing was used as an eye catcher. Unfortunately, it was not always reserved for abnormal events but to indicate movement. One example was rotating fan blades. Unfortunately, the blades flashed regardless of the operation which defeated the intent of the flashing representing running."

  23. Zumatic, "SCADA Dashboard Design: 7 Rules Backed by Standards"
    What we used

    "ANSI/ISA-101.01-2015 covers the philosophy, design, implementation, operation and maintenance of human–machine interfaces across their lifecycle, and defines a display hierarchy in clause 6.3. IEC 63303:2024, derived from ISA-101, sets requirements for consistency of design (6.1.2) and a four-level display hierarchy (7.3.2–7.3.5)." "On colour, the ASM Consortium recommends that abnormal states and live data stand out while static context blends into a light grey background, which also reduces glare and helps operators with colour-vision deficiency."

  24. Control Design, "Don't be too alarmed with your automation"
    What we used

    "The alarm messages must be presented to an operator in a limited number, or it's likely they will be ignored or disabled." "Don't overload the operator with information. Alarms can be broken down into priorities and areas, which allows only high-priority alarms to be displayed." "What is an alarm, and what is a warning? An alarm is a detectable condition that keeps a machine from cycling, and it requires immediate operator intervention or the machine stops." "Using simple relay logic, if a raised gripper output solenoid is energized and the corresponding gripper-raised sensor is not activated after a preset period of time, a 'gripper did not raise' fault is latched on and displayed via an HMI as a fault indicator and message text." "Unfortunately, this simple logic often creates many alarms that don't exist, meaning it's not an alarm if the underlying control is not ready to run. For example, the machine power is off, the air pressure is low, and the emergency stop is pressed. All are valid alarms to display. However, the other alarms, such as 'gripper failed to open' and 'gripper failed to raise' are not alarm conditions until the machine is turned on and air pressure is present."

  25. Demo arithmetic verified in Python before writing: 30 days of log, 54,000 annunciated alarms = 1,800/day = 75/hour = 12.5 per 10 min; 12× the very-likely-acceptable 150/day, 6× the maximum manageable 300/day; time per alarm 1,800/day → 48 s, 300/day → 4.8 min, 150/day → 9.6 min; 1 per 10 min over 24 h = 144/day (commonly quoted "about 150"
    What we used

    ); Milford Haven 275 alarms in 11 min = 25/min overall, 1 per 2.4 s, per operator 12.5/min, 1 per 4.8 s; 25× the flood line overall, 12.5× per operator; top 10 tags 45% of 54,000 = 24,300 over 30 days = 810/day = 33.8/hour; worst tag 12% = 6,480 over 30 days = 216/day = 9/hour = 1.5 per 10 min; time in flood 4.2% of a day = 60.5 min/day, target <1% = 14.4 min/day; priority split on 400 configured: high 38% = 152 alarms (target 5% = 20), medium 27% = 108, low 35% = 140; Pass 1 (top 3 chattering tags 30% = 540/day + demote no-action alarms 25% = 450/day + suppress expected shutdown alarms 15% = 270/day) → 540/day = 22.5/hour = 3.75 per 10 min, 70% reduction; Pass 2 (remaining chattering + priority re-derivation, 240/day) → 300/day = 12.5/hour = 2.08 per 10 min, just over the maximum manageable line; overall reduction 1,800 → 300 = 83.3%, time per alarm 48 s → 4.8 min; 60 active of 400 configured = 15%; 30–60 alarms/hour common (New Scientist) = 5–10× the acceptable 6/hour; 2,000/day (ABB) = 83.3/hour = 13.9 per 10 min = 13.3× the acceptable rate

More in software meets hardware

Design for TestA board that passes the bench and fails in the field is usually untestable by design, no test points, no JTAG chain, no fixture. The IPC-9252A rules, the four test methods, the coverage union, and what a fixture is worth on a real board.20 minTest It to DestructionHALT is a discovery test that steps stress far beyond specification to find weaknesses and measure operating and destruct limits, it is not a pass/fail qualification test. A complete HALT needs 4–8 units and 3–5 days; a field failure costs 100 times what a design-stage fix costs.21 minDigital twinThe prototype costs real money and weeks, and the failure it hides costs a hundred times more to find in the field than at validation. A digital twin is not a 3D model, it is a model that behaves like the physical system, fed by data from it, whose answers change the physical system. Example Labs' twin of its sensor hub predicted the rail sag that killed four of ten hubs in the field, as a band that straddled the sensor's minimum voltage, which is exactly what 'no margin' looks like before the hardware is built.21 minWhy Did the Prototype FailThe prototype worked on the bench and died in the field, and the team is about to replace the radio, which is the symptom, not the cause. The post-mortem runs three tools in order: the fishbone to find suspects, the Five Whys to reach the root of the best ones, and the FMEA to decide what to fix first. Example Labs' sensor hub stopped after 21 days because the power rail was sized for a fresh cell, not an aged one, and finding it in the field cost 100 times what finding it at validation would have.25 minRTOS vs Bare MetalA super-loop runs the jobs one after another, so one long job delays every other, and the moment a blocking operation can stall the one task that must run every millisecond, the loop is no longer a schedule. It is a hope. Use an RTOS when you have 3+ independent concurrent activities with different timing requirements; stick with bare metal for simple, single-purpose devices. On the Example Labs hub, control loop at 1 kHz, BLE every 7.5 ms, the on-device model once a second, the OTA flash writes, the super-loop leaves the control loop 2.6x late, 40.6x late when the model runs, at 385 samples a second instead of 1,000. The RTOS costs ~9 kB of flash and 0.03% of the CPU, and the control loop is never late.16 min