Menu

Why Did the Prototype Fail

Why Did the Prototype Fail? The Post-Mortem That Finds the Root Cause

Notes · Testing, safety and certificationNovember 19, 202625 min readSenova Systems Research

The prototype worked on the bench and died in the field, and the team is about to replace the radio, which is the symptom, not the cause. The post-mortem runs three tools in order: the fishbone to find suspects, the Five Whys to reach the root of the best ones, and the FMEA to decide what to fix first. Example Labs' sensor hub stopped after 21 days because the power rail was sized for a fresh cell, not an aged one, and finding it in the field cost 100 times what finding it at validation would have.

The prototype worked on the bench. It worked in the lab, on the bench supply, at room temperature, for three days straight. Then it went into the field, and three weeks later it stopped. Four of the ten units, between day 19 and day 23. The team is standing around the bench, and somebody has already said the words: "The radio must be bad. Let's replace the radio."

The radio is not bad. The radio is the symptom. And if the team replaces the radio, ships the same fault again, and learns nothing, the next ten units will stop in three weeks too, and the team will have spent the money twice and learned nothing either time.

A post-mortem is the thing that stops that from happening. It is not a meeting where people explain what went wrong. It is a method, with three tools, in a specific order, and the method is the whole article.

The distinction everything else depends on

The distinction between a symptom and a root cause is fundamental to quality management. A symptom describes what happened. A root cause explains why it happened. That is the whole distinction, and most teams never make it, they fix the symptom, which is what stopped working, and leave the root cause, which is the deepest reason, untouched. Fixing the root cause stops the failure from coming back. Fixing the symptom does not.

The worked example that makes the distinction concrete comes from Semishops, and it is worth reading slowly, because the four layers are the shape of every post-mortem in this article:

Investigation layerObservation
SymptomFPGA stopped responding
Immediate causePower rail instability
Contributing causeCapacitor degradation
Root causeSupplier process variation affecting capacitor quality

The FPGA is the thing that stopped working. The power rail is the thing that stopped it. The capacitor is the thing that made the rail unstable.

And the supplier's process variation is the thing that made the capacitor degrade. The team that replaces the FPGA fixes the symptom. The team that changes the supplier, or the capacitor's specification, or the rail's margin, fixes the root cause.

The ability to identify root causes accurately is, in Semishops' words, "a strategic business capability rather than merely a technical exercise", because the team that can do it stops paying for the same failure twice.

There is a number behind that claim, and it is the reason the post-mortem exists at all. Industry reliability investigations, as Semishops tabulates them, find that electronic system failures break down roughly like this: design issues 35%, manufacturing defects 25%, environmental factors 15%, power integrity problems 10%, component defects 8%, counterfeit components 4%, unknown causes 3%. And nearly 55% of field-returned electronic assemblies contain no intrinsic semiconductor defect when subjected to comprehensive laboratory analysis, the failure came from thermal fatigue, PCB assembly defects, mechanical stress, moisture, electrostatic discharge, electrical overstress, contamination, or design incompatibilities.

"In most cases, the semiconductor itself is not the primary cause of failure." The team that replaces the chip is, more often than not, replacing the wrong thing.

The three tools, in order

A post-mortem runs three tools, in this order, and the order is the method:

  1. The fishbone, to find the suspects. It spreads the possible causes wide, across six categories, so the loudest person in the room does not decide the answer.
  2. The Five Whys, to reach the root of the best suspects. It drills down one chain at a time, each answer forming the basis of the next question, until it reaches something you can change in the design or the process.
  3. The FMEA, to decide what to fix first, and to prevent the next failure before it happens. It scores each failure mode and ranks them, so the team fixes the highest risk first instead of the most obvious symptom.

The sequence, as one guide puts it, is: fishbone to find suspects, 5 Whys to reach the root of the best ones, Pareto to decide which roots are worth fixing first. Skip the fishbone and you chase the first cause that comes to mind. Skip the Five Whys and you fix a contributing cause and call it the root. Skip the FMEA and you fix the easiest thing first instead of the riskiest. The three tools are not three options. They are three steps.

The Five Whys

The Five Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a problem. The goal is to determine the root cause by repeating the question "Why?", and each answer forms the basis of the next question. The "5" in the name derives from an anecdotal observation on the number of iterations needed to resolve a problem.

The technique was originally developed by Sakichi Toyoda and later used at Toyota during the evolution of its manufacturing methodologies, and it is now used within Kaizen, lean manufacturing, and Six Sigma.

Taiichi Ohno, the architect of the Toyota Production System in the 1950s, describes the method in his book Toyota Production System: Beyond Large-Scale Production as "the basis of Toyota's scientific approach . . . by repeating why five times, the nature of the problem as well as its solution becomes clear." He encouraged his team to dig into each problem that arose until they found the root cause. "Observe the production floor without preconceptions," he would advise. "Ask 'why' five times about every matter." And the purpose is not blame: "The purpose of the Five Whys isn't to place blame, but rather to uncover the root cause of why something unexpected occurred."

Ohno's classic example, from his book, is the machine that stopped:

  1. Why did the machine stop? There was an overload and the fuse blew.
  2. Why was there an overload? The bearing was not sufficiently lubricated.
  3. Why was it not lubricated sufficiently? The lubrication pump was not pumping sufficiently.
  4. Why was it not pumping sufficiently? The shaft of the pump was worn and rattling.
  5. Why was the shaft worn out? There was no strainer attached and metal scrap got in.

Watch the chain move. It starts at the machine and the fuse, shifts to the bearing and its lubrication, then to the pump, then to the shaft, then to the protection of the shaft. In order to solve the problem, the team no longer controls the overload of the machine, it controls the inflow of metal scrap in the pump. The root cause is the missing strainer, and the fix is to fit one. A fuse is a symptom. A strainer is a root cause.

There are four rules, and the fourth is the one teams break.

First, five is a target, not a law. There is nothing magical about the number 5, but it seems to be an effective target to get to an actionable item, and the closer you get to the fifth "why," the more specific the response. Some chains end at three. Some need seven. The chain ends when it reaches something you can change in the design or the process, a strainer, a specification, a test, not when you have asked five times.

Second, each "why" must be answered by a measurement, not by an opinion. This is the step that separates analysis from theater. A "why" answered by "probably" is a guess with a number on it. The Lean Thinker, writing about Ohno's example, is blunt about this: "I have heard 'Just ask why five times' as a method, proposed in contrast to more rigorous methods. It ain't that simple, folks." Each "why" question triggers another round of investigation, a measurement, a test, a log, a scope trace. The chain is only as strong as the evidence at each link.

Third, the root cause points toward a process that is not working well or does not exist. The machine stopped because there was no strainer, because nobody's job was to protect the pump. The car would not start because it was not maintained according to the recommended service schedule. The worker slipped because the compressor was not in the maintenance system. The root cause is almost always a missing step, a missing check, or a missing specification, something the team can add, not something the team can blame.

Fourth, the chain is one chain. Toyota-style problem solving is about turning a multi-cause, multi-impact problem into a one-by-one cause-effect problem, by identifying which cause, of all potential factors, is responsible for the largest impact. The fishbone finds the candidates. The Five Whys follows the biggest one. If two chains compete, follow both, but follow them one at a time, and let the measurements decide which one is real.

The fishbone: six bones, and the suspects on them

The fishbone diagram is one of the most widely used tools in quality management and is considered one of the Seven basic quality tools. Also known as a cause-and-effect diagram or an Ishikawa diagram, after its creator Kaoru Ishikawa, it helps users identify the many possible causes for a problem by sorting ideas into useful categories, and it is especially useful in structuring brainstorming sessions, because it stops the loudest person in the room from deciding the answer.

Ishikawa popularised the diagram in the 1960s while developing quality processes at the Kawasaki shipyards, and formalised it in his 1968 book Guide to Quality Control. The mechanics take five minutes. Doing it well takes discipline. You write the problem, the effect you want to eliminate, on the right side, at the fish's head. A horizontal line runs left into it: the spine. Off that spine you draw diagonal branches, the ribs, and each rib is a category of causes.

The standard categories for manufacturing are the 6Ms. The list below is the one Semishops uses for semiconductor failures, the same source as the FPGA example, and it is the list this article uses, because it matches the failure the demo investigates:

  • Materials, parts, ingredients, supplies. Solder paste quality. Component aging. The capacitor whose electrolyte degraded.
  • Methods, procedures, techniques, processes. The reflow profile that varied. The test plan that never ran the hub on an aged cell.
  • Machines, production-related equipment, materials handling equipment, software. Placement accuracy issues. The worn tool. The pump with no strainer.
  • Measurement, key indicators, measurement devices, data capture. Inadequate inspection criteria. The gauge that was never calibrated. The rail voltage measured at rest, never under load.
  • Environment, the surroundings. Humidity exposure. The field site running 10 °C colder than the bench. Temperature, vibration, dust.
  • Personnel, people and human resources, with their training, skills, and competences. The test engineer who never saw the brownout because the bench supply holds the rail stiff.

Two notes on the list. First, the categories are training wheels, not scripture, Ishikawa introduced the 6Ms as generic labels and encouraged creativity in naming them, and other sources use different sets: ASQ's list swaps in Manpower and Mother Nature, and MetalForming's list adds Management and Maintenance. Use the categories that fit the failure; the point is to cover the space, not to recite the letters.

Second, a finished fishbone is a list of suspects, and every suspect is a hypothesis you can test against data. The fishbone does not find the root cause. It finds the candidates, and the Five Whys and the measurements do the rest.

The FMEA

Failure Mode and Effects Analysis is a structured, team-based method for systematically identifying the ways in which a product or process can fail, estimating the risk associated with each failure mode, and prioritizing actions to reduce that risk before failures occur. The operative word is before. FMEA is a preventive tool, not a corrective one.

It belongs at the design and planning stages, not in the post-mortem after a customer complaint arrives. But the post-mortem feeds it: every failure the post-mortem finds becomes a row in the next FMEA, so the next prototype does not fail the same way.

FMEA's origins trace back to the early 1950s, when the U.S. Navy Bureau of Aeronautics developed it as a reliability engineering technique for military aircraft and weapon systems. It was used by NASA in the 1960s, adopted by the automotive industry in the 1970s, and today the main references are IEC 60812 (the international standard) and the AIAG & VDA FMEA Handbook used in the automotive sector.

There are two primary variants, and the post-mortem needs both. Design FMEA (DFMEA) looks at how a product design could fail, wrong materials, weak parts, poor tolerances, missing features, and it is done during design, before the product is released. Process FMEA (PFMEA) looks at how a manufacturing or assembly process could fail, wrong torque, missed steps, wrong parts, wrong settings, and it is done before a process starts production.

The same failure can appear in both, with different causes, and the cause is where the two variants differ, which is the point of the next section.

At its core, FMEA asks three disciplined questions about every potential failure: how severe would the consequence be (Severity)? How likely is this failure to occur (Occurrence)? How well can our current controls detect it before it reaches the customer (Detection)? The answers are scored, multiplied together to produce a Risk Priority Number (RPN), and used to prioritize where corrective action is most urgent.

Each rating runs 1 to 10, so the RPN runs from 1 to 1,000. A worked example from a template guide: Severity 8 × Occurrence 6 × Detection 5 = RPN 240. Always review the underlying ratings, not only the total, and the RPN should support judgement rather than act as the only trigger for action, especially where severity is high. The newer AIAG & VDA method uses an Action Priority (High, Medium, Low) instead of the RPN, for exactly that reason: a failure with a severity of 9 or 10 demands action even if its RPN is moderate, because the consequence is the thing that matters.

One more rule, and it is the one that decides whether the occurrence rating means anything: most FMEA standards, AIAG, SAE J1739, VDA, attribute the occurrence ranking to the cause of the failure, not to the failure mode. Two different causes for the same failure mode can have very different likelihoods, and ranking them separately is what lets the team prioritise the right corrective action.

How to write a cause: the design deficiency, not the symptom

The most common mistake in an FMEA is writing the failure mode again in the cause column. Accendo Reliability's guide on FMEA causes is the clearest statement of the rule, and its two examples are the standard every cause in this article is judged against.

A cause is the specific reason for the failure, preferably found by asking "why" until the root cause is determined. For Design FMEAs, the cause is the design deficiency that results in the failure mode. For Process FMEAs, the cause is the manufacturing or assembly deficiency (or source of variation) that results in the failure mode. In Design FMEAs, root causes are described in terms of product characteristics, dimensions, weight, orientation, hardness, strength, and the design deficiency relates to the product characteristic, such as "material too soft."

In Process FMEAs, root causes are described in terms of process characteristics, oven temperature, tool wear, part position, weld-device current, pressure, flow rate, and the manufacturing deficiency relates to the process characteristic, such as "insufficient or intermittent weld-device current."

The two examples:

Design FMEA, the power steering pump. Function: delivers hydraulic power for steering by transforming oil pressure at the inlet into higher oil pressure at the outlet. Failure mode: inadequate outlet pressure. Effect on the end user: increased steering effort with potential for an accident during steering maneuvers. Cause: fluid incorrectly specified (viscosity too low). Poorly worded example of cause: "outlet pressure too low", which is the failure mode restated, not the cause.

Process FMEA, induction hardening shafts. Function: induction harden shafts using the induction-hardening machine, with minimum hardness Brinell Hardness Number "X", according to specification #123. Failure mode: shaft hardness less than BHN "X". Effect in the plant: 100% scrap. Effect on the end user: potential shaft fracture with complete loss of performance. Cause: induction machine electrical voltage/current settings are incorrect for the part number. Poorly worded example of cause: "operator error."

"Operator error" is a poorly worded cause, not because operators never make errors, but because it names nobody's fix. It does not say what was wrong with the design or the process that let the error happen, and it does not say what to change. The cause that replaces it names the deficiency: the settings were wrong for the part number, the fixture allowed the part to be loaded backwards, the procedure had no check.

The test for any cause is the same as the test for a root cause: can you change it in the design or the process? "Fluid incorrectly specified" can be changed, specify the right fluid. "Voltage/current settings incorrect for the part number" can be changed, lock the settings to the part number. "Operator error" cannot be changed, because it is not a cause.

It is a shrug.

The same test applies to the post-mortem's chain. A "why" whose answer is a person is not finished. Ask why the person could make the error, and keep asking until the answer is a design or a process.

The cost of finding it late

The post-mortem is not free. It costs engineer-days, and the team that is already behind schedule will ask whether it can be skipped. The answer is a table.

Semishops' failure-cost escalation, from the defective-component investigation process, runs like this: a failure found at engineering validation costs 1×; at the prototype stage, 5×; at pilot production, 20×; at mass production, 50×; at field deployment, 100× or more. "The earlier the investigation occurs, the greater the potential savings." A second model, from the technical failure investigation services, runs the same shape from the other end: incoming inspection 1×, manufacturing line 10×, system integration 25×, customer installation 100×, product recall 500× or more.

And the FMEA business case is the same shape compressed into three numbers, the 1-10-100 rule: prevention costs 1, correction costs 10, failure costs 100. FMEA is not a quality-department exercise. It is a financial risk management decision made at the design stage, when the cost of change is still low.

The demo below puts the escalation in engineer-days, so the team can see it: a fix that costs one engineer-day at validation costs five at prototype, twenty at pilot, fifty at mass production, and a hundred or more in the field. The hub's failure was found in the field. That is the most expensive place to find anything.

The four files the post-mortem reads first

A prototype most often fails in four places, and the cluster has a file for each of them. The post-mortem reads them first, because the failure is usually in one of the four, and the file tells the team what to measure.

#5, PCB DFM, the board. A board that cannot be tested is a board whose faults reach the customer. Test coverage is the percentage of possible defects the test can detect, and it decides the defect escape rate. Visual inspection only: 40–60% coverage, 500+ ppm escape. Flying probe: 85–92%, 50–100 ppm. ICT: 95–98%, 10–30 ppm. ICT plus boundary scan: 98–99.5%, 2–10 ppm. Plus functional test: 99.5–99.9%, under 5 ppm. At a 2% board defect rate, 90% coverage still sends 2,000 bad boards per million to customers. 99% sends 200. The test points are the evidence: per IPC-9252A, the pad must be at least 0.9 mm across, 1.0 mm recommended, with 1.27 mm minimum centre-to-centre pitch and 2.54 mm recommended, on the bottom side, with a target of ≥95% net accessibility. And the DFT failures are themselves the yield problem: one vendor-cited figure, attributed to IPC research, puts testability issues at 35–40% of PCB manufacturing yield losses.

#8, tolerance stack-up, the assembly. Every part is a little off, and the little offs add up. Ten parts at ±0.1 mm stack to ±1.00 mm worst case and ±0.32 mm statistically (RSS), and the difference between those two answers is the difference between a product that assembles and a product that doesn't. The fix is rarely a tighter tolerance. It is usually a shorter chain.

#17, calibration and drift, the sensor. A gyroscope measures rate, not angle, the angle is the integral of the rate, so any bias is integrated too: a bias of 0.1°/s adds 6° of error every minute, 360° in an hour. Bias is the offset the sensor reads at rest; it shifts with temperature and time, and a single zero subtraction cannot correct bias, noise, scale factor, and misalignment. Sea-Bird temperature sensors drift about 1 m°C per year, and a smoothly drifting sensor can be corrected for drift, while biofouling drift cannot be properly corrected for. The post-mortem asks whether the failure is drift, slow, steady, and predictable, or a fault, and the answer decides whether the fix is a calibration or a redesign.

#27, CNC DFM, the lid. A drawing with ±0.01 mm on every dimension costs eight times the same part at ±0.5 mm. The machined 6061-T6 hub lid, as drawn, a 30 mm deep pocket with R2 corners, a 1.0 mm thick flange, a Ø3 mm hole 30 mm deep, four M4 tapped holes with 12 mm of thread, twenty-eight dimensions at ±0.01 mm, features on six faces, costs $1,370 a part at a batch of ten. Fixed, ±0.01 mm on the two bearing bores only, ISO 2768-m everywhere else, it costs $430 a part. The saving is $940 a part, 69%, from one CAD edit: deleting twenty-six tolerances. The post-mortem asks whether the failure is in the drawing, and the drawing is the cheapest place to look.

The decision test

Before the post-mortem is written, five questions. Each one has an answer that can be checked, and the post-mortem is not done until all five are answered.

1. What is the symptom, stated as an observation, not a diagnosis? "The hub stops transmitting after 21 days" is a symptom. "The radio is bad" is a diagnosis, and a diagnosis written before the investigation is a guess. Write the symptom. Measure it. Then ask why.

2. What is the chain, and what is the evidence at each link? Symptom, immediate cause, contributing cause, root cause, four layers, each one supported by a measurement, a log, or a test. A link with no evidence is a guess, and a guess in the chain is a place the next failure will hide.

3. Which bone does the root cause sit on? Materials, Methods, Machines, Measurement, Environment, Personnel. If the root cause is on the Measurement bone, the test never measured the rail under load, then the fix is a test, not a part. If it is on the Methods bone, the test plan never ran the hub on an aged cell, then the fix is a procedure.

4. What is the RPN, and what is the highest one? Severity × Occurrence × Detection, each 1 to 10. The highest RPN gets fixed first. But a severity of 9 or 10 demands action even if the RPN is moderate, the RPN supports judgement, it does not replace it.

5. What is the fix, and what is the margin? Fix the root cause and strengthen the failure point, the fix that stops the return, plus the margin that survives the next one. And the fix is not done when the code is written or the part is changed. It is done when the failure cannot be reproduced.

Objections

"We don't have time for a post-mortem." A failure found in the field costs 100 times what the same failure costs at validation. The post-mortem costs engineer-days. The field failure costs engineer-days, plus the recall, plus the rework, plus the reship, plus the customer. The team that skips the post-mortem does not save the time. It spends it later, at a hundred times the rate.

"The root cause is obvious." The obvious cause is usually the symptom. The FPGA stopped responding, obvious. The power rail was unstable, less obvious. The capacitor degraded, less obvious still. The supplier's process varied, invisible until the fourth layer. Four layers deep is where the root cause lives, and the obvious answer is almost always the first layer.

"We fixed it, it works now." A fix is not done when the failure stops on the bench. It is done when the failure cannot be reproduced, on an aged cell, in the cold, under load, for thirty days. The bench is where the failure hid the first time. Testing the fix on the bench proves nothing.

"It was operator error." "Operator error" is a poorly worded cause. It names nobody's fix. Ask why the operator could make the error, the fixture allowed the part to be loaded backwards, the procedure had no check, the settings were not locked to the part number, and keep asking until the answer is a design or a process.

"Five Whys is enough." Five Whys is a tool, not a method. Each "why" triggers another round of investigation, and a "why" answered by an opinion is a guess with a number on it. The fishbone finds the suspects, the Five Whys follows the biggest one, and the measurements decide which chain is real.

Example Labs is a made-up company we use for worked examples.

DEMO

The firm is made up. Call it Example Labs again: 60 staff, and now a prototype in the field, the field sensor hub from Articles #5, #6, #7, #17, #27, and #37: a 100 × 60 × 40 mm enclosure, a 4-layer board, 100 × 60 mm, the event camera from Article #2, the IMU from Article #4, the STM32H7 running the Kalman filter and the PID loop, the BLE radio at 2.4 GHz, and a CR2032, 230 mAh = 2,484 J.

Ten units deployed. Four of the ten stopped transmitting between day 19 and day 23, 40% of the fleet, in three weeks. The demo's arithmetic is the article's own, verified before writing, on the published figures cited above; the firm, the hub, the failure, and the measurements are the worked example, not a measured cell.

Step 1, the symptom, stated as an observation. "Four of ten hubs stopped transmitting between day 19 and day 23." Not "the radio is bad." Not "the firmware crashed." The symptom, measured: the hubs stopped transmitting. The team wants to replace the radio. The post-mortem starts by freezing the evidence instead: the four dead hubs are quarantined, their date codes and lot numbers recorded, their .noinit SRAM logs read before anything is powered on again, because the reset-cause register and the SRAM log are the evidence, and powering the hub on first destroys them. The log shows the reset cause was the watchdog, and the BLE stack never completed the rejoin, the article's own construction, labelled.

Step 2, the fishbone. Six bones, one suspect on each, for the hub that stopped:

  • Materials: the CR2032's internal resistance rises as the cell ages, and faster in the cold. The field site runs 10 °C colder than the bench.
  • Methods: the test plan ran the hub on a fresh cell, at room temperature, on the bench supply. It never ran the hub on an aged cell, in the cold, under transmit load.
  • Machines: the reflow profile varied, the Semishops memory-device example lists reflow profile variation as a Methods suspect and placement accuracy as a Machines suspect, and the hub's board carries both.
  • Measurement: the test measured the rail voltage at rest, never under BLE transmit load. The rail sags only when the radio transmits, and the test never transmitted.
  • Environment: the field site is colder than the bench, and the cell's internal resistance rises in the cold. The bench is where the failure hid.
  • Personnel: the test engineer never saw the brownout, because the bench supply holds the rail stiff. The bench supply is not the battery.

A finished fishbone is a list of suspects. The Five Whys picks the best one, Materials and Measurement, together, and the measurements decide.

Step 3, the Five Whys, each answered by a measurement. The chain, with the evidence at each link, the article's own construction, labelled, built from the cluster's facts:

  1. Why did the hub stop transmitting? The firmware hung, the watchdog reset it, and the BLE stack never completed the rejoin. Evidence: the .noinit log shows the watchdog reset cause, and the BLE stack's rejoin never completed.
  2. Why did the firmware hang? The I2C sensor held SDA low, and the firmware waited for the bus forever, it has no timeout on the I2C read. Evidence: Article #37, the sensor holds SDA low after a brownout, "waiting for clocks that never came"; the firmware's I2C read has no timeout.
  3. Why did the sensor hold SDA low? The rail sagged below the sensor's minimum voltage during a BLE transmit burst, and the sensor latched up. Evidence: the rail, measured under load on an aged cell, sags below the sensor's minimum during the transmit pulse, a measurement the bench test never made, because it never loaded the rail.
  4. Why did the rail sag? The CR2032's internal resistance had risen as the cell aged, and the BLE transmit pulse pulled the rail down. Evidence: the rail sag, measured on a cell discharged to 50% of its rated capacity, in the cold, the condition the field units were in at day 21.
  5. Why was the rail sized so it could sag? The power budget was calculated for a fresh cell at room temperature, and the test plan never ran the hub on an aged cell, in the cold, under transmit load. The design assumed the fresh-cell voltage. The test never challenged the assumption.

The root cause is not the radio, and it is not the sensor. It is the assumption, the power budget calculated for a fresh cell, and the test plan that never challenged it. Both are things the team can change in the design and the process. The chain ends there, because the fifth why reaches something changeable: size the rail for the end-of-life cell, and test on an aged cell.

Step 4, the FMEA, for the next build. Three failure modes for the hub's power rail, scored Severity × Occurrence × Detection on the 1–10 scales, RPN = S × O × D:

Failure modeSODRPN
Rail sags below the sensor's minimum voltage during BLE transmit → sensor latches SDA low → hub stops reporting968432
Tolerance stack-up, the lid's sensor pocket is 0.32 mm off (RSS) and the board flexes, cracking a solder joint745140
Gyro bias drifts 6°/min (0.1°/s) and the heading walks, the hub reports the wrong orientation573105

The ranking: the rail sag first (432), the stack-up second (140), the drift third (105). The drift's occurrence is 7, drift is certain, it is physics, but its detection is 3, because the calibration routine catches it at startup, and its severity is 5, because a wrong heading is a wrong reading, not a silent hub. The rail sag's detection is 8, because the bench test never measures the rail under load, so the failure escapes, and that is the Measurement bone again: the test is the control, and the control cannot detect the cause.

The highest RPN gets fixed first. And the severity of 9 on the rail sag would demand action even if the RPN were moderate, the hub going silent in the field is the product's whole job failing.

Step 5, the cost of finding it late, and the fix with its margin. The failure was found in the field, the most expensive place to find anything. On Semishops' escalation, a fix that costs 1 engineer-day at engineering validation costs 5 at prototype, 20 at pilot, 50 at mass production, and 100 or more in the field. The hub's post-mortem itself costs 3 engineer-days. The field fix, recall the ten units, rework, retest, reship, costs 100 engineer-days or more, before the customer is counted. The fix, and the margin that survives the next one: size the rail for the end-of-life cell voltage under load, with a bulk capacitor on the rail; raise the brownout threshold above the sensor's latch-up voltage, so the supervisor resets the board before the sensor can latch; add the nine-SCL-pulse I2C recovery from Article #37, and a timeout on the I2C read, so a latched bus cannot hang the firmware; and change the test plan to run the hub on a cell discharged to 50% of its rated capacity, in the cold, under BLE transmit load, with the rail voltage logged. The verification: run ten units for thirty days on aged cells in the cold, with the rail voltage logged under load. The fix is not done when the code is written. It is done when the failure cannot be reproduced, and the post-mortem's last line is the FMEA row for the next build, so the next prototype does not fail the same way.

Sources

  1. Semishops, "Root Cause Identification Methods" (semishops.com, checked 2026-11-20)
    What we used

    "The distinction between a symptom and a root cause is fundamental to quality management. A symptom describes what happened. A root cause explains why it happened."; the four-layer table: Symptom, FPGA stopped responding; Immediate Cause, Power rail instability; Contributing Cause, Capacitor degradation; Root Cause, Supplier process variation affecting capacitor quality; "The ability to identify root causes accurately therefore becomes a strategic business capability rather than merely a technical exercise"; "In the electronics and semiconductor industries, where a single component may contain billions of transistors and operate within highly complex systems, identifying the true root cause of a failure is often considerably more challenging than detecting the failure itself"; the Five Whys example (industrial controller): Problem, unexpected shutdowns; Why? Power supply voltage dropped; Why? Input capacitor performance deteriorated; Why? Capacitor ESR exceeded specification; Why? Electrolyte degradation occurred prematurely; Why? Supplier manufacturing process deviated from approved parameters; Root Cause: Supplier process control failure; the fishbone categories: Materials, Methods, Machines, Measurement, Environment, Personnel; the memory-device example: Materials, solder paste quality, component aging; Methods, reflow profile variation; Machines, placement accuracy issues; Environment, humidity exposure; Measurement, inadequate inspection criteria

  2. Semishops, "Semiconductor Troubleshooting Support" (semishops.com, checked 2026-11-20)
    What we used

    "The relationship between observed symptoms and actual failure mechanisms is often indirect. A processor reset, for example, may result from: power rail instability, thermal overload, firmware timing conflicts, electromagnetic interference, damaged memory devices, clock distribution problems, counterfeit or degraded components"; the failure-source distribution ("Industry reliability investigations frequently reveal"): Design Issues 35%, Manufacturing Defects 25%, Environmental Factors 15%, Power Integrity Problems 10%, Component Defects 8%, Counterfeit Components 4%, Unknown Causes 3%; "The data illustrates a critical reality: in most cases, the semiconductor itself is not the primary cause of failure."

  3. Semishops, "Defective Component Investigation Process" (semishops.com, checked 2026-11-20)
    What we used

    "Detecting a defect answers one question: What failed? Investigation answers a more important question: Why did it fail?"; the failure-cost escalation: Engineering Validation 1×, Prototype Stage 5×, Pilot Production 20×, Mass Production 50×, Field Deployment 100×+; "The earlier the investigation occurs, the greater the potential savings."

  4. Semishops, "Technical Failure Investigation Services" (semishops.com, checked 2026-11-20)
    What we used

    "Industry reliability studies suggest that nearly 55% of field-returned electronic assemblies contain no intrinsic semiconductor defect when subjected to comprehensive laboratory analysis. Instead, failures often originate from: thermal fatigue, PCB assembly defects, mechanical stress, moisture ingress, electrostatic discharge (ESD), electrical overstress (EOS), contamination, design incompatibilities"; the failure-cost escalation model: Incoming Inspection 1×, Manufacturing Line 10×, System Integration 25×, Customer Installation 100×, Product Recall 500×+

  5. FastCompany, "The Formula To Better Problem Solving" (fastcompany.com, checked 2026-11-20)
    What we used

    "The Five Whys technique was developed and fine-tuned within the Toyota Motor Corporation as a critical component of its problem-solving training. Taiichi Ohno, the architect of the Toyota Production System in the 1950s, describes the method in his book *Toyota Production System: Beyond Large-Scale Production* as 'the basis of Toyota's scientific approach . . . by repeating why five times, the nature of the problem as well as its solution becomes clear.'"; "Ohno encouraged his team to dig into each problem that arose until they found the root cause. 'Observe the production floor without preconceptions,' he would advise. 'Ask \"why\" five times about every matter.'"; "It's important to note that the purpose of the Five Whys isn't to place blame, but rather to uncover the root cause of why something unexpected occurred."

  6. Lean Enterprise Institute, "The Five Whys" (lean.org, checked 2026-11-20)
    What we used

    Ohno's classic example (from his book on the *Toyota Production System*): "1. Why did the machine stop? There was an overload and the fuse blew. 2. Why was there an overload? The bearing was not sufficiently lubricated. 3. Why was it not lubricated sufficiently? The lubrication pump was not pumping sufficiently. 4. Why was it not pumping sufficiently? The shaft of the pump was worn and rattling. 5. Why was the shaft worn out? There was no strainer attached and metal scrap got in."; "We can see him shifting from the demand on the machine to the lubrication of the bearing, to the working of the pump, to wearing of the shaft, to the protection of the shaft: in order to solve the problem, we no longer control the overload of the machine, but the inflow of metal scrap in the pump."; "Toyota-style problem solving is about learning to turn a multi-cause multi-impact problem in a one-by-one cause-effect problem by identifying which cause (of all potential factors) is responsible for the largest impact (of all potential impacts)."; "The key to a successful '5 why?' is a correct grasp on the current state."

  7. Wolters Kluwer, "Safetip #109: '5 Whys' Method to Identify Root Causes of Incidents" (wolterskluwer.com, checked 2026-11-20)
    What we used

    "5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a problem. The goal is to determine the root cause by repeating the question 'Why?'. Each answer forms the basis of the next question. The '5' in the name derives from an anecdotal observation on the number of iterations needed to resolve a problem. The technique was originally developed by Sakichi Toyoda and later used at Toyota during the evolution of its manufacturing methodologies. The technique is now used within Kaizen, lean manufacturing, and Six Sigma."; the car example: battery dead → alternator not working → alternator belt broken → belt beyond its useful service life and never replaced → car not maintained according to the recommended service schedule (root cause); the OSHA example: worker slips and falls → oil puddle → oil spilled from a compressor → leak not detected → compressor not inspected regularly → compressor not in the maintenance system (root cause)

  8. The Lean Thinker, "Not Just Asking Why?, Five Investigations" (theleanthinker.com, checked 2026-11-20)
    What we used

    "I have heard 'Just ask why five times' as a *method,* proposed in contrast to more rigorous methods. It ain't that simple, folks."; "In the text, Shook's coverage of root cause investigation is very thorough. He tells the story of each 'Why?' question triggering another round of investigation."; "The conclusion is that the lack of a strainer is the root cause of the machine stoppage. ('For the want of a nail…')"

  9. Wikipedia, "Five whys" (en.wikipedia.org, checked 2026-11-20)
    What we used

    "Five whys (or 5 whys) is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question"; "The technique was described by Taiichi Ohno at Toyota Motor Corporation"; "The *real* root cause points toward a process that is not working well or does not exist"; "The modern technique was originally developed by Sakichi Toyoda and was used within the Toyota Motor Corporation during the evolution of its manufacturing methodologies"; "Two primary techniques are used to perform a five whys analysis: the fishbone (or Ishikawa) diagram and a tabular format."

  10. ASQ, "What is a Fishbone Diagram? Ishikawa Cause & Effect Diagram" (asq.org, checked 2026-11-20)
    What we used

    "A fishbone diagram is one of the most widely used tools in quality management and is considered one of the Seven basic quality tools. Also known as a cause and effect diagram or an Ishikawa diagram (after its creator, Kaoru Ishikawa), the fishbone diagram can help users identify the many possible causes for a problem by sorting ideas into useful categories and is especially useful in structuring brainstorming sessions."; "Ishikawa introduced the '6 M's' as generic labels to describe the core categories... The 6M's stand for: Materials, parts, ingredients, supplies; Machinery, production-related equipment, materials handling equipment, software; Methods, procedures, techniques, processes, regulations; Measurement, key indicators, measurement devices, key data capture or collection points; Manpower, people and human resources, with their associated training, skills, competences; Mother Nature, environment and externalities; Less commonly included but useful for contemporary diagrams is a seventh M: Money, operating expenses and capital investments."

  11. Rework, "Fishbone Diagram (Ishikawa): Root Cause Analysis Explained" (resources.rework.com, checked 2026-11-20)
    What we used

    "The tool was created by Japanese quality control expert Kaoru Ishikawa in the 1960s during his work at Kawasaki Heavy Industries shipyards."

  12. FlowFuse, "Ishikawa Fishbone Diagram: The 6Ms & Examples" (flowfuse.com, checked 2026-11-20)
    What we used

    "It's named after Kaoru Ishikawa (1915–1989), the Japanese engineer who popularized it in the 1960s while developing quality processes at the Kawasaki shipyards and formalized it in his 1968 book *Guide to Quality Control*."; "The sequence is: fishbone to find suspects, 5 Whys to reach the root of the best ones, Pareto to decide which roots are worth fixing first."; "A finished fishbone is a list of *suspects*, and every suspect is a hypothesis you can test against data."; "The 6Ms are training wheels, not scripture."

  13. MetalForming Magazine, "Be Wise: Ask the 'Five Whys' & Create a Fishbone for Faster Problem Solving"
    What we used

    (metalformingmagazine.com, checked 2026-11-20), "Action: Replace the damaged screen. This will prevent additional metal shavings from getting into the coolant. But, even with the new screen, the seal remains damaged and leaking will continue. Therefore, the corrective action is to replace the damaged screen and the damaged seal, to stop the coolant from leaking. However, the screen likely will become damaged again, which in turn will lead to the seal being damaged and more leaking coolant. So, the permanent corrective action: Redesign the machine or add a guard to cover the screen and prevent damage from occurring."; the fishbone categories: Manpower, Machines, Metals, Methods, Measurements, Mother nature, Management, Maintenance; "For manufacturing, the largest branches highlighting the major contributing factors likely fall into a category beginning with the letter M, these are known as the 4Ms, 5Ms, etc."

  14. The Fabricator, "Problem solving in metal fabrication: Look past the symptom to find the root cause" (thefabricator.com, checked 2026-11-20)
    What we used

    "The root cause is the fundamental reason the problem occurred. Another definition describes root cause as a factor that caused a nonconformance and provides an opportunity for a permanent fix."; "There is nothing magical about the number 5, but it seems to be an effective target to get to an actionable item. The closer you get to the fifth 'why,' the more specific the response."; the takt-time example: machine not meeting takt time → pneumatic pick-and-place moving too slowly → debris in the machine's air filter → breakdown of the plant's main air filter, which contaminated the parts-handling device and could affect the plant's entire air system; "Fixing the symptoms (not meeting takt time, handler moving too slowly, debris in the machine's filter, etc.) was not going to make the problem go away."

  15. OE Consulting, "FMEA: A Practitioner's Guide to Failure Mode & Effects Analysis" (oeconsulting.com.sg, checked 2026-11-20)
    What we used

    "Failure Mode and Effects Analysis is a structured, team-based method for systematically identifying the ways in which a product or process can fail, estimating the risk associated with each failure mode, and prioritizing actions to reduce that risk before failures occur. The operative word is *before*. FMEA is a preventive tool, not a corrective one."; "At its core, FMEA asks three disciplined questions about every potential failure: 1. How severe would the consequence be? (Severity) 2. How likely is this failure to occur? (Occurrence) 3. How well can our current controls detect it before it reaches the customer? (Detection) The answers to these three questions are scored, multiplied together to produce a Risk Priority Number (RPN), and used to prioritize where corrective action is most urgent."; "FMEA's origins trace back to the early 1950s, when the U.S. Navy Bureau of Aeronautics developed it as a reliability engineering technique for military aircraft and weapon systems."; the 1-10-100 rule; "FMEA is not a quality department exercise. It is a financial risk management decision made at the design stage, when the cost of change is still low."

  16. LearnLeanSigma, "Free FMEA Template For Excel And Google Sheets" (learnleansigma.com, checked 2026-11-20)
    What we used

    "It uses 1–10 ratings for Severity (S), Occurrence (O) and Detection (D), then calculates the Risk Priority Number as RPN = S × O × D."; "Example only: Severity 8 × Occurrence 6 × Detection 5 = RPN 240. Always review the underlying ratings, not only the total."; "Write one failure mode per row: Use precise descriptions such as 'seal fitted backwards,' not vague labels such as 'operator error.'"; "RPN should support judgement rather than act as the only trigger for action, especially where severity is high."

  17. Ahsan Rahman, "What Is FMEA? Failure Mode and Effects Analysis Explained With Example" (ahsanrahman.com, checked 2026-11-20)
    What we used

    "FMEA started in the US military in the late 1940s, was used by NASA in the 1960s, and was adopted by the automotive industry in the 1970s. Today the main references are IEC 60812 (the international standard) and the AIAG & VDA FMEA Handbook used in the automotive sector."; "The newer AIAG & VDA method uses an Action Priority (High, Medium, Low) instead of RPN."

  18. Accendo Reliability, "Understanding FMEA Causes"
    What we used

    (accendoreliability.com, checked 2026-11-20), "A 'cause' is the specific reason for the failure, preferably found by asking 'why' until the root cause is determined. For Design FMEAs, the cause is the design deficiency that results in the failure mode. For Process FMEAs, the cause is the manufacturing or assembly deficiency (or source of variation) that results in the failure mode."; "In Design FMEAs, root causes are often described in terms of product characteristics, such as dimensions, weight, orientation, hardness, strength, etc. The design deficiency relates to the product characteristic, such as 'material too soft.' In Process FMEAs, root causes are often described in terms of process characteristics, such as oven temperature, tool wear, part position, weld-device current, pressure, flow rate, etc. The manufacturing deficiency relates to the process characteristic, such as 'insufficient or intermittent weld-device current.'"; Design FMEA example (power steering pump): Failure Mode, inadequate outlet pressure; Effect (end user), increased steering effort with potential for an accident; Cause: Fluid incorrectly specified (viscosity too low); Poorly worded example of Cause: Outlet pressure too low; Process FMEA example (induction harden shafts): Failure Mode, shaft hardness less than BHN "X"; Effect (in plant), 100% scrap; Effect (end user), potential shaft fracture with complete loss of performance; Cause: Induction machine electrical voltage/current settings are incorrect for the part number; Poorly worded example of Cause: Operator error; "Many practitioners use repeated questioning of the FMEA team to ensure that the basic 'why' is determined as the cause of a failure mode. This technique, called the Five Whys, can be very helpful."

  19. Accendo Reliability, "Understanding FMEA Occurrence - Part 1"
    What we used

    (accendoreliability.com, checked 2026-11-20), "The occurrence ranking has a relative meaning rather than an absolute value and is determined without regard to the severity or likelihood of detection."; "Most FMEA standards, such as AIAG, SAE J1739 and VDA, attribute the occurrence ranking to the cause of the failure."

  20. Cluster facts re-checked on disk (2026-11-20): #5 pcb-design-for-manufacturing, test coverage: visual inspection only 40–60% coverage, 500+ ppm escape; flying probe 85–92%, 50–100 ppm; ICT 95–98%, 10–30 ppm; ICT + boundary scan 98–99.5%, 2–10 ppm; plus functional test 99.5–99.9%, under 5 ppm; at a 2% board defect rate, 90% coverage sends 2,000 bad boards per million, 99% sends 200; IPC-9252A test points: 0.9 mm minimum, 1.0 mm recommended, 1.27 mm minimum pitch, 2.54 mm recommended, ≥95% net accessibility; DFT failures at 35–40% of PCB manufacturing yield losses (vendor-cited, attributed to IPC research). #8 tolerance-stack-up-analysis, ten parts at ±0.1 mm stack to ±1.00 mm worst case and ±0.32 mm statistically (RSS); the fix is usually a shorter chain. #17 sensor-calibration-drift, gyroscope measures rate, not angle; bias 0.1°/s adds 6°/min, 360° in 1 h; bias stability is a 1σ spread from the Allan-deviation minimum and cannot be calibrated away; a single zero subtraction cannot correct bias, noise, scale factor, and misalignment; Sea-Bird temperature sensors drift about 1 m°C/yr. #27 cnc-machining-dfm, ISO 2768-m: ±0.1 mm up to 6 mm, ±0.2 mm to 30 mm, ±0.3 mm to 120 mm, ±0.5 mm to 400 mm; the machined 6061-T6 hub lid: as drawn $1,370/part, fixed $430/part at a batch of ten, saving $940/part (69%), from deleting twenty-six tolerances; baseline $160 at ±0.5 mm; 6061-T6 310 MPa tensile; 7075-T6 570 MPa. #37 watchdog-brownout-recovery, the I2C sensor holds SDA low after a brownout ("The target is not broken. It is waiting for clocks that never came"
    What we used

    ); nine SCL pulses then STOP per NXP UM10204 §3.1.16; the .noinit SRAM log survives reset; reset-cause capture in RCC_CSR. #6 power-design-battery-products, CR2032 230 mAh = 2,484 J. #32 design-for-test, IPC-9252A test points; ICT full access 95–98% coverage, 10–30 ppm escapes; fixture $3,000–$15,000 (Andwin) to $10,000–$50,000 complex (JLCPCB); field fixes cost 10–100× factory fixes

More in software meets hardware

RTOS vs Bare MetalA super-loop runs the jobs one after another, so one long job delays every other, and the moment a blocking operation can stall the one task that must run every millisecond, the loop is no longer a schedule. It is a hope. Use an RTOS when you have 3+ independent concurrent activities with different timing requirements; stick with bare metal for simple, single-purpose devices. On the Example Labs hub, control loop at 1 kHz, BLE every 7.5 ms, the on-device model once a second, the OTA flash writes, the super-loop leaves the control loop 2.6x late, 40.6x late when the model runs, at 385 samples a second instead of 1,000. The RTOS costs ~9 kB of flash and 0.03% of the CPU, and the control loop is never late.16 minSecure BootA device cannot verify its own first instruction with software. The chain starts in immutable ROM, each stage verifies a signature on the next, and a monotonic counter in fuses refuses every old image, even one signed with your own key. The CRA's main obligations apply from 11 December 2027, and its reporting duty is already live.22 minThe Firmware That Doesn't Get LostMost reactive programs start out simple, but as features are grafted on, more and more flags and variables are introduced, until no human being really has a good idea what part of the code gets executed in response to any given event. The fix is one variable that can only hold one value at a time. This is the method: the state machine, the table, and the rules that keep it from becoming spaghetti again.26 minWatchdog timer embeddedA watchdog is a counter on its own clock that resets the processor when it reaches its limit, but a watchdog kicked from a timer interrupt is faithfully serviced by a dead product, and a reset that returns the device to the state that hung it produces a boot loop, which is worse than a freeze. The internal watchdog is free and shares the die; the external supervisor IC costs $0.30–$1.50 and cannot be disabled by confused firmware. This is the watchdog, the brownout, and the recovery ladder, worked through on the Example Labs field sensor hub.28 minThe Update That Cannot Brick the DeviceAn over-the-air firmware update can either extend the life of an embedded product or brick an entire fleet in the field. The difference is not the transport, it is the architecture: A/B partitions so the old firmware is never touched until the new one proves itself, signing and secure boot so the image is yours and unmodified, trial-then-confirm so a bad build reverts automatically, and a staged rollout with a kill switch. The safety costs one extra flash slot, 512 kB on a 2 MB chip, and its payoff is that a bad build costs one reboot instead of one scrapped unit, one truck roll, and one lost customer.11 min