Software meets hardware
Firmware that survives the field, updates that can roll back, and testing a design before and after it is built.
Notes 10
RTOS vs Bare MetalA super-loop runs the jobs one after another, so one long job delays every other, and the moment a blocking operation can stall the one task that must run every millisecond, the loop is no longer a schedule. It is a hope. Use an RTOS when you have 3+ independent concurrent activities with different timing requirements; stick with bare metal for simple, single-purpose devices. On the Example Labs hub, control loop at 1 kHz, BLE every 7.5 ms, the on-device model once a second, the OTA flash writes, the super-loop leaves the control loop 2.6x late, 40.6x late when the model runs, at 385 samples a second instead of 1,000. The RTOS costs ~9 kB of flash and 0.03% of the CPU, and the control loop is never late.16 minSecure BootA device cannot verify its own first instruction with software. The chain starts in immutable ROM, each stage verifies a signature on the next, and a monotonic counter in fuses refuses every old image, even one signed with your own key. The CRA's main obligations apply from 11 December 2027, and its reporting duty is already live.22 minThe Firmware That Doesn't Get LostMost reactive programs start out simple, but as features are grafted on, more and more flags and variables are introduced, until no human being really has a good idea what part of the code gets executed in response to any given event. The fix is one variable that can only hold one value at a time. This is the method: the state machine, the table, and the rules that keep it from becoming spaghetti again.26 minWatchdog timer embeddedA watchdog is a counter on its own clock that resets the processor when it reaches its limit, but a watchdog kicked from a timer interrupt is faithfully serviced by a dead product, and a reset that returns the device to the state that hung it produces a boot loop, which is worse than a freeze. The internal watchdog is free and shares the die; the external supervisor IC costs $0.30–$1.50 and cannot be disabled by confused firmware. This is the watchdog, the brownout, and the recovery ladder, worked through on the Example Labs field sensor hub.28 minThe Update That Cannot Brick the DeviceAn over-the-air firmware update can either extend the life of an embedded product or brick an entire fleet in the field. The difference is not the transport, it is the architecture: A/B partitions so the old firmware is never touched until the new one proves itself, signing and secure boot so the image is yours and unmodified, trial-then-confirm so a bad build reverts automatically, and a staged rollout with a kill switch. The safety costs one extra flash slot, 512 kB on a 2 MB chip, and its payoff is that a bad build costs one reboot instead of one scrapped unit, one truck roll, and one lost customer.11 minAlarm managementA plant with 400 configured alarms, 60 active at any moment, and operators who acknowledge the horn without reading the screen. EEMUA 191 says an operator can handle about one alarm every ten minutes, and at Milford Haven, two operators had to recognise, acknowledge, and act on 275 alarms in 11 minutes. The warning is a design problem, not a discipline problem.27 minDesign for TestA board that passes the bench and fails in the field is usually untestable by design, no test points, no JTAG chain, no fixture. The IPC-9252A rules, the four test methods, the coverage union, and what a fixture is worth on a real board.20 minTest It to DestructionHALT is a discovery test that steps stress far beyond specification to find weaknesses and measure operating and destruct limits, it is not a pass/fail qualification test. A complete HALT needs 4–8 units and 3–5 days; a field failure costs 100 times what a design-stage fix costs.21 minDigital twinThe prototype costs real money and weeks, and the failure it hides costs a hundred times more to find in the field than at validation. A digital twin is not a 3D model, it is a model that behaves like the physical system, fed by data from it, whose answers change the physical system. Example Labs' twin of its sensor hub predicted the rail sag that killed four of ten hubs in the field, as a band that straddled the sensor's minimum voltage, which is exactly what 'no margin' looks like before the hardware is built.21 minWhy Did the Prototype FailThe prototype worked on the bench and died in the field, and the team is about to replace the radio, which is the symptom, not the cause. The post-mortem runs three tools in order: the fishbone to find suspects, the Five Whys to reach the root of the best ones, and the FMEA to decide what to fix first. Example Labs' sensor hub stopped after 21 days because the power rail was sized for a fresh cell, not an aged one, and finding it in the field cost 100 times what finding it at validation would have.25 min