Menu

Watchdog timer embedded

The Firmware That Recovers Instead of Bricking: Watchdogs, Brownouts, and the Boot Loop

Notes · FirmwareNovember 19, 202628 min readSenova Systems Research

A watchdog is a counter on its own clock that resets the processor when it reaches its limit, but a watchdog kicked from a timer interrupt is faithfully serviced by a dead product, and a reset that returns the device to the state that hung it produces a boot loop, which is worse than a freeze. The internal watchdog is free and shares the die; the external supervisor IC costs $0.30–$1.50 and cannot be disabled by confused firmware. This is the watchdog, the brownout, and the recovery ladder, worked through on the Example Labs field sensor hub.

Picture the firmware lead whose devices freeze in the field. The board works for hours on the bench, then a unit on a customer's wall stops answering, no HardFault, no watchdog log, no pattern anyone can see from the firmware side. Or a bad update bricks the fleet, and the only fix is a truck roll to every unit.

Or a device comes back from a customer with a fault nobody can reproduce, because the fault only happens after three weeks of uptime, or only when the supply sags, or only when a sensor holds a bus line low after a brownout. The product is not broken in the way a broken product is broken. It is alive enough to misbehave and dead enough to stop making sense, and the fix is not a better debugger.

It is a design that assumes the device will hang, and recovers instead of bricking.

Example Labs is a made-up company we use for worked examples.

A watchdog is the last line of defence that turns a bricked unit on a customer's wall into a two-second reboot. The trouble is that many watchdogs are configured in a way that hides faults rather than recovering from them, and a reset that returns the device to the state that hung it produces a boot loop, which is worse than a freeze. This article is the watchdog, the brownout, and the recovery ladder, the three layers that keep a device alive in the field, worked through on the Example Labs field sensor hub.

The words, in plain English

Watchdog. A hardware counter that resets the processor unless the firmware "kicks" it, also called feeding or refreshing, before it expires. It is the last line of defence in embedded firmware.

Kick / feed / refresh. The act of reloading the watchdog's counter before it reaches its limit, so the reset does not happen. On an STM32 the independent watchdog is refreshed by writing 0xAAAA to its key register.

Timeout. The time the watchdog gives the firmware to kick it. If the kick does not arrive within the timeout, the watchdog assumes something has gone wrong and resets the microcontroller.

Reset. The processor restarting from the beginning, loading two words from the vector table (the initial stack pointer and the Reset Handler address) and starting to execute again. Everything between that hardware moment and main() is firmware: clock tree bring-up, peripheral initialisation, and the C-runtime preparation.

Brownout. A supply voltage that falls slowly, sags under load, rebounds, or chatters around a threshold, present, but below the range where one or more components are guaranteed to operate correctly. A clean loss of power is obvious. A brownout is messier.

BOR / BOD. Brown-out reset / brown-out detect, the mechanism that watches the supply and holds or forces the device into reset when the voltage falls through a configured threshold. The point is to fail earlier and more cleanly.

Reset-cause register. The register that latches the cause of the last reset, power-on, brown-out, watchdog, software, external pin, so the firmware can read it at boot, log it, and then clear it. On an STM32 it is RCC_CSR.

Boot loop. A device that resets, boots, hangs in the same state that hung it, resets again, and never completes a single measurement, spending its battery on reboots instead of work. A boot loop is worse than a freeze, because a frozen device can at least be power-cycled once and understood; a looping device looks like intermittent firmware failure.

Safe state. The state the device returns to after a recovery, the outputs off, the motor stopped, the radio quiet, the previous known-good firmware running, instead of the state that hung it.

Supervisor. The task (or the external IC) that owns the watchdog: it checks that every critical task has made progress, and only then kicks. The supervisor is the thing that decides what "alive" means.

Windowed mode. A watchdog that rejects a kick that arrives too early as well as too late, it catches runaway code that loops through the kick instruction.

LSI. The low-speed internal RC oscillator, around 32 kHz, that runs the independent watchdog on its own clock, independent of the main clock tree, so it keeps running even if the main clock fails.

IWDG / WWDG. The STM32's two watchdogs: the independent watchdog (IWDG), clocked from the LSI, and the window watchdog (WWDG), clocked from the APB bus clock.

I2C / SDA / SCL. The two-wire bus the hub's sensors sit on: SDA carries the data, SCL carries the clock, and both lines are open-drain with pull-up resistors, a device releases a line rather than driving it high.

Heap fragmentation. The free memory of a device splitting into smaller and smaller pieces over months of uptime, until no single piece is big enough for the allocation the firmware needs, and the allocation fails, or the firmware hangs waiting for memory that will never come.

Bit flip. A single bit in memory changing state, from a soft memory failure, or an EMI discharge into a memory device or its interface, so the program starts executing bad code, or a variable takes a value it should never have.

The watchdog: a counter on its own clock

A watchdog is a counter running off its own clock that resets the processor when it reaches its limit. Firmware must kick it periodically, and the kick is the proof. The question a good watchdog answers is not "is the CPU running?" but "is the device still doing its job?", and that means the feed should depend on evidence that each critical activity has made progress since the last feed.

A watchdog does not know whether your program is correct. It only knows one thing: has the kick happened before the timeout? So the real question is: which condition does the kick prove? If the kick happens only when the important work has been done, the watchdog checks that the work is being done.

If it happens regardless, it checks nothing.

A watchdog cannot detect a fault instantaneously. By definition, the watchdog must reach the end of a predetermined time interval before it resets the processor, which is why the shortest possible timeout interval should be selected that still expires before the watchdog can correct an out-of-control condition. And a watchdog is particularly well-suited to detecting bit errors: momentary bit errors are caused by interruptions, including soft memory failures and EMI discharges into memory devices and their interfaces, and when the processor begins executing that bad code, it will not properly execute the code that services the watchdog, so after the timeout interval, the watchdog causes a processor reset, and in a properly designed system that reset corrects the original error.

Two kinds: internal and external

Two kinds matter. The internal watchdog is a peripheral inside the microcontroller, free, and configured in a few lines. Its weakness is that it shares the die: a clock failure or a latched-up core can take the watchdog with it. The external watchdog is a separate supervisor IC costing $0.30 to $1.50 that holds the reset line and cannot be disabled by confused firmware. For anything unattended, safety-relevant, or expensive to visit, fit the external part. It is the cheapest reliability insurance on the bill of materials.

The internal timer saves money, but can be affected by runaway code. The external watchdog has a separate clock source that gives it better reliability, and if properly configured it cannot be bypassed or disabled by runaway code. The choice is not a cost decision, it is a distance decision: how far away is the device when it freezes, and what does it cost to visit it?

A device on a bench is a different product from a device on a customer's wall, and the $0.30–$1.50 supervisor IC is the difference between a two-second reboot and a truck roll.

The timeout: long enough to fit, short enough not to notice

Timeout selection decides whether the watchdog helps or hurts. Long enough that the slowest legitimate loop iteration, including a flash write or a blocking radio call, comfortably fits. Short enough that a customer does not notice.

One to eight seconds covers most products; a motion-control loop may need tens of milliseconds. The slowest legitimate operation in the system, typically a flash erase, a modem attach, or an OTA write, sets the floor. Instead of lengthening the timeout to fit the slowest operation, break the work into chunks and feed between them, or have the task check in with a "still working" flag.

The timeout is not the number you programmed. On an STM32, the independent watchdog runs off the LSI, which is nominally 32 kHz but can vary from approximately 17–47 kHz across operating temperature and supply voltage ranges (STM32F4, reference manual RM0090). The timeout formula is timeout = (prescaler × reload) / LSI frequency, so a prescaler of 64 and a reload of 500 gives a nominal 1,000 ms at 32 kHz, but 681 ms at 47 kHz and 1,882 ms at 17 kHz, a 2.76× variation between the two ends.

The timeout is not 1 second. It is 0.68–1.88 seconds, and the slowest legitimate iteration must fit inside 681 ms, not 1,000 ms. The demo works this through on the hub.

Windowed mode: too early is also a fault

A window watchdog rejects a kick that arrives too early as well as too late, which catches runaway code that happens to loop through the kick instruction. Use it where the part supports it. Why punish an early kick?

Because a program that runs too fast is also broken: a loop that skips its work because of a corrupted variable, or code that jumped into a tight loop that happens to contain a kick. Functional safety standards such as ISO 26262 and IEC 61508 recognise this, and safety-related designs often use windowed or external watchdog ICs with this property. A windowed timer enhances the validation of proper operation by signalling a reset when the watchdog is serviced either too slowly or too quickly, and it can also reset the processor if the crystal oscillator begins to run the wrong frequency.

On the STM32, the window watchdog (WWDG) is clocked from the APB bus clock, and software must refresh the counter within a specific time band, not before and not after. Refresh too early, too late, or not at all, and the system resets. The counter is 7 bits, counting down from 0x7F toward 0x40; when bit T6 clears, the counter passes below 0x40, the WWDG resets the microcontroller.

The valid kick window is W[6:0] ≥ counter > 0x40, where W is the upper window limit set in the WWDG configuration register. If firmware kicks while the counter is still above W, that is also treated as a fault, and the WWDG resets the microcontroller immediately. The early wakeup interrupt fires at counter 0x40, one count before reset at 0x3F, giving the firmware a last chance to save debugging information before the reset.

The WWDG is typically used alongside the IWDG rather than instead of it: the IWDG provides a long-timeout safety net that survives low-power modes, and the WWDG provides tight timing enforcement on the main control loop.

Never kick from a timer interrupt

This is the single most common mistake in production firmware. An interrupt keeps firing while the main application is deadlocked, so the watchdog is faithfully serviced by a dead product. It is tempting to kick the watchdog in a periodic timer interrupt: it is regular, it is simple, and the watchdog never fires during development.

That last point is the problem. A main loop stuck waiting for a flag that never comes, a deadlock between two tasks, a driver spinning on a status bit after a peripheral error: in all those cases interrupts keep working. A watchdog kicked from an ISR only catches the case where the CPU itself has stopped, which is the rarest one.

Kick where the work is proven. In a super-loop, the minimum is to kick once per loop, after every job has run, so the loop stops, and the watchdog resets the device. With an RTOS, several tasks run independently, and one of them can starve while the others are fine.

The pattern is a watchdog task, a supervisor, that collects a check-in from every monitored task and only kicks when all of them have reported. Each task proves it went through its loop, and the hardware watchdog timeout is set longer than the check period (for example, a 100 ms check and a 500 ms timeout) so that normal jitter never triggers a reset. If a task misses, the supervisor records which one into memory that survives reset, then deliberately stops feeding, because the supervisor's job is to let the watchdog fire, not to hide the fault.

The STM32 pair: IWDG and WWDG

The STM32 family is a good example, because it carries both kinds. The independent watchdog (IWDG) runs from its own low-speed internal oscillator (around 32 kHz). It keeps running even if the main clock fails, and once started it cannot be stopped by software.

That is what you want for a last-resort reset. Since its clock is an independent 32-kHz low-speed internal RC oscillator (LSI), it remains active even if the main clock fails, and once enabled, it forces the activation of the low-speed internal oscillator, and it can only be disabled by a reset. The timeout range is from 125 microseconds to 32 seconds (the range depends on the device family and the prescaler/reload, the STM32L4's maximum is roughly 30 seconds).

It is possible to automatically enable the independent watchdog after a system reset thanks to the device option bits, and in hardware mode, after every system reset, the watchdog automatically loads the down-counter with 0xFFF and starts to count down. Once running, the IWDG cannot be stopped, and it remains active in Stop and Standby modes. Safety standards like IEC 61508 (industrial) and ISO 26262 (automotive) explicitly require independent hardware watchdog timers for any system where failure could cause harm.

The window watchdog (WWDG) runs from the main clock and adds a lower limit: a kick that comes too early also causes a reset. A window watchdog only accepts the kick inside a time window after the previous one. The two are used together: the IWDG is the long-timeout safety net that survives low-power modes and a main-clock failure, and the WWDG is the tight timing enforcement on the main control loop.

And the enablement matters as much as the configuration: enable it at boot, in hardware if possible. Many parts offer a fuse or option byte that starts the watchdog before the first instruction, closing the window where a corrupt image never enables it.

The brownout: stranger than a blackout

A clean loss of power is obvious. A supply that falls slowly, sags under load, rebounds, chatters around a threshold, or powers different rails at different times can leave electronics alive enough to misbehave and dead enough to stop making sense. Undervoltage is not merely "less power."

It can move digital and analogue subsystems through undefined operating regions where timing, memory, buses, clocks, peripherals, and reset logic stop sharing the same assumptions. A microcontroller may execute instructions below its recommended minimum voltage. That does not mean it is executing them correctly.

The CPU core, flash array, SRAM, clock tree, ADC, GPIO, serial peripherals, and external components can each have different undervoltage sensitivities, which creates one of the ugliest diagnostic categories in embedded work: plausible continued operation outside the guaranteed envelope. The board appears alive.

The result can look less like "power failed" and more like a software bug, damaged peripheral, random bus fault, corrupted configuration, impossible sensor value, or a device that only recovers after somebody physically removes the battery. Brown-out and power-on reset problems are behind many "random" MCU resets: the board runs for hours on the bench, then restarts in the field without a HardFault, without a watchdog log, and without any pattern you can see from the firmware side, and the usual first guess is a software bug.

Two details already point away from software: a HardFault or a watchdog reset leaves a trace if the firmware is written to leave one, and here there is none; and the resets cluster, which suggests something external that repeats, such as a radio transmitting.

The point of brownout reset is to fail earlier and more cleanly. Many microcontrollers include brownout reset, sometimes called BOR, BOD, low-voltage detect, voltage monitor, or a similar vendor-specific name. The mechanism watches the supply and holds or forces the device into reset when voltage falls through a configured threshold.

The philosophy is simple: if the processor cannot be trusted below a certain supply voltage, stop executing before reaching that region. The BOR module generates a device reset when a brown-out condition occurs to protect against code misexecution, and the BOR status bit is set to indicate that a BOR has occurred, and the BOR circuit continues to operate while in Sleep or Idle mode, resetting the device should VDD fall below the threshold.

A useful brownout implementation has more than a threshold. It also needs response time and hysteresis. Response time determines how quickly the reset logic reacts to a sag.

Hysteresis helps prevent rapid reset-release-reset chatter when the supply hovers around the threshold. Many devices also provide a power-up delay or oscillator-start delay so reset is not released the instant voltage first crosses the line. These details matter because a slow, noisy power ramp can cross the threshold many times, and a supervisor that releases reset too eagerly may create a boot loop that looks like intermittent firmware failure.

Recovery is more than a reset

A reset is not a recovery. Recovery is a ladder, and the reset is only the first rung. The goal is not to produce more reboots.

It is to recover the smallest failed layer, preserve enough evidence to diagnose the fault, and return the product to a safe operating state. Resetting the board for every fault creates a different reliability problem, and the recovery hierarchy is what decides which response each failure gets: a worker can time out a transaction, discard a damaged message, reopen a port, or recreate a client connection; a crashed process can be restarted; a kernel that is no longer scheduling gets a hardware watchdog reset, with an external power cycle if needed; a power rail latched incorrectly gets a board-level reset.

The first step is to read the reset-cause register, log it, and return to a safe state. Every STM32 latches the cause of the last reset in a register, RCC_CSR on the F4 family. The flags stay set until software writes the RMVF bit, so the first job of the startup code is to read them, store them, and clear them, before SystemInit() and before the HAL, which some projects let clear the flags.

A watchdog reset means firmware hung; a brown-out reset means the supply dipped; a power-on reset is a clean cold boot. The MCU records which source fired in a status register that firmware should read once at boot and then clear, and the flags are not exclusive, so reading only one of them leads to the wrong conclusion. The log itself lives in a .noinit section of SRAM, excluded from the zero-fill in the startup code: SRAM keeps its content across a reset as long as the supply did not fall below the retention voltage, and a magic word detects the case where it did.

For a field log that must survive full power loss, the entries are copied to flash or a backup register later in the boot, when the supply is stable.

The second step is the safe state. A reset that returns the device to the state that hung it produces a boot loop, which is worse than a freeze, and the boot loop is the failure the recovery ladder exists to break. The ladder, on the hub: the first watchdog reset is logged and the device reboots; the second within an hour boots the previous slot (the #11 trial-then-confirm, a bad image boots as "trial", never confirms, and the bootloader reverts to Slot A, so a bad build costs one reboot instead of one scrapped unit); the third stays in safe mode and signals for help.

The device that loops is the device that never completes a single measurement and spends its battery on reboots, and the counter in flash is what breaks the loop, because the counter is the memory the reset cannot erase.

The field-only failures

Three failures happen only in the field, and all three are the reason the watchdog exists. Bit flips: a system with a watchdog is particularly well-suited to detecting bit errors, because momentary bit errors are caused by interruptions, including soft memory failures and EMI discharges into memory devices and their interfaces, and when the processor begins executing that bad code, it will not properly execute the code that services the watchdog.

Heap fragmentation after months of uptime: every connected device will eventually hang in the field, a radio driver waits forever for an interrupt that never comes, a heap fragments after three weeks, a sensor holds the I²C bus low. The heap fragments because the allocations and frees over weeks of uptime split the free memory into pieces too small to use, and the allocation that fails is the one the firmware was waiting on.

An I2C sensor holding SDA low after a brownout: the board works for hours, then the microcontroller resets, maybe from a watchdog, a brownout, or a firmware update, and after the reboot every I2C read fails. The bus reads low. Power-cycling the whole board fixes it, so the bug is hard to reproduce and easy to ship.

What actually happens is a protocol state, not a broken part. I2C is a controller-driven protocol: a target only changes the data line while the controller is clocking it. During a read, the target drives each data bit onto SDA while SCL is low, and the controller releases SDA to let the target talk.

Now reset the microcontroller halfway through a byte. The controller stops clocking. The target is still in the middle of the transaction, and if the bit it was about to send is a zero, it holds SDA low and waits for the next clock edge.

The controller comes back up, sees SDA low, and cannot generate a START condition, because START needs SDA to fall while SCL is high. Most I2C peripherals report a busy bus or an arbitration loss and give up. **The target is not broken.

It is waiting for clocks that never came.** Nothing in the microcontroller's reset knows the target exists, and the target's own power never dropped.

The standard recovery is the bus clear, from NXP's I²C specification (UM10204, section 3.1.16): if SDA is stuck low, the controller sends nine clock pulses on SCL, and the target that was holding the line should release it within those nine clocks. Nine is not arbitrary: a byte is eight data bits plus an acknowledge bit, so nine clocks walk any target out of whatever byte it was in the middle of.

After the bus is released, the controller sends a STOP condition so every target returns to idle. If that does not work, use a hardware reset of the targets if the design has one, or cycle their power. And if SCL is stuck low, clock pulses cannot fix it, you cannot clock a line that someone else is holding low, so the preferred route is a hardware reset of the bus, if the design provides a reset signal.

Product firmware must include this recovery code and a timeout on every I2C operation.

The update the watchdog has to survive, and the image it has to trust

The watchdog does not live alone. It lives with the update and the image, and the three have to be designed together. The update, from #11: the OTA firmware update with A/B dual-slot partitioning, write to the inactive slot, verify, flip the boot pointer atomically, and trial-then-confirm, so the new image must confirm health (network up, sensors reading, watchdog happy) within a few boots, or the bootloader auto-reverts.

The watchdog is part of that confirmation: the new image must keep the watchdog happy, and the watchdog must be fed during the OTA write, because the OTA write is the slowest legitimate operation in the system and it sets the floor for the timeout. Break the write into chunks and feed between them, or the watchdog will reset the device in the middle of its own update, and the reset will return the device to the trial image, which will try the update again, which will reset again.

That is a boot loop with a radio attached.

The image, from #28: the secure boot chain of trust, ROM to bootloader to app, with the image signed (ECDSA P-256 or RSA-2048) and an anti-rollback monotonic counter. The watchdog has to trust the image it resets into: a watchdog that resets the device into an untrusted image has recovered the device into a compromise. Enable the watchdog at boot, in hardware if possible, so a corrupt image cannot skip it, and the bootloader is the thing that enables it, before the first instruction of the application runs.

And the tasks, from #13: the RTOS versus bare metal, the tasks the watchdog has to watch. In a super-loop, the minimum is to kick once per loop, after every job has run. With an RTOS, several tasks run independently, and one of them can starve while the others are fine, which is why the supervisor task exists, collecting a check-in from every monitored task and only kicking when all of them have reported.

The watchdog is the last line of defence, and the supervisor is the thing that decides what the line defends.

DEMO

The demo device, as always: the Example Labs field sensor hub, 100 × 60 × 40 mm, a 4-layer board, a BLE radio at 2.4 GHz, a plastic enclosure, an STM32H7 (512 kB SRAM, 2 MB flash, 48 MHz clock), a firmware image of 384 KiB (from #28), and a CR2032 cell (230 mAh, 2,484 J). The arithmetic was run in Python before writing.

Step 1, the timeout is not 1 second. The IWDG is configured with a prescaler of 64 and a reload of 500, which at the nominal LSI of 32 kHz gives a timeout of 1,000 ms. But the STM32F4 LSI varies from 17–47 kHz across temperature and supply voltage, so the actual timeout is 681 ms at 47 kHz and 1,882 ms at 17 kHz, a 2.76× variation between the two ends. The timeout is not 1 second. It is 0.68–1.88 seconds, and the slowest legitimate iteration must fit inside 681 ms, not 1,000 ms. A design that sizes its timeout against the nominal 1,000 ms has sized it against a number the hardware does not guarantee.

Step 2, the slowest legitimate iteration. The hub's main loop, in the worst case (the article's own construction, labelled): a sensor read at 40 ms, a BLE transmit at 60 ms, and a flash log write at 20 ms, 120 ms in total. Against the shortest timeout of 681 ms, the margin is 5.7×; against the nominal 1,000 ms, it is 8.3×. The margin is comfortable, and the margin is the point: the timeout is chosen against the shortest timeout the hardware can produce, not the nominal, and the slowest legitimate iteration is the number that has to fit inside it. If the flash log write grew to 600 ms, a sector erase, unchunked, the margin against the shortest timeout would be 1.1×, and the watchdog would fire on a legitimate operation. That is why the slowest operation is chunked, and the task checks in with a "still working" flag between chunks.

Step 3, the window. The WWDG is configured with a window of 0x5F (the article's own choice, labelled). The counter runs from 0x7F (127) down to 0x40 (64), and resets below 0x3F (63). The valid kick band is counter 0x5F (95) down to 0x41 (65), 31 counts out of 63, or 49% of the countdown. A kick above 0x5F (too early) resets the device; a kick at or below 0x40 (too late) resets the device; no kick at all resets the device. The early wakeup interrupt fires at 0x40, one count before reset at 0x3F, giving the firmware a last chance to save debugging information. The window catches the failure the IWDG cannot: runaway code that loops through the kick instruction, refreshing the IWDG as fast as it can, because the WWDG rejects a kick that arrives too early, and the runaway loop's kicks arrive too early by definition.

Step 4, the supervisor. The hub's tasks and their deadlines (the article's own construction, labelled): the sensor loop, 2 seconds; the BLE radio, 60 seconds for a blocking attach; the OTA write and the flash erase, chunked, with a "still working" check-in between chunks. The supervisor checks every 100 ms and feeds the hardware watchdog only when every registered task has reported in within its deadline. The hardware timeout is 1 second nominal (0.68–1.88 seconds actual), so the check period is 10× shorter than the timeout, normal jitter never triggers a reset. If a task misses, the supervisor records which one into memory that survives reset, then deliberately stops feeding. The supervisor's job is to let the watchdog fire, not to hide the fault, and the record of which task missed is the evidence the field log carries home.

Step 5, the brownout and the bus. The hub's supply is a CR2032, and the BLE radio's transmit current sags it, the brownout that the bench supply never shows, because the bench supply does not sag. The BOR is configured with a threshold above the voltage at which the processor cannot be trusted, with hysteresis so the supply hovering around the threshold does not chatter the reset line, and with a power-up delay so reset is not released the instant voltage first crosses the line. And the recovery the reset has to perform: after a watchdog or brownout reset, the firmware reads RCC_CSR before anything clears it, copies the cause into the .noinit log, and then runs the I2C bus clear, nine clock pulses on SCL, then a STOP, because the sensor may be holding SDA low, waiting for clocks that never came. The reset alone does not fix the bus. The bus clear does.

Step 6, the boot loop, and the ladder that breaks it. A 2-second boot that hangs at 1.5 seconds, then resets, then hangs again: the device never completes a single measurement, and the battery is spent on reboots. The ladder (the article's own construction, labelled): the first watchdog reset is logged and the device reboots; the second within an hour boots the previous slot, the #11 trial-then-confirm, so a bad build costs one reboot instead of one scrapped unit; the third stays in safe mode and signals for help. The counter lives in flash, because the counter is the memory the reset cannot erase, and the safe state is the outputs off, the radio quiet, and the previous known-good firmware running, instead of the state that hung it.

Step 7, the cost. The external supervisor IC costs $0.30–$1.50 per unit. On the hub's 1,000-unit fleet (from #11), that is $300–$1,500 for the fleet, against one truck roll per bricked unit, one scrapped unit per bad update, and one lost customer per device that freezes on their wall. The cheapest reliability insurance on the bill of materials is not a metaphor. It is a line item, and it is the smallest line item on the board.

The decision test: five questions

A firmware lead or founder whose devices freeze in the field can answer five questions in an afternoon. If the answer to three of them is no, the product has no recovery design, and the fix is the ladder, not the debugger.

  1. Is the watchdog kicked from evidence, or from habit? If the kick sits in a timer interrupt, the watchdog is faithfully serviced by a dead product, and it only catches the case where the CPU itself has stopped, which is the rarest one. The kick has to sit where the work is proven: at the end of the main loop, after every job has run, or in a supervisor that has checked every task's flag.
  2. Is there an external supervisor, and is the watchdog enabled in hardware? If the watchdog is internal only, a clock failure or a latched-up core can take it with it. If the watchdog is enabled in software, a corrupt image can skip it. The external IC costs $0.30–$1.50, and the option byte starts the watchdog before the first instruction.
  3. Is the timeout sized against the shortest timeout the hardware can produce? If the timeout was sized against the nominal LSI, the actual timeout is 0.68–1.88 seconds, not 1 second, and the slowest legitimate iteration has to fit inside the shortest timeout, with margin. If the slowest operation is a flash erase or an OTA write, it has to be chunked, and the task has to check in between chunks.
  4. Does the firmware read the reset-cause register, log it, and return to a safe state? If the reset cause is never read, the field log has no answer for why the device rebooted, and if the reset returns the device to the state that hung it, the device loops. The cause is read before anything clears it, the log survives the reset, and the recovery ladder breaks the loop.
  5. Does the recovery cover the bus, the supply, and the update? If the firmware resets without clearing the I2C bus, the sensor is still holding SDA low, waiting for clocks that never came. If the BOR has no hysteresis, the supply hovering around the threshold chatters the reset line. If the watchdog is not fed during the OTA write, it resets the device in the middle of its own update. The recovery is the ladder, not the reset.

Objections

"The watchdog never fires in development, so we don't need it." The watchdog never fires in development because the bench supply does not sag, the heap has not fragmented after three weeks, and the sensor has not held SDA low after a brownout. The field-only failures are the ones the watchdog exists for, and the fact that it never fires on the bench is the reason it has to be tested by fault injection, not by waiting. Deliberately hang a task, deliberately hold the bus low, deliberately sag the supply, and watch the recovery ladder run.

"We'll just add a bigger battery / a better regulator." A bigger battery does not fix a heap that fragments after three weeks, and a better regulator does not fix a sensor holding SDA low after a brownout. The brownout is a supply problem, and the BOR is the supply's answer, but the hang is a firmware problem, and the watchdog is the firmware's answer. The two layers are not substitutes. The device needs both.

"A reset is enough, the device comes back up." A reset is enough when the state that hung the device is gone. It is not enough when the state is still there, the sensor still holding SDA low, the heap still fragmented, the bad image still in the trial slot. A reset that returns the device to the state that hung it produces a boot loop, which is worse than a freeze. Recovery is the reset, the log, the safe state, and the ladder, and the ladder is what breaks the loop.

"The external supervisor IC costs too much." It costs $0.30–$1.50 per unit, $300–$1,500 for a 1,000-unit fleet, against one truck roll per bricked unit. The internal watchdog is free, and it shares the die with the thing it is supposed to watch. For anything unattended, safety-relevant, or expensive to visit, the external part is the cheapest reliability insurance on the bill of materials.

"Our code is well-tested, it doesn't hang." Every connected device will eventually hang in the field: a radio driver waits forever for an interrupt that never comes, a heap fragments after three weeks, a sensor holds the I²C bus low. The question is not whether the code hangs. The question is what the device does when it does, and whether it recovers, tells you why, and returns to a safe state, or whether it freezes on a customer's wall until someone visits it.

Sources

  1. Projects House, "Watchdog Timers and Firmware Recovery in Embedded Products"
    What we used

    A watchdog is a counter running off its own clock that resets the processor when it reaches its limit; the internal watchdog is a peripheral inside the microcontroller, free, and configured in a few lines, and its weakness is that it shares the die (a clock failure or a latched-up core can take the watchdog with it); the external watchdog is a separate supervisor IC costing $0.30 to $1.50 that holds the reset line and cannot be disabled by confused firmware, and for anything unattended, safety-relevant, or expensive to visit, fit the external part, it is the cheapest reliability insurance on the bill of materials; timeout selection (long enough that the slowest legitimate loop iteration, including a flash write or a blocking radio call, comfortably fits; short enough that a customer does not notice; one to eight seconds covers most products, a motion-control loop may need tens of milliseconds); windowed mode (a window watchdog rejects a kick that arrives too early as well as too late, which catches runaway code that happens to loop through the kick instruction); never kick from a timer interrupt (the single most common mistake in production firmware, an interrupt keeps firing while the main application is deadlocked, so the watchdog is faithfully serviced by a dead product); kick from evidence (in an RTOS, have each critical task set a flag on each successful cycle, and let one supervisor check that every flag has been set within its expected period before it kicks); enable it at boot, in hardware if possible (a fuse or option byte that starts the watchdog before the first instruction, closing the window where a corrupt image never enables it)

  2. Amansour, "Watchdog Timers Done Right in Embedded Firmware"
    What we used

    A watchdog timer is the last line of defence in embedded firmware: a hardware counter that resets the microcontroller unless the software "kicks" it regularly; a watchdog doesn't know whether your program is correct, it only knows one thing (has the kick happened before the timeout?), so the real question is which condition the kick proves; the classic mistake (kicking from a timer interrupt, a main loop stuck waiting for a flag that never comes, a deadlock between two tasks, a driver spinning on a status bit after a peripheral error: in all those cases interrupts keep working, and a watchdog kicked from an ISR only catches the case where the CPU itself has stopped, which is the rarest one); kick where the work is proven (in a super-loop, kick once per loop after every job has run; with an RTOS, a watchdog task collects a check-in from every monitored task and only kicks when all of them have reported; the hardware watchdog timeout is set longer than the check period, for example 100 ms check and 500 ms timeout, so normal jitter never triggers a reset); the IWDG runs from its own low-speed internal oscillator (around 32 kHz), keeps running even if the main clock fails, and once started it can't be stopped by software; the WWDG runs from the main clock and adds a lower limit (a kick that comes too early also causes a reset); why punish an early kick (a program that runs too fast is also broken, a loop that skips its work because of a corrupted variable, or code that jumped into a tight loop that happens to contain a kick; ISO 26262 and IEC 61508 recognise this)

  3. Maxim, Application Note 4229, "Comparison of Internal and External Watchdog Timers"
    What we used

    A watchdog timer forces an embedded microprocessor or microcontroller to reset (hardware reset) in response to an invalid software state; the internal timer saves money but can be affected by runaway code, while an external WDT has a separate clock source that gives it better reliability and, if properly configured, cannot be bypassed or disabled by runaway code; a system with a WDT is particularly well-suited to detecting bit errors (momentary bit errors caused by interruptions, including soft memory failures and EMI discharges into memory devices and their interfaces); a WDT cannot detect a fault instantaneously, by definition, the WDT must reach the end of a predetermined time interval before it resets the processor

  4. STMicroelectronics, "STM32 independent watchdog (IWDG)"
    What we used

    Presentation, the independent watchdog is used to detect and resolve malfunctions due to software failures; since its clock is an independent 32-kHz low-speed internal RC oscillator (LSI), it remains active even if the main clock fails; once enabled, it forces the activation of the low-speed internal oscillator, and it can only be disabled by a reset; the timeout range is from 125 microseconds to 32 seconds; it is possible to automatically enable the independent watchdog after a system reset thanks to the device option bits; once running, the IWDG cannot be stopped, and it remains active in Stop and Standby modes; in hardware mode, after every system reset, the watchdog automatically loads the down-count with 0xFFF and starts to count down; it generates a reset when the programmed timeout value elapses, or when a watchdog refresh occurs outside a programmed time-window

  5. EmbeddedSoft, "Watchdog Timer Design for Reliable Embedded Systems"
    What we used

    The IWDG is driven by the Low-Speed Internal (LSI) oscillator, nominally around 32 kHz on many STM32 devices, and this clock independence is its defining trait (even if the main oscillator fails or the PLL locks up, the IWDG continues counting); timeout (ms) = (prescaler × reload) / LSI frequency (kHz), with LSI = 32 kHz, prescaler = 64, reload = 500 giving a timeout of 1 second; once enabled, the IWDG cannot be disabled except by a system reset; the WWDG is clocked from an APB bus clock, and software must refresh the counter within a specific time band, not before and not after (refresh too early, too late, or not at all, and the system resets); the WWDG window (counter 0x7F max, window threshold 0x40, early wakeup interrupt at 0x40, reset threshold 0x3F; refresh too early (counter > W[6:0]) => reset; refresh too late (counter < 0x40) => reset); the WWDG's Early Wakeup Interrupt fires at counter 0x40, one count before reset at 0x3F; this catches failures a basic timeout-only IWDG cannot (a runaway loop refreshing the IWDG as fast as it can, or an ISR that keeps firing while the main application is dead)

  6. Electronics Design, "Watchdog Timer Explained: IWDG, WWDG, and Kick Strategy"
    What we used

    The STM32F4 LSI frequency is nominally 32 kHz but can vary from approximately 17–47 kHz across operating temperature and supply voltage ranges (RM0090); timeout_ms = (1000 × prescaler × (reload + 1)) / LSI_Hz; prescaler options 4, 8, 16, 32, 64, 128, 256; reload register 12-bit (0–4095); once started, the IWDG cannot be stopped in software (there is no disable register); the WWDG has a 7-bit down-counter (T[6:0]) that decrements from a loaded value toward 0x40, and when bit T6 clears (counter passes below 0x40) the WWDG resets the MCU; the upper window limit W[6:0] is set in the WWDG_CFR register, and if firmware kicks while the counter is still above W, the WWDG resets the MCU immediately; the valid kick window is W[6:0] ≥ counter > 0x40; the WWDG is typically used alongside the IWDG rather than instead of it (the IWDG provides a long-timeout safety net that survives low-power modes; the WWDG provides tight timing enforcement on the main control loop); the reset-cause register on STM32 is RCC_CSR

  7. Cyberdelia Engineering, "Brownouts Cause Stranger Failures Than Blackouts"
    What we used

    A clean loss of power is obvious, while a supply that falls slowly, sags under load, rebounds, chatters around a threshold, or powers different rails at different times can leave electronics alive enough to misbehave and dead enough to stop making sense; a microcontroller may execute instructions below its recommended minimum voltage, and that does not mean it is executing them correctly (the CPU core, flash array, SRAM, clock tree, ADC, GPIO, serial peripherals, and external components can each have different undervoltage sensitivities, plausible continued operation outside the guaranteed envelope); the point of brownout reset is to fail earlier and more cleanly (if the processor cannot be trusted below a certain supply voltage, stop executing before reaching that region); a useful brownout implementation has more than a threshold, it also needs response time and hysteresis (hysteresis helps prevent rapid reset-release-reset chatter when the supply hovers around the threshold), and many devices provide a power-up delay or oscillator-start delay so reset is not released the instant voltage first crosses the line; a supervisor that releases reset too eagerly may create a boot loop that looks like intermittent firmware failure

  8. Microchip, "12.3.6 Brown-out Reset (BOR)"
    What we used

    The BOR module generates a device Reset when a brown-out condition occurs to protect against code misexecution, based on an internal voltage reference circuit that monitors VDD; the BOR status bit (RCON[1]) is set to indicate that a BOR has occurred; the BOR circuit continues to operate while in Sleep or Idle mode and resets the device should VDD fall below the BOR threshold voltage

  9. EmbeddedInterviewLab, "Boot, vector table, and clock bring-up"
    What we used

    Multiple reset sources (power-on, brown-out, watchdog, software, external pin), each leaving a status flag for diagnostics; the status registers (RCC_CSR on STM32) record the reset source, and firmware should read once at boot and clear; a watchdog reset means firmware hung, a brown-out reset means the supply dipped, a power-on reset is a clean cold boot; in production firmware you should always check the reset-cause register early in SystemInit() or right after, copy it to RAM for telemetry, and then clear it

  10. Occasio005, "Brown-Out and Power-On Reset Problems: Debugging Random MCU Resets"
    What we used

    Brown-out and power-on reset problems are behind many "random" MCU resets, and the board runs for hours on the bench, then restarts in the field without a HardFault, without a watchdog log, and without any pattern you can see from the firmware side; every STM32 latches the cause of the last reset in a register, RCC_CSR on the F4 family, and the flags stay set until software writes the RMVF bit, so the first job of the startup code is to read them, store them, and clear them, before SystemInit() and before the HAL, which some projects let clear the flags; SRAM keeps its content across a reset as long as the supply did not fall below the retention voltage, and a magic word detects the case where it did; for a field log that must survive full power loss, copy the entries to flash or a backup register later in the boot, when the supply is stable; the flags are not exclusive, and reading only one of them leads to the wrong conclusion

  11. Miloperovic, "Firmware watchdog design that recovers devices in the field"
    What we used

    Every connected device will eventually hang in the field: a radio driver waits forever for an interrupt that never comes, a heap fragments after three weeks, a sensor holds the I²C bus low; a firmware watchdog is the last line of defence that turns a bricked unit on a customer's wall into a two-second reboot; the question a good watchdog answers is not "is the CPU running?" but "is the device still doing its job?", and the feed should depend on evidence that each critical activity has made progress since the last feed; the supervisor pattern (a supervisor task that owns the hardware watchdog, every critical task registers with it and must check in within its own deadline, perhaps 2 seconds for a sensor loop, 60 seconds for a cellular task that legitimately blocks on network attach, and the supervisor feeds the hardware only when all registered tasks have reported in); if a task misses, the supervisor records which one into memory that survives reset, then deliberately stops feeding; the slowest legitimate operation in the system, typically a flash erase, a modem attach or an OTA write, sets the floor; on an STM32L4 the IWDG maximum timeout is roughly 30 seconds

  12. Avontek, "Linux SBC Watchdog Design for Unattended Gateways"
    What we used

    The goal is not to produce more reboots, it is to recover the smallest failed layer, preserve enough evidence to diagnose the fault, and return the product to a safe operating state; resetting the board for every fault creates a different reliability problem; the recovery hierarchy (one application stops responding → restart that service, reboot if repeated; kernel or PID 1 no longer scheduling → hardware watchdog reset, external power cycle if needed; power rail latched incorrectly → board-level reset, power supervisor cycle)

  13. EmbeddedEra, "I2C bus stuck low after a reset"
    What we used

    The board works for hours, then the microcontroller resets, maybe from a watchdog, a brownout or a firmware update, and after the reboot every I2C read fails, the bus reads low, and power-cycling the whole board fixes it, so the bug is hard to reproduce and easy to ship; reset the microcontroller halfway through a byte, and the target is still in the middle of the transaction, if the bit it was about to send is a zero, it holds SDA low and waits for the next clock edge, and the controller comes back up, sees SDA low, and cannot generate a START condition because START needs SDA to fall while SCL is high; the target is not broken, it is waiting for clocks that never came, and nothing in the microcontroller's reset knows the target exists, and the target's own power never dropped; the standard recovery (NXP UM10204): if SDA is stuck low, the controller sends nine clock pulses on SCL, and the target that was holding the line should release it within those nine clocks, nine is not arbitrary, because a byte is eight data bits plus an acknowledge bit, so nine clocks walk any target out of whatever byte it was in the middle of, and after the bus is released the controller sends a STOP condition so every target returns to idle

  14. EmbeddedSoft, "Debugging Intermittent I2C Bus Hangs in Embedded Systems"
    What we used

    A slave may hold SDA low after a transaction due to a firmware bug in the slave state machine (missed STOP condition detection), a power glitch causing the slave to enter an undefined state with its output driver enabled, or a slave address collision; when SDA is stuck low, the master's hardware peripheral cannot generate START or STOP conditions, rendering the bus dead to standard API calls, but software can recover the bus by bit-banging clock pulses on SCL until SDA is released

  15. Example Labs figures from the cluster: the field sensor hub (100 × 60 × 40 mm, 4-layer board
    What we used

    BLE radio at 2.4 GHz, plastic enclosure, STM32H7 with 512 kB SRAM / 2 MB flash, 48 MHz clock, firmware image 384 KiB) (#28); the CR2032 cell (230 mAh, 2,484 J) (#6); the OTA design (A/B dual-slot, bootloader 64 kB + slot A 512 + slot B 512 + metadata 4 + config/calibration/logs 64 = 1,156 kB used, 892 kB spare; 512 kB image in 20.5 s over BLE at 200 kbps, delta 51.2 kB in 2.1 s; a bad build costs one reboot instead of one scrapped unit) (#11); the 1,000-unit fleet (#11); the 48 V bench pack (13S1P NMC, 48.1 V nominal, 54.6 V full, 39.0 V empty, 168 Wh) (#20/#23); common-mode radiation accounting for 80–90% of emission failures (#19)

More in software meets hardware

The Update That Cannot Brick the DeviceAn over-the-air firmware update can either extend the life of an embedded product or brick an entire fleet in the field. The difference is not the transport, it is the architecture: A/B partitions so the old firmware is never touched until the new one proves itself, signing and secure boot so the image is yours and unmodified, trial-then-confirm so a bad build reverts automatically, and a staged rollout with a kill switch. The safety costs one extra flash slot, 512 kB on a 2 MB chip, and its payoff is that a bad build costs one reboot instead of one scrapped unit, one truck roll, and one lost customer.11 minAlarm managementA plant with 400 configured alarms, 60 active at any moment, and operators who acknowledge the horn without reading the screen. EEMUA 191 says an operator can handle about one alarm every ten minutes, and at Milford Haven, two operators had to recognise, acknowledge, and act on 275 alarms in 11 minutes. The warning is a design problem, not a discipline problem.27 minDesign for TestA board that passes the bench and fails in the field is usually untestable by design, no test points, no JTAG chain, no fixture. The IPC-9252A rules, the four test methods, the coverage union, and what a fixture is worth on a real board.20 minTest It to DestructionHALT is a discovery test that steps stress far beyond specification to find weaknesses and measure operating and destruct limits, it is not a pass/fail qualification test. A complete HALT needs 4–8 units and 3–5 days; a field failure costs 100 times what a design-stage fix costs.21 minDigital twinThe prototype costs real money and weeks, and the failure it hides costs a hundred times more to find in the field than at validation. A digital twin is not a 3D model, it is a model that behaves like the physical system, fed by data from it, whose answers change the physical system. Example Labs' twin of its sensor hub predicted the rail sag that killed four of ten hubs in the field, as a band that straddled the sensor's minimum voltage, which is exactly what 'no margin' looks like before the hardware is built.21 min