OTA firmware over ESP-NOW

TL;DR

• What: A custom OTA firmware-update protocol that pushes a new firmware image from one ESP32 to another over ESP-NOW (no WiFi, no BLE), with just a radio link and a USB cable feeding the initiator.
• Outcome: Protocol, partition layout, and end-to-end flow all proven on small payloads (1 byte, 100 bytes, 1 KB) at 100% reliability. The full 742 KB firmware push reaches 97% and then crashes on a memory issue, not a protocol issue.
• Why interesting: Most OTA tutorials use WiFi and Espressif’s prebuilt esp_https_ota. Designing the protocol from scratch over a non-IP transport forced me to understand every layer (frame format, partition table, bootloader handoff, ack semantics) instead of treating OTA as a black box.
• Lesson: Knowing when to stop is a real engineering skill. The remaining 3% is a memory-management optimization, not a design problem, and I made a deliberate call to document the diagnosis and move on rather than spend another week chasing it.

What I built

The goal was simple to state and hard to execute: one ESP32 (the initiator) reads a new firmware image from my laptop over USB serial and pushes it over ESP-NOW to a second ESP32 (the responder), which writes it to a staging partition, validates it, and reboots into the new firmware. ESP-NOW is Espressif’s proprietary peer-to-peer radio protocol. It lets two ESP32s talk directly without joining a WiFi network, but it caps payloads at 250 bytes per frame, which means a 742 KB firmware turns into thousands of frames.

I designed a four-message protocol: OTA_BEGIN (carries total size and version string), OTA_CHUNK (sequence number plus up to 200 bytes of payload, leaving room for a small header), OTA_END (CRC32 over the whole image), and OTA_ACK (sent back from responder to initiator after each chunk). The responder uses ESP-IDF’s standard OTA partition layout: a factory slot, two OTA slots (ota_0 and ota_1), and a small otadata partition that the bootloader reads to decide which slot to boot next.

Most of the three days went to build-system and partition-table work, not protocol logic. ESP-IDF defaults to a 2 MB flash size and a single-app partition table; getting it to a 4 MB layout with two OTA slots required setting CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y first, before any other partition-table edits, because idf.py set-target esp32 silently resets partition-related config values. I also hit a build bug in components/esp_https_ota/CMakeLists.txt (missing bootloader_support in its REQUIRES list, fixed with a one-line patch to the IDF) and a UART driver conflict where my initiator’s uart_read_bytes() calls fought the console driver for UART0. Fixed by switching to fgetc(stdin), which goes through the VFS layer and shares the UART cleanly.

What worked, what didn’t

• Worked: Both devices boot, discover each other, and complete the protocol handshake. OTA_BEGIN and OTA_ACK round-trip cleanly. The staging partition is found, erased (~3 seconds), and written to. Payloads of 1 byte, 100 bytes, and 1 KB stage with 100% reliability across repeated runs.
• Didn’t work: The full 742 KB push reaches ~720,896 bytes written (97.1%) and then the responder crashes and reboots before OTA_END arrives.
• Diagnosis (educated guess, not confirmed): Memory pressure in the staging loop. Each iteration allocates a 4 KB receive buffer, reads into it, writes via esp_partition_write(), and frees it. After roughly 700 KB of these cycles, a subsequent allocation likely fails. I did not instrument with heap_caps_get_free_size() before stopping, so this is consistent with the symptoms but not proven. Probable fixes: move the staging loop to a dedicated FreeRTOS task with explicit yields, or switch from raw esp_partition_write() to the buffered esp_ota_write() path that Espressif uses in their own OTA examples.

What I learned

The most useful habit I picked up was logging the input, the output, and the byte count at every protocol boundary from the start, not as a debugging step after something breaks. ESP-NOW’s error messages are uniformly unhelpful (“Receive error data” can mean CRC mismatch, length mismatch, or struct-layout mismatch with no way to tell), and the only reliable diagnostic is comparing hex dumps on both ends. Adding that logging once cost an hour; not having it cost most of day one.

The bigger lesson was procedural. I went straight to a 742 KB payload because that’s the “real” test, and the resulting crash was hard to diagnose because the protocol logic, the partition setup, and the memory behavior were all unproven at the same time. A 1 KB smoke test first would have isolated the memory problem cleanly. Next OTA project: validate the smallest end-to-end path before scaling up the payload.