"Flash-verified: how our reproducible build pipeline works"
When you buy ESP32 hardware for security research, the firmware on the chip is a guess unless someone proves otherwise. Most vendors flash an image they downloaded from somewhere, ship the board, and hope. We don't think that's good enough for hardware people intend to use for lawful security work, so we built a pipeline where every unit we ship can be checked against a published artifact — by you, on your bench, without trusting anything we say in a product description.
Here's how it works, end to end. And here's where it still falls short.
Why reproducibility matters more than open source alone
Open-source firmware is necessary but not sufficient. The source sitting in a Git repository proves nothing about the binary written to your flash chip. The build that produced it could have included different toolchain versions, injected debug hooks, or a modified bootloader, and the repository would look perfectly clean. A build is reproducible when, given the same source code, build environment and build instructions, anyone can recreate bit-for-bit identical copies of the specified artifacts — the definition used by the reproducible-builds.org project, which has documented this discipline across Debian, Tor, and Bitcoin Core for years.
Reproducibility converts a trust problem into a comparison problem. Instead of asking you to believe our firmware is unmodified, we let you run the same build we ran and diff the hash. If your SHA-256 matches ours, the binary you verify is the binary we flash. If it doesn't match, something is wrong — in our build, in yours, or in the source — and that mismatch is itself useful information.
Step 1: incoming inspection and BOM freezing
Every build-to-order unit starts as parts on a bench in Melbourne. Before anything gets built, components pass through incoming inspection: we verify the ESP32 module markings against Espressif's datasheet expectations, check the flash chip's JEDEC ID and capacity, and record the lot numbers of anything RF-relevant — the SX1262 or SX1276 front ends on our mesh nodes especially. Counterfeit RF parts are common enough that this step earns its cost.
The verified bill of materials is then frozen and hashed. The BOM file, the firmware source commit, the toolchain version, and the board revision all go into a single manifest input file. From that point the unit has an identity: a serial number tied to a specific BOM hash. If we later change a supplier for a passive component, that produces a new BOM hash, and the manifest will say so. We do not silently swap parts.
This is the part of the pipeline we'd defend hardest. A reproducible build of the wrong hardware is still the wrong hardware.
Step 2: pinned builds, not vibes
Our firmware builds run in a container with pinned toolchain versions — GCC from the xtensa-esp32-elf toolchain at a specific release, ESP-IDF locked to a named tag, and Python dependencies hashed in a lockfile. Nothing floats. A floating dependency is a reproducibility hole: pip install pulling today's version of a library gives you a different binary than last week's build of identical source, and you'd have no way to know.
We also strip nondeterminism where it sneaks in. Build timestamps get replaced with the commit date. Any generated file that embeds absolute paths is patched or excluded. On the ESP32 side, this matters because Espressif's own tooling has historically embedded build metadata into images; the current ESP-IDF build system is more disciplined than it was in IDF 4.x days, but we verify the result rather than assume it — two builds of the same commit must produce identical .bin files or the release is blocked.
Not everything is reproducible yet. Our degoogled phone images, which ship with a different supply chain entirely, are only partially reproducible — the Android build system drags in enough vendor blobs that we publish hashes but can't yet claim bit-identical rebuilds for third parties. We say so in the manifest rather than pretend. The ESP32 products — the StealthDeck Pro and the StealthDeck Lite — are where the pipeline is genuinely end-to-end reproducible.
Step 3: hashing and signing the manifest
Once a release build passes, we generate a SHA-256 manifest: one line per artifact — bootloader, partition table, application image, and for mesh nodes the exact Meshtastic firmware version we branch from. Meshtastic's own flashing documentation shows how community firmware is normally distributed through a web flasher; useful, but it asks you to trust the flasher. Our manifest exists so you don't have to.
The manifest is then PGP-signed with a detached signature. We use GnuPG's --detach-sign flag, the standard mechanism for signing software archives described in the GnuPG manual. The public key fingerprint is published on the StealthOz site and cross-posted to our GitLab, so a compromised website alone can't substitute both the manifest and a forged key without someone noticing the fingerprint mismatch.
Detached signatures matter here. A signed manifest is only as trustworthy as your ability to verify it offline, later, with your own keyring. If you keep the manifest and signature from the day you ordered, you can re-verify in five years even if our site is gone.
Step 4: per-unit flash and QA
Nothing ships from a build directory. Each unit is flashed individually on the bench with esptool, from the artifact the manifest references, and then read back. The readback is hashed again and compared against the manifest. If flash-then-readback doesn't match — bad flash chip, marginal power, a flaky USB adapter — the unit doesn't ship. We see a failure rate of roughly 2-3% on flash verification across batches, almost always traceable to marginal flash chips that passed the JEDEC ID check but failed a full write-readback cycle. Those boards get stripped and the flash chip replaced.
For units where we enable Espressif's Secure Boot v2, the verification chain continues at runtime: the chip checks the RSA-3072 signature block on each boot, exactly as documented in Espressif's Secure Boot v2 guide. Note the tradeoff we make deliberately: enabling hardware Secure Boot burns eFuses permanently. For research hardware, that permanence cuts both ways — it protects against flash tampering, but it also locks you out of freely replacing the firmware with your own experimental build. That's why Secure Boot is off by default on our research decks and only enabled on request. We'd rather you have the option to brick your own experiments than force permanence on you.
Step 5: verifying your own unit
This is the part customers actually do. When your StealthDeck Pro arrives, the package includes the serial number and the release tag it was flashed with.
- Download the manifest and detached signature for that release tag. Verify the signature against our published fingerprint with
gpg --verify manifest.sha256.asc manifest.sha256. If the fingerprint doesn't match what you recorded from a second channel, stop. - Connect the unit over USB and read back the full flash image with
esptool.py read-flash 0 ALL-flash.bin— the exact command depends on your flash size, and our manifest states it per board revision. - Hash the readback:
sha256sum flash.bin, and compare against the application-image hash in the manifest.
If the hashes match, the flash contents you are holding are the ones we built and signed. If they don't match, contact us — but also check your own procedure first, because readback offsets and flash sizes are the most common source of false mismatches.
One honest limitation: this proves the flash contents match our manifest. It does not prove the ESP32 silicon itself is genuine, only that the module passed our marking inspection. A sufficiently sophisticated hardware implant below the module level is out of scope for this pipeline, and we're not going to claim otherwise.
Where we're taking it next
The pipeline works, but it's manual where it should be scripted. Our next step is publishing the full container recipe and a single verification script that automates steps 2 and 3 above, so checking a unit takes one command instead of five. We're also working toward a witness-signing setup where an independent second party rebuilds each release and countersigns the manifest — that closes the last gap where you're trusting one party's hashes.
Until then, the manifest and signature files for every release are on the site, the fingerprints are published in more than one place, and the comparison is yours to run. That's the whole point.