Two OpenWrt releases, analysed and compared
A MAGDOX demonstration on two public OpenWrt release images. It shows what one real run of the current FDIE development build found in each release, what it reported changed between them, how one candidate finding was decided and what the analysis cannot tell you. It is one pair, not a benchmark, and says nothing about detection rates on other firmware.
- Date
- 10 October 2026
- FDIE
- Development build 65d4ae8 plus three fixes, since committed as df68c25. The images were built without a build label, so FDIE records the build as unknown
- Machine
- macOS on Apple silicon; FDIE's Docker Compose stack in a Linux VM with 6 CPUs and 10 GiB
- Analysis options
- Dynamic analysis on (tier 1, binary emulation); frameworks ETSI EN 303 645 and OWASP FSTM
- Vulnerability data
- NVD last synced 10 October 2026 at 06:32:43 UTC, before the first upload, so no sync ran during the pair. EPSS snapshot epss_scores-2026-10-09.csv.gz, SHA-256 2470f5eb...432e
- Network
- Both analyses ran on this machine; nothing was sent to the hosted service
The pair
Two consecutive OpenWrt releases of the same generic Arm64 target, published by OpenWrt for anyone to download as free software under the GPL. Both hashes match the sha256sums file each release publishes. The ground truth is each release's own package manifest, which FDIE never reads: between the releases, 40 of the 122 packages changed.
| Release | File | Size | SHA-256 |
|---|---|---|---|
| 23.05.4 | openwrt-23.05.4-armsr-armv8-generic-squashfs-rootfs.img.gz | 4,142,320 bytes | 922980e521f39f959888c994f29c47712bf5dbd086d6b99b652da2f183e5c76f |
| 23.05.5 | openwrt-23.05.5-armsr-armv8-generic-squashfs-rootfs.img.gz | 4,142,444 bytes | 00bd253bfce5ea2d87c430dbbddc21528757b059d5b67b57b8f31397ddc7abbb |
- The kernel and 28 kernel-module packages, 5.15.162 to 5.15.167.
- Mbed TLS (libmbedtls12), 2.28.8 to 2.28.9, a security release.
- ucode and five of its modules, package revision 1 to 3, from the same upstream source.
- Three LuCI packages (luci-base, luci-mod-network and luci-mod-status) and base-files.
1. Analyse each release
One analysis at a time. 23.05.4 was uploaded at 06:35:20 UTC and its last assessment was written at 06:46:50; 23.05.5 was uploaded at 06:46:36, overlapping the first for 14 seconds, and finished at 06:55:55.
| Measure | 23.05.4 | 23.05.5 |
|---|---|---|
| Extraction | Complete, no warnings | Complete, no warnings |
| Files recorded | 354: 191 ELF, 79 scripts, 70 text, 13 configuration, 1 certificate | Same |
| ELF files analysed | 191 of the 206 on disk, 0 degraded | Same |
| Catalogue products the manifest lists, found | 7 of 10 (95% interval 40% to 89%) | Same |
| Versions that match the manifest | 3 of 3 (95% interval 44% to 100%) | Same |
| Findings | 451: 14 critical, 197 high, 120 medium, 108 low, 12 informational | Same |
| CVE candidates | 61 unique CVEs across 86 CVE rows; 0 reachable by a static call path; 0 in CISA KEV | Same |
| Risk score | 10.0 of 10 | 10.0 of 10 |
| ETSI EN 303 645 | 64% (41 of 45 mapped checks assessed) | 64% |
| OWASP FSTM | 68.2% (45 of 47 assessed) | 68.7% |
- The intervals are Wilson score intervals. They treat each product as an independent trial, which products in one image are not, so the real uncertainty is wider. With 10 items they mostly show how little one image can prove.
- The 15 ELF files on disk that were not analysed are all kernel modules. FDIE harvests at most 200 executables per image and takes kernel modules last, and it did not say that it stopped.
2. One component, its version and its evidence
BusyBox 1.36.1 in 23.05.5: FDIE read the version from the version string inside /bin/busybox (SBOM entry pkg:generic/busybox@1.36.1, licence GPL-2.0-only), and the release manifest lists busybox 1.36.1-1, upstream 1.36.1 with OpenWrt package revision 1: a match. dnsmasq 2.90 and Dropbear 2022.82 were identified the same way and also match. The gaps, from the same comparison:
- Mbed TLS has no version. FDIE lists it only by the sonames other files link against (libmbedcrypto.so.7, libmbedtls.so.14 and libmbedx509.so.1), so CVE matching cannot see the 2.28.9 security fix.
- Missed: the kernel (a root filesystem has no kernel image), musl (its libc.so is not mapped to a product) and ppp (pppd is present but not identified).
- Found but not in the manifest: openwrt 23.05.x itself, from /etc/banner and /etc/openwrt_release.
3. Compare the releases
Delta Analysis with 23.05.4 as the baseline and 23.05.5 to compare against.
| What changed | Result |
|---|---|
| Binaries | 102 modified, 89 unchanged, 0 added, 0 removed |
| CVE candidates | 0 new, 0 no longer matched, 6 critical persisting |
| Components | 1 change: openwrt 23.05.4 to 23.05.5 |
| Configuration | 4 files modified: /etc/banner, /etc/openwrt_release, /etc/openwrt_version, /etc/opkg/distfeeds.conf |
| Findings by type | 21 types, every count the same in both releases |
| Patch verification | 39 rows: 23 binary changed, CVE still matches; 16 binary unchanged, CVE still matches |
| Security tests | Grade F in both; pass rate 50.0% to 51.3% |
| OWASP FSTM | 68.2% to 68.7% (28 to 29 of 47 passed) |
- The only test that changed is CVE regression from prior firmware version: not applicable to the first release, which has no earlier version, and passed for the second. That one test explains the security-test and OWASP FSTM movement.
- No result here is a regression or a verified fix. FDIE says so on the page, and the Mbed TLS fix in the next step is the case in point.
4. Compare functions
FDIE compared every function of the 102 changed binaries: 5,181 functions in the older release; 2,740 unchanged, 11 modified, 5 added, 5 removed and 2,425 renamed or moved. Renamed or moved is mostly stripped functions whose address-based names changed: 1,974 have identical code and 451 are near matches. Code changed in six binaries only, and each belongs to a package the manifest says changed.
| Binary | Functions | Changes |
|---|---|---|
| libmbedcrypto.so.2.28.9 (was .2.28.8) | 758 | 5 modified (psa_sign_hash, psa_verify_hash, mbedtls_psa_rsa_sign_hash, mbedtls_psa_rsa_export_public_key, sub_0x34dd0); 21 moved with small changes (91% to 97% similar); 1 added; 1 removed |
| luci.so | 35 | 1 modified |
| nf_tables.ko | 238 | 3 added, 3 removed |
| rvu_af.ko | 381 | 3 modified, 1 added, 1 removed |
| nf_flow_table.ko and nft_counter.ko | 54 and 5 | 1 modified each |
- libmbedtls (205 functions), libmbedx509 (87), ucode (68) and libucode (293) were rebuilt, but their code did not change, which fits package revisions with the same upstream source.
- Every decision in the Mbed TLS library was made by code comparison, none by size alone.
- The comparison shows where the code changed. It does not say which Mbed TLS advisory those changes fix, and we do not claim one.
5. One candidate taken to a decision
FDIE matched CVE-2024-54143 (CVSS 9.3, critical) to openwrt 23.05.4 twice, from /etc/banner and from /etc/openwrt_release, at medium confidence: a product-name match. FDIE's CVE record describes openwrt/asu, the image-on-demand server that builds firmware for users, with no affected CPE listed. The 23.05.4 manifest lists no asu package, and this image is a device root filesystem, not a build server.
- Decision: not affected, component not present. Both findings were set to Not required with this reasoning, and a VEX statement was recorded: CycloneDX VEX not_affected and code_not_present, OpenVEX not_affected and component_not_present, each in its format's own vocabulary.
- Who decided: the MAGDOX engineering assistant, an AI agent, for this evidence package. A person has not yet reviewed the decision.
- It is recorded on the 23.05.4 analysis only. Carrying it to 23.05.5 is a separate triage step that this run did not take, so the comparison still counts CVE-2024-54143 among the six persisting critical candidates.
6. Runtime, reconciled
Tier 1 runs each runnable program once under user-mode emulation (qemu-aarch64), with --help or, for daemons, with no arguments, for at most 5 seconds and without fuzzing. Both releases gave the same counts; the run took 1 min 24 s for 23.05.4.
| Count | Files |
|---|---|
| ELF files considered | 206 |
| Kernel modules, not runnable on their own | 112 |
| Shared libraries, not runnable on their own (Unsupported) | 48 |
| Runnable | 46 |
| Ran | 46 = 45 finished + 1 timed out |
- Finished means the process exited within 5 seconds, not that it worked: exit code 0 for 11, 1 for 22, 255 for 8, and 127, 2, 16 and 3 for one each.
- The one timeout is /sbin/init (procd), which never exits by design.
- FDIE recorded 27 libraries loading, no crashes and two behaviour observations: a shell interpreter executed by /sbin/init (low) and dnsmasq listening on a network socket in the sandbox (medium). Neither is evidence of a vulnerability.
Problems found and fixed during this run
- Committed to FDIE dev as df68c25 after this run, each with a test.
- Compliance seeding failed on every start: a version label was 39 characters for a 32-character column, so framework and clause updates never reached an existing database.
- Renamed libraries were not compared: libmbedcrypto.so.2.28.8 and .2.28.9 showed as one removed and one added binary. A removed and an added library with the same name stem are now paired, when exactly one of each exists.
- The CycloneDX VEX export ignored analyst decisions: it served a stored copy made before the decision. OpenVEX was already correct. The stored copy is now cleared when a statement is recorded.
What this does not show
- One pair, one target, one architecture. A sample, not a detection rate, and no comparison with any other tool.
- A candidate is not a vulnerability. The 451 findings are candidates: none has a static call path from an entry point, none is in CISA KEV, and most CVE matches are by product name or version. The runtime observations do not show exploitability either.
- A mapping is not certification. The ETSI EN 303 645 and OWASP FSTM percentages come from automated checks mapped to clauses. They are not a conformity assessment, and the organisational clauses need a person.
- The engine build is not recorded. The code is FDIE dev 65d4ae8 plus the three fixes above (df68c25); a production image records its build.
- Feed state is current, not per analysis. FDIE reports the feed state when asked. The NVD timing shows nothing changed during the pair; the EPSS load time is not recorded.
- 15 kernel modules per image were not analysed, and FDIE did not say so.
- The Mbed TLS security update shows only in the function comparison. Without a version, CVE matching cannot see it.
- The CVE-2024-54143 decision was made by an AI agent for this package and awaits human review.
- Timings are from this machine.
Download the evidence
- 23.05.4: PDF report25 pages
- 23.05.5: PDF report26 pages
- 23.05.4: report (JSON)Scores, findings (the first 50 of 451), SBOM summary and threat model
- 23.05.5: report (JSON)Scores, findings (the first 50 of 451), SBOM summary and threat model
- 23.05.4: SBOM (CycloneDX 1.5)33 components, each with its detection method and CVE candidates
- 23.05.5: SBOM (CycloneDX 1.5)33 components, each with its detection method and CVE candidates
- 23.05.4: VEX (CycloneDX)54 statements, including the CVE-2024-54143 decision
- 23.05.4: VEX (OpenVEX 0.2.0)The same statements as OpenVEX
- 23.05.5: VEX (CycloneDX)54 statements
- 23.05.4: analysis manifestStage dates, settings, extraction quality, service build and feed state
- 23.05.5: analysis manifestStage dates, settings, extraction quality, service build and feed state
- 23.05.4: dynamic runPer binary: emulator, exit code, timeout, libraries and syscalls
- 23.05.5: dynamic runPer binary: emulator, exit code, timeout, libraries and syscalls
- Benchmark: the pinned pairBoth images and manifests, pinned by SHA-256
- Benchmark: summary (JSON)Every count with its denominator, and what was held constant
- Benchmark: summary (Markdown)The same summary, readable
B=https://downloads.openwrt.org/releases
for r in 23.05.4 23.05.5; do
curl -O $B/$r/targets/armsr/armv8/openwrt-$r-armsr-armv8-generic-squashfs-rootfs.img.gz
curl -O $B/$r/targets/armsr/armv8/openwrt-$r-armsr-armv8.manifest
curl -o sha256sums-$r $B/$r/targets/armsr/armv8/sha256sums
grep squashfs-rootfs.img.gz sha256sums-$r
shasum -a 256 openwrt-$r-armsr-armv8-generic-squashfs-rootfs.img.gz
doneThen upload both images to FDIE with dynamic analysis and both frameworks on, open Delta Analysis with 23.05.4 as the baseline and 23.05.5 to compare against, and open Functions on libmbedcrypto.so.2.28.9.