For engineers

Can a firmware scanner find injection bugs? Our Juliet test

By Manish SharmaPublished 10 min read

Founder and CEO, MAGDOX. Writes on firmware security and device regulation.

A scanner that reports many injection leads is only useful if you know how often it is right. On 24 October 2026 we scored FDIE's binary taint on stripped, size-optimised NIST Juliet C test cases for ARM and MIPS, plus a router CGI built for six architectures, the way firmware ships. It caught 583 of 984 flawed cases (59.2%), up from 241, and flagged 2 of 2,431 safe twins at medium or high confidence (0.08%). On a slice we never looked at while tuning, it caught 538 of 930 (57.8%). These are measurements on a synthetic test suite, not a detection rate for your firmware.

Results as of . We keep improving FDIE’s engine for accuracy, so later runs may score differently; this post records what we measured on that date.

In short

  • On 24 October 2026 we scored FDIE's binary taint on stripped, size-optimised NIST Juliet builds for ARM and MIPS, plus a router CGI on six architectures. It caught 583 of 984 flawed cases (59.2%) and flagged 2 of 2,431 safe twins at medium or high confidence (0.08%).
  • On a slice of Juliet we never looked at while tuning, it caught 538 of 930 (57.8%), within 1.4 points of the main set. Some cases cannot be caught from a binary at all, and we say which.
  • This is a synthetic test suite, not your firmware. On real router images, most medium and high leads we sampled by hand were false positives, so treat them as leads.

Almost every firmware security product says it finds memory corruption and command injection in compiled programs. Very few say how often it is right. We think a buyer deserves the number, the test it came from and the limits, so here are ours.

FDIE by Magdox reports binary taint leads: places where untrusted input appears to reach a risky call, or to overrun a buffer, inside a program in your firmware. This post gives the method and the results of our 24 October 2026 round. It also covers where we started on 19 October, the two bugs that made our own early numbers wrong, and what the numbers mean when you choose a tool.

What is binary taint analysis?

It reads a compiled program, follows untrusted input to a risky call or a size it can overrun, and reports the path as a lead with a confidence level.

In FDIE, the sources are network reads, environment variables, NVRAM settings and web form fields. The sinks include copy functions, command execution, format strings, file paths and allocation sizes. Since 24 October a value-flow pass also follows values through each function's registers and stack, through branches and loops, and between functions through arguments, return values and globals. It finds what a call graph alone cannot see:

  • arithmetic on input that overflows into a size
  • an index taken from input
  • a copy longer than a buffer of known size
  • a NULL pointer used, a pointer used after it was freed, or a stack value read before it was written

It runs on ARM and Thumb, MIPS in both byte orders, AArch64 and SuperH programs. Detection never looks at function or file names. The only names it uses are imported library calls such as recv, memcpy and system. Each lead says why it was raised.

How do you measure a firmware scanner fairly?

We test the way firmware ships: small, optimised and stripped.

  • Test cases. The public NIST Juliet C/C++ 1.3 suite, one test case per Juliet test (flow variant 01), in the classes FDIE claims to find: command injection, format string, path traversal, buffer overflows, integer overflow, NULL dereference, uninitialised stack, use after free and time of check to time of use. That is 465 cases per architecture.
  • Builds. ARM and MIPS big endian, with GCC at -Os as OpenWrt builds, position independent, without fortified string functions. Every program is stripped and packed into a firmware file system, 160 programs to an image. Each image goes through the same analysis as a firmware image a customer uploads.
  • A router CGI. A small CGI program with nine planted flaws, built into router images for AArch64, ARM, MIPS in both byte orders, SuperH and 64-bit PowerPC: 54 more cases.
  • Scoring. A case is caught when a lead of its class names one of its flawed functions, by start address, since the programs are stripped. A safe twin flagged with that class at medium or high confidence is a false flag. At low confidence, it is a low lead.
  • A held-back slice. For each case, one other Juliet flow variant of the same test, chosen by a hash of its name: 930 cases. We read only its totals and never tuned against it. Tuning used two other sets of flow variants.

Results, 24 October 2026

Results, 24 October 2026
24 October 2026Start of the roundAfter
Flawed cases caught, main set241 of 984 (24.5%)583 of 984 (59.2%)
Of those, at medium or high confidence420 of 984
Safe twins flagged at medium or high2 of 2,4312 of 2,431 (0.08%)
Flawed cases caught, held-back slice236 of 930538 of 930 (57.8%)
Safe twins flagged at medium or high, held-back slice7 of 3,4877 of 3,487 (0.2%)

The held-back slice moved with the main set, so the gain is not fitted to the cases we tuned on. The seven safe twins it flags were all flagged before this round. Every image in our test corpus had to pass a gate: no image could miss anything it found before, or gain false-flag candidates, and none did.

By class:

Results, 24 October 2026
ClassStart of the roundAfterAt medium or high
Command injection40 of 5249 of 5249
Format string40 of 5653 of 5653
Path traversal40 of 4645 of 4645
Buffer overflow114 of 360235 of 360210
Integer overflow0 of 324140 of 3242
NULL dereference3 of 5417 of 5417
Uninitialised stack0 of 6020 of 6020
Use after free0 of 2820 of 2820
Time of check to time of use4 of 44 of 44

The same table is available as a spreadsheet for your own comparisons.

Integer overflow leads stay at low confidence by design, as leads to check.

By architecture, the Juliet cases caught were 255 of 465 on ARM and 292 of 465 on MIPS big endian. On the router CGI, FDIE caught 36 of 54: 7 on AArch64, 7 on ARM, 8 on each MIPS byte order and 6 on SuperH, of 9 each. It caught none on 64-bit PowerPC, where FDIE cannot yet name library calls or follow values.

Why can't a binary scanner catch every case?

About 60% of the main set is caught, and most of the rest cannot be caught from the binary at all. We do not hide these cases by dropping them from the set.

  • Compiled away, 88 cases. At -Os, the compiler turns the flawed function into the same bytes as its safe twin. No tool reading the binary can tell them apart.
  • No outside input, 186 cases. In 176 integer cases and 10 index cases, the "input" is rand() or a constant such as INT_MAX. Treating rand() as an attacker would score well on Juliet and badly on firmware, so we do not.
  • Dead code. Some flawed functions are never called, so the stripped program does not contain them.

That leaves about 700 cases a binary can show. FDIE catches 583 of them, about 83%. What it still misses: the 64-bit PowerPC CGI, uninitialised values kept in registers rather than on the stack, and array sizes it cannot yet pin down.

Where we started, and two bugs in our own measurement

On 19 October we measured an earlier engine on a gentler build: Juliet's own Linux build at -O0, not stripped, 1,358 cases per architecture in 47 classes, many of which FDIE does not claim. It found about 35% of the flawed cases on x86-64, ARM and MIPS. We changed the test because shipped firmware is optimised and stripped, and a score on classes a product does not claim says little.

Two bugs taught us more than the numbers did.

MIPS scored zero. Position-independent MIPS programs call every library function through the global offset table (GOT). A MIPS o32 program does not call strcpy directly: it loads the address with lw $t9, %call16(strcpy)($gp) and jumps with jalr $t9. Until FDIE's call graph named those calls, the taint pass had no strcpy, system or fgets to reason about. Naming them took MIPS from zero findings to 1,806, with x86-64 and ARM unchanged byte for byte. "Supports MIPS" in a feature table told you nothing here, so ask for numbers per architecture.

Our scorer undercounted. The first score of the stripped test said 91 cases caught. A stripped MIPS program still keeps the names of its exported functions, and our scorer only counted leads that named a function by address. Matching every lead by the start address in its explanation gave 241 for the same engine. Every figure in this post uses the corrected scorer.

Against an established open-source checker

On 19 October, on the earlier unstripped -O0 build and the earlier engine, we ran cwe_checker 0.9.0 from Fraunhofer FKIE on an even sample of 92 binaries across the classes its checks cover.

Against an established open-source checker
19 October, same 92 binariesRecallPrecision
cwe_checker 0.9.083.7%58.8%
FDIE, then53.3%59.0%

cwe_checker found more of the flawed functions at about the same precision, and took about a hundred times longer per binary at the median: 48 seconds against 0.43. That is the trade FDIE is built for, a fast pass over every program in a firmware image. We have not yet run cwe_checker on the stripped test set, so this comparison does not include the 24 October engine.

Known bugs in deliberately vulnerable firmware

On 19 October, on the six DVRF challenge programs and the IoTGoat shellback service, FDIE reported all seven known bugs, three of them at low confidence.

Known bugs in deliberately vulnerable firmware
ProgramKnown bugFDIE reports
DVRF stack_bof_01stack overflowbuffer overflow, low
DVRF stack_bof_02stack overflowbuffer overflow, low
DVRF heap_overflow_01heap overflowbuffer overflow, low
DVRF socket_bofsocket buffer overflowbuffer overflow, high
DVRF socket_cmdcommand injectioncommand injection, high
DVRF uaf_01use after freeuse after free, medium
IoTGoat shellbackcommand injectioncommand injection, low

Low does not mean wrong. In two of these programs the input comes from the program's arguments rather than the network, and the confidence reflects that weaker evidence. A team that reads only high findings would miss three of these seven.

A command injection lead in the IoTGoat shellback program, with its confidence and the path from input to the risky call.

On real firmware: the noise

On real router images, most medium and high leads we sampled by hand were false positives for a device maker. Much of our work has been removing those patterns.

In a 19 October review we read 35 medium and high leads from OpenWrt images against the instructions behind them. About 25 were false positives, about 8 were plausible leads needing the surrounding code, and 2 could not be settled from the binary. The patterns were general: the C library's own code pairing its sources and sinks, a time-of-check lead where the check was fstat on an already open file, correctly sized snprintf calls, and logging wrappers. A second review read 40 new leads; 2 were real, 3 plausible, 3 unsettled and 32 false.

The 24 October round added about 4% more findings across the Linux images in our test corpus, 47,221 to 49,123 in all. That count is not yet gated or reviewed by hand, so read new leads the same way.

The firmware findings list for the DVRF image, with buffer overflow, command injection and use after free leads in its pwnable programs.

What it costs

The value-flow pass made analysis slower on the images it runs on: 57% longer on an OpenIPC camera image for ARM and 68% longer on an ARM router image, with no change on x86-64, where it does not run. That is seconds per image, far inside our time limit for one analysis.

What do these numbers mean for a buyer?

These are honest numbers on a synthetic suite, not a detection rate for your firmware. Our product pages say the same.

Juliet programs are small and simple next to a vendor's web server or daemon. Optimised and stripped builds bring the test closer to shipped firmware, but scores on real firmware will differ. To use the leads, read high and medium first, and read the stated reason on low ones. Confirm each with the code or a test before it becomes a ticket.

When you compare tools, ask each vendor the same questions we answered here:

  • which test set, which build flags, and whether the binaries were stripped
  • which scoring rule, and how safe twins count
  • results per architecture and per class
  • a held-back set they did not tune on
  • which cases they left out, and why
  • run time per image
  • what a hand review of real firmware showed

Questions

What is binary taint analysis?

It follows untrusted input, such as network data, environment variables, NVRAM settings or web form fields, through a compiled program without the source code. It looks for a path to a risky call such as strcpy, system or printf, or a value that overruns a buffer, and FDIE reports each one as a lead with a confidence level.

Why measure on NIST Juliet?

Every Juliet test case has a function with a known flaw and safe twins that look almost the same, so caught cases and false flags can be counted exactly. The suite is public, so anyone can repeat the measurement.

Is 59% good?

It depends on the class. FDIE catches 49 of 52 command injection and 53 of 56 format string cases in this set, and 235 of 360 buffer overflows. Some cases cannot be caught from the binary at all, because at -Os the flawed function compiles to the same bytes as its safe twin, or its input is rand() or a constant. Ask any vendor for per-class numbers and what their set left out.

Does FDIE find real bugs in firmware?

In our 19 October check, on the deliberately vulnerable DVRF programs and the IoTGoat shellback service, it reported all seven known bugs, three of them at low confidence. On real router images most of the medium and high leads we sampled by hand were false positives for a device maker, which is why we call them leads.

Sources

  1. NIST SARD, Juliet C/C++ 1.3 test suite samate.nist.gov
  2. NIST Software Assurance Reference Dataset samate.nist.gov
  3. cwe_checker (Fraunhofer FKIE), official repository github.com
  4. Damn Vulnerable Router Firmware (DVRF) github.com
  5. OWASP IoTGoat github.com
  6. MITRE Common Weakness Enumeration cwe.mitre.org

OpenWrt is a registered trademark of Software Freedom Conservancy; see openwrt.org. Product and project names are trademarks of their owners, used only to identify the public firmware, software and standards discussed. MAGDOX is not affiliated with or endorsed by them.

  • For engineers

    Is your UEFI Secure Boot real? PKfail, dbx and KEK checks

    Secure Boot can be switched on and still trust test keys or a stale revocation list. FDIE runs four Secure Boot checks on UEFI firmware: test platform keys such as PKfail, the revocation list (dbx) and KEK, revoked boot modules, and SMM callouts at low confidence. On eight UEFI images it found outdated default dbx lists on two consumer motherboards and missing AArch64 revocations in a Debian AArch64 variable store. It also found the Debian SnakeOil test key in its test variant, and marked checks not assessed wherever the image gave it nothing to examine.

    · 8 min read

  • For engineers

    What happens to your firmware after you upload it to FDIE

    Before you upload firmware anywhere, you should know what happens to it. FDIE unpacks a firmware image with its own readers, disassembles the programs inside and lists the components and their versions as an SBOM. It matches them to NVD and OSV with known exploited vulnerabilities first, and runs its device checks. Whatever it could not read or decide is shown as a coverage gap, never as a pass, and every result carries the evidence behind it.

    · 9 min read

  • For engineers

    One firmware analysis for India, the EU and the US

    India, the EU and the US ask device makers the same technical questions in different words: no default passwords, signed updates, encrypted traffic, only the services the device needs, and no known exploitable vulnerabilities. One firmware analysis can answer those questions once and map each answer to the provision that asks it, from MeitY ER: 01 and NCCS ITSAR to the EU Cyber Resilience Act for products sold in the EU, EN 18031-1, NIST IR 8425 and the California and Oregon laws. Lab tests, procedures and policy documents still come from you.

    · 6 min read

See what is inside your next release

Book a 30 or 45 minute demo with the team that builds FDIE, or request access for your organisation.

Request access