FuzzBench: oss-fuzz-benchmarks report

(experiment incomplete/still running...)
warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus_optimal 97.38
honggfuzz 93.87
entropic 93.53
lafintel 89.44
aflcc 88.70
fastcgs_lm 87.12
aflplusplus_qemu 86.53
mopt 86.47
aflplusplus 86.44
afl 86.32
aflsmart 86.15
aflfast 84.80
libfuzzer 84.66
ankou 82.22
fairfuzz 79.31
eclipser 75.33
afl_qemu 74.22
manul 66.94
honggfuzz_qemu 45.68
By avg. rank
average rank
fuzzer
aflplusplus_optimal 3.45
honggfuzz 4.60
entropic 5.39
afl 6.10
fastcgs_lm 6.10
aflplusplus 6.15
aflsmart 7.30
mopt 7.40
aflcc 8.85
aflfast 10.20
libfuzzer 10.45
lafintel 10.80
ankou 10.86
fairfuzz 11.15
aflplusplus_qemu 12.00
afl_qemu 15.20
eclipser 15.20
manul 15.90
honggfuzz_qemu 18.50
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    fuzzer afl afl_qemu aflcc aflfast aflplusplus aflplusplus_optimal aflplusplus_qemu aflsmart ankou eclipser entropic fairfuzz fastcgs_lm honggfuzz honggfuzz_qemu lafintel libfuzzer manul mopt
    benchmark
    bloaty_fuzz_target 5391.5 4254.0 5186.5 5181.0 5381.0 5680.0 NaN 5314.0 5201.5 4113.0 NaN 5149.5 5695.5 5515.5 3087.0 5037.5 4568.5 3927.0 5613.5
    curl_curl_fuzzer_http 5412.0 4629.0 5542.5 5324.0 5442.5 5620.0 5315.0 5422.0 5242.0 4399.0 5411.5 4838.0 5402.5 5449.0 3427.0 5319.0 4776.0 3510.0 5378.0
    jsoncpp_jsoncpp_fuzzer 634.0 627.0 632.0 634.0 634.0 634.0 629.0 634.0 612.5 591.0 633.0 634.0 634.0 635.0 163.5 631.0 635.0 474.5 634.0
    libpcap_fuzz_both 21.0 18.0 1993.5 21.0 21.0 2122.0 1403.0 18.0 22.0 884.0 1952.5 125.5 20.0 1952.5 3.0 1673.0 1515.5 120.0 21.0
    mbedtls_fuzz_dtlsclient 1676.5 1481.0 979.0 1607.5 1679.5 1698.0 1617.5 1672.0 2113.0 1409.0 1644.0 1663.5 1684.0 1674.5 975.0 1590.5 1607.0 1368.0 1678.5
    openssl_x509 4077.0 4057.0 4076.5 4073.0 4079.0 4078.0 4072.0 4076.0 NaN 4055.0 4080.0 4054.0 4077.0 4070.0 4044.0 4072.0 4069.0 4050.0 4075.0
    php_php-fuzz-parser 11224.0 10167.0 11268.5 11130.0 11185.0 11182.0 10914.0 11231.0 NaN 9868.5 11587.0 10832.5 11313.0 11492.5 9017.0 10972.0 10237.5 10487.0 11191.0
    sqlite3_ossfuzz 17261.0 9654.5 14977.5 16640.5 17496.0 17431.5 13188.0 17249.5 NaN 5426.0 13783.0 9116.0 17664.5 13247.0 5382.0 11865.0 8758.0 8221.5 17059.5
    systemd_fuzz-link-parser 990.0 757.5 774.0 987.0 988.0 1002.0 759.0 990.0 971.5 925.0 991.0 891.0 989.0 1004.0 537.5 979.0 967.0 890.0 989.0
    zlib_zlib_uncompress_fuzzer 330.5 328.0 329.0 324.0 330.0 330.5 328.0 330.0 323.0 314.0 331.5 330.5 329.0 333.0 29.0 329.5 334.0 227.0 329.5

bloaty_fuzz_target summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 82800 20.0 5695.950000 58.420864 5575.0 5665.25 5695.5 5751.25 5769.0
    aflplusplus_optimal 82800 20.0 5679.350000 57.037638 5581.0 5645.75 5680.0 5695.00 5851.0
    mopt 82800 20.0 5614.850000 64.548860 5507.0 5572.00 5613.5 5640.50 5781.0
    honggfuzz 82800 20.0 5509.500000 43.325573 5429.0 5482.00 5515.5 5547.25 5574.0
    afl 82800 20.0 5402.350000 142.340687 5194.0 5316.25 5391.5 5487.00 5690.0
    aflplusplus 82800 20.0 5323.100000 119.153108 5091.0 5277.75 5381.0 5401.00 5442.0
    aflsmart 82800 20.0 5343.050000 156.138088 5155.0 5197.50 5314.0 5440.75 5622.0
    ankou 82800 20.0 5208.450000 62.379547 5109.0 5178.25 5201.5 5255.75 5311.0
    aflcc 82800 20.0 5183.800000 84.697356 4985.0 5138.50 5186.5 5245.00 5318.0
    aflfast 82800 20.0 5240.700000 165.885852 5028.0 5128.75 5181.0 5289.25 5602.0
    fairfuzz 82800 20.0 5111.150000 342.558988 4488.0 4806.50 5149.5 5447.25 5538.0
    lafintel 82800 20.0 5057.850000 184.212199 4832.0 4898.50 5037.5 5140.25 5400.0
    libfuzzer 82800 20.0 4456.550000 218.647391 4066.0 4279.75 4568.5 4604.25 4737.0
    afl_qemu 82800 20.0 4266.800000 75.512704 4115.0 4215.25 4254.0 4320.25 4404.0
    eclipser 82800 19.0 4135.578947 111.032686 3926.0 4064.50 4113.0 4186.50 4344.0
    manul 82800 19.0 3922.210526 37.776505 3827.0 3908.00 3927.0 3945.00 3995.0
    honggfuzz_qemu 82800 20.0 3085.850000 1.694418 3080.0 3085.00 3087.0 3087.00 3087.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 20.0 5614.850000 33.768991 5565.0 5590.25 5620.0 5642.25 5676.0
    aflcc 82800 20.0 5538.500000 21.080048 5495.0 5520.75 5542.5 5558.50 5570.0
    honggfuzz 82800 20.0 5439.850000 52.189205 5322.0 5417.75 5449.0 5481.00 5500.0
    aflplusplus 82800 20.0 5446.950000 29.222333 5394.0 5431.75 5442.5 5456.00 5519.0
    aflsmart 82800 20.0 5423.600000 29.193366 5360.0 5410.75 5422.0 5431.00 5504.0
    afl 82800 20.0 5417.850000 26.190145 5373.0 5400.50 5412.0 5435.50 5472.0
    entropic 82800 20.0 5408.800000 27.277521 5355.0 5393.75 5411.5 5423.25 5469.0
    fastcgs_lm 82800 20.0 5407.400000 26.804163 5363.0 5388.50 5402.5 5427.75 5467.0
    mopt 82800 19.0 5380.894737 23.715665 5342.0 5360.00 5378.0 5398.50 5423.0
    aflfast 82800 19.0 5317.736842 33.916306 5240.0 5297.50 5324.0 5337.50 5382.0
    lafintel 82800 20.0 5319.900000 30.426616 5264.0 5305.00 5319.0 5332.25 5402.0
    aflplusplus_qemu 82800 20.0 5318.050000 31.453349 5276.0 5305.50 5315.0 5324.25 5415.0
    ankou 82800 20.0 5243.750000 27.962145 5180.0 5227.25 5242.0 5263.75 5285.0
    fairfuzz 82800 19.0 4882.210526 165.115306 4514.0 4809.50 4838.0 4963.00 5324.0
    libfuzzer 82800 20.0 4726.550000 356.897372 4143.0 4470.25 4776.0 5054.25 5295.0
    afl_qemu 82800 20.0 4589.000000 160.150916 4175.0 4577.00 4629.0 4683.25 4763.0
    eclipser 82800 20.0 4398.500000 45.308881 4305.0 4375.75 4399.0 4429.25 4484.0
    manul 82800 20.0 3519.350000 30.112552 3499.0 3505.75 3510.0 3520.00 3627.0
    honggfuzz_qemu 82800 20.0 3427.000000 0.000000 3427.0 3427.00 3427.0 3427.00 3427.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 635.000000 0.000000 635.0 635.00 635.0 635.00 635.0
    honggfuzz 82800 19.0 635.000000 0.000000 635.0 635.00 635.0 635.00 635.0
    afl 82800 20.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflsmart 82800 20.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    fastcgs_lm 82800 19.0 633.894737 0.315302 633.0 634.00 634.0 634.00 634.0
    fairfuzz 82800 20.0 633.900000 0.307794 633.0 634.00 634.0 634.00 634.0
    mopt 82800 20.0 633.900000 0.307794 633.0 634.00 634.0 634.00 634.0
    aflplusplus_optimal 82800 19.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflplusplus 82800 20.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflfast 82800 20.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    entropic 82800 20.0 631.750000 2.173404 627.0 631.00 633.0 633.00 634.0
    aflcc 82800 20.0 631.650000 1.980829 626.0 632.00 632.0 633.00 634.0
    lafintel 82800 20.0 631.250000 0.444262 631.0 631.00 631.0 631.25 632.0
    aflplusplus_qemu 82800 20.0 629.400000 1.095445 628.0 629.00 629.0 630.00 633.0
    afl_qemu 82800 20.0 626.600000 0.994723 624.0 626.00 627.0 627.00 628.0
    ankou 82800 20.0 613.250000 7.217486 601.0 609.50 612.5 617.25 627.0
    eclipser 82800 20.0 588.200000 8.994735 570.0 581.00 591.0 595.25 600.0
    manul 82800 20.0 479.450000 18.537231 453.0 467.50 474.5 484.75 526.0
    honggfuzz_qemu 82800 20.0 151.100000 38.924894 77.0 161.75 163.5 174.50 192.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 20.0 2114.75 166.472971 1716.0 2025.25 2122.0 2244.25 2339.0
    aflcc 82800 20.0 1969.85 159.624980 1688.0 1865.00 1993.5 2077.00 2246.0
    honggfuzz 82800 20.0 1961.70 146.499183 1713.0 1865.25 1952.5 2079.75 2212.0
    entropic 82800 20.0 1985.15 184.944167 1718.0 1827.25 1952.5 2133.75 2313.0
    lafintel 82800 20.0 1675.75 191.655907 1087.0 1607.00 1673.0 1740.25 2004.0
    libfuzzer 82800 20.0 1559.30 105.596102 1458.0 1475.50 1515.5 1614.50 1772.0
    aflplusplus_qemu 82800 20.0 1414.15 211.667882 1065.0 1341.50 1403.0 1504.25 1855.0
    eclipser 82800 20.0 941.05 142.903160 812.0 848.75 884.0 975.75 1226.0
    fairfuzz 82800 20.0 101.75 48.297189 19.0 95.25 125.5 131.00 140.0
    manul 82800 20.0 78.45 56.250825 17.0 18.00 120.0 127.25 138.0
    ankou 82800 20.0 68.60 54.988420 18.0 21.00 22.0 125.50 136.0
    afl 82800 20.0 25.50 22.776950 16.0 21.00 21.0 21.00 122.0
    aflplusplus 82800 20.0 25.95 24.047048 18.0 21.00 21.0 21.00 128.0
    aflfast 82800 20.0 19.55 1.669384 17.0 18.00 21.0 21.00 21.0
    mopt 82800 20.0 99.70 310.614010 18.0 18.00 21.0 21.00 1412.0
    fastcgs_lm 82800 20.0 25.10 25.692309 16.0 18.00 20.0 20.00 134.0
    afl_qemu 82800 20.0 18.60 1.231174 18.0 18.00 18.0 18.00 21.0
    aflsmart 82800 20.0 18.50 1.235442 18.0 18.00 18.0 18.00 22.0
    honggfuzz_qemu 82800 20.0 3.60 2.683282 3.0 3.00 3.0 3.00 15.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    ankou 82800 19.0 2102.736842 217.870207 1621.0 2017.50 2113.0 2309.50 2332.0
    aflplusplus_optimal 82800 20.0 1762.200000 207.893090 1678.0 1687.75 1698.0 1705.00 2373.0
    fastcgs_lm 82800 20.0 1768.600000 208.693782 1667.0 1676.75 1684.0 1699.25 2303.0
    aflplusplus 82800 20.0 1734.600000 165.385483 1660.0 1674.75 1679.5 1695.75 2261.0
    mopt 82800 20.0 1737.900000 163.378121 1662.0 1673.00 1678.5 1683.25 2257.0
    afl 82800 20.0 1735.650000 196.608153 1654.0 1664.50 1676.5 1680.50 2352.0
    honggfuzz 82800 20.0 1678.900000 13.912887 1662.0 1669.25 1674.5 1688.75 1713.0
    aflsmart 82800 20.0 1715.250000 139.192738 1655.0 1664.75 1672.0 1675.75 2157.0
    fairfuzz 82800 20.0 1690.050000 131.224787 1638.0 1650.50 1663.5 1673.00 2245.0
    entropic 82800 20.0 1644.400000 11.771509 1610.0 1638.75 1644.0 1653.50 1661.0
    aflplusplus_qemu 82800 20.0 1617.050000 9.023216 1599.0 1612.00 1617.5 1621.75 1633.0
    aflfast 82800 20.0 1604.850000 13.433168 1565.0 1599.75 1607.5 1614.00 1622.0
    libfuzzer 82800 20.0 1565.600000 63.477638 1464.0 1499.50 1607.0 1615.00 1634.0
    lafintel 82800 20.0 1589.850000 8.368330 1569.0 1585.75 1590.5 1596.25 1604.0
    afl_qemu 82800 19.0 1486.421053 19.562764 1464.0 1471.50 1481.0 1495.00 1532.0
    eclipser 82800 20.0 1410.600000 20.858925 1383.0 1396.25 1409.0 1415.75 1465.0
    manul 82800 20.0 1368.600000 14.117552 1341.0 1359.00 1368.0 1380.75 1391.0
    aflcc 82800 19.0 978.736842 0.561951 978.0 978.00 979.0 979.00 980.0
    honggfuzz_qemu 82800 20.0 975.000000 0.000000 975.0 975.00 975.0 975.00 975.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 4080.250000 1.712954 4078.0 4079.00 4080.0 4081.00 4085.0
    aflplusplus 82800 20.0 4079.450000 1.820208 4077.0 4078.75 4079.0 4080.00 4085.0
    aflplusplus_optimal 82800 20.0 4078.350000 1.424411 4076.0 4077.00 4078.0 4079.00 4082.0
    fastcgs_lm 82800 20.0 4077.150000 0.933302 4076.0 4076.00 4077.0 4078.00 4079.0
    afl 82800 20.0 4076.700000 1.128576 4074.0 4076.00 4077.0 4077.25 4078.0
    aflcc 82800 20.0 4076.650000 1.598519 4074.0 4075.75 4076.5 4077.00 4080.0
    aflsmart 82800 20.0 4076.700000 1.218282 4075.0 4076.00 4076.0 4078.00 4079.0
    mopt 82800 20.0 4075.650000 1.039990 4075.0 4075.00 4075.0 4076.00 4079.0
    aflfast 82800 18.0 4073.000000 1.940285 4069.0 4072.00 4073.0 4074.00 4077.0
    aflplusplus_qemu 82800 20.0 4072.200000 2.261811 4068.0 4071.00 4072.0 4073.25 4079.0
    lafintel 82800 19.0 4071.684211 2.667763 4064.0 4070.50 4072.0 4074.00 4075.0
    honggfuzz 82800 20.0 4070.000000 2.316985 4066.0 4068.75 4070.0 4071.00 4077.0
    libfuzzer 82800 20.0 4068.650000 2.230766 4064.0 4067.75 4069.0 4070.00 4073.0
    afl_qemu 82800 20.0 4056.350000 3.065513 4050.0 4055.00 4057.0 4058.00 4061.0
    eclipser 82800 20.0 4055.300000 5.100052 4046.0 4051.00 4055.0 4058.25 4066.0
    fairfuzz 82800 20.0 4057.600000 10.023131 4044.0 4050.75 4054.0 4068.25 4074.0
    manul 82800 19.0 4049.578947 2.063325 4045.0 4049.00 4050.0 4051.00 4053.0
    honggfuzz_qemu 82800 20.0 4044.000000 0.000000 4044.0 4044.00 4044.0 4044.00 4044.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 19.0 11584.842105 21.899425 11533.0 11578.50 11587.0 11598.00 11619.0
    honggfuzz 82800 20.0 11492.800000 27.385359 11435.0 11477.00 11492.5 11509.50 11551.0
    fastcgs_lm 82800 20.0 11316.100000 33.872672 11262.0 11288.00 11313.0 11346.75 11378.0
    aflcc 82800 20.0 11273.800000 41.688822 11177.0 11249.00 11268.5 11292.00 11377.0
    aflsmart 82800 20.0 11227.900000 41.764881 11163.0 11181.00 11231.0 11257.50 11293.0
    afl 82800 20.0 11219.200000 31.458578 11160.0 11194.75 11224.0 11243.25 11277.0
    mopt 82800 20.0 11194.400000 29.265392 11122.0 11178.75 11191.0 11221.75 11242.0
    aflplusplus 82800 19.0 11197.263158 50.190572 11105.0 11170.00 11185.0 11217.50 11307.0
    aflplusplus_optimal 82800 19.0 11168.631579 43.228348 11106.0 11128.50 11182.0 11188.50 11257.0
    aflfast 82800 20.0 11122.250000 44.046477 11029.0 11105.50 11130.0 11143.25 11198.0
    lafintel 82800 19.0 10968.631579 39.361869 10864.0 10946.50 10972.0 10991.00 11032.0
    aflplusplus_qemu 82800 20.0 10909.150000 64.423169 10718.0 10877.25 10914.0 10946.00 11033.0
    fairfuzz 82800 20.0 10769.250000 256.650054 10273.0 10607.50 10832.5 10952.75 11140.0
    manul 82800 20.0 10493.300000 23.362363 10451.0 10477.75 10487.0 10508.75 10534.0
    libfuzzer 82800 20.0 10253.750000 78.144182 10072.0 10214.25 10237.5 10276.75 10417.0
    afl_qemu 82800 20.0 10180.300000 55.637076 10098.0 10144.25 10167.0 10192.75 10324.0
    eclipser 82800 20.0 9872.650000 20.308865 9844.0 9856.75 9868.5 9888.50 9909.0
    honggfuzz_qemu 82800 20.0 9017.000000 0.000000 9017.0 9017.00 9017.0 9017.00 9017.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 82800 20.0 17596.600000 199.322378 17199.0 17515.50 17664.5 17742.75 17853.0
    aflplusplus 82800 20.0 17528.750000 249.891740 16996.0 17401.00 17496.0 17668.50 18124.0
    aflplusplus_optimal 82800 20.0 17419.850000 240.865605 16989.0 17251.50 17431.5 17599.00 17814.0
    afl 82800 20.0 17268.100000 202.750015 16825.0 17166.25 17261.0 17379.25 17774.0
    aflsmart 82800 20.0 17249.650000 205.790204 16944.0 17090.75 17249.5 17354.25 17666.0
    mopt 82800 20.0 17098.750000 240.460058 16562.0 16932.50 17059.5 17296.75 17545.0
    aflfast 82800 20.0 16578.900000 249.108072 15860.0 16451.25 16640.5 16737.00 16875.0
    aflcc 82800 20.0 14983.900000 428.153059 14183.0 14704.50 14977.5 15305.00 15725.0
    entropic 82800 19.0 13852.052632 481.878324 12796.0 13506.00 13783.0 14257.50 14591.0
    honggfuzz 82800 19.0 13185.631579 327.979879 12539.0 12943.50 13247.0 13472.50 13636.0
    aflplusplus_qemu 82800 19.0 13343.684211 935.480688 11623.0 12756.00 13188.0 14090.00 15015.0
    lafintel 82800 20.0 11865.700000 600.312208 10903.0 11294.00 11865.0 12248.00 13006.0
    afl_qemu 82800 20.0 9725.550000 251.154906 9389.0 9542.50 9654.5 9936.25 10220.0
    fairfuzz 82800 20.0 10014.200000 1493.416909 8505.0 8823.50 9116.0 11325.50 13093.0
    libfuzzer 82800 20.0 8822.450000 464.170735 7495.0 8635.25 8758.0 9052.00 9792.0
    manul 82800 20.0 8193.100000 337.733140 7687.0 7938.50 8221.5 8387.75 9117.0
    eclipser 82800 19.0 5527.473684 441.355408 5421.0 5424.50 5426.0 5429.00 7350.0
    honggfuzz_qemu 82800 20.0 5382.000000 0.000000 5382.0 5382.00 5382.0 5382.00 5382.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 334.200000 2.284962 330.0 333.00 334.0 335.25 338.0
    honggfuzz 82800 20.0 332.100000 2.174009 329.0 330.00 333.0 333.00 337.0
    entropic 82800 20.0 330.500000 3.734618 322.0 328.00 331.5 332.25 336.0
    afl 82800 20.0 331.050000 2.089447 326.0 330.00 330.5 333.00 334.0
    aflplusplus_optimal 82800 20.0 331.100000 2.049390 328.0 329.75 330.5 333.00 334.0
    fairfuzz 82800 20.0 331.250000 2.314144 328.0 329.00 330.5 333.00 337.0
    aflplusplus 82800 20.0 330.900000 2.049390 329.0 329.00 330.0 333.00 336.0
    aflsmart 82800 19.0 331.210526 2.507299 329.0 329.50 330.0 333.00 337.0
    lafintel 82800 20.0 329.700000 1.625455 327.0 328.75 329.5 330.25 333.0
    mopt 82800 20.0 329.500000 2.800376 324.0 329.00 329.5 330.00 334.0
    fastcgs_lm 82800 20.0 329.800000 2.330575 324.0 328.00 329.0 332.00 333.0
    aflcc 82800 20.0 328.700000 3.113637 322.0 326.00 329.0 331.00 333.0
    afl_qemu 82800 20.0 326.550000 3.153528 320.0 324.00 328.0 329.00 331.0
    aflplusplus_qemu 82800 20.0 328.500000 2.982405 322.0 327.00 328.0 330.00 334.0
    aflfast 82800 20.0 324.100000 4.076893 318.0 320.75 324.0 328.00 330.0
    ankou 82800 18.0 322.222222 3.828001 312.0 321.25 323.0 324.00 328.0
    eclipser 82800 20.0 313.650000 3.048295 308.0 312.00 314.0 316.00 319.0
    manul 82800 19.0 227.210526 15.450922 209.0 214.50 227.0 234.00 268.0
    honggfuzz_qemu 82800 20.0 29.000000 0.000000 29.0 29.00 29.0 29.00 29.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 2dfe596dcdb3666eeaeddf3fe975cb1ea9554eb4