FuzzBench: 2020-07-17 report

(experiment incomplete/still running...)
warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggreage (cross-benchmark) ranking of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus_optimal 99.24
fastcgs_lm 96.29
aflplusplus_ctx_nozerosingle 94.84
entropic 94.68
aflplusplus_ctx_default 94.66
aflplusplus_ctx_nosingle 94.62
ankou 94.23
aflplusplus_ngram4 93.89
aflplusplus_ctx_nozero 93.85
honggfuzz 93.35
aflplusplus_ngram8 91.44
aflplusplus_ngram6 91.44
lafintel 88.89
fairfuzz 87.15
aflplusplus 86.88
mopt 86.72
aflsmart 86.54
aflcc 86.52
aflplusplus_qemu 86.50
afl 86.49
aflfast 85.29
libfuzzer 84.64
eclipser 75.47
afl_qemu 73.69
manul 66.02
honggfuzz_qemu 46.73
By avg. rank
average rank
fuzzer
aflplusplus_optimal 4.07
fastcgs_lm 5.81
afl 6.90
mopt 7.05
aflplusplus 7.20
honggfuzz 7.31
entropic 7.67
aflplusplus_ngram4 7.70
aflsmart 7.90
aflplusplus_ctx_default 9.65
aflplusplus_ngram6 9.80
aflplusplus_ctx_nozero 10.40
aflplusplus_ctx_nosingle 10.55
aflplusplus_ctx_nozerosingle 10.75
aflfast 13.15
aflplusplus_ngram8 14.00
aflcc 14.17
libfuzzer 15.15
ankou 15.60
lafintel 16.65
fairfuzz 17.60
aflplusplus_qemu 17.61
eclipser 20.40
afl_qemu 20.55
manul 21.00
honggfuzz_qemu 23.50
  • Critical difference diagram
    Too many fuzzers to render the diagram. The critical difference plot currently only supports up to 20 fuzzers.
  • Median coverages on each benchmark
    fuzzer afl afl_qemu aflcc aflfast aflplusplus aflplusplus_ctx_default aflplusplus_ctx_nosingle aflplusplus_ctx_nozero aflplusplus_ctx_nozerosingle aflplusplus_ngram4 aflplusplus_ngram6 aflplusplus_ngram8 aflplusplus_optimal aflplusplus_qemu aflsmart ankou eclipser entropic fairfuzz fastcgs_lm honggfuzz honggfuzz_qemu lafintel libfuzzer manul mopt
    benchmark
    bloaty_fuzz_target 5135.0 4014.0 4903.0 5123.5 5196.5 5247.0 5328.0 5047.5 5191.0 5196.0 4992.0 4886.0 5452.5 NaN 5223.0 4980.0 3847.0 NaN 5064.0 5459.0 5304.0 3087.0 4802.0 4423.5 3754.0 5437.0
    curl_curl_fuzzer_http 5329.5 4465.0 5311.0 5261.5 5364.0 5269.0 5250.0 5277.0 5270.0 5277.0 5287.0 5276.0 5500.0 5236.5 5346.0 5135.0 4097.0 5305.0 4874.0 5323.0 5318.5 3427.0 5223.0 4513.0 3507.0 5335.0
    jsoncpp_jsoncpp_fuzzer 634.0 626.5 629.5 634.0 634.0 633.0 633.0 633.0 632.0 634.0 634.0 634.0 634.0 629.5 634.0 612.0 583.0 632.0 NaN 634.0 635.0 166.0 631.0 635.0 464.0 634.0
    libpcap_fuzz_both 21.0 18.0 1458.0 18.0 21.0 1467.0 1431.0 1401.0 NaN 1557.0 1449.0 NaN NaN 1306.0 18.0 NaN 827.0 1633.0 NaN NaN NaN 3.0 1387.5 1493.0 123.0 18.0
    mbedtls_fuzz_dtlsclient 1650.0 1426.0 978.0 1599.0 1657.0 1642.0 1632.5 1642.0 1642.0 1614.5 1610.0 NaN 2014.5 1588.0 1646.0 1901.0 1351.0 1624.5 NaN 1649.0 1640.0 975.0 1560.5 1477.0 1304.5 1652.0
    openssl_x509 4075.0 4048.5 4069.0 4072.0 4077.5 4073.0 4071.5 4073.5 4074.0 4075.0 4073.5 4073.0 NaN 4067.0 4075.0 NaN 4051.0 4078.0 NaN 4076.0 4068.0 4044.0 4065.5 4062.0 4045.0 4075.0
    php_php-fuzz-parser 11030.0 9979.0 11033.0 10931.0 10923.0 11016.0 11018.0 11016.0 11008.0 11082.0 11094.0 11010.5 11012.5 10768.0 11013.5 NaN 9560.0 11400.5 10847.5 11127.0 11330.0 9017.0 10780.0 10059.5 10177.5 11021.0
    sqlite3_ossfuzz 15836.0 8436.0 13251.0 15002.5 16307.0 14715.0 14922.0 14623.0 14934.0 12671.0 10272.0 9852.0 16130.0 9415.5 15715.0 NaN 5426.0 12778.0 11063.0 15691.0 11807.0 5382.0 8902.5 8908.0 7442.0 15377.0
    systemd_fuzz-link-parser 988.0 762.0 775.0 982.0 982.0 986.0 984.5 986.0 987.0 987.0 983.0 979.5 994.5 756.5 986.0 958.0 917.0 985.0 914.0 981.0 1000.0 537.5 970.0 761.0 839.0 987.5
    zlib_zlib_uncompress_fuzzer 329.0 324.0 NaN 325.0 330.0 329.0 329.0 329.0 NaN 331.0 334.0 NaN NaN 327.0 329.0 NaN 308.0 330.0 NaN NaN NaN 29.0 329.0 334.5 215.0 328.0

bloaty_fuzz_target summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 27900 15.0 5442.866667 53.653474 5347.0 5421.00 5459.0 5482.50 5527.0
    aflplusplus_optimal 27900 16.0 5469.125000 74.705533 5409.0 5432.25 5452.5 5478.50 5728.0
    mopt 27900 7.0 5391.000000 67.808554 5282.0 5342.00 5437.0 5440.00 5454.0
    aflplusplus_ctx_nosingle 27900 11.0 5310.363636 227.283203 4983.0 5091.00 5328.0 5503.50 5578.0
    honggfuzz 27900 17.0 5277.588235 98.940044 4978.0 5268.00 5304.0 5333.00 5399.0
    aflplusplus_ctx_default 27900 17.0 5281.352941 155.573512 5059.0 5161.00 5247.0 5418.00 5583.0
    aflsmart 27900 15.0 5180.933333 131.188777 4965.0 5060.50 5223.0 5265.50 5410.0
    aflplusplus 27900 8.0 5181.125000 108.523121 4994.0 5126.25 5196.5 5239.25 5340.0
    aflplusplus_ngram4 27900 7.0 5166.000000 119.532422 4961.0 5105.50 5196.0 5248.00 5298.0
    aflplusplus_ctx_nozerosingle 27900 13.0 5179.461538 130.301327 5012.0 5063.00 5191.0 5284.00 5365.0
    afl 27900 16.0 5160.937500 93.738977 5022.0 5108.25 5135.0 5216.75 5336.0
    aflfast 27900 4.0 5175.250000 247.650796 4955.0 4997.00 5123.5 5301.75 5499.0
    fairfuzz 27900 13.0 5010.769231 228.241522 4624.0 4818.00 5064.0 5131.00 5457.0
    aflplusplus_ctx_nozero 27900 14.0 5134.428571 174.452424 4973.0 5030.00 5047.5 5181.00 5546.0
    aflplusplus_ngram6 27900 9.0 5009.888889 82.101225 4928.0 4959.00 4992.0 5037.00 5177.0
    ankou 27900 16.0 4973.500000 52.536971 4900.0 4918.75 4980.0 5014.00 5051.0
    aflcc 27900 6.0 4917.666667 75.746067 4816.0 4881.50 4903.0 4960.50 5030.0
    aflplusplus_ngram8 27900 10.0 4888.700000 98.688117 4737.0 4810.75 4886.0 4965.75 5025.0
    lafintel 27900 16.0 4812.562500 122.081377 4618.0 4716.00 4802.0 4917.00 4990.0
    libfuzzer 27900 4.0 4423.500000 85.027447 4320.0 4389.00 4423.5 4458.00 4527.0
    afl_qemu 27900 6.0 4024.000000 41.718102 3983.0 3990.00 4014.0 4062.75 4072.0
    eclipser 27900 15.0 3832.133333 89.762279 3569.0 3823.00 3847.0 3864.00 4005.0
    manul 27900 13.0 3755.000000 30.569593 3705.0 3734.00 3754.0 3780.00 3812.0
    honggfuzz_qemu 27900 13.0 3085.461538 2.221688 3080.0 3085.00 3087.0 3087.00 3087.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 28800 4.0 5498.750000 23.669601 5469.0 5488.50 5500.0 5510.25 5526.0
    aflplusplus 28800 10.0 5363.900000 21.548137 5323.0 5352.25 5364.0 5373.00 5399.0
    aflsmart 28800 14.0 5335.857143 36.683379 5266.0 5313.50 5346.0 5359.50 5396.0
    mopt 28800 2.0 5335.000000 9.899495 5328.0 5331.50 5335.0 5338.50 5342.0
    afl 28800 14.0 5337.285714 25.076586 5298.0 5319.25 5329.5 5350.25 5379.0
    fastcgs_lm 28800 10.0 5325.500000 30.653077 5257.0 5312.50 5323.0 5350.75 5360.0
    honggfuzz 28800 2.0 5318.500000 12.020815 5310.0 5314.25 5318.5 5322.75 5327.0
    aflcc 28800 3.0 5298.000000 27.874720 5266.0 5288.50 5311.0 5314.00 5317.0
    entropic 28800 16.0 5310.312500 31.146361 5265.0 5290.75 5305.0 5337.25 5376.0
    aflplusplus_ngram6 28800 11.0 5296.272727 25.310436 5245.0 5285.00 5287.0 5313.00 5331.0
    aflplusplus_ctx_nozero 28800 14.0 5275.857143 31.456581 5226.0 5245.75 5277.0 5308.75 5315.0
    aflplusplus_ngram4 28800 6.0 5278.333333 19.211108 5260.0 5263.75 5277.0 5282.75 5312.0
    aflplusplus_ngram8 28800 3.0 5277.333333 9.073772 5269.0 5272.50 5276.0 5281.50 5287.0
    aflplusplus_ctx_nozerosingle 28800 12.0 5273.916667 30.666708 5231.0 5248.25 5270.0 5297.00 5323.0
    aflplusplus_ctx_default 28800 15.0 5273.066667 35.740067 5216.0 5247.50 5269.0 5286.50 5349.0
    aflfast 28800 14.0 5260.357143 24.818130 5218.0 5245.25 5261.5 5278.00 5296.0
    aflplusplus_ctx_nosingle 28800 11.0 5249.272727 22.908911 5209.0 5237.50 5250.0 5267.50 5280.0
    aflplusplus_qemu 28800 18.0 5232.666667 29.397879 5153.0 5219.00 5236.5 5252.75 5284.0
    lafintel 28800 13.0 5206.846154 34.638240 5150.0 5193.00 5223.0 5230.00 5259.0
    ankou 28800 4.0 5129.250000 25.708300 5096.0 5115.50 5135.0 5148.75 5151.0
    fairfuzz 28800 1.0 4874.000000 NaN 4874.0 4874.00 4874.0 4874.00 4874.0
    libfuzzer 28800 6.0 4615.333333 314.148797 4347.0 4422.75 4513.0 4671.50 5202.0
    afl_qemu 28800 7.0 4423.285714 78.675948 4312.0 4358.50 4465.0 4475.00 4519.0
    eclipser 28800 13.0 4111.923077 52.098403 4034.0 4078.00 4097.0 4162.00 4218.0
    manul 28800 15.0 3512.000000 12.598186 3496.0 3503.50 3507.0 3516.50 3545.0
    honggfuzz_qemu 28800 13.0 3427.076923 0.277350 3427.0 3427.00 3427.0 3427.00 3428.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 28800 12.0 635.000000 0.000000 635.0 635.00 635.0 635.00 635.0
    honggfuzz 28800 2.0 635.000000 0.000000 635.0 635.00 635.0 635.00 635.0
    afl 28800 14.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflplusplus_ngram4 28800 8.0 633.375000 1.060660 631.0 633.00 634.0 634.00 634.0
    fastcgs_lm 28800 8.0 633.875000 0.353553 633.0 634.00 634.0 634.00 634.0
    aflsmart 28800 14.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflplusplus_ngram8 28800 1.0 634.000000 NaN 634.0 634.00 634.0 634.00 634.0
    aflplusplus_ngram6 28800 12.0 633.916667 0.288675 633.0 634.00 634.0 634.00 634.0
    aflplusplus_optimal 28800 1.0 634.000000 NaN 634.0 634.00 634.0 634.00 634.0
    aflplusplus 28800 12.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflfast 28800 14.0 633.785714 0.425815 633.0 634.00 634.0 634.00 634.0
    mopt 28800 5.0 633.600000 0.547723 633.0 633.00 634.0 634.00 634.0
    aflplusplus_ctx_nozero 28800 13.0 632.692308 1.109400 631.0 632.00 633.0 634.00 634.0
    aflplusplus_ctx_nosingle 28800 14.0 632.785714 1.121714 631.0 632.00 633.0 634.00 634.0
    aflplusplus_ctx_default 28800 13.0 633.230769 0.725011 632.0 633.00 633.0 634.00 634.0
    aflplusplus_ctx_nozerosingle 28800 8.0 632.125000 0.991031 631.0 631.75 632.0 632.25 634.0
    entropic 28800 15.0 631.333333 2.160247 627.0 630.00 632.0 633.00 634.0
    lafintel 28800 13.0 630.384615 0.767948 629.0 630.00 631.0 631.00 631.0
    aflplusplus_qemu 28800 12.0 630.416667 2.234373 628.0 629.00 629.5 631.00 634.0
    aflcc 28800 2.0 629.500000 3.535534 627.0 628.25 629.5 630.75 632.0
    afl_qemu 28800 6.0 625.666667 2.160247 623.0 623.75 626.5 627.00 628.0
    ankou 28800 4.0 611.500000 4.434712 606.0 609.00 612.0 614.50 616.0
    eclipser 28800 19.0 585.473684 7.939832 564.0 582.00 583.0 591.00 597.0
    manul 28800 15.0 461.066667 10.905219 439.0 455.00 464.0 466.00 481.0
    honggfuzz_qemu 28800 11.0 172.090909 10.681335 163.0 164.50 166.0 176.00 194.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 29700 13.0 1574.846154 213.543456 1147.0 1475.00 1633.0 1730.00 1815.0
    aflplusplus_ngram4 29700 9.0 1546.222222 118.630495 1372.0 1526.00 1557.0 1593.00 1779.0
    libfuzzer 29700 7.0 1504.714286 49.317726 1455.0 1471.50 1493.0 1529.50 1583.0
    aflplusplus_ctx_default 29700 15.0 1493.733333 118.123100 1353.0 1417.50 1467.0 1559.00 1816.0
    aflcc 29700 1.0 1458.000000 NaN 1458.0 1458.00 1458.0 1458.00 1458.0
    aflplusplus_ngram6 29700 14.0 1445.642857 98.955007 1294.0 1360.25 1449.0 1498.75 1646.0
    aflplusplus_ctx_nosingle 29700 12.0 1437.250000 137.997447 1098.0 1387.75 1431.0 1529.75 1659.0
    aflplusplus_ctx_nozero 29700 17.0 1401.823529 121.007353 1123.0 1378.00 1401.0 1464.00 1588.0
    lafintel 29700 16.0 1361.000000 181.540078 1053.0 1301.50 1387.5 1485.00 1594.0
    aflplusplus_qemu 29700 15.0 1271.200000 125.333954 1025.0 1221.00 1306.0 1363.50 1432.0
    eclipser 29700 12.0 867.500000 122.571315 727.0 799.50 827.0 877.00 1163.0
    manul 29700 12.0 82.333333 55.652058 17.0 20.25 123.0 127.50 132.0
    afl 29700 14.0 48.071429 47.896466 17.0 18.00 21.0 89.25 128.0
    aflplusplus 29700 15.0 33.866667 36.827526 18.0 18.00 21.0 21.00 125.0
    aflsmart 29700 15.0 24.400000 24.787093 18.0 18.00 18.0 18.00 114.0
    afl_qemu 29700 16.0 60.312500 83.215158 18.0 18.00 18.0 42.50 260.0
    aflfast 29700 14.0 19.000000 1.797434 16.0 18.00 18.0 21.00 22.0
    mopt 29700 6.0 17.833333 1.834848 16.0 16.50 18.0 18.00 21.0
    honggfuzz_qemu 29700 17.0 4.411765 3.985267 3.0 3.00 3.0 3.00 15.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 28800 2.0 2014.500000 447.598592 1698.0 1856.25 2014.5 2172.75 2331.0
    ankou 28800 2.0 1901.000000 459.619408 1576.0 1738.50 1901.0 2063.50 2226.0
    aflplusplus 28800 12.0 1655.916667 9.336715 1644.0 1647.50 1657.0 1660.50 1675.0
    mopt 28800 3.0 1644.333333 23.459184 1618.0 1635.00 1652.0 1657.50 1663.0
    afl 28800 13.0 1691.000000 170.726877 1608.0 1638.00 1650.0 1655.00 2257.0
    fastcgs_lm 28800 6.0 1731.333333 217.940971 1620.0 1633.75 1649.0 1662.00 2175.0
    aflsmart 28800 16.0 1645.437500 7.999740 1632.0 1639.50 1646.0 1650.25 1661.0
    aflplusplus_ctx_default 28800 13.0 1639.923077 8.911243 1622.0 1635.00 1642.0 1646.00 1651.0
    aflplusplus_ctx_nozero 28800 20.0 1639.900000 6.248368 1624.0 1636.50 1642.0 1644.00 1649.0
    aflplusplus_ctx_nozerosingle 28800 9.0 1637.222222 8.288211 1626.0 1630.00 1642.0 1644.00 1647.0
    honggfuzz 28800 5.0 1639.200000 4.868265 1631.0 1640.00 1640.0 1641.00 1644.0
    aflplusplus_ctx_nosingle 28800 12.0 1633.833333 11.002754 1618.0 1627.75 1632.5 1637.25 1654.0
    entropic 28800 14.0 1620.571429 14.542851 1585.0 1612.25 1624.5 1628.00 1643.0
    aflplusplus_ngram4 28800 10.0 1616.100000 7.880355 1605.0 1610.25 1614.5 1624.00 1627.0
    aflplusplus_ngram6 28800 12.0 1610.166667 9.759223 1587.0 1608.75 1610.0 1615.25 1625.0
    aflfast 28800 13.0 1594.769231 16.130478 1568.0 1586.00 1599.0 1602.00 1626.0
    aflplusplus_qemu 28800 18.0 1588.777778 9.570516 1576.0 1581.25 1588.0 1593.00 1612.0
    lafintel 28800 16.0 1558.312500 11.013439 1536.0 1552.75 1560.5 1567.00 1572.0
    libfuzzer 28800 5.0 1503.400000 86.239782 1413.0 1438.00 1477.0 1593.00 1596.0
    afl_qemu 28800 11.0 1432.818182 22.211790 1404.0 1419.50 1426.0 1449.50 1467.0
    eclipser 28800 13.0 1347.000000 14.781745 1328.0 1334.00 1351.0 1355.00 1374.0
    manul 28800 14.0 1305.642857 8.767134 1292.0 1300.00 1304.5 1309.75 1323.0
    aflcc 28800 1.0 978.000000 NaN 978.0 978.00 978.0 978.00 978.0
    honggfuzz_qemu 28800 15.0 975.000000 0.000000 975.0 975.00 975.0 975.00 975.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 28800 17.0 4077.647059 1.169464 4075.0 4077.00 4078.0 4078.00 4080.0
    aflplusplus 28800 16.0 4077.312500 1.887459 4075.0 4076.00 4077.5 4078.00 4082.0
    fastcgs_lm 28800 11.0 4076.090909 1.136182 4074.0 4075.50 4076.0 4077.00 4078.0
    afl 28800 16.0 4075.062500 1.652019 4072.0 4075.00 4075.0 4075.25 4079.0
    aflplusplus_ngram4 28800 7.0 4075.285714 1.112697 4074.0 4074.50 4075.0 4076.00 4077.0
    aflsmart 28800 16.0 4075.625000 1.784190 4073.0 4075.00 4075.0 4076.25 4080.0
    mopt 28800 3.0 4074.000000 1.732051 4072.0 4073.50 4075.0 4075.00 4075.0
    aflplusplus_ctx_nozerosingle 28800 11.0 4074.272727 1.793929 4072.0 4073.00 4074.0 4076.00 4077.0
    aflplusplus_ctx_nozero 28800 10.0 4073.600000 1.646545 4071.0 4072.25 4073.5 4075.00 4076.0
    aflplusplus_ngram6 28800 14.0 4073.857143 1.915810 4071.0 4072.25 4073.5 4075.00 4077.0
    aflplusplus_ctx_default 28800 15.0 4072.866667 2.166850 4069.0 4071.00 4073.0 4074.50 4076.0
    aflplusplus_ngram8 28800 1.0 4073.000000 NaN 4073.0 4073.00 4073.0 4073.00 4073.0
    aflfast 28800 13.0 4071.615385 2.180920 4067.0 4071.00 4072.0 4073.00 4075.0
    aflplusplus_ctx_nosingle 28800 10.0 4071.500000 1.269296 4070.0 4070.25 4071.5 4072.75 4073.0
    aflcc 28800 1.0 4069.000000 NaN 4069.0 4069.00 4069.0 4069.00 4069.0
    honggfuzz 28800 5.0 4068.000000 2.121320 4065.0 4068.00 4068.0 4068.00 4071.0
    aflplusplus_qemu 28800 14.0 4066.785714 2.082106 4064.0 4065.25 4067.0 4068.00 4070.0
    lafintel 28800 14.0 4065.785714 3.445813 4060.0 4063.25 4065.5 4068.75 4071.0
    libfuzzer 28800 11.0 4062.727273 2.935674 4058.0 4060.50 4062.0 4064.50 4067.0
    eclipser 28800 13.0 4051.923077 3.012793 4048.0 4049.00 4051.0 4055.00 4056.0
    afl_qemu 28800 6.0 4049.333333 2.250926 4047.0 4048.00 4048.5 4050.50 4053.0
    manul 28800 17.0 4044.941176 0.747545 4044.0 4044.00 4045.0 4045.00 4046.0
    honggfuzz_qemu 28800 15.0 4044.000000 0.000000 4044.0 4044.00 4044.0 4044.00 4044.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 28800 12.0 11401.083333 23.035381 11372.0 11382.50 11400.5 11413.50 11447.0
    honggfuzz 28800 8.0 11336.500000 23.880656 11305.0 11324.50 11330.0 11343.50 11373.0
    fastcgs_lm 28800 7.0 11122.571429 36.271463 11053.0 11113.50 11127.0 11141.50 11168.0
    aflplusplus_ngram6 28800 8.0 11097.750000 31.918423 11054.0 11077.25 11094.0 11110.00 11145.0
    aflplusplus_ngram4 28800 5.0 11088.600000 36.322170 11049.0 11061.00 11082.0 11116.00 11135.0
    aflcc 28800 3.0 11050.333333 39.004273 11023.0 11028.00 11033.0 11064.00 11095.0
    afl 28800 15.0 11025.066667 37.287429 10936.0 11003.50 11030.0 11045.00 11079.0
    mopt 28800 3.0 11024.666667 20.744477 11006.0 11013.50 11021.0 11034.00 11047.0
    aflplusplus_ctx_nosingle 28800 8.0 11014.250000 38.850446 10965.0 10982.50 11018.0 11041.75 11066.0
    aflplusplus_ctx_default 28800 16.0 11020.187500 35.678600 10956.0 10997.00 11016.0 11037.00 11088.0
    aflplusplus_ctx_nozero 28800 17.0 11004.529412 49.643879 10865.0 10984.00 11016.0 11034.00 11066.0
    aflsmart 28800 16.0 11011.812500 45.930700 10933.0 10983.25 11013.5 11034.00 11083.0
    aflplusplus_optimal 28800 6.0 11020.666667 54.157794 10969.0 10995.25 11012.5 11014.75 11125.0
    aflplusplus_ngram8 28800 8.0 11011.000000 34.690468 10968.0 10984.75 11010.5 11023.50 11076.0
    aflplusplus_ctx_nozerosingle 28800 11.0 11005.818182 40.303395 10957.0 10986.00 11008.0 11011.00 11109.0
    aflfast 28800 8.0 10937.000000 37.879508 10871.0 10920.50 10931.0 10961.25 10998.0
    aflplusplus 28800 8.0 10934.000000 60.038083 10862.0 10893.50 10923.0 10962.00 11050.0
    fairfuzz 28800 4.0 10852.000000 245.401711 10597.0 10671.25 10847.5 11028.25 11116.0
    lafintel 28800 14.0 10774.642857 47.960893 10670.0 10743.00 10780.0 10813.50 10844.0
    aflplusplus_qemu 28800 17.0 10790.235294 69.376265 10700.0 10743.00 10768.0 10849.00 10929.0
    manul 28800 16.0 10150.437500 97.226175 9799.0 10144.25 10177.5 10186.25 10234.0
    libfuzzer 28800 4.0 10055.500000 75.244047 9971.0 10004.00 10059.5 10111.00 10132.0
    afl_qemu 28800 7.0 9977.571429 20.313027 9939.0 9972.00 9979.0 9991.00 9999.0
    eclipser 28800 15.0 9555.333333 32.823482 9500.0 9538.00 9560.0 9577.50 9601.0
    honggfuzz_qemu 28800 17.0 9017.000000 0.000000 9017.0 9017.00 9017.0 9017.00 9017.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 27900 3.0 16181.333333 289.717679 15850.0 16078.50 16307.0 16347.00 16387.0
    aflplusplus_optimal 27900 15.0 16142.933333 439.423724 15036.0 15962.50 16130.0 16404.50 16802.0
    afl 27900 11.0 15826.363636 231.137739 15453.0 15695.50 15836.0 15980.00 16162.0
    aflsmart 27900 9.0 15736.777778 237.471987 15423.0 15530.00 15715.0 15933.00 16100.0
    fastcgs_lm 27900 16.0 15635.875000 428.376295 14874.0 15279.00 15691.0 15882.00 16252.0
    mopt 27900 9.0 15239.555556 361.995204 14444.0 15232.00 15377.0 15440.00 15635.0
    aflfast 27900 12.0 15046.750000 171.908604 14818.0 14942.25 15002.5 15187.25 15414.0
    aflplusplus_ctx_nozerosingle 27900 17.0 14991.176471 352.912389 14434.0 14733.00 14934.0 15264.00 15635.0
    aflplusplus_ctx_nosingle 27900 13.0 14787.076923 424.829468 13931.0 14436.00 14922.0 15111.00 15298.0
    aflplusplus_ctx_default 27900 11.0 14698.909091 358.651489 13878.0 14557.50 14715.0 14948.00 15111.0
    aflplusplus_ctx_nozero 27900 13.0 14602.846154 440.774479 13918.0 14438.00 14623.0 14949.00 15328.0
    aflcc 27900 9.0 12937.888889 853.354329 10795.0 13006.00 13251.0 13419.00 13551.0
    entropic 27900 13.0 12675.384615 385.974209 12005.0 12367.00 12778.0 12874.00 13241.0
    aflplusplus_ngram4 27900 5.0 12413.800000 1009.202011 10914.0 11971.00 12671.0 12990.00 13523.0
    honggfuzz 27900 14.0 11595.142857 435.370551 10738.0 11317.00 11807.0 11946.00 12116.0
    fairfuzz 27900 13.0 10516.615385 1859.729306 8442.0 8763.00 11063.0 12210.00 13111.0
    aflplusplus_ngram6 27900 8.0 10402.250000 289.330631 10146.0 10226.25 10272.0 10489.50 10999.0
    aflplusplus_ngram8 27900 12.0 9895.500000 263.070506 9540.0 9714.00 9852.0 10091.50 10273.0
    aflplusplus_qemu 27900 18.0 9466.500000 310.674456 9033.0 9253.50 9415.5 9605.25 10284.0
    libfuzzer 27900 5.0 8706.400000 496.199859 7827.0 8828.00 8908.0 8958.00 9011.0
    lafintel 27900 10.0 8916.400000 148.238696 8618.0 8852.00 8902.5 9025.75 9110.0
    afl_qemu 27900 5.0 8421.600000 44.774993 8344.0 8432.00 8436.0 8436.00 8460.0
    manul 27900 11.0 7360.000000 295.659940 6978.0 7098.00 7442.0 7585.50 7840.0
    eclipser 27900 15.0 5680.600000 531.334412 5423.0 5423.50 5426.0 5430.00 6879.0
    honggfuzz_qemu 27900 17.0 5382.000000 0.000000 5382.0 5382.00 5382.0 5382.00 5382.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 29700 10.0 335.200000 2.529822 331.0 334.00 334.5 337.50 339.0
    aflplusplus_ngram6 29700 14.0 334.071429 2.335818 330.0 333.00 334.0 336.00 337.0
    aflplusplus_ngram4 29700 10.0 331.600000 2.011080 329.0 330.25 331.0 333.75 334.0
    entropic 29700 10.0 328.500000 3.689324 322.0 326.50 330.0 331.00 332.0
    aflplusplus 29700 17.0 331.294118 3.057055 328.0 329.00 330.0 333.00 338.0
    afl 29700 10.0 329.700000 2.359378 327.0 328.25 329.0 330.75 334.0
    aflplusplus_ctx_default 29700 17.0 329.294118 2.023756 324.0 328.00 329.0 330.00 333.0
    aflplusplus_ctx_nosingle 29700 16.0 328.812500 2.315707 324.0 328.00 329.0 329.00 333.0
    aflplusplus_ctx_nozero 29700 16.0 329.312500 2.701080 324.0 328.00 329.0 330.75 334.0
    lafintel 29700 16.0 329.125000 2.061553 326.0 328.00 329.0 329.25 334.0
    aflsmart 29700 14.0 329.785714 2.913591 323.0 328.25 329.0 331.75 334.0
    mopt 29700 3.0 328.000000 1.000000 327.0 327.50 328.0 328.50 329.0
    aflplusplus_qemu 29700 18.0 327.333333 2.700762 322.0 326.00 327.0 328.75 333.0
    aflfast 29700 13.0 324.692308 3.902662 319.0 321.00 325.0 327.00 332.0
    afl_qemu 29700 12.0 324.500000 2.746899 320.0 323.00 324.0 327.00 329.0
    eclipser 29700 13.0 307.846154 4.140172 301.0 306.00 308.0 311.00 313.0
    manul 29700 15.0 216.466667 11.993252 191.0 211.50 215.0 218.50 247.0
    honggfuzz_qemu 29700 16.0 29.000000 0.000000 29.0 29.00 29.0 29.00 29.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 49ee62feb010b69eb936c7b7ac8c021cccf884b8