FuzzBench: 2020-08-12 report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus_same1 99.97
aflplusplus_same3 99.86
aflplusplus_same2 99.75
By avg. rank
average rank
fuzzer
aflplusplus_same1 1.88
aflplusplus_same3 2.00
aflplusplus_same2 2.12
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    fuzzer aflplusplus_same1 aflplusplus_same2 aflplusplus_same3
    benchmark
    lcms-2017-03-21 1319 1311 1317
    libpng-1.2.56 1508 1509 1509
    vorbis-2017-12-11 2106 2098 2099
    woff2-2016-05-06 1700 1701 1700

lcms-2017-03-21 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_same1 900 20.0 1443.250000 415.324334 1175.0 1256.0 1319.0 1328.0 2408.0
    aflplusplus_same3 900 18.0 1542.944444 459.428606 1171.0 1311.0 1317.0 1335.5 2402.0
    aflplusplus_same2 900 19.0 1443.526316 419.344643 1171.0 1222.0 1311.0 1323.5 2396.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng-1.2.56 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_same2 900 20.0 1508.600000 0.598243 1507.0 1508.0 1509.0 1509.00 1509.0
    aflplusplus_same3 900 19.0 1508.473684 0.611775 1507.0 1508.0 1509.0 1509.00 1509.0
    aflplusplus_same1 900 18.0 1508.222222 0.548319 1507.0 1508.0 1508.0 1508.75 1509.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis-2017-12-11 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_same1 900 19.0 2106.947368 19.831159 2068.0 2090.50 2106.0 2118.50 2140.0
    aflplusplus_same3 900 20.0 2099.550000 18.588267 2076.0 2085.25 2099.0 2107.25 2142.0
    aflplusplus_same2 900 19.0 2101.315789 25.560500 2033.0 2087.00 2098.0 2122.50 2141.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2-2016-05-06 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_same2 900 20.0 1704.150000 19.677064 1671.0 1696.0 1701.0 1708.5 1745.0
    aflplusplus_same1 900 20.0 1617.950000 381.527743 0.0 1682.0 1700.0 1708.0 1770.0
    aflplusplus_same3 900 19.0 1704.263158 25.352716 1673.0 1696.0 1700.0 1705.5 1772.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 0dc59a26ba262ddcdac58d70ce92384780197104