--- created_at: '2017-02-21T11:28:54.000Z' title: Fooling the masses with performance results on parallel computers (2010) url: https://blogs.fau.de/hager/archives/5260 author: jsnell points: 53 story_text: comment_text: num_comments: 15 story_id: story_title: story_url: parent_id: created_at_i: 1487676534 _tags: - story - author_jsnell - story_13694809 objectID: '13694809' --- [Source](https://blogs.fau.de/hager/archives/5260 "Permalink to Georg Hager's Blog | Fooling the masses with performance results on parallel computers – prelude") # Georg Hager's Blog | Fooling the masses with performance results on parallel computers – prelude * [Skip to Content][1] * [Go to area menu][2] * [Jump to the general information area][3] # [Georg Hager's Blog][4] ### Random thoughts on High Performance Computing ## Area menu * [Home][5] ### Breadcrumb * You are here: * [Home][6]» * Fooling the masses with performance results on parallel computers – prelude ## Kurzinfo ###### Pages * [Contact info][7] * [Publications][8] * [HPC Book][9] * [Talks & Teaching][10] * [Node-Level Performance Engineering: Tutorials & Lectures][11] * [Conference Tutorials][12] * [SC10][13] * [PPoPP11][14] * [ISC11][15] * [HPC 2012][16] * [ISC12][17] * [SC12][18] * [SAHPC 2012][19] * [ISC13][20] * [PPAM 2013][21] * [SC13][22] * [PPoPP 2014][23] * [ISC14][24] * [SC14][25] * [PPoPP 2015][26] * [SC15][27] * [SC16][28] * [QDyn16][29] * [SC17][30] * [Workshops, Symposia, BoFs][31] * [SC11 BoF][32] * [ISC13 BoF][33] * [SIAM PP14 MS][34] * [SC14 BoF][35] * [ISC15 Workshop][36] * [ISC16 Workshop][37] * [ISC17 Workshop][38] ###### Categories * [Cplusplus (3)][39] * [Fooling the masses (18)][40] * [HPC (43)][41] * [Miscellaneous (9)][42] * [RRZE (3)][43] * [Talks (1)][44] * [Tools (1)][45] ###### Tags [benchmarking][46] [C++][47] [ccNUMA][48] [Compilers][49] [energy][50] [Fooling the masses][51] [GPU][52] [LIKWID][53] [MPI][54] [OpenMP][55] [Optimization][56] [performance modeling][57] [SC][58] [TBB][59] [ ![Friedrich-Alexander-Universität Erlangen-Nürnberg][60] ][61] [ ![FAU-Blogdienst][62] ][63] ## Content # Fooling the masses with performance results on parallel computers – prelude [April 30, 2010][64] In 1991, [David H. Bailey][65] published his insightful “[Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers][66].” In that humorous article, Bailey pinpointed typical “evade and disguise” techniques for presenting mediocre performance results in the best possible light. These are the original 12 ways: 1. Quote only 32-bit performance results, not 64-bit results. 2. Present performance figures for an inner kernel, and then represent these figures as the performance of the entire application. 3. Quietly employ assembly code and other low-level language constructs. 4. Scale up the problem size with the number of processors, but omit any mention of this fact. 5. Quote performance results projected to a full system. 6. Compare your results against scalar, unoptimized code on Crays. 7. When direct run time comparisons are required, compare with an old code on an obsolete system. 8. If MFLOPS rates must be quoted, base the operation count on the parallel implementation, not on the best sequential implementation. 9. Quote performance in terms of processor utilization, parallel speedups or MFLOPS per dollar. 10. Mutilate the algorithm used in the parallel implementation to match the architecture. 11. Measure parallel run times on a dedicated system, but measure conventional run times in a busy environment. 12. If all else fails, show pretty pictures and animated videos, and don’t talk about performance. There are further explanations in the original paper for each item. After two decades, it’s high time for an update. In 1991 the supercomputing landscape was governed by the “chicken vs. oxen” debate: The famous question “If you were plowing a field, which would you rather use?… Two strong oxen or 1024 chickens?” is attributed to Seymour Cray who couldn’t have said it better. Cray’s machines were certainly dominating in the oxen department, but competition from massively parallel systems like the Connection Machine was building up. At that time, users were much more used to dive into system-specific optimizations — with no MPI and OpenMP standards, portability of parallel programs was pretty much restricted to a certain vendor. And the use of double precision floating point was probably not as much a matter of course as it is today. In the past two decades, hybrid, hierarchical systems, multi-core processors, accelerator technology, and the dominating presence of commodity hardware have reshaped the landscape of High Performance Computing. It’s also not so much oxen vs. chickens anymore; _ants_ have received more than their share of hype. However, some things never change. My points (which I prefer to call “stunts”) are derived from Bailey’s original collection, and some are identical or merely reformulated. Others are new, reflecting today’s boundary conditions. Although these musings are certainly inspired by experience with many publications and talks in HPC, I wish to point out that (i) no offense is intended, (ii) I am not immune to the inherent temptations myself and (iii) this all still just meant to be fun. This is the list of stunts. It will be extended along the way: 1. [Report speedup instead of absolute performance!][67] 2. [Slow down code execution!][68] 3. [The log scale is your friend!][69] 4. [Quietly employ weak scaling to show off!][70] 5. [Instead of performance, plot absolute runtime versus CPU count!][71] 6. [Ignore affinity and topology issues! ][72] 7. [Be creative when comparing scaled performance!][73] 8. [Impress your audience with awe-inspiring accuracy!][74] 9. [Boast massive speedups with accelerators!][75] 10. [Always emphasize the "interesting" part of your work!][76] 11. [Show data! Plenty. And then some.][77] 12. [Redefine “performance” to suit your needs!][78] 13. [If they get you cornered, blame it all on OS jitter!][79] 14. [Secretly use fancy hardware setups and software tricks!][80] 15. [Play mysterious!][81] 16. [Worship the God of Automation!][82]     This entry was posted in [Fooling the masses][40] and tagged [Fooling the masses][51] by [Georg Hager][83].[Permanenter Link zum Eintrag][84] ### Post navigation ←[ Previous post link][85] [Next post link →][86] ## Technical menu * [Blogs@FAU][63] * [Forum][87] * [Help][88] * [Contact][89] * [Imprint][90] * [Terms of Use][91] [1]: https://blogs.fau.de#contentmarke [2]: https://blogs.fau.de#bereichsmenumarke [3]: https://blogs.fau.de#hilfemarke [4]: https://blogs.fau.de/hager/ [5]: http://blogs.fau.de/hager [6]: https://blogs.fau.de/hager [7]: https://blogs.fau.de/hager/contact-info [8]: https://blogs.fau.de/hager/publications [9]: https://blogs.fau.de/hager/hpc-book [10]: https://blogs.fau.de/hager/talks [11]: https://blogs.fau.de/hager/talks/nlpe [12]: https://blogs.fau.de/hager/tutorials [13]: https://blogs.fau.de/hager/tutorials/sc10 [14]: https://blogs.fau.de/hager/tutorials/ppopp11 [15]: https://blogs.fau.de/hager/tutorials/isc11 [16]: https://blogs.fau.de/hager/tutorials/hpc-2012 [17]: https://blogs.fau.de/hager/tutorials/isc12 [18]: https://blogs.fau.de/hager/tutorials/sc12 [19]: https://blogs.fau.de/hager/tutorials/sahpc-2012 [20]: https://blogs.fau.de/hager/tutorials/isc13 [21]: https://blogs.fau.de/hager/tutorials/ppam-2013 [22]: https://blogs.fau.de/hager/tutorials/sc13 [23]: https://blogs.fau.de/hager/tutorials/ppopp-2014 [24]: https://blogs.fau.de/hager/tutorials/isc14 [25]: https://blogs.fau.de/hager/tutorials/sc14 [26]: https://blogs.fau.de/hager/tutorials/ppopp-2015 [27]: https://blogs.fau.de/hager/tutorials/sc15 [28]: https://blogs.fau.de/hager/tutorials/sc16 [29]: https://blogs.fau.de/hager/tutorials/qdyn16 [30]: https://blogs.fau.de/hager/tutorials/sc17 [31]: https://blogs.fau.de/hager/bofs [32]: https://blogs.fau.de/hager/bofs/singlenode-sc11 [33]: https://blogs.fau.de/hager/bofs/isc13 [34]: https://blogs.fau.de/hager/bofs/siam-pp14-ms [35]: https://blogs.fau.de/hager/bofs/sc14-bof [36]: https://blogs.fau.de/hager/bofs/isc15-workshop-on-performance-modeling [37]: https://blogs.fau.de/hager/bofs/isc16-workshop [38]: https://blogs.fau.de/hager/bofs/isc17-workshop [39]: https://blogs.fau.de/hager/archives/category/cplusplus [40]: https://blogs.fau.de/hager/archives/category/fooling-the-masses [41]: https://blogs.fau.de/hager/archives/category/hpc [42]: https://blogs.fau.de/hager/archives/category/allgemeines [43]: https://blogs.fau.de/hager/archives/category/rrze [44]: https://blogs.fau.de/hager/archives/category/talks [45]: https://blogs.fau.de/hager/archives/category/tools [46]: https://blogs.fau.de/hager/archives/tag/benchmarking [47]: https://blogs.fau.de/hager/archives/tag/c [48]: https://blogs.fau.de/hager/archives/tag/ccnuma [49]: https://blogs.fau.de/hager/archives/tag/compilers [50]: https://blogs.fau.de/hager/archives/tag/energy [51]: https://blogs.fau.de/hager/archives/tag/fooling-the-masses [52]: https://blogs.fau.de/hager/archives/tag/gpu [53]: https://blogs.fau.de/hager/archives/tag/likwid [54]: https://blogs.fau.de/hager/archives/tag/mpi [55]: https://blogs.fau.de/hager/archives/tag/openmp [56]: https://blogs.fau.de/hager/archives/tag/optimization [57]: https://blogs.fau.de/hager/archives/tag/performance-modeling [58]: https://blogs.fau.de/hager/archives/tag/sc [59]: https://blogs.fau.de/hager/archives/tag/tbb [60]: https://blogs.fau.de/hager/wp-content/themes/blue-edgy/images/fau-logo.png [61]: http://www.fau.de [62]: https://blogs.fau.de/hager/wp-content/themes/blue-edgy/images/network-logo.png [63]: http://blogs.fau.de/ [64]: https://blogs.fau.de/hager/archives/5260 "4:30 pm" [65]: http://crd.lbl.gov/~dhbailey/ [66]: http://crd.lbl.gov/~dhbailey/dhbpapers/twelve-ways.pdf [67]: http://blogs.fau.de/hager/stories/5299/ "Link to Stunt 1" [68]: http://blogs.fau.de/hager/stories/5429/ "Link to Stunt 2" [69]: http://blogs.fau.de/hager/stories/5835/ "Link to Stunt 3" [70]: http://blogs.fau.de/hager/?p=5864 "Link to Stunt 4" [71]: http://blogs.fau.de/hager/?p=6130 [72]: http://blogs.fau.de/hager/2012/05/11/fooling-the-masses-stunt-6-ignore-affinity-and-topology-issues/ "Fooling the masses – Stunt 6: Ignore affinity and topology issues." [73]: http://blogs.fau.de/hager/2012/06/02/fooling-the-masses-stunt-7-be-creative-when-comparing-scaled-performance/ "Fooling the masses – Stunt 7: Be creative when comparing scaled performance" [74]: http://blogs.fau.de/hager/2012/10/23/fooling-the-masses-stunt-8-impress-your-audience-with-awe-inspiring-accuracy/ "Fooling the masses – Stunt 8: Impress your audience with awe-inspiring accuracy" [75]: http://blogs.fau.de/hager/2012/11/03/fooling-the-masses-stunt-9-boast-massive-speedups-with-accelerators/ "Fooling the masses – Stunt 9: Boast massive speedups with accelerators!" [76]: http://blogs.fau.de/hager/2013/02/11/fooling-the-masses-stunt-10-always-emphasize-the-interesting-part-of-your-work/ "Fooling the masses – Stunt 10: Always emphasize the " [77]: http://blogs.fau.de/hager/2013/02/18/fooling-the-masses-stunt-11-show-data-plenty-and-then-some/ "Fooling the masses – Stunt 11: Show data! Plenty. And then some." [78]: http://blogs.fau.de/hager/2013/03/23/fooling-the-masses-stunt-12-redefine-performance-to-suit-your-needs/ "Fooling the masses – Stunt 12: Redefine " [79]: http://blogs.fau.de/hager/2013/05/16/fooling-the-masses-stunt-13-if-they-get-you-cornered-blame-it-all-on-os-jitter/ "Fooling the masses – Stunt 13: If they get you cornered, blame it all on OS jitter!" [80]: http://blogs.fau.de/hager/2013/05/26/fooling-the-masses-stunt-14-secretly-use-fancy-hardware-setups-and-software-tricks/ "Fooling the masses – Stunt 14: Secretly use fancy hardware setups and software tricks!" [81]: http://blogs.fau.de/hager/archives/7312 "Fooling the masses – Stunt 15: Play mysterious!" [82]: http://blogs.fau.de/hager/archives/6695 "Fooling the masses – Stunt 15: Play mysterious!" [83]: https://blogs.fau.de/hager/archives/author/unrz55 [84]: https://blogs.fau.de/hager/archives/5260 "Permalink zu Fooling the masses with performance results on parallel computers – prelude" [85]: https://blogs.fau.de/hager/archives/4119 [86]: https://blogs.fau.de/hager/archives/5299 [87]: http://www.portal.uni-erlangen.de/forums/viewforum/94 [88]: http://blogs.fau.de/hilfe/ [89]: http://blogs.fau.de/hager/kontakt/ [90]: http://blogs.fau.de/impressum/ [91]: http://blogs.fau.de/nutzungsbedingungen/