LibSciBench is a framework developed to facilitate the adoption of statistically sound measurements of parallel computations. It automates many of the guidelines and processes described in .
The main component of the framework is LibLSB, a performance measurement library that can be linked to any application to either measure execution times or it can be used as a building block for a new benchmark suite. The library seamlessly integrates with MPI and OpenMP applications and can easily be extended to new models of parallelism.
The datasets produced by LibLSB can be read directly with established statistical tools such as GNU R. LibSciBench is coupled with an R script for the postprocessing of the gathered data. It allows to check for normality, compute CIs, perform ANOVA tests, and quantile regression. Furthermore, LibSciBench’s R script supports the generation of Q-Q plots, box plots and violin plots.
LibSciBench offers high-resolution timers for many architectures (currently x86, x86-64, PowerPC, and Sparc). The library automatically reports the timer resolution and overhead on the target architecture. LibSciBench has support for arbitrary PAPI counters and, as showed in the example below, allows to create group of measurements identified by an arbitrary number of parameters.
Most of today’s parallel systems are asynchronous and do not have a common clock source. Furthermore, clock drift between processes could impact measurements and network latency variations make time synchronization tricky .
Many evaluations use an (MPI or OpenMP) barrier to synchronize processes for time measurement. This may be unreliable because neither MPI nor OpenMP provides timing guarantees for their barrier calls. While a barrier commonly synchronizes processes enough, we recommend checking the implementation. For accurate time synchronization, LibLSB offers a simple delay window scheme .In this scheme, a master synchronizes the clocks of all processes and broadcasts a common start time for the operation. The start time is sufficiently far in the future that the broadcast will arrive before the time itself. Each process then waits until this time and the operations starts synchronously.
Post analysisNormality Check In order to check if a dataset is normally distributed the following methods are offered: Shapiro-Wilk test, QQ-plot and histogram/density plot. Options for log- or k-normalizing the data are provided.
Comparing Statistical Data ANOVA and Kruskal-Waillis one-way ANOVA tests are provided in order to check the statistically significance of the mean or the median, respectively. Quantile Regression plots can be produced in order to observe the effect of varying factor on arbitrary quantiles.
Graphing Results Box and violin plots offer a rich set of statistical information for arbitrary distributions. They can be produced starting from the provided data or its summary.
Filtering & Summarizing The input dataset can be filtered using a R-style boolean expression on the parameter indicated to LibLSB during the measurement phase. A summary of the data can be computed specifying the grouping columns (i.e., parameters specified to LibLSB) and the measurement column (i.e., the measured time or a PAPI counter).
DownloadLatest version: LibLSB GitHub repository
|liblsb-0.2.2.tar.gz - (698 KB)||Aug 21, 2016||Fix window-based synchronization|
|liblsb-0.2.1.tar.gz - (389 KB)||May 20, 2016||Minor fixes|
|liblsb-0.2.tar.gz - (432.2 KB)||November 13, 2015||First release|
| T. Hoefler, R. Belli:|
|Scientific Benchmarking of Parallel Computing Systems
presented in Austin, TX, USA, pages 73:1--73:12, ACM, ISBN: 978-1-4503-3723-6, Nov. 2015, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15) (acceptance rate: 22%, 79/358) |
| T. Hoefler, T. Schneider, A. Lumsdaine:|
|Accurately Measuring Overhead, Communication Time and Progression of Blocking and Nonblocking Collective Operations at Massive Scale
International Journal of Parallel, Emergent and Distributed Systems. Vol 25, Nr. 4, pages 241-258, Taylor & Francis Group, ISSN: 1744-5779, Jul. 2010, |