Netgauge - A Network Performance Measurement Toolkit

Netgauge is a high-precision network parameter measurement tool. It supports benchmarking of many different network protocols and communication patterns. The main focus lies on accuracy, statistical analysis and easy extensibility. Download Netgauge:

See README file in package!

Building Netgauge

If something is wrong with the high-precision timer (e.g., frequency scaling invalidates the results), MPI_Wtime() can be selected at configure time by passing "HRT_ARCH=6" to configure!

On Linux Systems:
  • ./configure MPICC=mpicc MPICXX=mpicxx # set appropriately
  • make
  • mpirun -n 2 ./netgauge (-- help)

On Cray XT-4/5 (tested on Jaguar@ORNL):
  • ./configure MPICC=cc MPICXX=CC
  • make
  • aprun -n 2 -N 1 ./netgauge (-- help)

On Blue Gene/P (tested on Eugene@ORNL):
  • ./configure HRT_ARCH=3 # needs override because cross-compilation can't execute
  • make
  • submit job to batch system

On AIX and Linux/POE (tested on BluePrint@NCSA and BlueDrop@NCSA):
  • edit llscript.ll for your setting (if necessary)
  • export MP_LLFILE=$(pwd)/llscript.ll
  • ./configure MPICC=mpcc CFLAGS="-qlanglvl=extc99" MPICXX=mpCC ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes HRT_ARCH=6 # inline assembler not gcc compatible
  • make
  • submit job to batch system

Netgauge is developed in the Computer Architecture Group at TU Chemnitz and the Open Systems Lab at Indiana University. Many students contributed to netgauge, see the "Authors"-file in the distribution.

Netgauge's Main Goals

  • enable portability (a single tool measures many networks and communication patterns)
  • avoid common benchmarking problems (pipeline-effects, influence of outliers to the average values)
  • enable detailed measurements (more than rount trip time (RTT) and bandwidth
  • parametrize network models
  • offer a simple and extensible framework to implement different ideas
  • combine efforts of network developers (nwho write network modules) and application scientists (who are interested in communication patterns)
  • offer a portable high-precision timer
  • support many networks and patterns "out of the box"

Netgauge's Patterns and Modules

The Netgauge framework consist mainly of commmunication patterns that implement benchmarks and communication modules that implement low-level network drivers. Other parts are the netgauge framework itself which interact with the user to read and process command line arguments and passes them on to the modules and patterns. A statistics and output framework can be used by the pattern implementor to easily achieve common tasks (e.g., get average/median or print data to a file). The general framework is shown in the following figure:


The variable P is used to represent the number of nodes in MPI_COMM_WORLD!
  • one_one - simple ping-pong patterns, if started with more than 2 processes, half of the processes (P/2)communicate with the other half in a ping-pong scheme
  • 1toN - one to many communication scheme, a randomly chosen root communicates data to P-1 processes and they send it back
  • Nto1 - one to many communication scheme, P-1 processes communicate data to a randomly chosen root it sends it back
  • meas_o - measures the overhead in the pLogP model as described in "Fast Measurement of LogP Parameters for Message Passing Platforms"
  • loggp - measures LogGP parameters with the method described in [2]


Different communication modules for many networks are available out of the box. Many of them are experimental which means that the implementation is not well tested. Netgauge is a research tool, so users should expect errors (patches are generally welcome).

Two Sided Protocols
  • MPI - Message Passing Interface (MPI) communication
  • TCP - Transmission Control Protocol (TCP) comunication, implemented with standard POSIX sockets
  • UDP - Unreliable Datagram Protocol (UDP) communication, implemented with standard POSIX sockets
  • ETH - low-level Ethernet socket communication. Using RAW sockets (Linux) and forging customized Ethernet packets
  • EDP/ESP - Ethernet Datagram/Stream Protocol implementation (experimental)
  • IB - experimental InfiniBand implementation

One Sided Protocols
  • MPI2OS - experimental MPI-2 One Sided implementation
  • ARMCI - Aggregate Remote Memory Copy Interface implementation

The internal interface of Netgauge is two-sided. Thus all one-sided operations are "mapped" to two-sided operations. The scheme is depicted in the following figure. The module is supposed to issue the one-sided data-transfer and after that increment a counter on the remote receiver side. The network has to guarantee that those two operations do not pass each other! Both partners keep track of the appropriate peer's counter values.


IP over IB (ofed 1.1) compared with Open MPI 1.1:


[1] T. Hoefler, T. Mehlan, A. Lumsdaine, W. Rehm:
 Netgauge: A Network Performance Measurement Framework Vol 4782, In Proceedings of High Performance Computing and Communications, HPCC'07, presented in Houston, USA, pages 659-671, Springer, ISBN: 978-3-540-75443-5, Sep. 2007,
[2] T. Hoefler, A. Lichei, W. Rehm:
 Low-Overhead LogGP Parameter Assessment for Modern Interconnection Networks TU Chemnitz. In Proceedings of the 21st IEEE International Parallel & Distributed Processing Symposium, PMEO'07 Workshop, presented in Long Beach, CA, USA, IEEE Computer Society, ISBN: 1-4244-0909-8, Mar. 2007,