Graph500 Benchmark: MPI Reference Implementation
Jeremiah Willcock and Andrew Lumsdaine

This directory contains an MPI-based reference implementation for the
Graph500 benchmark.  The Makefile will automatically build the
supporting code, along with two reference implementations of
breadth-first search and a skeleton version for users to modify.  The
Makefile may need to be modified with your MPI location and compiler
flags.  The generated programs, named graph500_mpi_*, take up to two
parameters: the scale of the problem and the edge factor.  The problem
scale is the logarithm, base 2, of the number of vertices in the graph;
only graphs with power-of-2 vertex counts are supported without source
code modification.  The edge factor is the ratio of the number of edges
to the number of vertices; i.e., it is half the average vertex degree in
the graph.  The scale parameter is mandatory; the edge factor is
optional and defaults to 16 (the value specified by the benchmark).
Running any of the graph500_mpi_* programs without any arguments will
produce a usage message.

The raw graph tuples produced by the graph generator can be stored
either in memory or in a file (using MPI file I/O).  If the TMPFILE
environment variable is set, it should point to a filename (in the sense
of section 13.2.1 of the MPI 2.2 standard) whose corresponding file does
not exist and which points to a filesystem that is globally accessible
and consistent on all ranks and has enough storage space for the graph
data.  The amount of space required is given in
<URL:http://www.graph500.org/Specifications.html#tbl:classes> or can be
computed as 256*(2**SCALE) bytes.  If GENERATOR_USE_PACKED_EDGE_TYPE is
#define'd in ../generator/user_settings.h, the space required is reduced
to 192*(2**SCALE) bytes.

The code is written in C; the code compiles with GCC's default gnu89
language setting, but should be valid C99 and C++ (except for the use of
a few C99 headers).  The main non-C89 features used are variable
declarations after statements in a block and the <stdint.h> and
<inttypes.h> headers.  The code assumes that your MPI implementation is
compliant with version 2.2 of the MPI standard; it uses some MPI
datatypes that were added in that version, and the result validator uses
MPI-2 RMA (one-sided) operations.  If your MPI does not have the new
datatypes, you can edit mpi_workarounds.h to define them manually.  If
your MPI does not have RMA support, or its RMA support does not work
correctly, there is a section in mpi_workarounds.h that enables an
emulation of one-sided functionality; the emulation is not as fast as a
native implementation, however.

A template for writing your own BFS using the reference data structures
and infrastructure is in bfs_custom.c.  You can either modify that file
in place or copy it (adjusting the Makefile) to create your own version.
The documentation for what data structures are available and how to use
them is in comments in bfs_custom.c.  The reference implementation also
contains code to convert from a distributed list of graph edges into a
distributed compressed sparse row data structure, as well as code for
timing the BFS run, validating the correctness of the results, and
printing the timings in the Graph500-required format.

------------------------------------------------------------------------
Troubleshooting:

- A compile error about your MPI implementation not being MPI
  2.2-compliant is triggered from mpi_workarounds.h.  If your MPI is
  older than that, it will not define certain MPI datatypes such as
  MPI_INT64_T that were introduced in MPI 2.2.  The file
  mpi_workarounds.h contains #defines for those types; after you ensure
  that the versions in that file are correct, you can remove the #error
  line that triggers the compile error.

- Some MPI implementations use excessive memory when using MPI-2 one-sided
  (RMA) operations, leading to out-of-memory errors or system swapping.
  Enabling one-sided operation emulation (EMULATE_ONE_SIDED) in
  mpi_workarounds.h works around the problem.

- Segmentation faults or "sreq != NULL" assertion failures in the MPI
  implementation, triggered from inside validate.c, are caused by MPICH2
  bug 1156 at <URL:https://trac.mcs.anl.gov/projects/mpich2/ticket/1156>
  (or similar code in other MPI implementations).  Enabling one-sided
  operation emulation (EMULATE_ONE_SIDED) in mpi_workarounds.h works
  around the problem.

- Undefined symbol errors when using the IBM compilers are due to a
  compiler bug triggered by two source files having the same name, even
  though they are in different directories.  Renaming one of the utils.c
  files and updating the Makefile in the mpi/ directory works around the
  problem.

------------------------------------------------------------------------
Copyright (C) 2009-2011 The Trustees of Indiana University.

Use, modification and distribution is subject to the Boost Software License,
Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at
http://www.boost.org/LICENSE_1_0.txt)
