Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

T. Hoefler, J. Domke:

 Fail-in-Place Network Design

(presented in Oslo, Norway, May 2015)

Publisher Reference


Hardware failures in networks are omnipresent and often inevitable. The growing system size of high performance computers results in a steady decrease of the mean time between failures. Exchanging network components often requires whole system downtime which increases the cost of failures. In this work, we study a fail-in-place strategy where broken network elements remain untouched for longer periods of time. Each hardware failure of a network component will decrease the overall bandwidth of the network and the rerouting delay will cause disconnects for a subset of the paths. We show, that a fail-in-place strategy is feasible for today’s networks and the degradation is manageable, and provide guidelines for the design. Our network failure simulation toolchain allows system designers to extrapolate the performance degradation based on expected failure rates, and it can be used to evaluate the current state of a system. In a case study of real-world HPC systems, we will analyze the performance degradation throughout the system’s lifetime under the assumption that faulty network components are not repaired, which results in a recommendation to change the used routing algorithm to improve the network performance as well as the fail-in-place characteristic. We firmly believe that fail-in-place is a practical strategy to move towards the next-generation large-scale networks.


download slides:


  author={Torsten Hoefler and Jens Domke},
  title={{Fail-in-Place Network Design}},
  location={Oslo, Norway},