CLaMPI allows to set the default value of these parameters at compile time. Such values can be overridden at running time by using specific environment variables. So they can be adjusted for each application/settings.

CLaMPI implements an adaptive scheme that adjusts the index and memory buffer sizes dynamically, taking track of a set of performance/efficiency indicators. Here we show the results of a benchmark comparing the same access pattern with and without adaptive scheme. Details of this benchmark can be found in the paper (Section IV.A). The fixed strategy presents poor performance when the index size is small w.r.t. the number of distinct gets due to high number of conflicts. This does not happen with the adaptive strategy, which is able to adjust the hash table size at runtime, minimizing the number of conflicts.

We modified the UPC implementation by Larkins et al. introducing MPI One Sided operations for retrieving/storing cells and leaves of the tree. We enable CLaMPI with the user-defined mode, that allows us to invalidate the cache after the termination of the force computation phase.

We compare the two CLaMPI strategies, adaptive and fixed, with an ad-hoc caching system, referred to as native, that was included in our reference UPC implementation. The adaptive strategy is annotated with the number of performed invalidations/adjustments. The Force Computation time required by foMPI is $1.53$ms. The experiment fixes the number of processing elements to $P=16$ and the total number of bodies to $n=20$K. The CLaMPI parameters - such as index and memory buffer size - are varied in order to find the best setting. The memory size of the native caching solution is set to the same value of the CLaMPI's one. The adaptive strategy is the one presenting the best performance: in all the cases it converges to $1MB$ as memory buffer size, while it sets the length of the index to $20$K entries (averaged on all the processes). With an index of $1K$ entries, the performance of the fixed strategy is limited by the high number of conflicts, The native solution performance heavily depends on its memory size: it ranges from $820us$ with $1$MB to $400us$ with $4$MB memory buffer size. This behavior can be explained by the fact that this system is a block-based software cache with direct mapping, hence the number of conflicts is strictly related to the available memory size.

At the right we show the results of a weak scaling experiment. The index and the memory buffer sizes are set to 30K entries and 2MB, respectively.These are the initial parameters of the adaptive strategy. The number of bodies is set to $N = P \cdot 8K$ and $P$, that is the number of processing elements, varies between $16$ and $128$. Both the CLaMPI strategies outperform the native and foMPI solutions up to a factor of $\sim3$x and $\sim5$x, respectively. With this settings, the adaptive strategy does not perform any adjustment of the initial parameters.

We introduce CLaMPI in an LCC algorithm leveraging one-sided operations. Given an input graph, to compute the LCC of a local vertex $v$, a process $p_i$ has to retrieve the adjacency list of every incident vertex $u$. If the owner of $u$ is not $p_i$, the retrieve operation can be performed as an RMA get. The number of gets issued by a single process depends on the size of its own partition and on the degree of the vertices in such partition. The size of the each issued get depends on the degree of each neighbor of the vertex $v$. The LCC computation exposes data reuse since the adjacency list of the same vertex $u$ can be accessed several times by a single process: every time $u$ appears in the adjacency list of an owned node.

On the right, we compare the vertex processing time of the CLaMPI adaptive and fixed configurations with respect to foMPI. The adaptive strategy is annotated with the number of performed invalidations/adjustments. The CLaMPI fixed strategy with a memory buffer of $64$MB is limited by the significant number of evictions and failed accesses. Increasing the memory buffer size the number of evictions and failed accesses decreases, explaining the $5$x speedup presented by the fixed strategy with respect to foMPI when the memory buffer size is set to $128M$MB and the index size to $256$K entries. The adaptive strategy achieves a speedup similar to the one presented by the best fixed configuration, independently from the starting parameters.

On the left we report the results of the weak scaling experiment, where the problem size per processing element stays constant. A detailed description of the benchmark can be found in the paper. The adaptive strategy is annotated with the total number of automatically performed invalidations/adjustments. Increasing the graph scale with the number of processes leads to a situation in which the number of get per process stays constant but the average get size increases. As a consequence, the fixed strategy suffers of an higher number of evictions and failed accesses when increasing the number of processing elements. This explains the gap between the fixed and the adaptive strategy when $P>32$. In fact, the adaptive strategy is able to resize the memory buffer size in order to accommodate the larger gets,

CMPI_Win cmpi_win;
MPI_Info info;
MPI_Info_create(&info);
MPI_Info_set(info, CLAMPI_MODE, CLAMPI_USER_DEFINED);

int CMPI_Win_create(void * base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, CMPI_Win * win);

int CMPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, void * baseptr, CMPI_Win * win);

int CMPI_Get(void * origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, CMPI_Win win);

int CMPI_Win_flush(int rank, CMPI_Win win);

int CMPI_Win_invalidate(CMPI_Win win);

int CMPI_Win_free(CMPI_Win * win);