cache-intel using RP temporary Intel MPI installation,
cache-intel{2,3} using /cm/shared/... version

For cache-intel{2,3} other network traffic may have slowed down MPI:

[rperry@Node-001 mpi]$ squeue -a
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              8557    shared benchmar   yqian1  R      28:55      4 Node-[005-008]
              8558      fast benchmar   yqian1  R       9:01      4 Node-[009-012]
[rperry@Node-001 mpi]$

% paste cache-intel.out cache-intel{2,3}.out
21  7 1.11	21  7 1.3	21  7 0.93
21  9 0.82	21  9 0.85	21  9 0.86
21 11 0.83	21 11 0.84	21 11 0.85
21 13 0.81	21 13 0.84	21 13 0.86
21 15 0.81	21 15 0.84	21 15 0.86
25  7 2.34	25  7 2.31	25  7 2.5
25  9 2.1	25  9 2.01	25  9 2.07
25 11 2.01	25 11 2.17	25 11 2.09
25 13 1.99	25 13 2.07	25 13 2.04
25 15 1.76	25 15 1.84	25 15 2.01
31  7 105.13	31  7 110.92	31  7 112.48
31  9 91.29	31  9 91.35	31  9 91.93
31 11 90.35	31 11 94.54	31 11 89.06
31 13 86.49	31 13 87.84	31 13 107.84
31 15 82.1	31 15 98.81	31 15 96.35