I do not know if this problem have been solved yet, but the variable I_MPI_PIN_PROCESSOR_LIST=0,16,32,48 should do the trick. Vitali p.s. The question remains whether Intel MPI uses the same processor ids as KMP_AFFINITY, but I_MPI_DEBUG should help to determine that. On 04/22/2016 02:02 PM, Harms, Kevin wrote:

I found this environment variable that prints out the process bound list. I_MPI_DEBUG=5

[harms@ftlogin1 tests]$ I_MPI_DEBUG=5 mpirun -n 2 ./mpi_test [0] MPI startup(): Multi-threaded optimized library [0] MPI startup(): shm data transfer mode [1] MPI startup(): shm data transfer mode [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 29236 ftlogin1 {0,1,2,3,8,9,10,11} [0] MPI startup(): 1 29237 ftlogin1 {4,5,6,7,12,13,14,15} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 4

If Intel MPI pins the processes sanely on KNL, then coupling that with KMP_AFFINITY=balanced,granularity=thread should give you a good result. Here's an explanation of balanced: https://software.intel.com/en-us/node/522518

If Intel MPI doesn't know KNL and does something weird then you will have to try to understand KNL's tile/core numbering scheme which appears complex.

kevin

...

KMP_AFFINITY only works for thread affinity though, no? If I have four MPI processes, each of the processes will assign threads to the same cores rather than cores in a different sub-NUMA cluster. I need to be able to specify process affinity and my problem is that 1) numactl can only control a single process and 2) the usual mpirun -bind-to and -map-by options don't really have enough levels of abstraction to match the KNL's hierarchy. For what it's worth, I tried mpirun -bind-to numa -map-by numa ... and that resulted in an error. It's not even clear, to me at least, what to expect if you say 'numa' -- one could interpret it to mean a tile (that shares L2 cache) or a cluster (that shares MCDRAM).

On Fri, Apr 22, 2016 at 9:46 AM, Harms, Kevin <[email protected]> wrote:

KMP_AFFINITY=balanced is supposed to do that but if it's not working, it might be that the MPI library does not have proper tuning for KNL yet.

kevin

...

I'm trying to test a code using hybrid MPI/OpenMP on the KNL node but haven't been able to successfully set process affinity. I'd like to run four processes in SNC-4 mode with each process being bound to a cluster. Does anyone know if this is possible?

Thanks, Paul

_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda