September 2016 - Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] MKL KNL BLAS performance
by Kenneth E Jansen 11 Sep '16

11 Sep '16

Does anyone know where I can find what the expected performance of level 1 and level 2 BLAS on KNL using MKL should be? For now lets keep it simple DDOT n=O(10k) DGEMV n=O(10k) and m O(100) called in “T” mode so that the fast/first index of the passed matrix is the one that is contracted with the vector x to create an m length y result. Benchmark or even personal experience would be useful. I know somewhere on one of the Wiki’s someone posted something like 1500GF/node for the 7210 for DGEMM so if anyone knows the comparable number for DDOT or DGEMV that would be great. If it is known how big n, and need to be to achieve the quoted number that would also be great. In a related question, is there anything aside from adding -mkl to the compiler flags to get the code to use the AVX512 MKL code? This was enough to get correct results but is it enough to get best performance? Further followup, does Advisor or VTUNE have knowledge of what MKL is doing? I ask this because Advisor is telling me that dgemv and ddot are scalar and tells me I should use the vector form so I am kind of hoping that I am missing something but I recognize that Advisor may just not have the ability to gather that information and assumes external stuff is scalar? I have asked this in the nda list. Is this sort of information still nda? Thanks for any help with even one of these questions. Best regards, Ken ---------------------------------------------------------------- Kenneth E. Jansen, Professor of Aerospace Engineering Sciences ECAE 161 OFFICE (303) 492-4359 429 UCB FAX (303) 492-4990 University of Colorado at Boulder jansenke(a)colorado.edu Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace ----------------------------------------------------------------

1 0

Re: [JLSE-Intel-NDA] MCRAM/Memkind/Vtune questions
by Vitali Morozov 07 Sep '16

07 Sep '16

Hi Fabian, let us meet tomorrow at 10 am and try to understand the output. Vitali On 09/07/2016 05:22 PM, Delalondre Fabien Jonathan wrote: > > Hi All, > > > I am trying to benchmark and understand performance using Vtune memory > bandwidth analysis when allocating memory in DRAM and MCRAM explicitly > using both memkind and hbwmalloc interface (on a toy example). > > > The KNL config I used is FLAT/Quadrant. I am just using a single core > with 4 omp threads and I am not sure to understand the results I am > getting from vtune. > > > Who would feel kind enough and comfortable enough with Vtune to spend > 15-20min with me maybe tomorrow ? > > > Few questions: > > 1) Even though I allocate memory first in DRAM, then in MCDRAM (adding > a barrier between the two), Vtune seems to indicate DRAM/MCDRAM memory > bandwidth utilisation is overlapping as if it would read data from > both MCDRAM/DRAM at the same time. > > Note that I make sure to explicitly initialise memory to force the > allocation. > > 2) What's the difference between package and channel memory in this > context. I would expect the channel to be the memory channel > monitored. But then... if I am using a single core, why is the DRAM > package memory bandwidth different from the channel one... > > > Thanks for your help. > > > Fabien > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] MCRAM/Memkind/Vtune questions
by Delalondre Fabien Jonathan 07 Sep '16

07 Sep '16

Hi All, I am trying to benchmark and understand performance using Vtune memory bandwidth analysis when allocating memory in DRAM and MCRAM explicitly using both memkind and hbwmalloc interface (on a toy example). The KNL config I used is FLAT/Quadrant. I am just using a single core with 4 omp threads and I am not sure to understand the results I am getting from vtune. Who would feel kind enough and comfortable enough with Vtune to spend 15-20min with me maybe tomorrow ? Few questions: 1) Even though I allocate memory first in DRAM, then in MCDRAM (adding a barrier between the two), Vtune seems to indicate DRAM/MCDRAM memory bandwidth utilisation is overlapping as if it would read data from both MCDRAM/DRAM at the same time. Note that I make sure to explicitly initialise memory to force the allocation. 2) What's the difference between package and channel memory in this context. I would expect the channel to be the memory channel monitored. But then... if I am using a single core, why is the DRAM package memory bandwidth different from the channel one... Thanks for your help. Fabien

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Tom LeCompte 02 Sep '16

02 Sep '16

Hi Jeff- This seems to be it. What happens is at the beginning of the code, when I am only using MPI, everything is fine. As soon as it launches the pthreads (one per rank), they all land on the same core. I'm now poking around in that section of the code. Thanks everyone for your help! Cheers, Tom On 9/2/2016 8:53 AM, Hammond, Jeff R wrote: > In the past, MVAPICH2 plus Pthreads would do this because of default > affinity mask being the location of the process. > > You are using Intel MPI and calling MPI_Init_thread with > MPI_THREAD_SERIALIZED or MPI_THREAD_MULTIPLE, right? > > Intel MPI might set affinity mask and assume OpenMP runtime changes it > later for threading codes. You should explicitly disable Intel MPI > affinity environment variables for debugging purposes to see if that > is the issue. > > In any case, it might be productive to explicitly set your Pthread > affinity mask with the appropriate Linux API. > > Jeff > > Sent from my iPhone > > On Sep 2, 2016, at 6:48 AM, Luo, Ye <yeluo(a)anl.gov > <mailto:[email protected]>> wrote: > >> Just tried it. I can do it. >> You must have your job running. >> qstat -lf your_job_id >> find your node name and ssh to it. >> >> Ye >> >> =================== >> Ye Luo, Ph.D. >> Leadership Computing Facility >> Argonne National Laboratory >> ------------------------------------------------------------------------ >> *From:* Tom LeCompte [lecompte(a)anl.gov <mailto:[email protected]>] >> *Sent:* Friday, September 02, 2016 8:31 AM >> *To:* Luo, Ye; intel-nda(a)jlse.anl.gov <mailto:[email protected]> >> *Subject:* Re: [JLSE-Intel-NDA] MPI vs Threading Performance >> >> Unfortunately, I don't think that works any more. ssh knl08 returns >> Connection closed by 10.17.6.112. >> >> >> On 9/2/2016 8:28 AM, Luo, Ye wrote: >>> There is quick way to check what is going on in the node. >>> start your job with 8 ranks. >>> ssh to the node you job is assigned to. >>> top and see if there are 8 processes of your application running at >>> about 100% CPU. >>> >>> Ye >>> >>> =================== >>> Ye Luo, Ph.D. >>> Leadership Computing Facility >>> Argonne National Laboratory >>> ------------------------------------------------------------------------ >>> *From:* intel-nda-bounces(a)lists.jlse.anl.gov >>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Tom LeCompte >>> [lecompte(a)anl.gov] >>> *Sent:* Friday, September 02, 2016 7:31 AM >>> *To:* intel-nda(a)jlse.anl.gov >>> *Subject:* [JLSE-Intel-NDA] MPI vs Threading Performance >>> >>> Hi everyone- >>> >>> I am investigating the relative performance of threading vs. MPI, >>> and don't understand the results: I expect threading to be better, >>> but not that much better. >>> >>> I have a fixed amount of work per thread or rank, so I expect with >>> perfect scaling run times to be roughly constant. With threading, >>> that's roughly what I see, and even the deviations make sense: there >>> are inflections at 64, 256 and a dropoff around 280-300 threads. >>> With MPI, I see the following: >>> >>> * 1 rank: 77s reported by the code, 213s wallclock >>> * 2 ranks: 152s reported by the code, 288s wallclock >>> * 8 ranks: 635s reported by the code, 769s wallclock >>> * 32 ranks 2504s reported by the code, 2648s wallclock >>> * 64 ranks 5463s reported by the code, 5613s wallclock >>> >>> >>> So it takes about twice as much time to do twice as much work with >>> twice as many ranks. This pattern persists up to at least 64 >>> ranks. The natural explanation would be that I am really running on >>> only one core. I ran with I_MPI_DEBUG set to 5, and the results are >>> below. This looks right to me, but I am not an expert. >>> >>> Any thoughts? >>> >>> Cheers, >>> Tom >>> >>> >>> >>> ========================================================= >>> >>> 1 rank: >>> [0] MPI startup(): Rank Pid Node name Pin cpu >>> [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov >>> <http://knl08.ftm.alcf.anl.gov> >>> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >>> 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 >>> ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 >>> 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 >>> 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 >>> 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 >>> 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 >>> 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 >>> 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 >>> 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 >>> 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 >>> 7,248,249,250,251,252,253,254,255} >>> [0] MPI startup(): I_MPI_DEBUG=5 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >>> [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 >>> >>> >>> 2 ranks: >>> [0] MPI startup(): Rank Pid Node name Pin cpu >>> [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov >>> <http://knl07.ftm.alcf.anl.gov> >>> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >>> 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 >>> ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 >>> 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 >>> 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 >>> 4,215,216,217,218,219,220,221,222,223} >>> [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov >>> <http://knl07.ftm.alcf.anl.gov> >>> {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 >>> ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, >>> 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, >>> 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, >>> 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, >>> 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} >>> [0] MPI startup(): I_MPI_DEBUG=5 >>> [0] MPI startup(): >>> I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 >>> [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 >>> Configuration : >>> >>> >>> 8 ranks: >>> [0] MPI startup(): Rank Pid Node name Pin cpu >>> [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} >>> [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} >>> [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} >>> [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} >>> [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} >>> [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} >>> [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} >>> [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> >>> {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} >>> [0] MPI startup(): I_MPI_DEBUG=5 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >>> [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 >>> 48,7 56 >>> >>> >>> >>> 32 ranks: >>> [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {0,1,64,65,128,129,192,193} >>> [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {2,3,66,67,130,131,194,195} >>> [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {4,5,68,69,132,133,196,197} >>> [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {6,7,70,71,134,135,198,199} >>> [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {8,9,72,73,136,137,200,201} >>> [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {10,11,74,75,138,139,202,203} >>> [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {12,13,76,77,140,141,204,205} >>> [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {14,15,78,79,142,143,206,207} >>> [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {16,17,80,81,144,145,208,209} >>> [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {18,19,82,83,146,147,210,211} >>> [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {20,21,84,85,148,149,212,213} >>> [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {22,23,86,87,150,151,214,215} >>> [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {24,25,88,89,152,153,216,217} >>> [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {26,27,90,91,154,155,218,219} >>> [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {28,29,92,93,156,157,220,221} >>> [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {30,31,94,95,158,159,222,223} >>> [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {32,33,96,97,160,161,224,225} >>> [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {34,35,98,99,162,163,226,227} >>> [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {36,37,100,101,164,165,228,229} >>> [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {38,39,102,103,166,167,230,231} >>> [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {40,41,104,105,168,169,232,233} >>> [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {42,43,106,107,170,171,234,235} >>> [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {44,45,108,109,172,173,236,237} >>> [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {46,47,110,111,174,175,238,239} >>> [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {48,49,112,113,176,177,240,241} >>> [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {50,51,114,115,178,179,242,243} >>> [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {52,53,116,117,180,181,244,245} >>> [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {54,55,118,119,182,183,246,247} >>> [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {56,57,120,121,184,185,248,249} >>> [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {58,59,122,123,186,187,250,251} >>> [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {60,61,124,125,188,189,252,253} >>> [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov >>> <http://knl04.ftm.alcf.anl.gov> {62,63,126,127,190,191,254,255} >>> [0] MPI startup(): I_MPI_DEBUG=5 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >>> [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 >>> 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 >>> 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 >>> 58,30 60,31 62 >>> >>> 64 ranks: >>> [0] MPI startup(): Rank Pid Node name Pin cpu >>> [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {0,64,128,192} >>> [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {1,65,129,193} >>> [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {2,66,130,194} >>> [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {3,67,131,195} >>> [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {4,68,132,196} >>> [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {5,69,133,197} >>> [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {6,70,134,198} >>> [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {7,71,135,199} >>> [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {8,72,136,200} >>> [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {9,73,137,201} >>> [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {10,74,138,202} >>> [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {11,75,139,203} >>> [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {12,76,140,204} >>> [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {13,77,141,205} >>> [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {14,78,142,206} >>> [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {15,79,143,207} >>> [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {16,80,144,208} >>> [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {17,81,145,209} >>> [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {18,82,146,210} >>> [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {19,83,147,211} >>> [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {20,84,148,212} >>> [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {21,85,149,213} >>> [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {22,86,150,214} >>> [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {23,87,151,215} >>> [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {24,88,152,216} >>> [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {25,89,153,217} >>> [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {26,90,154,218} >>> [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {27,91,155,219} >>> [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {28,92,156,220} >>> [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {29,93,157,221} >>> [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {30,94,158,222} >>> [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {31,95,159,223} >>> [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {32,96,160,224} >>> [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {33,97,161,225} >>> [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {34,98,162,226} >>> [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {35,99,163,227} >>> [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {36,100,164,228} >>> [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {37,101,165,229} >>> [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {38,102,166,230} >>> [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {39,103,167,231} >>> [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {40,104,168,232} >>> [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {41,105,169,233} >>> [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {42,106,170,234} >>> [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {43,107,171,235} >>> [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {44,108,172,236} >>> [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {45,109,173,237} >>> [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {46,110,174,238} >>> [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {47,111,175,239} >>> [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {48,112,176,240} >>> [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {49,113,177,241} >>> [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {50,114,178,242} >>> [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {51,115,179,243} >>> [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {52,116,180,244} >>> [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {53,117,181,245} >>> [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {54,118,182,246} >>> [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {55,119,183,247} >>> [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {56,120,184,248} >>> [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {57,121,185,249} >>> [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {58,122,186,250} >>> [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {59,123,187,251} >>> [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {60,124,188,252} >>> [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {61,125,189,253} >>> [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {62,126,190,254} >>> [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov >>> <http://knl06.ftm.alcf.anl.gov> {63,127,191,255} >>> [0] MPI startup(): I_MPI_DEBUG=5 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >>> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >>> [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 >>> 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 >>> 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 >>> 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 >>> 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 >>> 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 >>> 62,63 63 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov <mailto:[email protected]> >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Luo, Ye 02 Sep '16

02 Sep '16

I was using mpich-intel-3.0.4 first which gave me hard time. Later the 3.1.4 is installed. I don't remember if I have tested it or not. Since the Intel compiler on JLSE has everything including compiler wrapper and runtime. I just use the consistent set all the time now. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Hammond, Jeff R [jeff.r.hammond(a)intel.com] Sent: Friday, September 02, 2016 9:18 AM To: Luo, Ye Cc: LeCompte, Thomas J.; intel-nda(a)jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance I_MPI_VERBOSE>0 is a good way to verify mpirun is from Intel MPI but MPICH Hydra is used by Intel MPI so I would be surprised if mpirun is the problem here. Sent from my iPhone On Sep 2, 2016, at 7:02 AM, Luo, Ye <yeluo(a)anl.gov<mailto:[email protected]>> wrote: After top, click 1 on your keyboard, you can see how much workload each cpu has. Jeff's email remind one thing on jlse machine. pay attention to the mpirun, I had a hard time using the one under /soft/libraries/mpi/mpich. Building code with the compiler wrapper there is fine but I ran codes with the one from intel compiler like /soft/compilers/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpirun choose the one correspond to your compiler. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:56 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Hi Ye, It was the running node thing that tripped me up. So what is coming up is that N tasks are showing in top, each taking 25%. So we're definitely on one core. Jeff's giving me some things to check, so I am going to go down that path for now. Thanks everyone for your help! Cheers, Tom On 9/2/2016 8:48 AM, Luo, Ye wrote: Just tried it. I can do it. You must have your job running. qstat -lf your_job_id find your node name and ssh to it. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:31 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. On 9/2/2016 8:28 AM, Luo, Ye wrote: There is quick way to check what is going on in the node. start your job with 8 ranks. ssh to the node you job is assigned to. top and see if there are 8 processes of your application running at about 100% CPU. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 7:31 AM To: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: [JLSE-Intel-NDA] MPI vs Threading Performance Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov<http://knl08.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Hammond, Jeff R 02 Sep '16

02 Sep '16

I_MPI_VERBOSE>0 is a good way to verify mpirun is from Intel MPI but MPICH Hydra is used by Intel MPI so I would be surprised if mpirun is the problem here. Sent from my iPhone On Sep 2, 2016, at 7:02 AM, Luo, Ye <yeluo(a)anl.gov<mailto:[email protected]>> wrote: After top, click 1 on your keyboard, you can see how much workload each cpu has. Jeff's email remind one thing on jlse machine. pay attention to the mpirun, I had a hard time using the one under /soft/libraries/mpi/mpich. Building code with the compiler wrapper there is fine but I ran codes with the one from intel compiler like /soft/compilers/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpirun choose the one correspond to your compiler. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:56 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Hi Ye, It was the running node thing that tripped me up. So what is coming up is that N tasks are showing in top, each taking 25%. So we're definitely on one core. Jeff's giving me some things to check, so I am going to go down that path for now. Thanks everyone for your help! Cheers, Tom On 9/2/2016 8:48 AM, Luo, Ye wrote: Just tried it. I can do it. You must have your job running. qstat -lf your_job_id find your node name and ssh to it. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:31 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. On 9/2/2016 8:28 AM, Luo, Ye wrote: There is quick way to check what is going on in the node. start your job with 8 ranks. ssh to the node you job is assigned to. top and see if there are 8 processes of your application running at about 100% CPU. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 7:31 AM To: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: [JLSE-Intel-NDA] MPI vs Threading Performance Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov<http://knl08.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Luo, Ye 02 Sep '16

02 Sep '16

After top, click 1 on your keyboard, you can see how much workload each cpu has. Jeff's email remind one thing on jlse machine. pay attention to the mpirun, I had a hard time using the one under /soft/libraries/mpi/mpich. Building code with the compiler wrapper there is fine but I ran codes with the one from intel compiler like /soft/compilers/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpirun choose the one correspond to your compiler. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov] Sent: Friday, September 02, 2016 8:56 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Hi Ye, It was the running node thing that tripped me up. So what is coming up is that N tasks are showing in top, each taking 25%. So we're definitely on one core. Jeff's giving me some things to check, so I am going to go down that path for now. Thanks everyone for your help! Cheers, Tom On 9/2/2016 8:48 AM, Luo, Ye wrote: Just tried it. I can do it. You must have your job running. qstat -lf your_job_id find your node name and ssh to it. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:31 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. On 9/2/2016 8:28 AM, Luo, Ye wrote: There is quick way to check what is going on in the node. start your job with 8 ranks. ssh to the node you job is assigned to. top and see if there are 8 processes of your application running at about 100% CPU. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 7:31 AM To: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: [JLSE-Intel-NDA] MPI vs Threading Performance Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Tom LeCompte 02 Sep '16

02 Sep '16

Hi Ye, It was the running node thing that tripped me up. So what is coming up is that N tasks are showing in top, each taking 25%. So we're definitely on one core. Jeff's giving me some things to check, so I am going to go down that path for now. Thanks everyone for your help! Cheers, Tom On 9/2/2016 8:48 AM, Luo, Ye wrote: > Just tried it. I can do it. > You must have your job running. > qstat -lf your_job_id > find your node name and ssh to it. > > Ye > > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > ------------------------------------------------------------------------ > *From:* Tom LeCompte [lecompte(a)anl.gov] > *Sent:* Friday, September 02, 2016 8:31 AM > *To:* Luo, Ye; intel-nda(a)jlse.anl.gov > *Subject:* Re: [JLSE-Intel-NDA] MPI vs Threading Performance > > Unfortunately, I don't think that works any more. ssh knl08 returns > Connection closed by 10.17.6.112. > > > On 9/2/2016 8:28 AM, Luo, Ye wrote: >> There is quick way to check what is going on in the node. >> start your job with 8 ranks. >> ssh to the node you job is assigned to. >> top and see if there are 8 processes of your application running at >> about 100% CPU. >> >> Ye >> >> =================== >> Ye Luo, Ph.D. >> Leadership Computing Facility >> Argonne National Laboratory >> ------------------------------------------------------------------------ >> *From:* intel-nda-bounces(a)lists.jlse.anl.gov >> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Tom LeCompte >> [lecompte(a)anl.gov] >> *Sent:* Friday, September 02, 2016 7:31 AM >> *To:* intel-nda(a)jlse.anl.gov >> *Subject:* [JLSE-Intel-NDA] MPI vs Threading Performance >> >> Hi everyone- >> >> I am investigating the relative performance of threading vs. MPI, and >> don't understand the results: I expect threading to be better, but >> not that much better. >> >> I have a fixed amount of work per thread or rank, so I expect with >> perfect scaling run times to be roughly constant. With threading, >> that's roughly what I see, and even the deviations make sense: there >> are inflections at 64, 256 and a dropoff around 280-300 threads. >> With MPI, I see the following: >> >> * 1 rank: 77s reported by the code, 213s wallclock >> * 2 ranks: 152s reported by the code, 288s wallclock >> * 8 ranks: 635s reported by the code, 769s wallclock >> * 32 ranks 2504s reported by the code, 2648s wallclock >> * 64 ranks 5463s reported by the code, 5613s wallclock >> >> >> So it takes about twice as much time to do twice as much work with >> twice as many ranks. This pattern persists up to at least 64 ranks. >> The natural explanation would be that I am really running on only one >> core. I ran with I_MPI_DEBUG set to 5, and the results are below. >> This looks right to me, but I am not an expert. >> >> Any thoughts? >> >> Cheers, >> Tom >> >> >> >> ========================================================= >> >> 1 rank: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov >> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >> 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 >> ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 >> 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 >> 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 >> 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 >> 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 >> 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 >> 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 >> 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 >> 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 >> 7,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 >> >> >> 2 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov >> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >> 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 >> ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 >> 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 >> 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 >> 4,215,216,217,218,219,220,221,222,223} >> [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov >> {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 >> ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, >> 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, >> 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, >> 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, >> 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): >> I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 >> [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 >> Configuration : >> >> >> 8 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov >> {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} >> [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov >> {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} >> [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov >> {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} >> [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov >> {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} >> [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov >> {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} >> [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov >> {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} >> [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov >> {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} >> [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov >> {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 >> 48,7 56 >> >> >> >> 32 ranks: >> [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov >> {0,1,64,65,128,129,192,193} >> [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov >> {2,3,66,67,130,131,194,195} >> [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov >> {4,5,68,69,132,133,196,197} >> [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov >> {6,7,70,71,134,135,198,199} >> [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov >> {8,9,72,73,136,137,200,201} >> [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov >> {10,11,74,75,138,139,202,203} >> [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov >> {12,13,76,77,140,141,204,205} >> [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov >> {14,15,78,79,142,143,206,207} >> [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov >> {16,17,80,81,144,145,208,209} >> [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov >> {18,19,82,83,146,147,210,211} >> [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov >> {20,21,84,85,148,149,212,213} >> [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov >> {22,23,86,87,150,151,214,215} >> [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov >> {24,25,88,89,152,153,216,217} >> [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov >> {26,27,90,91,154,155,218,219} >> [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov >> {28,29,92,93,156,157,220,221} >> [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov >> {30,31,94,95,158,159,222,223} >> [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov >> {32,33,96,97,160,161,224,225} >> [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov >> {34,35,98,99,162,163,226,227} >> [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov >> {36,37,100,101,164,165,228,229} >> [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov >> {38,39,102,103,166,167,230,231} >> [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov >> {40,41,104,105,168,169,232,233} >> [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov >> {42,43,106,107,170,171,234,235} >> [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov >> {44,45,108,109,172,173,236,237} >> [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov >> {46,47,110,111,174,175,238,239} >> [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov >> {48,49,112,113,176,177,240,241} >> [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov >> {50,51,114,115,178,179,242,243} >> [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov >> {52,53,116,117,180,181,244,245} >> [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov >> {54,55,118,119,182,183,246,247} >> [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov >> {56,57,120,121,184,185,248,249} >> [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov >> {58,59,122,123,186,187,250,251} >> [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov >> {60,61,124,125,188,189,252,253} >> [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov >> {62,63,126,127,190,191,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 >> 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 >> 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 >> 58,30 60,31 62 >> >> 64 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} >> [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} >> [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} >> [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} >> [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} >> [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} >> [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} >> [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} >> [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} >> [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} >> [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} >> [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} >> [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} >> [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} >> [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} >> [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} >> [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} >> [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} >> [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} >> [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} >> [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} >> [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} >> [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} >> [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} >> [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} >> [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} >> [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} >> [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} >> [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} >> [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} >> [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} >> [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} >> [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} >> [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} >> [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} >> [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} >> [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228} >> [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229} >> [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230} >> [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231} >> [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232} >> [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233} >> [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234} >> [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235} >> [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236} >> [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237} >> [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238} >> [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239} >> [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240} >> [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241} >> [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242} >> [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243} >> [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244} >> [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245} >> [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246} >> [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247} >> [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248} >> [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249} >> [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250} >> [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251} >> [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252} >> [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253} >> [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254} >> [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 >> 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 >> 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 >> 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 >> 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 >> 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 >> >> >> >> >> >> >> >> >> >> >

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Hammond, Jeff R 02 Sep '16

02 Sep '16

In the past, MVAPICH2 plus Pthreads would do this because of default affinity mask being the location of the process. You are using Intel MPI and calling MPI_Init_thread with MPI_THREAD_SERIALIZED or MPI_THREAD_MULTIPLE, right? Intel MPI might set affinity mask and assume OpenMP runtime changes it later for threading codes. You should explicitly disable Intel MPI affinity environment variables for debugging purposes to see if that is the issue. In any case, it might be productive to explicitly set your Pthread affinity mask with the appropriate Linux API. Jeff Sent from my iPhone On Sep 2, 2016, at 6:48 AM, Luo, Ye <yeluo(a)anl.gov<mailto:[email protected]>> wrote: Just tried it. I can do it. You must have your job running. qstat -lf your_job_id find your node name and ssh to it. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:31 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. On 9/2/2016 8:28 AM, Luo, Ye wrote: There is quick way to check what is going on in the node. start your job with 8 ranks. ssh to the node you job is assigned to. top and see if there are 8 processes of your application running at about 100% CPU. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 7:31 AM To: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: [JLSE-Intel-NDA] MPI vs Threading Performance Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov<http://knl08.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Luo, Ye 02 Sep '16

02 Sep '16

Just tried it. I can do it. You must have your job running. qstat -lf your_job_id find your node name and ssh to it. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [lecompte(a)anl.gov] Sent: Friday, September 02, 2016 8:31 AM To: Luo, Ye; intel-nda(a)jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. On 9/2/2016 8:28 AM, Luo, Ye wrote: There is quick way to check what is going on in the node. start your job with 8 ranks. ssh to the node you job is assigned to. top and see if there are 8 processes of your application running at about 100% CPU. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Tom LeCompte [lecompte(a)anl.gov<mailto:[email protected]>] Sent: Friday, September 02, 2016 7:31 AM To: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: [JLSE-Intel-NDA] MPI vs Threading Performance Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63

1 0