September 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Allen, Benjamin S. 02 Sep '16

02 Sep '16

Hi Tom, Your job needs to be running on a node to be allowed to ssh into that node. If thats not work please let me know. Ben > On Sep 2, 2016, at 8:31 AM, Tom LeCompte <lecompte(a)anl.gov> wrote: > > Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. > > On 9/2/2016 8:28 AM, Luo, Ye wrote: >> There is quick way to check what is going on in the node. >> start your job with 8 ranks. >> ssh to the node you job is assigned to. >> top and see if there are 8 processes of your application running at about 100% CPU. >> >> Ye >> >> =================== >> Ye Luo, Ph.D. >> Leadership Computing Facility >> Argonne National Laboratory >> From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Tom LeCompte [lecompte(a)anl.gov] >> Sent: Friday, September 02, 2016 7:31 AM >> To: intel-nda(a)jlse.anl.gov >> Subject: [JLSE-Intel-NDA] MPI vs Threading Performance >> >> Hi everyone- >> >> I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. >> >> I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: >> >> • 1 rank: 77s reported by the code, 213s wallclock >> • 2 ranks: 152s reported by the code, 288s wallclock >> • 8 ranks: 635s reported by the code, 769s wallclock >> • 32 ranks 2504s reported by the code, 2648s wallclock >> • 64 ranks 5463s reported by the code, 5613s wallclock >> >> So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. >> >> Any thoughts? >> >> Cheers, >> Tom >> >> >> >> ========================================================= >> >> 1 rank: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >> 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 >> ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 >> 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 >> 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 >> 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 >> 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 >> 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 >> 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 >> 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 >> 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 >> 7,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 >> >> >> 2 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >> 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 >> ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 >> 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 >> 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 >> 4,215,216,217,218,219,220,221,222,223} >> [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 >> ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, >> 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, >> 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, >> 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, >> 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 >> [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 >> Configuration : >> >> >> 8 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} >> [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} >> [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} >> [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} >> [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} >> [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} >> [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} >> [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 >> >> >> >> 32 ranks: >> [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov {0,1,64,65,128,129,192,193} >> [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov {2,3,66,67,130,131,194,195} >> [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov {4,5,68,69,132,133,196,197} >> [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov {6,7,70,71,134,135,198,199} >> [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov {8,9,72,73,136,137,200,201} >> [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov {10,11,74,75,138,139,202,203} >> [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov {12,13,76,77,140,141,204,205} >> [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov {14,15,78,79,142,143,206,207} >> [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov {16,17,80,81,144,145,208,209} >> [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov {18,19,82,83,146,147,210,211} >> [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov {20,21,84,85,148,149,212,213} >> [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov {22,23,86,87,150,151,214,215} >> [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov {24,25,88,89,152,153,216,217} >> [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov {26,27,90,91,154,155,218,219} >> [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov {28,29,92,93,156,157,220,221} >> [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov {30,31,94,95,158,159,222,223} >> [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov {32,33,96,97,160,161,224,225} >> [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov {34,35,98,99,162,163,226,227} >> [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov {36,37,100,101,164,165,228,229} >> [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov {38,39,102,103,166,167,230,231} >> [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov {40,41,104,105,168,169,232,233} >> [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov {42,43,106,107,170,171,234,235} >> [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov {44,45,108,109,172,173,236,237} >> [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov {46,47,110,111,174,175,238,239} >> [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov {48,49,112,113,176,177,240,241} >> [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov {50,51,114,115,178,179,242,243} >> [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov {52,53,116,117,180,181,244,245} >> [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov {54,55,118,119,182,183,246,247} >> [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov {56,57,120,121,184,185,248,249} >> [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov {58,59,122,123,186,187,250,251} >> [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov {60,61,124,125,188,189,252,253} >> [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov {62,63,126,127,190,191,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 >> >> 64 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} >> [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} >> [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} >> [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} >> [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} >> [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} >> [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} >> [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} >> [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} >> [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} >> [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} >> [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} >> [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} >> [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} >> [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} >> [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} >> [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} >> [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} >> [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} >> [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} >> [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} >> [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} >> [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} >> [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} >> [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} >> [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} >> [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} >> [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} >> [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} >> [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} >> [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} >> [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} >> [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} >> [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} >> [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} >> [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} >> [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228} >> [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229} >> [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230} >> [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231} >> [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232} >> [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233} >> [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234} >> [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235} >> [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236} >> [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237} >> [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238} >> [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239} >> [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240} >> [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241} >> [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242} >> [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243} >> [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244} >> [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245} >> [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246} >> [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247} >> [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248} >> [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249} >> [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250} >> [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251} >> [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252} >> [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253} >> [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254} >> [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 >> >> >> >> >> >> >> >> >> >> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Tom LeCompte 02 Sep '16

02 Sep '16

The code is in /home/lecompte/g4/Geant4HepExpMTBenchmark/run HepExpMT-mpi is the executable mpi.sh the run script run_mpi.sh is the submit script I am not using OpenMP at all. I do use pthreads, but the number of threads is set to 1 per rank. Cheers, Tom On 9/2/2016 8:00 AM, Jim Phillips wrote: > > Perhaps MPI sets the cores correctly at firset, but OpenMP sets them > differently later, assuming that there is only one rank per node. > > We would need the whole job script and command line to diagnose this. > > Jim > > > On Fri, 2 Sep 2016, Tom LeCompte wrote: > >> Hi everyone- >> >> I am investigating the relative performance of threading vs. MPI, and >> don't understand the results: I expect threading to be better, but >> not that much better. >> >> I have a fixed amount of work per thread or rank, so I expect with >> perfect scaling run times to be roughly constant. With threading, >> that's roughly what I see, and even the deviations make sense: there >> are inflections at 64, 256 and a dropoff around 280-300 threads. With >> MPI, I see the following: >> >> * 1 rank: 77s reported by the code, 213s wallclock >> * 2 ranks: 152s reported by the code, 288s wallclock >> * 8 ranks: 635s reported by the code, 769s wallclock >> * 32 ranks 2504s reported by the code, 2648s wallclock >> * 64 ranks 5463s reported by the code, 5613s wallclock >> >> >> So it takes about twice as much time to do twice as much work with >> twice as many ranks. This pattern persists up to at least 64 ranks. >> The natural explanation would be that I am really running on only one >> core. I ran with I_MPI_DEBUG set to 5, and the results are below. >> This looks right to me, but I am not an expert. >> >> Any thoughts? >> >> Cheers, >> Tom >> >> >> >> ========================================================= >> >> 1 rank: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov >> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >> 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 >> >> ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 >> >> 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 >> >> 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 >> >> 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 >> >> 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 >> >> 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 >> >> 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 >> >> 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 >> >> 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 >> >> 7,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 >> >> >> 2 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov >> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, >> 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 >> >> ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 >> >> 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 >> >> 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 >> >> 4,215,216,217,218,219,220,221,222,223} >> [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov >> {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 >> ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, >> >> 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, >> >> 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, >> >> 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, >> >> 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} >> >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): >> I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 >> [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 >> Configuration : >> >> >> 8 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov >> {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} >> [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov >> {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} >> [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov >> {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} >> [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov >> {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} >> [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov >> {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} >> [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov >> {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} >> [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov >> {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} >> [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov >> {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 >> 48,7 56 >> >> >> >> 32 ranks: >> [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov >> {0,1,64,65,128,129,192,193} >> [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov >> {2,3,66,67,130,131,194,195} >> [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov >> {4,5,68,69,132,133,196,197} >> [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov >> {6,7,70,71,134,135,198,199} >> [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov >> {8,9,72,73,136,137,200,201} >> [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov >> {10,11,74,75,138,139,202,203} >> [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov >> {12,13,76,77,140,141,204,205} >> [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov >> {14,15,78,79,142,143,206,207} >> [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov >> {16,17,80,81,144,145,208,209} >> [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov >> {18,19,82,83,146,147,210,211} >> [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov >> {20,21,84,85,148,149,212,213} >> [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov >> {22,23,86,87,150,151,214,215} >> [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov >> {24,25,88,89,152,153,216,217} >> [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov >> {26,27,90,91,154,155,218,219} >> [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov >> {28,29,92,93,156,157,220,221} >> [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov >> {30,31,94,95,158,159,222,223} >> [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov >> {32,33,96,97,160,161,224,225} >> [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov >> {34,35,98,99,162,163,226,227} >> [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov >> {36,37,100,101,164,165,228,229} >> [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov >> {38,39,102,103,166,167,230,231} >> [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov >> {40,41,104,105,168,169,232,233} >> [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov >> {42,43,106,107,170,171,234,235} >> [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov >> {44,45,108,109,172,173,236,237} >> [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov >> {46,47,110,111,174,175,238,239} >> [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov >> {48,49,112,113,176,177,240,241} >> [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov >> {50,51,114,115,178,179,242,243} >> [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov >> {52,53,116,117,180,181,244,245} >> [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov >> {54,55,118,119,182,183,246,247} >> [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov >> {56,57,120,121,184,185,248,249} >> [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov >> {58,59,122,123,186,187,250,251} >> [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov >> {60,61,124,125,188,189,252,253} >> [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov >> {62,63,126,127,190,191,254,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 >> 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 >> 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 >> 58,30 60,31 62 >> >> 64 ranks: >> [0] MPI startup(): Rank Pid Node name Pin cpu >> [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov >> {0,64,128,192} >> [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov >> {1,65,129,193} >> [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov >> {2,66,130,194} >> [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov >> {3,67,131,195} >> [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov >> {4,68,132,196} >> [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov >> {5,69,133,197} >> [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov >> {6,70,134,198} >> [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov >> {7,71,135,199} >> [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov >> {8,72,136,200} >> [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov >> {9,73,137,201} >> [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov >> {10,74,138,202} >> [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov >> {11,75,139,203} >> [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov >> {12,76,140,204} >> [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov >> {13,77,141,205} >> [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov >> {14,78,142,206} >> [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov >> {15,79,143,207} >> [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov >> {16,80,144,208} >> [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov >> {17,81,145,209} >> [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov >> {18,82,146,210} >> [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov >> {19,83,147,211} >> [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov >> {20,84,148,212} >> [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov >> {21,85,149,213} >> [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov >> {22,86,150,214} >> [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov >> {23,87,151,215} >> [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov >> {24,88,152,216} >> [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov >> {25,89,153,217} >> [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov >> {26,90,154,218} >> [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov >> {27,91,155,219} >> [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov >> {28,92,156,220} >> [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov >> {29,93,157,221} >> [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov >> {30,94,158,222} >> [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov >> {31,95,159,223} >> [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov >> {32,96,160,224} >> [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov >> {33,97,161,225} >> [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov >> {34,98,162,226} >> [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov >> {35,99,163,227} >> [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov >> {36,100,164,228} >> [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov >> {37,101,165,229} >> [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov >> {38,102,166,230} >> [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov >> {39,103,167,231} >> [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov >> {40,104,168,232} >> [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov >> {41,105,169,233} >> [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov >> {42,106,170,234} >> [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov >> {43,107,171,235} >> [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov >> {44,108,172,236} >> [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov >> {45,109,173,237} >> [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov >> {46,110,174,238} >> [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov >> {47,111,175,239} >> [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov >> {48,112,176,240} >> [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov >> {49,113,177,241} >> [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov >> {50,114,178,242} >> [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov >> {51,115,179,243} >> [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov >> {52,116,180,244} >> [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov >> {53,117,181,245} >> [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov >> {54,118,182,246} >> [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov >> {55,119,183,247} >> [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov >> {56,120,184,248} >> [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov >> {57,121,185,249} >> [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov >> {58,122,186,250} >> [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov >> {59,123,187,251} >> [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov >> {60,124,188,252} >> [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov >> {61,125,189,253} >> [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov >> {62,126,190,254} >> [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov >> {63,127,191,255} >> [0] MPI startup(): I_MPI_DEBUG=5 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 >> [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 >> [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 >> 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 >> 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 >> 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 >> 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 >> 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 >> >> >> >> >> >> >> >> >> >> >>

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Tom LeCompte 02 Sep '16

02 Sep '16

Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. On 9/2/2016 8:28 AM, Luo, Ye wrote: > There is quick way to check what is going on in the node. > start your job with 8 ranks. > ssh to the node you job is assigned to. > top and see if there are 8 processes of your application running at > about 100% CPU. > > Ye > > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > ------------------------------------------------------------------------ > *From:* intel-nda-bounces(a)lists.jlse.anl.gov > [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Tom LeCompte > [lecompte(a)anl.gov] > *Sent:* Friday, September 02, 2016 7:31 AM > *To:* intel-nda(a)jlse.anl.gov > *Subject:* [JLSE-Intel-NDA] MPI vs Threading Performance > > Hi everyone- > > I am investigating the relative performance of threading vs. MPI, and > don't understand the results: I expect threading to be better, but not > that much better. > > I have a fixed amount of work per thread or rank, so I expect with > perfect scaling run times to be roughly constant. With threading, > that's roughly what I see, and even the deviations make sense: there > are inflections at 64, 256 and a dropoff around 280-300 threads. With > MPI, I see the following: > > * 1 rank: 77s reported by the code, 213s wallclock > * 2 ranks: 152s reported by the code, 288s wallclock > * 8 ranks: 635s reported by the code, 769s wallclock > * 32 ranks 2504s reported by the code, 2648s wallclock > * 64 ranks 5463s reported by the code, 5613s wallclock > > > So it takes about twice as much time to do twice as much work with > twice as many ranks. This pattern persists up to at least 64 ranks. > The natural explanation would be that I am really running on only one > core. I ran with I_MPI_DEBUG set to 5, and the results are below. > This looks right to me, but I am not an expert. > > Any thoughts? > > Cheers, > Tom > > > > ========================================================= > > 1 rank: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, > 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 > ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 > 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 > 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 > 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 > 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 > 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 > 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 > 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 > 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 > 7,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 > > > 2 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, > 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 > ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 > 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 > 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 > 4,215,216,217,218,219,220,221,222,223} > [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov > {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 > ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, > 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, > 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, > 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, > 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): > I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 > [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 > Configuration : > > > 8 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} > [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov > {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} > [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov > {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} > [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov > {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} > [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov > {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} > [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov > {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} > [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov > {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} > [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov > {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 > 48,7 56 > > > > 32 ranks: > [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov > {0,1,64,65,128,129,192,193} > [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov > {2,3,66,67,130,131,194,195} > [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov > {4,5,68,69,132,133,196,197} > [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov > {6,7,70,71,134,135,198,199} > [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov > {8,9,72,73,136,137,200,201} > [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov > {10,11,74,75,138,139,202,203} > [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov > {12,13,76,77,140,141,204,205} > [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov > {14,15,78,79,142,143,206,207} > [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov > {16,17,80,81,144,145,208,209} > [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov > {18,19,82,83,146,147,210,211} > [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov > {20,21,84,85,148,149,212,213} > [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov > {22,23,86,87,150,151,214,215} > [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov > {24,25,88,89,152,153,216,217} > [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov > {26,27,90,91,154,155,218,219} > [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov > {28,29,92,93,156,157,220,221} > [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov > {30,31,94,95,158,159,222,223} > [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov > {32,33,96,97,160,161,224,225} > [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov > {34,35,98,99,162,163,226,227} > [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov > {36,37,100,101,164,165,228,229} > [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov > {38,39,102,103,166,167,230,231} > [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov > {40,41,104,105,168,169,232,233} > [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov > {42,43,106,107,170,171,234,235} > [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov > {44,45,108,109,172,173,236,237} > [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov > {46,47,110,111,174,175,238,239} > [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov > {48,49,112,113,176,177,240,241} > [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov > {50,51,114,115,178,179,242,243} > [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov > {52,53,116,117,180,181,244,245} > [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov > {54,55,118,119,182,183,246,247} > [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov > {56,57,120,121,184,185,248,249} > [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov > {58,59,122,123,186,187,250,251} > [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov > {60,61,124,125,188,189,252,253} > [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov > {62,63,126,127,190,191,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 > 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 > 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 > 58,30 60,31 62 > > 64 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} > [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} > [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} > [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} > [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} > [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} > [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} > [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} > [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} > [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} > [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} > [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} > [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} > [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} > [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} > [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} > [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} > [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} > [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} > [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} > [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} > [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} > [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} > [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} > [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} > [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} > [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} > [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} > [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} > [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} > [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} > [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} > [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} > [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} > [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} > [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} > [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov > {36,100,164,228} > [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov > {37,101,165,229} > [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov > {38,102,166,230} > [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov > {39,103,167,231} > [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov > {40,104,168,232} > [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov > {41,105,169,233} > [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov > {42,106,170,234} > [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov > {43,107,171,235} > [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov > {44,108,172,236} > [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov > {45,109,173,237} > [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov > {46,110,174,238} > [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov > {47,111,175,239} > [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov > {48,112,176,240} > [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov > {49,113,177,241} > [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov > {50,114,178,242} > [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov > {51,115,179,243} > [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov > {52,116,180,244} > [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov > {53,117,181,245} > [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov > {54,118,182,246} > [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov > {55,119,183,247} > [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov > {56,120,184,248} > [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov > {57,121,185,249} > [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov > {58,122,186,250} > [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov > {59,123,187,251} > [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov > {60,124,188,252} > [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov > {61,125,189,253} > [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov > {62,126,190,254} > [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov > {63,127,191,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 > 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 > 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 > 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 > 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 > 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 > > > > > > > > > >

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Luo, Ye 02 Sep '16

02 Sep '16

There is quick way to check what is going on in the node. start your job with 8 ranks. ssh to the node you job is assigned to. top and see if there are 8 processes of your application running at about 100% CPU. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Tom LeCompte [lecompte(a)anl.gov] Sent: Friday, September 02, 2016 7:31 AM To: intel-nda(a)jlse.anl.gov Subject: [JLSE-Intel-NDA] MPI vs Threading Performance Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Vitali Morozov 02 Sep '16

02 Sep '16

To me, this looks like everything is running on 1 core - then it explains the run time roughly growing with the number of MPI processes. To debug why this might be happening requires more info on how you run a job and how you set up OpenMP thread affinity. Vitali On 09/02/2016 07:31 AM, Tom LeCompte wrote: > Hi everyone- > > I am investigating the relative performance of threading vs. MPI, and > don't understand the results: I expect threading to be better, but not > that much better. > > I have a fixed amount of work per thread or rank, so I expect with > perfect scaling run times to be roughly constant. With threading, > that's roughly what I see, and even the deviations make sense: there > are inflections at 64, 256 and a dropoff around 280-300 threads. With > MPI, I see the following: > > * 1 rank: 77s reported by the code, 213s wallclock > * 2 ranks: 152s reported by the code, 288s wallclock > * 8 ranks: 635s reported by the code, 769s wallclock > * 32 ranks 2504s reported by the code, 2648s wallclock > * 64 ranks 5463s reported by the code, 5613s wallclock > > > So it takes about twice as much time to do twice as much work with > twice as many ranks. This pattern persists up to at least 64 ranks. > The natural explanation would be that I am really running on only one > core. I ran with I_MPI_DEBUG set to 5, and the results are below. > This looks right to me, but I am not an expert. > > Any thoughts? > > Cheers, > Tom > > > > ========================================================= > > 1 rank: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, > 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 > ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 > 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 > 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 > 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 > 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 > 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 > 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 > 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 > 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 > 7,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 > > > 2 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, > 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 > ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 > 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 > 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 > 4,215,216,217,218,219,220,221,222,223} > [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov > {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 > ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, > 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, > 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, > 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, > 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): > I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 > [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 > Configuration : > > > 8 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} > [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov > {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} > [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov > {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} > [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov > {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} > [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov > {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} > [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov > {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} > [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov > {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} > [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov > {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 > 48,7 56 > > > > 32 ranks: > [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov > {0,1,64,65,128,129,192,193} > [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov > {2,3,66,67,130,131,194,195} > [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov > {4,5,68,69,132,133,196,197} > [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov > {6,7,70,71,134,135,198,199} > [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov > {8,9,72,73,136,137,200,201} > [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov > {10,11,74,75,138,139,202,203} > [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov > {12,13,76,77,140,141,204,205} > [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov > {14,15,78,79,142,143,206,207} > [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov > {16,17,80,81,144,145,208,209} > [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov > {18,19,82,83,146,147,210,211} > [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov > {20,21,84,85,148,149,212,213} > [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov > {22,23,86,87,150,151,214,215} > [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov > {24,25,88,89,152,153,216,217} > [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov > {26,27,90,91,154,155,218,219} > [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov > {28,29,92,93,156,157,220,221} > [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov > {30,31,94,95,158,159,222,223} > [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov > {32,33,96,97,160,161,224,225} > [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov > {34,35,98,99,162,163,226,227} > [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov > {36,37,100,101,164,165,228,229} > [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov > {38,39,102,103,166,167,230,231} > [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov > {40,41,104,105,168,169,232,233} > [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov > {42,43,106,107,170,171,234,235} > [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov > {44,45,108,109,172,173,236,237} > [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov > {46,47,110,111,174,175,238,239} > [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov > {48,49,112,113,176,177,240,241} > [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov > {50,51,114,115,178,179,242,243} > [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov > {52,53,116,117,180,181,244,245} > [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov > {54,55,118,119,182,183,246,247} > [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov > {56,57,120,121,184,185,248,249} > [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov > {58,59,122,123,186,187,250,251} > [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov > {60,61,124,125,188,189,252,253} > [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov > {62,63,126,127,190,191,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 > 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 > 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 > 58,30 60,31 62 > > 64 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} > [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} > [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} > [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} > [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} > [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} > [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} > [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} > [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} > [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} > [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} > [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} > [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} > [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} > [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} > [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} > [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} > [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} > [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} > [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} > [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} > [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} > [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} > [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} > [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} > [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} > [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} > [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} > [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} > [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} > [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} > [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} > [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} > [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} > [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} > [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} > [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov > {36,100,164,228} > [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov > {37,101,165,229} > [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov > {38,102,166,230} > [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov > {39,103,167,231} > [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov > {40,104,168,232} > [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov > {41,105,169,233} > [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov > {42,106,170,234} > [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov > {43,107,171,235} > [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov > {44,108,172,236} > [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov > {45,109,173,237} > [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov > {46,110,174,238} > [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov > {47,111,175,239} > [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov > {48,112,176,240} > [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov > {49,113,177,241} > [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov > {50,114,178,242} > [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov > {51,115,179,243} > [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov > {52,116,180,244} > [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov > {53,117,181,245} > [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov > {54,118,182,246} > [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov > {55,119,183,247} > [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov > {56,120,184,248} > [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov > {57,121,185,249} > [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov > {58,122,186,250} > [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov > {59,123,187,251} > [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov > {60,124,188,252} > [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov > {61,125,189,253} > [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov > {62,126,190,254} > [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov > {63,127,191,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 > 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 > 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 > 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 > 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 > 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 > > > > > > > > > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] MPI vs Threading Performance
by Jim Phillips 02 Sep '16

02 Sep '16

Perhaps MPI sets the cores correctly at firset, but OpenMP sets them differently later, assuming that there is only one rank per node. We would need the whole job script and command line to diagnose this. Jim On Fri, 2 Sep 2016, Tom LeCompte wrote: > Hi everyone- > > I am investigating the relative performance of threading vs. MPI, and don't > understand the results: I expect threading to be better, but not that much > better. > > I have a fixed amount of work per thread or rank, so I expect with perfect > scaling run times to be roughly constant. With threading, that's roughly > what I see, and even the deviations make sense: there are inflections at 64, > 256 and a dropoff around 280-300 threads. With MPI, I see the following: > > * 1 rank: 77s reported by the code, 213s wallclock > * 2 ranks: 152s reported by the code, 288s wallclock > * 8 ranks: 635s reported by the code, 769s wallclock > * 32 ranks 2504s reported by the code, 2648s wallclock > * 64 ranks 5463s reported by the code, 5613s wallclock > > > So it takes about twice as much time to do twice as much work with twice as > many ranks. This pattern persists up to at least 64 ranks. The natural > explanation would be that I am really running on only one core. I ran with > I_MPI_DEBUG set to 5, and the results are below. This looks right to me, > but I am not an expert. > > Any thoughts? > > Cheers, > Tom > > > > ========================================================= > > 1 rank: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, > 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 > ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 > 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 > 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 > 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 > 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 > 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 > 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 > 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 > 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 > 7,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 > > > 2 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, > 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 > ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 > 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 > 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 > 4,215,216,217,218,219,220,221,222,223} > [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov > {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 > ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, > 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, > 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, > 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, > 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): > I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 > [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 > Configuration : > > > 8 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov > {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} > [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov > {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} > [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov > {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} > [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov > {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} > [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov > {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} > [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov > {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} > [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov > {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} > [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov > {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 > > > > 32 ranks: > [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov > {0,1,64,65,128,129,192,193} > [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov > {2,3,66,67,130,131,194,195} > [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov > {4,5,68,69,132,133,196,197} > [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov > {6,7,70,71,134,135,198,199} > [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov > {8,9,72,73,136,137,200,201} > [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov > {10,11,74,75,138,139,202,203} > [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov > {12,13,76,77,140,141,204,205} > [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov > {14,15,78,79,142,143,206,207} > [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov > {16,17,80,81,144,145,208,209} > [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov > {18,19,82,83,146,147,210,211} > [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov > {20,21,84,85,148,149,212,213} > [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov > {22,23,86,87,150,151,214,215} > [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov > {24,25,88,89,152,153,216,217} > [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov > {26,27,90,91,154,155,218,219} > [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov > {28,29,92,93,156,157,220,221} > [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov > {30,31,94,95,158,159,222,223} > [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov > {32,33,96,97,160,161,224,225} > [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov > {34,35,98,99,162,163,226,227} > [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov > {36,37,100,101,164,165,228,229} > [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov > {38,39,102,103,166,167,230,231} > [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov > {40,41,104,105,168,169,232,233} > [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov > {42,43,106,107,170,171,234,235} > [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov > {44,45,108,109,172,173,236,237} > [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov > {46,47,110,111,174,175,238,239} > [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov > {48,49,112,113,176,177,240,241} > [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov > {50,51,114,115,178,179,242,243} > [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov > {52,53,116,117,180,181,244,245} > [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov > {54,55,118,119,182,183,246,247} > [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov > {56,57,120,121,184,185,248,249} > [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov > {58,59,122,123,186,187,250,251} > [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov > {60,61,124,125,188,189,252,253} > [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov > {62,63,126,127,190,191,254,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 > 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 > 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 > > 64 ranks: > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} > [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} > [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} > [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} > [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} > [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} > [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} > [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} > [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} > [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} > [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} > [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} > [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} > [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} > [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} > [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} > [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} > [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} > [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} > [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} > [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} > [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} > [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} > [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} > [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} > [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} > [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} > [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} > [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} > [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} > [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} > [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} > [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} > [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} > [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} > [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} > [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228} > [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229} > [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230} > [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231} > [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232} > [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233} > [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234} > [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235} > [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236} > [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237} > [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238} > [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239} > [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240} > [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241} > [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242} > [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243} > [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244} > [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245} > [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246} > [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247} > [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248} > [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249} > [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250} > [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251} > [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252} > [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253} > [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254} > [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 > 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 > 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 > 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 > 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 > 61,62 62,63 63 > > > > > > > > > > >

1 0

[JLSE-Intel-NDA] MPI vs Threading Performance
by Tom LeCompte 02 Sep '16

02 Sep '16

Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63

1 0