Hi everyone-
I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better.
I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300
threads. With MPI, I see the following:
- 1 rank: 77s reported by the code, 213s wallclock
- 2 ranks: 152s reported by the code, 288s wallclock
- 8 ranks: 635s reported by the code, 769s wallclock
- 32 ranks 2504s reported by the code, 2648s wallclock
- 64 ranks 5463s reported by the code, 5613s wallclock
So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results
are below. This looks right to me, but I am not an expert.
Any thoughts?
Cheers,
Tom
=========================================================
1 rank:
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56
,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8
3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10
7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12
7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14
7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16
7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18
7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20
7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22
7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24
7,248,249,250,251,252,253,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0
2 ranks:
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88
,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14
2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19
4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21
4,215,216,217,218,219,220,221,222,223}
[0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58
,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,
113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164,
165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,
185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236,
237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4
[0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32
Configuration :
8 ranks:
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199}
[0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207}
[0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215}
[0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223}
[0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231}
[0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239}
[0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247}
[0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56
32 ranks:
[0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov {0,1,64,65,128,129,192,193}
[0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov {2,3,66,67,130,131,194,195}
[0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov {4,5,68,69,132,133,196,197}
[0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov {6,7,70,71,134,135,198,199}
[0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov {8,9,72,73,136,137,200,201}
[0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov {10,11,74,75,138,139,202,203}
[0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov {12,13,76,77,140,141,204,205}
[0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov {14,15,78,79,142,143,206,207}
[0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov {16,17,80,81,144,145,208,209}
[0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov {18,19,82,83,146,147,210,211}
[0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov {20,21,84,85,148,149,212,213}
[0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov {22,23,86,87,150,151,214,215}
[0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov {24,25,88,89,152,153,216,217}
[0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov {26,27,90,91,154,155,218,219}
[0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov {28,29,92,93,156,157,220,221}
[0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov {30,31,94,95,158,159,222,223}
[0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov {32,33,96,97,160,161,224,225}
[0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov {34,35,98,99,162,163,226,227}
[0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov {36,37,100,101,164,165,228,229}
[0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov {38,39,102,103,166,167,230,231}
[0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov {40,41,104,105,168,169,232,233}
[0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov {42,43,106,107,170,171,234,235}
[0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov {44,45,108,109,172,173,236,237}
[0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov {46,47,110,111,174,175,238,239}
[0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov {48,49,112,113,176,177,240,241}
[0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov {50,51,114,115,178,179,242,243}
[0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov {52,53,116,117,180,181,244,245}
[0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov {54,55,118,119,182,183,246,247}
[0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov {56,57,120,121,184,185,248,249}
[0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov {58,59,122,123,186,187,250,251}
[0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov {60,61,124,125,188,189,252,253}
[0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov {62,63,126,127,190,191,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62
64 ranks:
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov {0,64,128,192}
[0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov {1,65,129,193}
[0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov {2,66,130,194}
[0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov {3,67,131,195}
[0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov {4,68,132,196}
[0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov {5,69,133,197}
[0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov {6,70,134,198}
[0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov {7,71,135,199}
[0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov {8,72,136,200}
[0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov {9,73,137,201}
[0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov {10,74,138,202}
[0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov {11,75,139,203}
[0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov {12,76,140,204}
[0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov {13,77,141,205}
[0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov {14,78,142,206}
[0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov {15,79,143,207}
[0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov {16,80,144,208}
[0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov {17,81,145,209}
[0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov {18,82,146,210}
[0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov {19,83,147,211}
[0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov {20,84,148,212}
[0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov {21,85,149,213}
[0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov {22,86,150,214}
[0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov {23,87,151,215}
[0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov {24,88,152,216}
[0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov {25,89,153,217}
[0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov {26,90,154,218}
[0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov {27,91,155,219}
[0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov {28,92,156,220}
[0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov {29,93,157,221}
[0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov {30,94,158,222}
[0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov {31,95,159,223}
[0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov {32,96,160,224}
[0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov {33,97,161,225}
[0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov {34,98,162,226}
[0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov {35,99,163,227}
[0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov {36,100,164,228}
[0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov {37,101,165,229}
[0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov {38,102,166,230}
[0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov {39,103,167,231}
[0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov {40,104,168,232}
[0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov {41,105,169,233}
[0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov {42,106,170,234}
[0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov {43,107,171,235}
[0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov {44,108,172,236}
[0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov {45,109,173,237}
[0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov {46,110,174,238}
[0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov {47,111,175,239}
[0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov {48,112,176,240}
[0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov {49,113,177,241}
[0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov {50,114,178,242}
[0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov {51,115,179,243}
[0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov {52,116,180,244}
[0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov {53,117,181,245}
[0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov {54,118,182,246}
[0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov {55,119,183,247}
[0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov {56,120,184,248}
[0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov {57,121,185,249}
[0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov {58,122,186,250}
[0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov {59,123,187,251}
[0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov {60,124,188,252}
[0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov {61,125,189,253}
[0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov {62,126,190,254}
[0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov {63,127,191,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39
39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63