There is quick way to check what is going on in the node.
start your job with 8 ranks.
ssh to the node you job is assigned to.
top and see if there are 8 processes of your application running at about 100% CPU.

Ye

===================

Ye Luo, Ph.D.
Leadership Computing Facility
Argonne National Laboratory

From: intel-nda-bounces@lists.jlse.anl.gov [intel-nda-bounces@lists.jlse.anl.gov] on behalf of Tom LeCompte [lecompte@anl.gov]
Sent: Friday, September 02, 2016 7:31 AM
To: intel-nda@jlse.anl.gov
Subject: [JLSE-Intel-NDA] MPI vs Threading Performance

Hi everyone-

I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better.�

I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant.� With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads.� With MPI, I see the following:

1 rank:�� 77s reported by the code, 213s wallclock
2 ranks: 152s reported by the code, 288s wallclock
8 ranks: 635s reported by the code, 769s wallclock
32 ranks 2504s reported by the code, 2648s wallclock
64 ranks 5463s reported by the code, 5613s wallclock

So it takes about twice as much time to do twice as much work with twice as many ranks.� This pattern persists up to at least 64 ranks.� The natural explanation would be that I am really running on only one core.� I ran with� I_MPI_DEBUG set to 5, and the results are below.�� This looks right to me, but I am not an expert.

Any thoughts?

Cheers,
Tom

=========================================================

1 rank:
[0] MPI startup(): Rank�� Pid�� Node name�� Pin cpu
[0] MPI startup(): 0�� 23191�� knl08.ftm.alcf.anl.gov� {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
�� 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56
�� ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8
�� 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10
�� 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12
�� 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14
�� 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16
�� 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18
�� 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20
�� 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22
�� 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24
�� 7,248,249,250,251,252,253,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0

2 ranks:
[0] MPI startup(): Rank�� Pid�� Node name�� Pin cpu
[0] MPI startup(): 0�� 113057�� knl07.ftm.alcf.anl.gov� {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
�� 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88
�� ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14
�� 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19
�� 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21
�� 4,215,216,217,218,219,220,221,222,223}
[0] MPI startup(): 1�� 113058�� knl07.ftm.alcf.anl.gov� {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58
�� ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,
�� 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164,
�� 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,
�� 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236,
�� 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4
[0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32
�Configuration :

8 ranks:
[0] MPI startup(): Rank�� Pid�� Node name�� Pin cpu
[0] MPI startup(): 0�� 23089�� knl04.ftm.alcf.anl.gov� {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199}
[0] MPI startup(): 1�� 23090�� knl04.ftm.alcf.anl.gov� {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207}
[0] MPI startup(): 2�� 23091�� knl04.ftm.alcf.anl.gov� {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215}
[0] MPI startup(): 3�� 23092�� knl04.ftm.alcf.anl.gov� {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223}
[0] MPI startup(): 4�� 23093�� knl04.ftm.alcf.anl.gov� {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231}
[0] MPI startup(): 5�� 23094�� knl04.ftm.alcf.anl.gov� {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239}
[0] MPI startup(): 6�� 23095�� knl04.ftm.alcf.anl.gov� {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247}
[0] MPI startup(): 7�� 23096�� knl04.ftm.alcf.anl.gov� {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56

32 ranks:
[0] MPI startup(): 0�� 90622�� knl04.ftm.alcf.anl.gov� {0,1,64,65,128,129,192,193}
[0] MPI startup(): 1�� 90623�� knl04.ftm.alcf.anl.gov� {2,3,66,67,130,131,194,195}
[0] MPI startup(): 2�� 90624�� knl04.ftm.alcf.anl.gov� {4,5,68,69,132,133,196,197}
[0] MPI startup(): 3�� 90625�� knl04.ftm.alcf.anl.gov� {6,7,70,71,134,135,198,199}
[0] MPI startup(): 4�� 90626�� knl04.ftm.alcf.anl.gov� {8,9,72,73,136,137,200,201}
[0] MPI startup(): 5�� 90627�� knl04.ftm.alcf.anl.gov� {10,11,74,75,138,139,202,203}
[0] MPI startup(): 6�� 90628�� knl04.ftm.alcf.anl.gov� {12,13,76,77,140,141,204,205}
[0] MPI startup(): 7�� 90629�� knl04.ftm.alcf.anl.gov� {14,15,78,79,142,143,206,207}
[0] MPI startup(): 8�� 90630�� knl04.ftm.alcf.anl.gov� {16,17,80,81,144,145,208,209}
[0] MPI startup(): 9�� 90631�� knl04.ftm.alcf.anl.gov� {18,19,82,83,146,147,210,211}
[0] MPI startup(): 10�� 90632�� knl04.ftm.alcf.anl.gov� {20,21,84,85,148,149,212,213}
[0] MPI startup(): 11�� 90633�� knl04.ftm.alcf.anl.gov� {22,23,86,87,150,151,214,215}
[0] MPI startup(): 12�� 90634�� knl04.ftm.alcf.anl.gov� {24,25,88,89,152,153,216,217}
[0] MPI startup(): 13�� 90635�� knl04.ftm.alcf.anl.gov� {26,27,90,91,154,155,218,219}
[0] MPI startup(): 14�� 90636�� knl04.ftm.alcf.anl.gov� {28,29,92,93,156,157,220,221}
[0] MPI startup(): 15�� 90637�� knl04.ftm.alcf.anl.gov� {30,31,94,95,158,159,222,223}
[0] MPI startup(): 16�� 90638�� knl04.ftm.alcf.anl.gov� {32,33,96,97,160,161,224,225}
[0] MPI startup(): 17�� 90639�� knl04.ftm.alcf.anl.gov� {34,35,98,99,162,163,226,227}
[0] MPI startup(): 18�� 90640�� knl04.ftm.alcf.anl.gov� {36,37,100,101,164,165,228,229}
[0] MPI startup(): 19�� 90641�� knl04.ftm.alcf.anl.gov� {38,39,102,103,166,167,230,231}
[0] MPI startup(): 20�� 90642�� knl04.ftm.alcf.anl.gov� {40,41,104,105,168,169,232,233}
[0] MPI startup(): 21�� 90643�� knl04.ftm.alcf.anl.gov� {42,43,106,107,170,171,234,235}
[0] MPI startup(): 22�� 90644�� knl04.ftm.alcf.anl.gov� {44,45,108,109,172,173,236,237}
[0] MPI startup(): 23�� 90645�� knl04.ftm.alcf.anl.gov� {46,47,110,111,174,175,238,239}
[0] MPI startup(): 24�� 90646�� knl04.ftm.alcf.anl.gov� {48,49,112,113,176,177,240,241}
[0] MPI startup(): 25�� 90647�� knl04.ftm.alcf.anl.gov� {50,51,114,115,178,179,242,243}
[0] MPI startup(): 26�� 90648�� knl04.ftm.alcf.anl.gov� {52,53,116,117,180,181,244,245}
[0] MPI startup(): 27�� 90649�� knl04.ftm.alcf.anl.gov� {54,55,118,119,182,183,246,247}
[0] MPI startup(): 28�� 90650�� knl04.ftm.alcf.anl.gov� {56,57,120,121,184,185,248,249}
[0] MPI startup(): 29�� 90651�� knl04.ftm.alcf.anl.gov� {58,59,122,123,186,187,250,251}
[0] MPI startup(): 30�� 90652�� knl04.ftm.alcf.anl.gov� {60,61,124,125,188,189,252,253}
[0] MPI startup(): 31�� 90653�� knl04.ftm.alcf.anl.gov� {62,63,126,127,190,191,254,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62

64 ranks:
[0] MPI startup(): Rank�� Pid�� Node name�� Pin cpu
[0] MPI startup(): 0�� 23076�� knl06.ftm.alcf.anl.gov� {0,64,128,192}
[0] MPI startup(): 1�� 23077�� knl06.ftm.alcf.anl.gov� {1,65,129,193}
[0] MPI startup(): 2�� 23078�� knl06.ftm.alcf.anl.gov� {2,66,130,194}
[0] MPI startup(): 3�� 23079�� knl06.ftm.alcf.anl.gov� {3,67,131,195}
[0] MPI startup(): 4�� 23080�� knl06.ftm.alcf.anl.gov� {4,68,132,196}
[0] MPI startup(): 5�� 23081�� knl06.ftm.alcf.anl.gov� {5,69,133,197}
[0] MPI startup(): 6�� 23082�� knl06.ftm.alcf.anl.gov� {6,70,134,198}
[0] MPI startup(): 7�� 23083�� knl06.ftm.alcf.anl.gov� {7,71,135,199}
[0] MPI startup(): 8�� 23084�� knl06.ftm.alcf.anl.gov� {8,72,136,200}
[0] MPI startup(): 9�� 23087�� knl06.ftm.alcf.anl.gov� {9,73,137,201}
[0] MPI startup(): 10�� 23088�� knl06.ftm.alcf.anl.gov� {10,74,138,202}
[0] MPI startup(): 11�� 23089�� knl06.ftm.alcf.anl.gov� {11,75,139,203}
[0] MPI startup(): 12�� 23090�� knl06.ftm.alcf.anl.gov� {12,76,140,204}
[0] MPI startup(): 13�� 23091�� knl06.ftm.alcf.anl.gov� {13,77,141,205}
[0] MPI startup(): 14�� 23092�� knl06.ftm.alcf.anl.gov� {14,78,142,206}
[0] MPI startup(): 15�� 23093�� knl06.ftm.alcf.anl.gov� {15,79,143,207}
[0] MPI startup(): 16�� 23099�� knl06.ftm.alcf.anl.gov� {16,80,144,208}
[0] MPI startup(): 17�� 23101�� knl06.ftm.alcf.anl.gov� {17,81,145,209}
[0] MPI startup(): 18�� 23102�� knl06.ftm.alcf.anl.gov� {18,82,146,210}
[0] MPI startup(): 19�� 23103�� knl06.ftm.alcf.anl.gov� {19,83,147,211}
[0] MPI startup(): 20�� 23104�� knl06.ftm.alcf.anl.gov� {20,84,148,212}
[0] MPI startup(): 21�� 23105�� knl06.ftm.alcf.anl.gov� {21,85,149,213}
[0] MPI startup(): 22�� 23108�� knl06.ftm.alcf.anl.gov� {22,86,150,214}
[0] MPI startup(): 23�� 23109�� knl06.ftm.alcf.anl.gov� {23,87,151,215}
[0] MPI startup(): 24�� 23111�� knl06.ftm.alcf.anl.gov� {24,88,152,216}
[0] MPI startup(): 25�� 23113�� knl06.ftm.alcf.anl.gov� {25,89,153,217}
[0] MPI startup(): 26�� 23114�� knl06.ftm.alcf.anl.gov� {26,90,154,218}
[0] MPI startup(): 27�� 23115�� knl06.ftm.alcf.anl.gov� {27,91,155,219}
[0] MPI startup(): 28�� 23116�� knl06.ftm.alcf.anl.gov� {28,92,156,220}
[0] MPI startup(): 29�� 23117�� knl06.ftm.alcf.anl.gov� {29,93,157,221}
[0] MPI startup(): 30�� 23118�� knl06.ftm.alcf.anl.gov� {30,94,158,222}
[0] MPI startup(): 31�� 23120�� knl06.ftm.alcf.anl.gov� {31,95,159,223}
[0] MPI startup(): 32�� 23121�� knl06.ftm.alcf.anl.gov� {32,96,160,224}
[0] MPI startup(): 33�� 23123�� knl06.ftm.alcf.anl.gov� {33,97,161,225}
[0] MPI startup(): 34�� 23124�� knl06.ftm.alcf.anl.gov� {34,98,162,226}
[0] MPI startup(): 35�� 23125�� knl06.ftm.alcf.anl.gov� {35,99,163,227}
[0] MPI startup(): 36�� 23126�� knl06.ftm.alcf.anl.gov� {36,100,164,228}
[0] MPI startup(): 37�� 23127�� knl06.ftm.alcf.anl.gov� {37,101,165,229}
[0] MPI startup(): 38�� 23128�� knl06.ftm.alcf.anl.gov� {38,102,166,230}
[0] MPI startup(): 39�� 23129�� knl06.ftm.alcf.anl.gov� {39,103,167,231}
[0] MPI startup(): 40�� 23130�� knl06.ftm.alcf.anl.gov� {40,104,168,232}
[0] MPI startup(): 41�� 23131�� knl06.ftm.alcf.anl.gov� {41,105,169,233}
[0] MPI startup(): 42�� 23132�� knl06.ftm.alcf.anl.gov� {42,106,170,234}
[0] MPI startup(): 43�� 23133�� knl06.ftm.alcf.anl.gov� {43,107,171,235}
[0] MPI startup(): 44�� 23134�� knl06.ftm.alcf.anl.gov� {44,108,172,236}
[0] MPI startup(): 45�� 23135�� knl06.ftm.alcf.anl.gov� {45,109,173,237}
[0] MPI startup(): 46�� 23136�� knl06.ftm.alcf.anl.gov� {46,110,174,238}
[0] MPI startup(): 47�� 23137�� knl06.ftm.alcf.anl.gov� {47,111,175,239}
[0] MPI startup(): 48�� 23138�� knl06.ftm.alcf.anl.gov� {48,112,176,240}
[0] MPI startup(): 49�� 23139�� knl06.ftm.alcf.anl.gov� {49,113,177,241}
[0] MPI startup(): 50�� 23140�� knl06.ftm.alcf.anl.gov� {50,114,178,242}
[0] MPI startup(): 51�� 23141�� knl06.ftm.alcf.anl.gov� {51,115,179,243}
[0] MPI startup(): 52�� 23147�� knl06.ftm.alcf.anl.gov� {52,116,180,244}
[0] MPI startup(): 53�� 23148�� knl06.ftm.alcf.anl.gov� {53,117,181,245}
[0] MPI startup(): 54�� 23149�� knl06.ftm.alcf.anl.gov� {54,118,182,246}
[0] MPI startup(): 55�� 23151�� knl06.ftm.alcf.anl.gov� {55,119,183,247}
[0] MPI startup(): 56�� 23153�� knl06.ftm.alcf.anl.gov� {56,120,184,248}
[0] MPI startup(): 57�� 23154�� knl06.ftm.alcf.anl.gov� {57,121,185,249}
[0] MPI startup(): 58�� 23156�� knl06.ftm.alcf.anl.gov� {58,122,186,250}
[0] MPI startup(): 59�� 23157�� knl06.ftm.alcf.anl.gov� {59,123,187,251}
[0] MPI startup(): 60�� 23158�� knl06.ftm.alcf.anl.gov� {60,124,188,252}
[0] MPI startup(): 61�� 23159�� knl06.ftm.alcf.anl.gov� {61,125,189,253}
[0] MPI startup(): 62�� 23160�� knl06.ftm.alcf.anl.gov� {62,126,190,254}
[0] MPI startup(): 63�� 23163�� knl06.ftm.alcf.anl.gov� {63,127,191,255}
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1
[0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63