I was using mpich-intel-3.0.4 first which gave me hard time. Later the 3.1.4 is installed. I don't remember if I have tested it or not. Since the Intel compiler on JLSE has everything including compiler wrapper and runtime. I just use the consistent set all the time now. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Hammond, Jeff R [[email protected]] Sent: Friday, September 02, 2016 9:18 AM To: Luo, Ye Cc: LeCompte, Thomas J.; [email protected] Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance I_MPI_VERBOSE>0 is a good way to verify mpirun is from Intel MPI but MPICH Hydra is used by Intel MPI so I would be surprised if mpirun is the problem here. Sent from my iPhone On Sep 2, 2016, at 7:02 AM, Luo, Ye <[email protected]<mailto:[email protected]>> wrote: After top, click 1 on your keyboard, you can see how much workload each cpu has. Jeff's email remind one thing on jlse machine. pay attention to the mpirun, I had a hard time using the one under /soft/libraries/mpi/mpich. Building code with the compiler wrapper there is fine but I ran codes with the one from intel compiler like /soft/compilers/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpirun choose the one correspond to your compiler. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [[email protected]<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:56 AM To: Luo, Ye; [email protected]<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Hi Ye, It was the running node thing that tripped me up. So what is coming up is that N tasks are showing in top, each taking 25%. So we're definitely on one core. Jeff's giving me some things to check, so I am going to go down that path for now. Thanks everyone for your help! Cheers, Tom On 9/2/2016 8:48 AM, Luo, Ye wrote: Just tried it. I can do it. You must have your job running. qstat -lf your_job_id find your node name and ssh to it. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: Tom LeCompte [[email protected]<mailto:[email protected]>] Sent: Friday, September 02, 2016 8:31 AM To: Luo, Ye; [email protected]<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] MPI vs Threading Performance Unfortunately, I don't think that works any more. ssh knl08 returns Connection closed by 10.17.6.112. On 9/2/2016 8:28 AM, Luo, Ye wrote: There is quick way to check what is going on in the node. start your job with 8 ranks. ssh to the node you job is assigned to. top and see if there are 8 processes of your application running at about 100% CPU. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: [email protected]<mailto:[email protected]> [[email protected]<mailto:[email protected]>] on behalf of Tom LeCompte [[email protected]<mailto:[email protected]>] Sent: Friday, September 02, 2016 7:31 AM To: [email protected]<mailto:[email protected]> Subject: [JLSE-Intel-NDA] MPI vs Threading Performance Hi everyone- I am investigating the relative performance of threading vs. MPI, and don't understand the results: I expect threading to be better, but not that much better. I have a fixed amount of work per thread or rank, so I expect with perfect scaling run times to be roughly constant. With threading, that's roughly what I see, and even the deviations make sense: there are inflections at 64, 256 and a dropoff around 280-300 threads. With MPI, I see the following: * 1 rank: 77s reported by the code, 213s wallclock * 2 ranks: 152s reported by the code, 288s wallclock * 8 ranks: 635s reported by the code, 769s wallclock * 32 ranks 2504s reported by the code, 2648s wallclock * 64 ranks 5463s reported by the code, 5613s wallclock So it takes about twice as much time to do twice as much work with twice as many ranks. This pattern persists up to at least 64 ranks. The natural explanation would be that I am really running on only one core. I ran with I_MPI_DEBUG set to 5, and the results are below. This looks right to me, but I am not an expert. Any thoughts? Cheers, Tom ========================================================= 1 rank: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23191 knl08.ftm.alcf.anl.gov<http://knl08.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56 ,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,8 3,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,10 7,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,12 7,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,14 7,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,16 7,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,18 7,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,20 7,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,22 7,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,24 7,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=1:0 0 2 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 113057 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29, 30,31,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88 ,89,90,91,92,93,94,95,128,129,130,131,132,133,134,135,136,137,138,139,140,141,14 2,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,192,193,19 4,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,21 4,215,216,217,218,219,220,221,222,223} [0] MPI startup(): 1 113058 knl07.ftm.alcf.anl.gov<http://knl07.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 ,59,60,61,62,63,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,160,161,162,163,164, 165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184, 185,186,187,188,189,190,191,224,225,226,227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10,21,31,41,21,10,41,31,31,41,10,41,41,31,41,10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=4 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 32 Configuration : 8 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23089 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,2,3,4,5,6,7,64,65,66,67,68,69,70,71,128,129,130,131,132,133,134,135,192,193,194,195,196,197,198,199} [0] MPI startup(): 1 23090 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,10,11,12,13,14,15,72,73,74,75,76,77,78,79,136,137,138,139,140,141,142,143,200,201,202,203,204,205,206,207} [0] MPI startup(): 2 23091 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,18,19,20,21,22,23,80,81,82,83,84,85,86,87,144,145,146,147,148,149,150,151,208,209,210,211,212,213,214,215} [0] MPI startup(): 3 23092 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,26,27,28,29,30,31,88,89,90,91,92,93,94,95,152,153,154,155,156,157,158,159,216,217,218,219,220,221,222,223} [0] MPI startup(): 4 23093 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,34,35,36,37,38,39,96,97,98,99,100,101,102,103,160,161,162,163,164,165,166,167,224,225,226,227,228,229,230,231} [0] MPI startup(): 5 23094 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,42,43,44,45,46,47,104,105,106,107,108,109,110,111,168,169,170,171,172,173,174,175,232,233,234,235,236,237,238,239} [0] MPI startup(): 6 23095 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,50,51,52,53,54,55,112,113,114,115,116,117,118,119,176,177,178,179,180,181,182,183,240,241,242,243,244,245,246,247} [0] MPI startup(): 7 23096 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,58,59,60,61,62,63,120,121,122,123,124,125,126,127,184,185,186,187,188,189,190,191,248,249,250,251,252,253,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=8:0 0,1 8,2 16,3 24,4 32,5 40,6 48,7 56 32 ranks: [0] MPI startup(): 0 90622 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {0,1,64,65,128,129,192,193} [0] MPI startup(): 1 90623 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {2,3,66,67,130,131,194,195} [0] MPI startup(): 2 90624 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {4,5,68,69,132,133,196,197} [0] MPI startup(): 3 90625 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {6,7,70,71,134,135,198,199} [0] MPI startup(): 4 90626 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {8,9,72,73,136,137,200,201} [0] MPI startup(): 5 90627 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {10,11,74,75,138,139,202,203} [0] MPI startup(): 6 90628 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {12,13,76,77,140,141,204,205} [0] MPI startup(): 7 90629 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {14,15,78,79,142,143,206,207} [0] MPI startup(): 8 90630 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {16,17,80,81,144,145,208,209} [0] MPI startup(): 9 90631 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {18,19,82,83,146,147,210,211} [0] MPI startup(): 10 90632 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {20,21,84,85,148,149,212,213} [0] MPI startup(): 11 90633 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {22,23,86,87,150,151,214,215} [0] MPI startup(): 12 90634 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {24,25,88,89,152,153,216,217} [0] MPI startup(): 13 90635 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {26,27,90,91,154,155,218,219} [0] MPI startup(): 14 90636 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {28,29,92,93,156,157,220,221} [0] MPI startup(): 15 90637 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {30,31,94,95,158,159,222,223} [0] MPI startup(): 16 90638 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {32,33,96,97,160,161,224,225} [0] MPI startup(): 17 90639 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {34,35,98,99,162,163,226,227} [0] MPI startup(): 18 90640 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {36,37,100,101,164,165,228,229} [0] MPI startup(): 19 90641 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {38,39,102,103,166,167,230,231} [0] MPI startup(): 20 90642 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {40,41,104,105,168,169,232,233} [0] MPI startup(): 21 90643 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {42,43,106,107,170,171,234,235} [0] MPI startup(): 22 90644 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {44,45,108,109,172,173,236,237} [0] MPI startup(): 23 90645 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {46,47,110,111,174,175,238,239} [0] MPI startup(): 24 90646 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {48,49,112,113,176,177,240,241} [0] MPI startup(): 25 90647 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {50,51,114,115,178,179,242,243} [0] MPI startup(): 26 90648 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {52,53,116,117,180,181,244,245} [0] MPI startup(): 27 90649 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {54,55,118,119,182,183,246,247} [0] MPI startup(): 28 90650 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {56,57,120,121,184,185,248,249} [0] MPI startup(): 29 90651 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {58,59,122,123,186,187,250,251} [0] MPI startup(): 30 90652 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {60,61,124,125,188,189,252,253} [0] MPI startup(): 31 90653 knl04.ftm.alcf.anl.gov<http://knl04.ftm.alcf.anl.gov> {62,63,126,127,190,191,254,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=32:0 0,1 2,2 4,3 6,4 8,5 10,6 12,7 14,8 16,9 18,10 20,11 22,12 24,13 26,14 28,15 30,16 32,17 34,18 36,19 38,20 40,21 42,22 44,23 46,24 48,25 50,26 52,27 54,28 56,29 58,30 60,31 62 64 ranks: [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 23076 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {0,64,128,192} [0] MPI startup(): 1 23077 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {1,65,129,193} [0] MPI startup(): 2 23078 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {2,66,130,194} [0] MPI startup(): 3 23079 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {3,67,131,195} [0] MPI startup(): 4 23080 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {4,68,132,196} [0] MPI startup(): 5 23081 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {5,69,133,197} [0] MPI startup(): 6 23082 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {6,70,134,198} [0] MPI startup(): 7 23083 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {7,71,135,199} [0] MPI startup(): 8 23084 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {8,72,136,200} [0] MPI startup(): 9 23087 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {9,73,137,201} [0] MPI startup(): 10 23088 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {10,74,138,202} [0] MPI startup(): 11 23089 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {11,75,139,203} [0] MPI startup(): 12 23090 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {12,76,140,204} [0] MPI startup(): 13 23091 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {13,77,141,205} [0] MPI startup(): 14 23092 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {14,78,142,206} [0] MPI startup(): 15 23093 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {15,79,143,207} [0] MPI startup(): 16 23099 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {16,80,144,208} [0] MPI startup(): 17 23101 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {17,81,145,209} [0] MPI startup(): 18 23102 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {18,82,146,210} [0] MPI startup(): 19 23103 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {19,83,147,211} [0] MPI startup(): 20 23104 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {20,84,148,212} [0] MPI startup(): 21 23105 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {21,85,149,213} [0] MPI startup(): 22 23108 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {22,86,150,214} [0] MPI startup(): 23 23109 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {23,87,151,215} [0] MPI startup(): 24 23111 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {24,88,152,216} [0] MPI startup(): 25 23113 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {25,89,153,217} [0] MPI startup(): 26 23114 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {26,90,154,218} [0] MPI startup(): 27 23115 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {27,91,155,219} [0] MPI startup(): 28 23116 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {28,92,156,220} [0] MPI startup(): 29 23117 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {29,93,157,221} [0] MPI startup(): 30 23118 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {30,94,158,222} [0] MPI startup(): 31 23120 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {31,95,159,223} [0] MPI startup(): 32 23121 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {32,96,160,224} [0] MPI startup(): 33 23123 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {33,97,161,225} [0] MPI startup(): 34 23124 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {34,98,162,226} [0] MPI startup(): 35 23125 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {35,99,163,227} [0] MPI startup(): 36 23126 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {36,100,164,228} [0] MPI startup(): 37 23127 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {37,101,165,229} [0] MPI startup(): 38 23128 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {38,102,166,230} [0] MPI startup(): 39 23129 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {39,103,167,231} [0] MPI startup(): 40 23130 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {40,104,168,232} [0] MPI startup(): 41 23131 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {41,105,169,233} [0] MPI startup(): 42 23132 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {42,106,170,234} [0] MPI startup(): 43 23133 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {43,107,171,235} [0] MPI startup(): 44 23134 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {44,108,172,236} [0] MPI startup(): 45 23135 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {45,109,173,237} [0] MPI startup(): 46 23136 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {46,110,174,238} [0] MPI startup(): 47 23137 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {47,111,175,239} [0] MPI startup(): 48 23138 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {48,112,176,240} [0] MPI startup(): 49 23139 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {49,113,177,241} [0] MPI startup(): 50 23140 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {50,114,178,242} [0] MPI startup(): 51 23141 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {51,115,179,243} [0] MPI startup(): 52 23147 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {52,116,180,244} [0] MPI startup(): 53 23148 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {53,117,181,245} [0] MPI startup(): 54 23149 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {54,118,182,246} [0] MPI startup(): 55 23151 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {55,119,183,247} [0] MPI startup(): 56 23153 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {56,120,184,248} [0] MPI startup(): 57 23154 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {57,121,185,249} [0] MPI startup(): 58 23156 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {58,122,186,250} [0] MPI startup(): 59 23157 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {59,123,187,251} [0] MPI startup(): 60 23158 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {60,124,188,252} [0] MPI startup(): 61 23159 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {61,125,189,253} [0] MPI startup(): 62 23160 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {62,126,190,254} [0] MPI startup(): 63 23163 knl06.ftm.alcf.anl.gov<http://knl06.ftm.alcf.anl.gov> {63,127,191,255} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=hfi1_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=64:0 0,1 1,2 2,3 3,4 4,5 5,6 6,7 7,8 8,9 9,10 10,11 11,12 12,13 13,14 14,15 15,16 16,17 17,18 18,19 19,20 20,21 21,22 22,23 23,24 24,25 25,26 26,27 27,28 28,29 29,30 30,31 31,32 32,33 33,34 34,35 35,36 36,37 37,38 38,39 39,40 40,41 41,42 42,43 43,44 44,45 45,46 46,47 47,48 48,49 49,50 50,51 51,52 52,53 53,54 54,55 55,56 56,57 57,58 58,59 59,60 60,61 61,62 62,63 63 _______________________________________________ Intel-nda mailing list [email protected]<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda