March 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] knl00 down
by Allen, Benjamin S. 18 Mar '16

18 Mar '16

Chris, The console was printing the following panic per core it appears: [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 [349243.709061] NMI backtrace for cpu 104 [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [349243.906788] Stack: [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b [349243.951521] Call Trace: [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? Ben > On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: > > Hi Ben, > > knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. > > Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? > > chris > >

1 0

Re: [JLSE-Intel-NDA] A0
by Allen, Benjamin S. 17 Mar '16

17 Mar '16

Sounds good. Ben > On Mar 17, 2016, at 12:23 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: > > No. We do not need an A0 for testing. There is no use. We should report problems on B0. Avinash is interested in talking to us and we should share just B0 results with him. > > -Kumar > > >> On Mar 17, 2016, at 11:58 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> All, >> >> Note, this A0 KNL is needed to upgrade our existing Adam Pass motherboard's BIOS evidently. We'll swap out the B0 currently installed, upgrade the BIOS, and swap back to the B0 KNL. This was the "quickest" option versus having Intel ship us a physical replacement BIOS flash chip for the board, or sending the system back to Intel for the upgrade. >> >> Not sure if Russ can make this happen or not, but is there any value in requesting another Adams Pass node, and installing this A0 KNL for testing in JLSE? >> >> Ben >> >>> On Mar 17, 2016, at 11:50 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>> >>> >>> >>>> Begin forwarded message: >>>> >>>> From: "Fromkin, Russ" <russ.fromkin(a)intel.com> >>>> Subject: A0 >>>> Date: March 17, 2016 at 11:47:42 AM CDT >>>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> >>>> Cc: "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com> >>>> >>>> Kumar, Ben, >>>> >>>> If all goes well, the A0 will ship to your attention today or tomorrow. My guess is that you will see it tomorrow or Monday. It should come with the instructions from the Adams Pass team. >>>> >>>> In addition, Avinash Sodani would like to discuss with you what you are seeing. I will add him to our other email string shortly. >>>> >>>> Russ >>>> >>>> Russ Fromkin >>>> Intel Corporation >>>> Director, Americas HPC Sales >>>> Office: (732) 790-0178 >>>> Mobile: (201) 532-1858 >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >

1 0

Re: [JLSE-Intel-NDA] A0
by Kumaran, Kalyan 17 Mar '16

17 Mar '16

No. We do not need an A0 for testing. There is no use. We should report problems on B0. Avinash is interested in talking to us and we should share just B0 results with him. -Kumar > On Mar 17, 2016, at 11:58 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > All, > > Note, this A0 KNL is needed to upgrade our existing Adam Pass motherboard's BIOS evidently. We'll swap out the B0 currently installed, upgrade the BIOS, and swap back to the B0 KNL. This was the "quickest" option versus having Intel ship us a physical replacement BIOS flash chip for the board, or sending the system back to Intel for the upgrade. > > Not sure if Russ can make this happen or not, but is there any value in requesting another Adams Pass node, and installing this A0 KNL for testing in JLSE? > > Ben > >> On Mar 17, 2016, at 11:50 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >> >> >> >>> Begin forwarded message: >>> >>> From: "Fromkin, Russ" <russ.fromkin(a)intel.com> >>> Subject: A0 >>> Date: March 17, 2016 at 11:47:42 AM CDT >>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> >>> Cc: "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com> >>> >>> Kumar, Ben, >>> >>> If all goes well, the A0 will ship to your attention today or tomorrow. My guess is that you will see it tomorrow or Monday. It should come with the instructions from the Adams Pass team. >>> >>> In addition, Avinash Sodani would like to discuss with you what you are seeing. I will add him to our other email string shortly. >>> >>> Russ >>> >>> Russ Fromkin >>> Intel Corporation >>> Director, Americas HPC Sales >>> Office: (732) 790-0178 >>> Mobile: (201) 532-1858 >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] A0
by Allen, Benjamin S. 17 Mar '16

17 Mar '16

All, Note, this A0 KNL is needed to upgrade our existing Adam Pass motherboard's BIOS evidently. We'll swap out the B0 currently installed, upgrade the BIOS, and swap back to the B0 KNL. This was the "quickest" option versus having Intel ship us a physical replacement BIOS flash chip for the board, or sending the system back to Intel for the upgrade. Not sure if Russ can make this happen or not, but is there any value in requesting another Adams Pass node, and installing this A0 KNL for testing in JLSE? Ben > On Mar 17, 2016, at 11:50 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: > > > >> Begin forwarded message: >> >> From: "Fromkin, Russ" <russ.fromkin(a)intel.com> >> Subject: A0 >> Date: March 17, 2016 at 11:47:42 AM CDT >> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> >> Cc: "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com> >> >> Kumar, Ben, >> >> If all goes well, the A0 will ship to your attention today or tomorrow. My guess is that you will see it tomorrow or Monday. It should come with the instructions from the Adams Pass team. >> >> In addition, Avinash Sodani would like to discuss with you what you are seeing. I will add him to our other email string shortly. >> >> Russ >> >> Russ Fromkin >> Intel Corporation >> Director, Americas HPC Sales >> Office: (732) 790-0178 >> Mobile: (201) 532-1858 > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Fwd: A0
by Kumaran, Kalyan 17 Mar '16

17 Mar '16

Begin forwarded message: From: "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>> Subject: A0 Date: March 17, 2016 at 11:47:42 AM CDT To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>> Kumar, Ben, If all goes well, the A0 will ship to your attention today or tomorrow. My guess is that you will see it tomorrow or Monday. It should come with the instructions from the Adams Pass team. In addition, Avinash Sodani would like to discuss with you what you are seeing. I will add him to our other email string shortly. Russ Russ Fromkin Intel Corporation Director, Americas HPC Sales Office: (732) 790-0178 Mobile: (201) 532-1858

1 0

[JLSE-Intel-NDA] Fwd: B0 box
by Kumaran, Kalyan 16 Mar '16

16 Mar '16

Begin forwarded message: From: "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>> Subject: RE: B0 box Date: March 16, 2016 at 10:46:18 AM CDT To: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Ben, So there are three options to upgrade the bios. 1) Return the system to be updated? 2) Send a new flash chip? 3) Find an A0 CPU and ship it to ANL? For #1, I don't trust having our system out of our hands. Possession in 9/10th the law. :-) For #2: I am being told that it might be a little wait for new flash chips. For #3: I don't have any A0s, but with the B0s rolling into the Intel internal systems, we think we can get one pretty quickly and send it out. For #3, this is the most work on the ANL side. We would need to pull the B0 and replace with the A0. Then we can update the bios. After it is update, we can pull the A0 and replace the B0. Are you okay with this? Russ

1 0

[JLSE-Intel-NDA] Access for Jeff Hammond
by Kumaran, Kalyan 16 Mar '16

16 Mar '16

Hi Ben, is it possible to provide Jeff access to our KNL box? Jeff might be able to help us with software and also explain some of the performance issues we have. Thanks, Kumar

1 0

[JLSE-Intel-NDA] Fwd: B0 box
by Kumaran, Kalyan 15 Mar '16

15 Mar '16

Begin forwarded message: From: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Date: March 15, 2016 at 5:43:27 PM CDT To: "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>> Cc: "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: B0 box Russ, Both the single 16GB DIMM and 6x 32GB DIMMs are coming up at the same speed 2134Mt/s, and are otherwise spec'ed the same. As such, I don't think thats the issue at hand. If anything with 6 DIMMs we should be getter better memory throughput, since KNL has 6 memory channels. Vitali has benchmarked with STREAM a decrease in memory throughput between having a single 16GB DIMM vs 6 32GB DIMMs. Even though the test should be small enough to stay on the MCDRAM cached, and we have the MCDRAM in cached mode. We're looking to discuss why that is as its unexpected, and ensure we didn't overlook something. Note, the goal with 6 x 32GB DIMMs is to simulate a Theta node. Thanks, Ben On Mar 15, 2016, at 4:34 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> wrote: Hi Russ, yes, it is the memory bus frequency issue. -Kumar On Mar 15, 2016, at 4:33 PM, Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> wrote: Kumar, I this the 2133 vs 2400 question or something different? I believe we need a bios upgrade, but i am hearing we may need to send an A0 for it to work. We are questioning why and will have an answer soon. This is our PCSC beta box, so there isnt regular support for it yet. Only a few people have this box. Russ Sent via the Samsung Galaxy S® 6, an AT&T 4G LTE smartphone -------- Original message -------- From: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Date: 3/15/2016 4:20 PM (GMT-05:00) To: "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>>, "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>> Cc: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: B0 box Russ, to whom do we send our questions to on the box? We removed the 16GB DIMM and added 192 GB of memory. Our Stream results are not good. We want to make sure we are doing the right thing. Thanks, Kumar

1 0

Re: [JLSE-Intel-NDA] MILCmk results on KNL b0
by Kumaran, Kalyan 14 Mar '16

14 Mar '16

Please use intel-nda list. -Kumar > On Mar 14, 2016, at 3:29 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: > > > ok. I will take a look at that one and see what I can find out. > > kevin > > > > >> Hi Kevin, >> >> Probably 65536 is the most relevant. This code is old now so I don't >> care about it too much now anyway. The performance is still slower than >> I would have expected even without any vectorization. It may be nice to >> understand what the bottleneck is for say the single precision, 65536, >> QLA_V_vpeq_M_times_pV case (or for simplicity, even the >> QLA_V_veq_Ma_times_V one since that one has a more regular memory access >> pattern). >> >> Cheers, >> James >> >> >> On 03/11/2016 12:10 PM, Harms, Kevin wrote: >>> James, >>> >>> I ran the MILCmk CORAL benchmarks on the KNL b0. I'm attaching the results. For larger sizes, there's a 2-3x speedup over mira. What sizes do you consider to be the important ones? >>> >>> kevin >>> >>> MILC-v1 benchmark from CORAL site >>> A2 results from running defaults in Makefile >>> KNL B0 results compiled with Intel using basic optimization >>> >>> icc -std=c99 -O3 -xMIC-AVX512 -openmp -Iqla-1.7.1 -DQLA_Precision=\'F\' -DQLA_Colors=3 -DQLA_Nc=3 -c qla-1.7.1/QLA_F3_c1_veq_V_dot_V.c -o qla-1.7.1/QLA_F3_c1_veq_V_dot_V.o >>> >>> Run with OMP_NUM_THREADS=64 (have not tried 128 yet) >>> >>> I could not run the larger sizes on KNL because lack of memory. >>> >>>

1 0

Re: [JLSE-Intel-NDA] Bringing KNL Machine Down for SSD Install
by Allen, Benjamin S. 14 Mar '16

14 Mar '16

This is done. The machine is back open for regular use. Ben > On Mar 14, 2016, at 1:56 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Adding in a 200GB SATA SSD. The machine will be down for 20-30minutes. > > Ben > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0