- Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] IPM size
by Kumaran, Kalyan 25 Mar '16

25 Mar '16

I’m checking on the IPM size for our Bin3 cpu. Vitali says his STREAM tests tell him it is 8GB and not 16GB. There are no details on this slide. -Kumar [cid:[email protected]]

1 0

Re: [JLSE-Intel-NDA] knl00 down
by Allen, Benjamin S. 23 Mar '16

23 Mar '16

I can get a similar kernel panic by running a 128 or 256 rank mpi_hello_world app, so I don't think this is actually related to LAMMPS. LAMMPS is just quicker at triggering it. I'm going to hold off doing additional testing and troubleshooting on this until we get the BIOS upgraded. We're waiting on a A0 KNL from Intel, which we need to swap out with the currently installed B0, as it's evidently needed for the BIOS update. After the update we'll swap the B0 back in. We're also still seeing numerous Machine Check Errors on the B0 CPU cores, eg. "Generic CACHE Level-2 Generic Error". Note, kdump is now setup to copy vmcore's on panics out to /var/crash, which is a persistent NFS mount. For whatever reason after it dumps and reboots the machine, the machine hangs in POST. As such it still needs to be cycled via IPMI after a panic. Thanks, Ben > On Mar 22, 2016, at 1:51 PM, Christopher Knight <knightc(a)anl.gov> wrote: > > Short runs with <= 64 MPI ranks ran once successfully. A run with 128 MPI ranks stalled after compute phase completed. > > mpirun -np 128 ../src/lmp_knl-mpi -in lmp.in -var NSTEPS 10 > > > > Do you want to have back-forth discussion in this email thread with list cc’d? Jabber? > > knightc(a)jabber.it.anl.gov > > > chris > > > > > On Mar 21, 2016, at 6:10 PM, Christopher Knight <knightc(a)anl.gov> wrote: > > 1:30pm tomorrow works for me. > > chris > > >> On Mar 21, 2016, at 5:55 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Sounds good. How's starting around 1:30pm tomorrow? >> >> Ben >> >>> On Mar 21, 2016, at 5:23 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>> >>> Yes, those were the settings used to compile the current executable. >>> >>> I can build a few different executables with same compiler, but different libraries linked and try running tests again to cause kernel panic. >>> >>> 1) Only MPI >>> 2) MPI + OpenMP >>> 3) MPI + MKL >>> 4) MPI + OpenMP + MKL (current executable) >>> >>> If #1 causes a kernel panic, then I could repeat tests with different compiler. I’m open to suggestions. >>> >>> >>> I could coordinate with you and start running these tomorrow during a specific block of time if that helps any? >>> >>> chris >>> >>> >>> >>> >>> On Mar 21, 2016, at 3:36 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct. >>> >>> Did you freshly compile lammps with the below compiler and MPI vars? >>> >>> Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel. >>> >>> Power cycling the node now, and bring back up with the 3.10 kernel. >>> Ben >>> >>>> On Mar 18, 2016, at 6:20 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>> >>>> <snaps fingers> it happened again. >>>> >>>> >>>> I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled. >>>> >>>> env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp >>>> >>>> >>>> One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right? >>>> >>>> source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>>> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>>> >>>> chris >>>> >>>> >>>> >>>>> On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based. >>>>> >>>>> I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again. >>>>> >>>>> Chris, >>>>> >>>>> I've now ran the following a few times successfully: >>>>> >>>>> source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 >>>>> source /soft/compilers/intel/impi_latest/bin64/mpivars.sh >>>>> >>>>> env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000 >>>>> >>>>> Please free to continue with your testing. >>>>> >>>>> Ben >>>>> >>>>>> On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>> >>>>>> Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler: >>>>>> >>>>>> BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa >>>>>> [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>>> [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 >>>>>> [ 1275.748524] Oops: 0002 [#1] SMP >>>>>> [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common >>>>>> [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>>> [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>>> [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 >>>>>> [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>>> [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 >>>>>> [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 >>>>>> [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 >>>>>> [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 >>>>>> [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 >>>>>> [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 >>>>>> [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 >>>>>> [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>> [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 >>>>>> [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>> [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>> [ 1275.998089] Stack: >>>>>> [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 >>>>>> [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 >>>>>> [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 >>>>>> [ 1276.054382] Call Trace: >>>>>> [ 1276.063947] <IRQ> >>>>>> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 >>>>>> [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 >>>>>> [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 >>>>>> [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 >>>>>> [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 >>>>>> [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 >>>>>> [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 >>>>>> [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 >>>>>> [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 >>>>>> [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 >>>>>> [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 >>>>>> [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 >>>>>> [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 >>>>>> [ 1276.225074] <EOI> >>>>>> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b >>>>>> [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 >>>>>> [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>>> [ 1276.286963] RSP <ffff882f7da63dd8> >>>>>> [ 1276.295181] CR2: 00000000000001fa >>>>>> >>>>>> The kernel then goes on to send NMI to the rest of the CPUs: >>>>>> >>>>>> [ 1344.258750] sending NMI to all CPUs: >>>>>> >>>>>> The NMIs were what I was seeing last go around. This is going to be a painful to fix. >>>>>> >>>>>> Ben >>>>>> >>>>>> >>>>>>> On Mar 18, 2016, at 3:48 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>> >>>>>>> I think I broke it again… >>>>>>> >>>>>>> This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file. >>>>>>> >>>>>>> chris >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>>> >>>>>>> Odd. The machine is back up, feel free to continue your testing. >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>>> On Mar 18, 2016, at 3:26 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>>> >>>>>>>> Hi Ben, >>>>>>>> >>>>>>>> If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs. >>>>>>>> >>>>>>>> >>>>>>>> The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following. >>>>>>>> >>>>>>>> [knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>>> … >>>>>>>> Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled. >>>>>>>> >>>>>>>> [knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>>> ... >>>>>>>> Setting up run ... >>>>>>>> Memory usage per processor = 18.0914 Mbytes >>>>>>>> ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- >>>>>>>> TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 >>>>>>>> PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 >>>>>>>> E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 >>>>>>>> E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562 >>>>>>>> >>>>>>>> … then sat here for little while doing nothing. >>>>>>>> >>>>>>>> >>>>>>>> chris >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>>>> >>>>>>>> Chris, >>>>>>>> >>>>>>>> The console was printing the following panic per core it appears: >>>>>>>> >>>>>>>> [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 >>>>>>>> [349243.709061] NMI backtrace for cpu 104 >>>>>>>> [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>>>>> [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>>>>> [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 >>>>>>>> [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 >>>>>>>> [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 >>>>>>>> [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 >>>>>>>> [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 >>>>>>>> [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f >>>>>>>> [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 >>>>>>>> [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 >>>>>>>> [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 >>>>>>>> [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>>> [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 >>>>>>>> [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>>>> [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>>>> [349243.906788] Stack: >>>>>>>> [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 >>>>>>>> [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 >>>>>>>> [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b >>>>>>>> [349243.951521] Call Trace: >>>>>>>> [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 >>>>>>>> [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 >>>>>>>> [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 >>>>>>>> [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 >>>>>>>> >>>>>>>> This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? >>>>>>>> >>>>>>>> Ben >>>>>>>> >>>>>>>>> On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>>>> >>>>>>>>> Hi Ben, >>>>>>>>> >>>>>>>>> knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. >>>>>>>>> >>>>>>>>> Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? >>>>>>>>> >>>>>>>>> chris >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] knl00 down
by Allen, Benjamin S. 22 Mar '16

22 Mar '16

Assume you started trying things, same panic: [78779.507861] Call Trace: [78779.517642] <IRQ> [78779.519968] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 [78779.540147] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 [78779.553513] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 [78779.566616] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 [78779.580873] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 [78779.594370] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 [78779.608295] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 [78779.621358] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 [78779.634818] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 [78779.647939] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 [78779.660620] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 [78779.674437] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 [78779.688008] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 [78779.701134] <EOI> [78779.703409] Code: Bad RIP value. Power cycling the box. Ben > On Mar 21, 2016, at 6:10 PM, Christopher Knight <knightc(a)anl.gov> wrote: > > 1:30pm tomorrow works for me. > > chris > > >> On Mar 21, 2016, at 5:55 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Sounds good. How's starting around 1:30pm tomorrow? >> >> Ben >> >>> On Mar 21, 2016, at 5:23 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>> >>> Yes, those were the settings used to compile the current executable. >>> >>> I can build a few different executables with same compiler, but different libraries linked and try running tests again to cause kernel panic. >>> >>> 1) Only MPI >>> 2) MPI + OpenMP >>> 3) MPI + MKL >>> 4) MPI + OpenMP + MKL (current executable) >>> >>> If #1 causes a kernel panic, then I could repeat tests with different compiler. I’m open to suggestions. >>> >>> >>> I could coordinate with you and start running these tomorrow during a specific block of time if that helps any? >>> >>> chris >>> >>> >>> >>> >>> On Mar 21, 2016, at 3:36 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct. >>> >>> Did you freshly compile lammps with the below compiler and MPI vars? >>> >>> Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel. >>> >>> Power cycling the node now, and bring back up with the 3.10 kernel. >>> Ben >>> >>>> On Mar 18, 2016, at 6:20 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>> >>>> <snaps fingers> it happened again. >>>> >>>> >>>> I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled. >>>> >>>> env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp >>>> >>>> >>>> One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right? >>>> >>>> source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>>> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>>> >>>> chris >>>> >>>> >>>> >>>>> On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based. >>>>> >>>>> I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again. >>>>> >>>>> Chris, >>>>> >>>>> I've now ran the following a few times successfully: >>>>> >>>>> source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 >>>>> source /soft/compilers/intel/impi_latest/bin64/mpivars.sh >>>>> >>>>> env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000 >>>>> >>>>> Please free to continue with your testing. >>>>> >>>>> Ben >>>>> >>>>>> On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>> >>>>>> Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler: >>>>>> >>>>>> BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa >>>>>> [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>>> [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 >>>>>> [ 1275.748524] Oops: 0002 [#1] SMP >>>>>> [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common >>>>>> [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>>> [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>>> [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 >>>>>> [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>>> [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 >>>>>> [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 >>>>>> [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 >>>>>> [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 >>>>>> [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 >>>>>> [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 >>>>>> [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 >>>>>> [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>> [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 >>>>>> [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>> [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>> [ 1275.998089] Stack: >>>>>> [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 >>>>>> [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 >>>>>> [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 >>>>>> [ 1276.054382] Call Trace: >>>>>> [ 1276.063947] <IRQ> >>>>>> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 >>>>>> [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 >>>>>> [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 >>>>>> [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 >>>>>> [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 >>>>>> [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 >>>>>> [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 >>>>>> [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 >>>>>> [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 >>>>>> [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 >>>>>> [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 >>>>>> [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 >>>>>> [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 >>>>>> [ 1276.225074] <EOI> >>>>>> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b >>>>>> [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 >>>>>> [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>>> [ 1276.286963] RSP <ffff882f7da63dd8> >>>>>> [ 1276.295181] CR2: 00000000000001fa >>>>>> >>>>>> The kernel then goes on to send NMI to the rest of the CPUs: >>>>>> >>>>>> [ 1344.258750] sending NMI to all CPUs: >>>>>> >>>>>> The NMIs were what I was seeing last go around. This is going to be a painful to fix. >>>>>> >>>>>> Ben >>>>>> >>>>>> >>>>>>> On Mar 18, 2016, at 3:48 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>> >>>>>>> I think I broke it again… >>>>>>> >>>>>>> This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file. >>>>>>> >>>>>>> chris >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>>> >>>>>>> Odd. The machine is back up, feel free to continue your testing. >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>>> On Mar 18, 2016, at 3:26 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>>> >>>>>>>> Hi Ben, >>>>>>>> >>>>>>>> If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs. >>>>>>>> >>>>>>>> >>>>>>>> The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following. >>>>>>>> >>>>>>>> [knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>>> … >>>>>>>> Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled. >>>>>>>> >>>>>>>> [knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>>> ... >>>>>>>> Setting up run ... >>>>>>>> Memory usage per processor = 18.0914 Mbytes >>>>>>>> ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- >>>>>>>> TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 >>>>>>>> PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 >>>>>>>> E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 >>>>>>>> E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562 >>>>>>>> >>>>>>>> … then sat here for little while doing nothing. >>>>>>>> >>>>>>>> >>>>>>>> chris >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>>>> >>>>>>>> Chris, >>>>>>>> >>>>>>>> The console was printing the following panic per core it appears: >>>>>>>> >>>>>>>> [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 >>>>>>>> [349243.709061] NMI backtrace for cpu 104 >>>>>>>> [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>>>>> [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>>>>> [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 >>>>>>>> [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 >>>>>>>> [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 >>>>>>>> [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 >>>>>>>> [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 >>>>>>>> [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f >>>>>>>> [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 >>>>>>>> [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 >>>>>>>> [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 >>>>>>>> [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>>> [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 >>>>>>>> [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>>>> [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>>>> [349243.906788] Stack: >>>>>>>> [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 >>>>>>>> [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 >>>>>>>> [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b >>>>>>>> [349243.951521] Call Trace: >>>>>>>> [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 >>>>>>>> [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 >>>>>>>> [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 >>>>>>>> [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 >>>>>>>> >>>>>>>> This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? >>>>>>>> >>>>>>>> Ben >>>>>>>> >>>>>>>>> On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>>>> >>>>>>>>> Hi Ben, >>>>>>>>> >>>>>>>>> knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. >>>>>>>>> >>>>>>>>> Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? >>>>>>>>> >>>>>>>>> chris >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> >

1 0

Re: [JLSE-Intel-NDA] knl00 down
by Christopher Knight 22 Mar '16

22 Mar '16

Short runs with <= 64 MPI ranks ran once successfully. A run with 128 MPI ranks stalled after compute phase completed. mpirun -np 128 ../src/lmp_knl-mpi -in lmp.in -var NSTEPS 10 Do you want to have back-forth discussion in this email thread with list cc’d? Jabber? knightc(a)jabber.it.anl.gov chris On Mar 21, 2016, at 6:10 PM, Christopher Knight <knightc(a)anl.gov> wrote: 1:30pm tomorrow works for me. chris > On Mar 21, 2016, at 5:55 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Sounds good. How's starting around 1:30pm tomorrow? > > Ben > >> On Mar 21, 2016, at 5:23 PM, Christopher Knight <knightc(a)anl.gov> wrote: >> >> Yes, those were the settings used to compile the current executable. >> >> I can build a few different executables with same compiler, but different libraries linked and try running tests again to cause kernel panic. >> >> 1) Only MPI >> 2) MPI + OpenMP >> 3) MPI + MKL >> 4) MPI + OpenMP + MKL (current executable) >> >> If #1 causes a kernel panic, then I could repeat tests with different compiler. I’m open to suggestions. >> >> >> I could coordinate with you and start running these tomorrow during a specific block of time if that helps any? >> >> chris >> >> >> >> >> On Mar 21, 2016, at 3:36 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct. >> >> Did you freshly compile lammps with the below compiler and MPI vars? >> >> Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel. >> >> Power cycling the node now, and bring back up with the 3.10 kernel. >> Ben >> >>> On Mar 18, 2016, at 6:20 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>> >>> <snaps fingers> it happened again. >>> >>> >>> I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled. >>> >>> env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp >>> >>> >>> One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right? >>> >>> source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>> >>> chris >>> >>> >>> >>>> On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based. >>>> >>>> I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again. >>>> >>>> Chris, >>>> >>>> I've now ran the following a few times successfully: >>>> >>>> source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 >>>> source /soft/compilers/intel/impi_latest/bin64/mpivars.sh >>>> >>>> env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000 >>>> >>>> Please free to continue with your testing. >>>> >>>> Ben >>>> >>>>> On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler: >>>>> >>>>> BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa >>>>> [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>> [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 >>>>> [ 1275.748524] Oops: 0002 [#1] SMP >>>>> [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common >>>>> [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>> [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>> [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 >>>>> [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>> [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 >>>>> [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 >>>>> [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 >>>>> [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 >>>>> [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 >>>>> [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 >>>>> [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 >>>>> [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 >>>>> [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>> [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>> [ 1275.998089] Stack: >>>>> [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 >>>>> [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 >>>>> [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 >>>>> [ 1276.054382] Call Trace: >>>>> [ 1276.063947] <IRQ> >>>>> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 >>>>> [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 >>>>> [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 >>>>> [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 >>>>> [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 >>>>> [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 >>>>> [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 >>>>> [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 >>>>> [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 >>>>> [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 >>>>> [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 >>>>> [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 >>>>> [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 >>>>> [ 1276.225074] <EOI> >>>>> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b >>>>> [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 >>>>> [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>> [ 1276.286963] RSP <ffff882f7da63dd8> >>>>> [ 1276.295181] CR2: 00000000000001fa >>>>> >>>>> The kernel then goes on to send NMI to the rest of the CPUs: >>>>> >>>>> [ 1344.258750] sending NMI to all CPUs: >>>>> >>>>> The NMIs were what I was seeing last go around. This is going to be a painful to fix. >>>>> >>>>> Ben >>>>> >>>>> >>>>>> On Mar 18, 2016, at 3:48 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>> >>>>>> I think I broke it again… >>>>>> >>>>>> This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file. >>>>>> >>>>>> chris >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>> >>>>>> Odd. The machine is back up, feel free to continue your testing. >>>>>> >>>>>> Ben >>>>>> >>>>>>> On Mar 18, 2016, at 3:26 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>> >>>>>>> Hi Ben, >>>>>>> >>>>>>> If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs. >>>>>>> >>>>>>> >>>>>>> The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following. >>>>>>> >>>>>>> [knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>> … >>>>>>> Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms >>>>>>> >>>>>>> >>>>>>> >>>>>>> When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled. >>>>>>> >>>>>>> [knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>> ... >>>>>>> Setting up run ... >>>>>>> Memory usage per processor = 18.0914 Mbytes >>>>>>> ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- >>>>>>> TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 >>>>>>> PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 >>>>>>> E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 >>>>>>> E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562 >>>>>>> >>>>>>> … then sat here for little while doing nothing. >>>>>>> >>>>>>> >>>>>>> chris >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>>> >>>>>>> Chris, >>>>>>> >>>>>>> The console was printing the following panic per core it appears: >>>>>>> >>>>>>> [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 >>>>>>> [349243.709061] NMI backtrace for cpu 104 >>>>>>> [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>>>> [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>>>> [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 >>>>>>> [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 >>>>>>> [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 >>>>>>> [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 >>>>>>> [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 >>>>>>> [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f >>>>>>> [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 >>>>>>> [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 >>>>>>> [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 >>>>>>> [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>> [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 >>>>>>> [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>>> [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>>> [349243.906788] Stack: >>>>>>> [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 >>>>>>> [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 >>>>>>> [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b >>>>>>> [349243.951521] Call Trace: >>>>>>> [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 >>>>>>> [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 >>>>>>> [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 >>>>>>> [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 >>>>>>> >>>>>>> This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>>> On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>>> >>>>>>>> Hi Ben, >>>>>>>> >>>>>>>> knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. >>>>>>>> >>>>>>>> Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? >>>>>>>> >>>>>>>> chris >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >> > _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] [Alcf3-acceptance] Access to our KNL box
by Hal Finkel 22 Mar '16

22 Mar '16

Hi Ben, Thanks! I'll play more with this later. Good news in the mean time is that Clang AVX-512 support seems at least minimally functional: /soft/compilers/llvm/nightly/bin/clang -O3 -ffast-math -march=knl -o test test.c -Hal ----- Original Message ----- > From: "Benjamin S. Allen" <bsallen(a)alcf.anl.gov> > To: "Hal J. Finkel" <hfinkel(a)anl.gov> > Cc: intel-nda(a)jlse.anl.gov > Sent: Tuesday, March 22, 2016 1:02:14 PM > Subject: Re: [Alcf3-acceptance] Access to our KNL box > > Also in the future CC intel-nda(a)jlse.anl.gov, so everyone is aware of > status on the machine. > > Ben > > > On Mar 22, 2016, at 1:01 PM, Allen, Benjamin S. > > <bsallen(a)alcf.anl.gov> wrote: > > > > nslcd segfaulted evidently, so the system wasn't aware of who you > > were. This machine isn't very stable. Note, at 1:30 I'm going to > > be work with Chris Knight to trial-and-error a kernel panic that > > LAMMPS can trigger. > > > > Ben > > > >> On Mar 22, 2016, at 12:57 PM, Hal Finkel <hfinkel(a)anl.gov> wrote: > >> > >> Hi Ben, > >> > >> Thanks. When I tried to ssh to knl00 from ftlogin2, it prompted me > >> for a password. How can I login? > >> > >> -Hal > >> > >> ----- Original Message ----- > >>> From: "Benjamin S. Allen" <bsallen(a)alcf.anl.gov> > >>> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov> > >>> Cc: "Hal J. Finkel" <hfinkel(a)anl.gov>, "James C. Osborn" > >>> <osborn(a)alcf.anl.gov> > >>> Sent: Monday, March 14, 2016 3:36:32 PM > >>> Subject: Re: [Alcf3-acceptance] Access to our KNL box > >>> > >>> Added Hal. > >>> > >>> Ben > >>> > >>>> On Mar 14, 2016, at 3:23 PM, Kumaran, Kalyan > >>>> <kumaran(a)alcf.anl.gov> > >>>> wrote: > >>>> > >>>> Hal, even though technically you are not part of theta > >>>> acceptance > >>>> you still have remote access to A0 hardware. So I’m fine with > >>>> you > >>>> accessing this system. Please note that we are keeping it as a > >>>> top > >>>> secret and not announcing beyond ALCF. > >>>> > >>>> -Kumar > >>>> > >>>>> Begin forwarded message: > >>>>> > >>>>> From: Kalyan Kumaran <kumaran(a)anl.gov> > >>>>> Subject: Re: [Alcf3-acceptance] Access to our KNL box > >>>>> Date: March 14, 2016 at 12:43:07 PM CDT > >>>>> To: "alcf3-acceptance(a)alcf.anl.gov" > >>>>> <alcf3-acceptance(a)alcf.anl.gov> > >>>>> Cc: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> > >>>>> > >>>>> Its intel-nda(a)jlse.anl.gov > >>>>> > >>>>> > >>>>>> On Mar 14, 2016, at 12:41 PM, Kumaran, Kalyan > >>>>>> <kumaran(a)alcf.anl.gov> wrote: > >>>>>> > >>>>>> We are ready to open up the KNL box to our staff preparing for > >>>>>> Theta acceptance. If you would like access to the box please > >>>>>> send an email to Ben with a cc to me. Please ask for access > >>>>>> only > >>>>>> if you are ready to run something and seriously test the > >>>>>> machine. We need testers to find and report issues. > >>>>>> > >>>>>> You will be added to the unix group and also the email list > >>>>>> (intel-ada(a)jlse.anl.gov) We will use the mailing list for > >>>>>> discussion and report issues. > >>>>>> > >>>>>> Considering that this box is being shared by several users we > >>>>>> will use a rather simplistic approach to scheduling. There is > >>>>>> a > >>>>>> file (/soft/scheduling/knl.txt) that will be used to announce > >>>>>> a > >>>>>> reservation or make one. > >>>>>> > >>>>>> Example: > >>>>>> > >>>>>> knl00 node status: in use by Scott Parker until 5pm > >>>>>> or > >>>>>> knl00 node status: free > >>>>>> > >>>>>> and reservation requests like > >>>>>> > >>>>>> Reservation: 3/17, 10am-2pm, Scott Parker > >>>>>> > >>>>>> Each user of the node should update the file before and after > >>>>>> every use > >>>>>> > >>>>>> -Kumar > >>>>>> > >>>>>> P.S.: we will be taking down the system sometime today to add > >>>>>> a > >>>>>> SSD. > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Alcf3-acceptance mailing list > >>>>>> Alcf3-acceptance(a)lists.alcf.anl.gov > >>>>>> https://lists.alcf.anl.gov/mailman/listinfo/alcf3-acceptance > >>>>> > >>>> > >>> > >>> > >> > >> -- > >> Hal Finkel > >> Assistant Computational Scientist > >> Leadership Computing Facility > >> Argonne National Laboratory > > > > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] [Alcf3-acceptance] Access to our KNL box
by Allen, Benjamin S. 22 Mar '16

22 Mar '16

Also in the future CC intel-nda(a)jlse.anl.gov, so everyone is aware of status on the machine. Ben > On Mar 22, 2016, at 1:01 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > nslcd segfaulted evidently, so the system wasn't aware of who you were. This machine isn't very stable. Note, at 1:30 I'm going to be work with Chris Knight to trial-and-error a kernel panic that LAMMPS can trigger. > > Ben > >> On Mar 22, 2016, at 12:57 PM, Hal Finkel <hfinkel(a)anl.gov> wrote: >> >> Hi Ben, >> >> Thanks. When I tried to ssh to knl00 from ftlogin2, it prompted me for a password. How can I login? >> >> -Hal >> >> ----- Original Message ----- >>> From: "Benjamin S. Allen" <bsallen(a)alcf.anl.gov> >>> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov> >>> Cc: "Hal J. Finkel" <hfinkel(a)anl.gov>, "James C. Osborn" <osborn(a)alcf.anl.gov> >>> Sent: Monday, March 14, 2016 3:36:32 PM >>> Subject: Re: [Alcf3-acceptance] Access to our KNL box >>> >>> Added Hal. >>> >>> Ben >>> >>>> On Mar 14, 2016, at 3:23 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>> wrote: >>>> >>>> Hal, even though technically you are not part of theta acceptance >>>> you still have remote access to A0 hardware. So I’m fine with you >>>> accessing this system. Please note that we are keeping it as a top >>>> secret and not announcing beyond ALCF. >>>> >>>> -Kumar >>>> >>>>> Begin forwarded message: >>>>> >>>>> From: Kalyan Kumaran <kumaran(a)anl.gov> >>>>> Subject: Re: [Alcf3-acceptance] Access to our KNL box >>>>> Date: March 14, 2016 at 12:43:07 PM CDT >>>>> To: "alcf3-acceptance(a)alcf.anl.gov" >>>>> <alcf3-acceptance(a)alcf.anl.gov> >>>>> Cc: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> >>>>> >>>>> Its intel-nda(a)jlse.anl.gov >>>>> >>>>> >>>>>> On Mar 14, 2016, at 12:41 PM, Kumaran, Kalyan >>>>>> <kumaran(a)alcf.anl.gov> wrote: >>>>>> >>>>>> We are ready to open up the KNL box to our staff preparing for >>>>>> Theta acceptance. If you would like access to the box please >>>>>> send an email to Ben with a cc to me. Please ask for access only >>>>>> if you are ready to run something and seriously test the >>>>>> machine. We need testers to find and report issues. >>>>>> >>>>>> You will be added to the unix group and also the email list >>>>>> (intel-ada(a)jlse.anl.gov) We will use the mailing list for >>>>>> discussion and report issues. >>>>>> >>>>>> Considering that this box is being shared by several users we >>>>>> will use a rather simplistic approach to scheduling. There is a >>>>>> file (/soft/scheduling/knl.txt) that will be used to announce a >>>>>> reservation or make one. >>>>>> >>>>>> Example: >>>>>> >>>>>> knl00 node status: in use by Scott Parker until 5pm >>>>>> or >>>>>> knl00 node status: free >>>>>> >>>>>> and reservation requests like >>>>>> >>>>>> Reservation: 3/17, 10am-2pm, Scott Parker >>>>>> >>>>>> Each user of the node should update the file before and after >>>>>> every use >>>>>> >>>>>> -Kumar >>>>>> >>>>>> P.S.: we will be taking down the system sometime today to add a >>>>>> SSD. >>>>>> >>>>>> _______________________________________________ >>>>>> Alcf3-acceptance mailing list >>>>>> Alcf3-acceptance(a)lists.alcf.anl.gov >>>>>> https://lists.alcf.anl.gov/mailman/listinfo/alcf3-acceptance >>>>> >>>> >>> >>> >> >> -- >> Hal Finkel >> Assistant Computational Scientist >> Leadership Computing Facility >> Argonne National Laboratory >

1 0

Re: [JLSE-Intel-NDA] knl00 down
by Christopher Knight 21 Mar '16

21 Mar '16

1:30pm tomorrow works for me. chris > On Mar 21, 2016, at 5:55 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Sounds good. How's starting around 1:30pm tomorrow? > > Ben > >> On Mar 21, 2016, at 5:23 PM, Christopher Knight <knightc(a)anl.gov> wrote: >> >> Yes, those were the settings used to compile the current executable. >> >> I can build a few different executables with same compiler, but different libraries linked and try running tests again to cause kernel panic. >> >> 1) Only MPI >> 2) MPI + OpenMP >> 3) MPI + MKL >> 4) MPI + OpenMP + MKL (current executable) >> >> If #1 causes a kernel panic, then I could repeat tests with different compiler. I’m open to suggestions. >> >> >> I could coordinate with you and start running these tomorrow during a specific block of time if that helps any? >> >> chris >> >> >> >> >> On Mar 21, 2016, at 3:36 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct. >> >> Did you freshly compile lammps with the below compiler and MPI vars? >> >> Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel. >> >> Power cycling the node now, and bring back up with the 3.10 kernel. >> Ben >> >>> On Mar 18, 2016, at 6:20 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>> >>> <snaps fingers> it happened again. >>> >>> >>> I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled. >>> >>> env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp >>> >>> >>> One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right? >>> >>> source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>> >>> chris >>> >>> >>> >>>> On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based. >>>> >>>> I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again. >>>> >>>> Chris, >>>> >>>> I've now ran the following a few times successfully: >>>> >>>> source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 >>>> source /soft/compilers/intel/impi_latest/bin64/mpivars.sh >>>> >>>> env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000 >>>> >>>> Please free to continue with your testing. >>>> >>>> Ben >>>> >>>>> On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler: >>>>> >>>>> BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa >>>>> [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>> [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 >>>>> [ 1275.748524] Oops: 0002 [#1] SMP >>>>> [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common >>>>> [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>> [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>> [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 >>>>> [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>> [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 >>>>> [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 >>>>> [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 >>>>> [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 >>>>> [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 >>>>> [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 >>>>> [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 >>>>> [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 >>>>> [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>> [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>> [ 1275.998089] Stack: >>>>> [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 >>>>> [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 >>>>> [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 >>>>> [ 1276.054382] Call Trace: >>>>> [ 1276.063947] <IRQ> >>>>> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 >>>>> [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 >>>>> [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 >>>>> [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 >>>>> [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 >>>>> [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 >>>>> [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 >>>>> [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 >>>>> [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 >>>>> [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 >>>>> [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 >>>>> [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 >>>>> [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 >>>>> [ 1276.225074] <EOI> >>>>> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b >>>>> [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 >>>>> [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>>> [ 1276.286963] RSP <ffff882f7da63dd8> >>>>> [ 1276.295181] CR2: 00000000000001fa >>>>> >>>>> The kernel then goes on to send NMI to the rest of the CPUs: >>>>> >>>>> [ 1344.258750] sending NMI to all CPUs: >>>>> >>>>> The NMIs were what I was seeing last go around. This is going to be a painful to fix. >>>>> >>>>> Ben >>>>> >>>>> >>>>>> On Mar 18, 2016, at 3:48 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>> >>>>>> I think I broke it again… >>>>>> >>>>>> This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file. >>>>>> >>>>>> chris >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>> >>>>>> Odd. The machine is back up, feel free to continue your testing. >>>>>> >>>>>> Ben >>>>>> >>>>>>> On Mar 18, 2016, at 3:26 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>> >>>>>>> Hi Ben, >>>>>>> >>>>>>> If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs. >>>>>>> >>>>>>> >>>>>>> The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following. >>>>>>> >>>>>>> [knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>> … >>>>>>> Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms >>>>>>> >>>>>>> >>>>>>> >>>>>>> When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled. >>>>>>> >>>>>>> [knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>>> ... >>>>>>> Setting up run ... >>>>>>> Memory usage per processor = 18.0914 Mbytes >>>>>>> ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- >>>>>>> TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 >>>>>>> PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 >>>>>>> E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 >>>>>>> E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562 >>>>>>> >>>>>>> … then sat here for little while doing nothing. >>>>>>> >>>>>>> >>>>>>> chris >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>>> >>>>>>> Chris, >>>>>>> >>>>>>> The console was printing the following panic per core it appears: >>>>>>> >>>>>>> [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 >>>>>>> [349243.709061] NMI backtrace for cpu 104 >>>>>>> [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>>>> [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>>>> [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 >>>>>>> [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 >>>>>>> [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 >>>>>>> [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 >>>>>>> [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 >>>>>>> [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f >>>>>>> [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 >>>>>>> [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 >>>>>>> [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 >>>>>>> [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>> [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 >>>>>>> [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>>> [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>>> [349243.906788] Stack: >>>>>>> [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 >>>>>>> [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 >>>>>>> [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b >>>>>>> [349243.951521] Call Trace: >>>>>>> [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 >>>>>>> [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 >>>>>>> [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 >>>>>>> [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 >>>>>>> >>>>>>> This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>>> On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>>> >>>>>>>> Hi Ben, >>>>>>>> >>>>>>>> knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. >>>>>>>> >>>>>>>> Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? >>>>>>>> >>>>>>>> chris >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >> >

1 0

Re: [JLSE-Intel-NDA] knl00 down
by Allen, Benjamin S. 21 Mar '16

21 Mar '16

Sounds good. How's starting around 1:30pm tomorrow? Ben > On Mar 21, 2016, at 5:23 PM, Christopher Knight <knightc(a)anl.gov> wrote: > > Yes, those were the settings used to compile the current executable. > > I can build a few different executables with same compiler, but different libraries linked and try running tests again to cause kernel panic. > > 1) Only MPI > 2) MPI + OpenMP > 3) MPI + MKL > 4) MPI + OpenMP + MKL (current executable) > > If #1 causes a kernel panic, then I could repeat tests with different compiler. I’m open to suggestions. > > > I could coordinate with you and start running these tomorrow during a specific block of time if that helps any? > > chris > > > > > On Mar 21, 2016, at 3:36 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct. > > Did you freshly compile lammps with the below compiler and MPI vars? > > Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel. > > Power cycling the node now, and bring back up with the 3.10 kernel. > Ben > >> On Mar 18, 2016, at 6:20 PM, Christopher Knight <knightc(a)anl.gov> wrote: >> >> <snaps fingers> it happened again. >> >> >> I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled. >> >> env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp >> >> >> One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right? >> >> source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 >> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >> >> chris >> >> >> >>> On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based. >>> >>> I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again. >>> >>> Chris, >>> >>> I've now ran the following a few times successfully: >>> >>> source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 >>> source /soft/compilers/intel/impi_latest/bin64/mpivars.sh >>> >>> env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000 >>> >>> Please free to continue with your testing. >>> >>> Ben >>> >>>> On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler: >>>> >>>> BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa >>>> [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>> [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 >>>> [ 1275.748524] Oops: 0002 [#1] SMP >>>> [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common >>>> [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>> [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>> [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 >>>> [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>> [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 >>>> [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 >>>> [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 >>>> [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 >>>> [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 >>>> [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 >>>> [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 >>>> [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>> [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 >>>> [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>> [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>> [ 1275.998089] Stack: >>>> [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 >>>> [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 >>>> [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 >>>> [ 1276.054382] Call Trace: >>>> [ 1276.063947] <IRQ> >>>> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 >>>> [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 >>>> [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 >>>> [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 >>>> [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 >>>> [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 >>>> [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 >>>> [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 >>>> [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 >>>> [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 >>>> [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 >>>> [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 >>>> [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 >>>> [ 1276.225074] <EOI> >>>> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b >>>> [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 >>>> [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>>> [ 1276.286963] RSP <ffff882f7da63dd8> >>>> [ 1276.295181] CR2: 00000000000001fa >>>> >>>> The kernel then goes on to send NMI to the rest of the CPUs: >>>> >>>> [ 1344.258750] sending NMI to all CPUs: >>>> >>>> The NMIs were what I was seeing last go around. This is going to be a painful to fix. >>>> >>>> Ben >>>> >>>> >>>>> On Mar 18, 2016, at 3:48 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>> >>>>> I think I broke it again… >>>>> >>>>> This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file. >>>>> >>>>> chris >>>>> >>>>> >>>>> >>>>> >>>>> On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Odd. The machine is back up, feel free to continue your testing. >>>>> >>>>> Ben >>>>> >>>>>> On Mar 18, 2016, at 3:26 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>> >>>>>> Hi Ben, >>>>>> >>>>>> If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs. >>>>>> >>>>>> >>>>>> The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following. >>>>>> >>>>>> [knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>> … >>>>>> Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms >>>>>> >>>>>> >>>>>> >>>>>> When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled. >>>>>> >>>>>> [knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>>> ... >>>>>> Setting up run ... >>>>>> Memory usage per processor = 18.0914 Mbytes >>>>>> ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- >>>>>> TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 >>>>>> PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 >>>>>> E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 >>>>>> E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562 >>>>>> >>>>>> … then sat here for little while doing nothing. >>>>>> >>>>>> >>>>>> chris >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>> >>>>>> Chris, >>>>>> >>>>>> The console was printing the following panic per core it appears: >>>>>> >>>>>> [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 >>>>>> [349243.709061] NMI backtrace for cpu 104 >>>>>> [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>>> [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>>> [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 >>>>>> [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 >>>>>> [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 >>>>>> [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 >>>>>> [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 >>>>>> [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f >>>>>> [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 >>>>>> [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 >>>>>> [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 >>>>>> [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>> [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 >>>>>> [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>>> [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>>> [349243.906788] Stack: >>>>>> [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 >>>>>> [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 >>>>>> [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b >>>>>> [349243.951521] Call Trace: >>>>>> [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 >>>>>> [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 >>>>>> [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 >>>>>> [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 >>>>>> >>>>>> This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? >>>>>> >>>>>> Ben >>>>>> >>>>>>> On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>>> >>>>>>> Hi Ben, >>>>>>> >>>>>>> knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. >>>>>>> >>>>>>> Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? >>>>>>> >>>>>>> chris >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> > >

1 0

Re: [JLSE-Intel-NDA] knl00 down
by Christopher Knight 21 Mar '16

21 Mar '16

Yes, those were the settings used to compile the current executable. I can build a few different executables with same compiler, but different libraries linked and try running tests again to cause kernel panic. 1) Only MPI 2) MPI + OpenMP 3) MPI + MKL 4) MPI + OpenMP + MKL (current executable) If #1 causes a kernel panic, then I could repeat tests with different compiler. I’m open to suggestions. I could coordinate with you and start running these tomorrow during a specific block of time if that helps any? chris On Mar 21, 2016, at 3:36 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct. Did you freshly compile lammps with the below compiler and MPI vars? Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel. Power cycling the node now, and bring back up with the 3.10 kernel. Ben > On Mar 18, 2016, at 6:20 PM, Christopher Knight <knightc(a)anl.gov> wrote: > > <snaps fingers> it happened again. > > > I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled. > > env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp > > > One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right? > > source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > > chris > > > >> On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based. >> >> I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again. >> >> Chris, >> >> I've now ran the following a few times successfully: >> >> source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 >> source /soft/compilers/intel/impi_latest/bin64/mpivars.sh >> >> env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000 >> >> Please free to continue with your testing. >> >> Ben >> >>> On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler: >>> >>> BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa >>> [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>> [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 >>> [ 1275.748524] Oops: 0002 [#1] SMP >>> [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common >>> [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>> [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>> [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 >>> [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>> [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 >>> [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 >>> [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 >>> [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 >>> [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 >>> [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 >>> [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 >>> [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 >>> [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>> [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>> [ 1275.998089] Stack: >>> [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 >>> [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 >>> [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 >>> [ 1276.054382] Call Trace: >>> [ 1276.063947] <IRQ> >>> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 >>> [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 >>> [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 >>> [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 >>> [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 >>> [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 >>> [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 >>> [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 >>> [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 >>> [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 >>> [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 >>> [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 >>> [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 >>> [ 1276.225074] <EOI> >>> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b >>> [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 >>> [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>> [ 1276.286963] RSP <ffff882f7da63dd8> >>> [ 1276.295181] CR2: 00000000000001fa >>> >>> The kernel then goes on to send NMI to the rest of the CPUs: >>> >>> [ 1344.258750] sending NMI to all CPUs: >>> >>> The NMIs were what I was seeing last go around. This is going to be a painful to fix. >>> >>> Ben >>> >>> >>>> On Mar 18, 2016, at 3:48 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>> >>>> I think I broke it again… >>>> >>>> This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file. >>>> >>>> chris >>>> >>>> >>>> >>>> >>>> On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> Odd. The machine is back up, feel free to continue your testing. >>>> >>>> Ben >>>> >>>>> On Mar 18, 2016, at 3:26 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>> >>>>> Hi Ben, >>>>> >>>>> If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs. >>>>> >>>>> >>>>> The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following. >>>>> >>>>> [knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>> … >>>>> Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms >>>>> >>>>> >>>>> >>>>> When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled. >>>>> >>>>> [knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>> ... >>>>> Setting up run ... >>>>> Memory usage per processor = 18.0914 Mbytes >>>>> ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- >>>>> TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 >>>>> PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 >>>>> E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 >>>>> E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562 >>>>> >>>>> … then sat here for little while doing nothing. >>>>> >>>>> >>>>> chris >>>>> >>>>> >>>>> >>>>> >>>>> On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Chris, >>>>> >>>>> The console was printing the following panic per core it appears: >>>>> >>>>> [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 >>>>> [349243.709061] NMI backtrace for cpu 104 >>>>> [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>> [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>> [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 >>>>> [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 >>>>> [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 >>>>> [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 >>>>> [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 >>>>> [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f >>>>> [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 >>>>> [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 >>>>> [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 >>>>> [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 >>>>> [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>> [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>> [349243.906788] Stack: >>>>> [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 >>>>> [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 >>>>> [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b >>>>> [349243.951521] Call Trace: >>>>> [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 >>>>> [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 >>>>> [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 >>>>> [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 >>>>> >>>>> This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? >>>>> >>>>> Ben >>>>> >>>>>> On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>> >>>>>> Hi Ben, >>>>>> >>>>>> knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. >>>>>> >>>>>> Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? >>>>>> >>>>>> chris >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >> >

1 0

Re: [JLSE-Intel-NDA] knl00 down
by Allen, Benjamin S. 21 Mar '16

21 Mar '16

I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct. Did you freshly compile lammps with the below compiler and MPI vars? Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel. Power cycling the node now, and bring back up with the 3.10 kernel. Ben > On Mar 18, 2016, at 6:20 PM, Christopher Knight <knightc(a)anl.gov> wrote: > > <snaps fingers> it happened again. > > > I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled. > > env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp > > > One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right? > > source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > > chris > > > >> On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based. >> >> I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again. >> >> Chris, >> >> I've now ran the following a few times successfully: >> >> source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 >> source /soft/compilers/intel/impi_latest/bin64/mpivars.sh >> >> env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000 >> >> Please free to continue with your testing. >> >> Ben >> >>> On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler: >>> >>> BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa >>> [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>> [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 >>> [ 1275.748524] Oops: 0002 [#1] SMP >>> [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common >>> [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>> [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>> [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 >>> [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>> [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 >>> [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 >>> [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 >>> [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 >>> [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 >>> [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 >>> [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 >>> [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 >>> [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>> [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>> [ 1275.998089] Stack: >>> [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 >>> [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 >>> [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 >>> [ 1276.054382] Call Trace: >>> [ 1276.063947] <IRQ> >>> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 >>> [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 >>> [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 >>> [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 >>> [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 >>> [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 >>> [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 >>> [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 >>> [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 >>> [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 >>> [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 >>> [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 >>> [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 >>> [ 1276.225074] <EOI> >>> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b >>> [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 >>> [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 >>> [ 1276.286963] RSP <ffff882f7da63dd8> >>> [ 1276.295181] CR2: 00000000000001fa >>> >>> The kernel then goes on to send NMI to the rest of the CPUs: >>> >>> [ 1344.258750] sending NMI to all CPUs: >>> >>> The NMIs were what I was seeing last go around. This is going to be a painful to fix. >>> >>> Ben >>> >>> >>>> On Mar 18, 2016, at 3:48 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>> >>>> I think I broke it again… >>>> >>>> This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file. >>>> >>>> chris >>>> >>>> >>>> >>>> >>>> On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> Odd. The machine is back up, feel free to continue your testing. >>>> >>>> Ben >>>> >>>>> On Mar 18, 2016, at 3:26 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>> >>>>> Hi Ben, >>>>> >>>>> If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs. >>>>> >>>>> >>>>> The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following. >>>>> >>>>> [knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>> … >>>>> Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms >>>>> >>>>> >>>>> >>>>> When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled. >>>>> >>>>> [knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 >>>>> ... >>>>> Setting up run ... >>>>> Memory usage per processor = 18.0914 Mbytes >>>>> ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- >>>>> TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 >>>>> PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 >>>>> E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 >>>>> E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562 >>>>> >>>>> … then sat here for little while doing nothing. >>>>> >>>>> >>>>> chris >>>>> >>>>> >>>>> >>>>> >>>>> On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Chris, >>>>> >>>>> The console was printing the following panic per core it appears: >>>>> >>>>> [349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 >>>>> [349243.709061] NMI backtrace for cpu 104 >>>>> [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 >>>>> [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 >>>>> [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 >>>>> [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 >>>>> [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 >>>>> [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 >>>>> [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 >>>>> [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f >>>>> [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 >>>>> [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 >>>>> [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 >>>>> [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 >>>>> [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>>> [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>>> [349243.906788] Stack: >>>>> [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 >>>>> [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 >>>>> [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b >>>>> [349243.951521] Call Trace: >>>>> [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 >>>>> [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 >>>>> [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 >>>>> [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 >>>>> >>>>> This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM? >>>>> >>>>> Ben >>>>> >>>>>> On Mar 18, 2016, at 2:49 PM, Christopher Knight <knightc(a)anl.gov> wrote: >>>>>> >>>>>> Hi Ben, >>>>>> >>>>>> knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error. >>>>>> >>>>>> Should I send emails like this to intel-ada(a)jlse.anl.gov to report this type of issue? >>>>>> >>>>>> chris >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >> >

1 0