1:30pm tomorrow works for me. chris

On Mar 21, 2016, at 5:55 PM, Allen, Benjamin S. <[email protected]> wrote:

Sounds good. How's starting around 1:30pm tomorrow?

Ben

...

On Mar 21, 2016, at 5:23 PM, Christopher Knight <[email protected]> wrote:

Yes, those were the settings used to compile the current executable.

I can build a few different executables with same compiler, but different libraries linked and try running tests again to cause kernel panic.

1) Only MPI 2) MPI + OpenMP 3) MPI + MKL 4) MPI + OpenMP + MKL (current executable)

If #1 causes a kernel panic, then I could repeat tests with different compiler. I’m open to suggestions.

I could coordinate with you and start running these tomorrow during a specific block of time if that helps any?

chris

On Mar 21, 2016, at 3:36 PM, Allen, Benjamin S. <[email protected]> wrote:

I was actually sourcing older Intel MKL and MPI installs, which should be incorrect. What you're doing is correct.

Did you freshly compile lammps with the below compiler and MPI vars?

Obviously its not great that a user app can cause the host to kernel panic. If we can narrow down what libraries/compilers/etc is causing it we should report back to Intel.

Power cycling the node now, and bring back up with the 3.10 kernel. Ben

...

On Mar 18, 2016, at 6:20 PM, Christopher Knight <[email protected]> wrote:

<snaps fingers> it happened again.

I tried the following combination, segfaulted with exit code 4 after zeroeth step, tried to repeat and it stalled.

env OMP_NUM_THREADS=4 mpirun -np 32 ../src/lmp_knl -in lmp.in -var NSTEPS 10 -sf omp

One difference I see is that I’ve been sourcing the following copied from the sniper setup I use. This shouldn’t be related, right?

source /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh

chris

...

On Mar 18, 2016, at 5:24 PM, Allen, Benjamin S. <[email protected]> wrote:

Ok, not the best solution, but I just rebooted the node into the 4.5.0 mainline kernel. For anyone that's run their code or benchmarks already, I'd suggest re-running your tests again to see if this breaks/slows/improves anything. 4.5.0 is also quite a bit newer kernel than we'll be running on Theta. Assuming Theta compute runs a SLES 12 derived kernel, it'll be 3.12 based.

I've also enabled kdump to write out a vmcore to /soft/var/crash/`hostname` on kernel panic so we'll actually have something to work with if this happens again.

Chris,

I've now ran the following a few times successfully:

source /soft/compilers/intel/mkl/bin/mklvars.sh intel64 source /soft/compilers/intel/impi_latest/bin64/mpivars.sh

env OMP_NUM_THREADS=4 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 1000

Please free to continue with your testing.

Ben

...

On Mar 18, 2016, at 3:51 PM, Allen, Benjamin S. <[email protected]> wrote:

Yea you hit the same issue again. I kept a console open this go around. It appears we're hitting a null pointer deref in the kernel's scheduler:

BUG: unable to handle kernel NULL pointer dereference at 00000000000001fa [ 1275.736095] IP: [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 [ 1275.742977] PGD 2f26429067 PUD 2f23ca6067 PMD 0 [ 1275.748524] Oops: 0002 [#1] SMP [ 1275.752409] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan ip6_udp_tunnel udp_tunnel mlx4_ib(OE) ib_sa(OE) mlx4_core(OE) iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel syscopyarea lrw sysfillrect gf128mul sysimgblt glue_helper ablk_helper ttm cryptd pcspkr drm_kms_helper ib_mad(OE) ib_core(OE) drm mei_me sg lpc_ich i2c_i801 ib_addr(OE) mfd_core mei mlx_compat(OE) shpchp wmi acpi_power_meter acpi_pad binfmt_misc nfsd knem(OE) auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 jbd2 mbcache ipmi_si ipmi_devintf ipmi_msghandler igb i2c_algo_bit dca ptp pps_core i2c_core ahci libahci libata sd_mod crc_t10dif crct10dif_common [ 1275.848733] CPU: 83 PID: 10669 Comm: lmp_knl Tainted: G OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 [ 1275.860566] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 [ 1275.872302] task: ffff882cb03f7300 ti: ffff882f13c54000 task.ti: ffff882f13c54000 [ 1275.880867] RIP: 0010:[<ffffffff810c2b74>] [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 [ 1275.890516] RSP: 0000:ffff882f7da63dd8 EFLAGS: 00010002 [ 1275.896611] RAX: 000038b712dcb4a2 RBX: 000000000000017a RCX: 0000000000000001 [ 1275.904786] RDX: 000000000000bc00 RSI: 00000000000039f5 RDI: 0000000000000000 [ 1275.912967] RBP: ffff882f7da63e28 R08: 0000000000000000 R09: 0000000000000000 [ 1275.921143] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 [ 1275.929325] R13: ffff882f7da74800 R14: ffff882f7da74780 R15: 0000000000000000 [ 1275.937506] FS: 00007f83194aab80(0000) GS:ffff882f7da60000(0000) knlGS:0000000000000000 [ 1275.946776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1275.953379] CR2: 00000000000001fa CR3: 0000002f26fe4000 CR4: 00000000001407e0 [ 1275.968365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1275.983284] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1275.998089] Stack: [ 1276.007173] ffff882f7da63de8 ffffffff8101cd69 ffff882f7da63e08 ffffffff810bb655 [ 1276.022943] ffff882cb03f7300 ffff882f7da74780 0000000000000053 0000000000000053 [ 1276.038667] 0000000000014780 ffff882cb03f7300 ffff882f7da63e60 ffffffff810b75e8 [ 1276.054382] Call Trace: [ 1276.063947] <IRQ> [ 1276.066275] [<ffffffff8101cd69>] ? sched_clock+0x9/0x10 [ 1276.085997] [<ffffffff810bb655>] ? sched_clock_cpu+0x85/0xc0 [ 1276.099140] [<ffffffff810b75e8>] scheduler_tick+0xc8/0x150 [ 1276.111712] [<ffffffff810e06e0>] ? tick_sched_handle.isra.14+0x60/0x60 [ 1276.123778] [<ffffffff8108e8c6>] update_process_times+0x66/0x80 [ 1276.135116] [<ffffffff810e06a5>] tick_sched_handle.isra.14+0x25/0x60 [ 1276.146903] [<ffffffff810e0721>] tick_sched_timer+0x41/0x70 [ 1276.157799] [<ffffffff810a9d12>] __hrtimer_run_queues+0xd2/0x260 [ 1276.169156] [<ffffffff810aa2b0>] hrtimer_interrupt+0xb0/0x1e0 [ 1276.180177] [<ffffffff8164735c>] ? call_softirq+0x1c/0x30 [ 1276.190766] [<ffffffff81049537>] local_apic_timer_interrupt+0x37/0x60 [ 1276.202484] [<ffffffff81647fcf>] smp_apic_timer_interrupt+0x3f/0x60 [ 1276.213984] [<ffffffff8164669d>] apic_timer_interrupt+0x6d/0x80 [ 1276.225074] <EOI> [ 1276.227325] [<ffffffff81645a49>] ? system_call_fastpath+0x16/0x1b [ 1276.243048] Code: 9e 1c 0a 00 00 41 8b 86 18 0a 00 00 e9 08 fe ff ff 49 8b 84 24 e8 00 00 00 e9 4a fd ff ff 48 c7 46 68 00 00 00 00 e9 69 fc ff ff <c7> 83 80 00 00 00 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 31 [ 1276.275761] RIP [<ffffffff810c2b74>] task_tick_fair+0x5a4/0x670 [ 1276.286963] RSP <ffff882f7da63dd8> [ 1276.295181] CR2: 00000000000001fa

The kernel then goes on to send NMI to the rest of the CPUs:

[ 1344.258750] sending NMI to all CPUs:

The NMIs were what I was seeing last go around. This is going to be a painful to fix.

Ben

...

On Mar 18, 2016, at 3:48 PM, Christopher Knight <[email protected]> wrote:

I think I broke it again…

This time I made my way up to 32 MPI ranks and 4 threads per rank. This run segfaulted at very end after main compute phase finished with an exit code of 4. I tried turning on core files with ‘ulimit -c unlimited’ and rerunning to reproduce and see what happened. It stalled right away while reading the input file.

chris

On Mar 18, 2016, at 3:29 PM, Allen, Benjamin S. <[email protected]> wrote:

Odd. The machine is back up, feel free to continue your testing.

Ben

...

On Mar 18, 2016, at 3:26 PM, Christopher Knight <[email protected]> wrote:

Hi Ben,

If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs.

The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following.

[knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 … Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms

When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled.

[knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 ... Setting up run ... Memory usage per processor = 18.0914 Mbytes ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562

… then sat here for little while doing nothing.

chris

On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <[email protected]> wrote:

Chris,

The console was printing the following panic per core it appears:

[349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 [349243.709061] NMI backtrace for cpu 104 [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [349243.906788] Stack: [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b [349243.951521] Call Trace: [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230

This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM?

Ben

...

On Mar 18, 2016, at 2:49 PM, Christopher Knight <[email protected]> wrote:

Hi Ben,

knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error.

Should I send emails like this to [email protected] to report this type of issue?

chris