Odd. The machine is back up, feel free to continue your testing. Ben

On Mar 18, 2016, at 3:26 PM, Christopher Knight <[email protected]> wrote:

Hi Ben,

If it did, it’s definitely not expected behavior for this small test. Memory footprint should have been relatively small, definitely less than several GBs.

The run just before that was fine and LAMMPS reported an (under) estimate of 18 MBs per MPI rank. The initial series of runs was from 1 to 64 MPI ranks like in the following.

[knightc@knl00 rhodo]$ mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 … Loop time of 0.480942 on 64 procs (64 MPI x 1 OpenMP) for 10 steps with 32000 atoms

When I subsequently tried increasing number of OpenMP threads to 2 for that last run, the code made it past step 0 and then stalled.

[knightc@knl00 rhodo]$ env OMP_NUM_THREADS=2 mpirun -np 64 ../src/lmp_knl -in lmp.in -var NSTEPS 10 ... Setting up run ... Memory usage per processor = 18.0914 Mbytes ---------------- Step 0 ----- CPU = 0.0000 (sec) ---------------- TotEng = -25691.3377 KinEng = 21444.8313 Temp = 299.0397 PotEng = -47136.1690 E_bond = 2537.9940 E_angle = 10921.3742 E_dihed = 5211.7865 E_impro = 213.5116 E_vdwl = -2637.3323 E_coul = 162089.4165 E_long = -225472.9195 Press = -223.9562

… then sat here for little while doing nothing.

chris

On Mar 18, 2016, at 3:16 PM, Allen, Benjamin S. <[email protected]> wrote:

Chris,

The console was printing the following panic per core it appears:

[349243.677483] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 [349243.709061] NMI backtrace for cpu 104 [349243.717125] CPU: 104 PID: 0 Comm: swapper/104 Tainted: G W OE ------------ 3.10.0-327.10.1.el7.x86_64 #1 [349243.732898] Hardware name: Intel Corporation S7200AP/S7200AP, BIOS S72C610.86B.01.01.0041.112720151659 11/27/2015 [349243.748571] task: ffff882f337bc500 ti: ffff882f337e8000 task.ti: ffff882f337e8000 [349243.761187] RIP: 0010:[<ffffffff81058e96>] [<ffffffff81058e96>] native_safe_halt+0x6/0x10 [349243.774763] RSP: 0018:ffff882f337ebe98 EFLAGS: 00000286 [349243.784946] RAX: 00000000ffffffed RBX: ffff882f337e8000 RCX: 0100000000000000 [349243.797189] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046 [349243.809426] RBP: ffff882f337ebe98 R08: 0000000000000000 R09: 0000000000000f9f [349243.821694] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000068 [349243.833914] R13: ffff882f337e8000 R14: ffff882f337e8000 R15: 0000000000000000 [349243.846094] FS: 0000000000000000(0000) GS:ffff882f7dd00000(0000) knlGS:0000000000000000 [349243.859415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [349243.870100] CR2: 00007f89e6a83185 CR3: 000000000194a000 CR4: 00000000001407e0 [349243.882346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [349243.894577] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [349243.906788] Stack: [349243.913242] ffff882f337ebeb8 ffffffff8101dbcf ffff882f337e8000 ffffffff81a68260 [349243.926105] ffff882f337ebec8 ffffffff8101e4d6 ffff882f337ebf20 ffffffff810d62c5 [349243.938850] ffff882f337ebfd8 ffff882f337e8000 6890e75d0dc16093 307f0504002da88b [349243.951521] Call Trace: [349243.958359] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [349243.968375] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [349243.978393] [<ffffffff810d62c5>] cpu_startup_entry+0x245/0x290 [349243.988910] [<ffffffff810475fa>] start_secondary+0x1ba/0x230

This has happened before when we only had 16GB installed. Any chance your app ran through the system's 192GB of RAM?

Ben

...

On Mar 18, 2016, at 2:49 PM, Christopher Knight <[email protected]> wrote:

Hi Ben,

knl00 is down. I was running some quick LAMMPS tests to get a feel for things and looks like something bad happened in middle of run. Trying to ssh from another shell gives me the “No route to host” error.

Should I send emails like this to [email protected] to report this type of issue?

chris