knl05 went down (panic), and Cobalt kept scheduling jobs on it. Same issue as Steve reported. Don't quite have a fix for this yet. You have access to nodes were scheduled as part of job 548, the next job that comes along will correctly set permissions for the new job. I'll add to my todos to cleanup access after a failed job launch. Thanks, Ben
On Aug 17, 2016, at 10:05 AM, Luo, Ye <[email protected]> wrote:
Hi JLSE support,
I'm not sure if this a known issue or not. I noticed some strange behaviour last night on the cobalt for KNL. My jobs submitted to knl_7210 exited without starting. I also noticed that I can ssh directly to several nodes without being allocated by cobalt for running jobs.
The cobalt log and my job script is attached.
Ye
log: Jobid: 548 qsub --mode script -q knl_7210 -n 4 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:48:30 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 548 submitted from terminal /dev/pts/9 Wed Aug 17 02:49:02 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 549 qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:49:45 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 549 submitted from terminal /dev/pts/9 Wed Aug 17 02:50:03 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 550 qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:53:31 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 550 submitted from terminal /dev/pts/9 Wed Aug 17 02:53:44 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 551 qsub -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant -I Wed Aug 17 02:54:08 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 551 submitted from terminal /dev/pts/9 Wed Aug 17 02:54:35 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts
script: #COBALT -q knl_7210 #COBALT -n 2 #COBALT -t 120 #COBALT -O vmc #COBALT --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant
=================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda