All,
We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue.
We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example:
qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh
The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage.
Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values:
knl_memory_mode:
Cache
Flat
Hybrid
Auto
knl_cluster_mode:
All2All
SNC-2
SNC-4
Hemisphere
Quadrant
Auto
Example submission script:
#!/bin/bash
source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh
mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello
Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes.
Ben
