Thanks for letting me know. I'll try to remember this link line in case someone else asks about it in the future. Best, Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 4:59 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Thanks Jeff. It worked for me when I dynamically linked the blacs library. So my compile line looked as: mpiicc -mkl ex.c /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,-Bstatic -Wl,--start-group /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_intel_lp64.a /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_core.a /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -Wl,-Bdynamic -o mult -lmkl_blacs_intelmpi_lp64 ​Best regards, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 7:01 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? You might post on the Intel Forum about this. It's going to be at least two weeks before I have time to do any debugging on JLSE. Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:59 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks for the links. So, its the MPI wrapper that needs to be resolved. I tried to follow the option that talks about adding -Wl,-Bstatic/dynamic and currently have this: MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,-Bstatic -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group -Wl,-Bdynamic Still I am not able to get the blacs_library* symbols below to appear with my executable: 21 00000000036a5f60 b blacs_library_loader_lock 22 00000000036a5f64 b blacs_library_lock_flag​ Beyond that it still is confusing. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 6:25 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? I have not seen that one before but I have a good idea where it comes from. Have you looked at https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/5903... or https://software.intel.com/en-us/articles/using-intel-mkl-mpi-wrapper-with-t... yet? Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:19 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks a lot for your help. I am trying to figure it out, but currently I am getting an error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. ​when I run my program. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 5:35 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania