- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Weird 'undefined reference to' error for an AVX512 intrinsic
by Luo, Ye 21 Jun '16

21 Jun '16

Hi Jiang, According to https://software.intel.com/sites/landingpage/IntrinsicsGuide/# _mm512_mask_extloadunpacklo_epi32 belongs to the ISA of KNC but it is not part of AVX512. So it seems not supported on KNL. Intel only says KNL is binary compatible with Xeon but never said KNC. I'm sorry that I can't help you more. An expert on intrinsic might know how to make a workaround. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Jiang, Wei [wjiang(a)alcf.anl.gov] Sent: Tuesday, June 21, 2016 5:20 PM To: intel-nda(a)lists.jlse.anl.gov Subject: [JLSE-Intel-NDA] Weird 'undefined reference to' error for an AVX512 intrinsic Hi, I am compiling a test code with icpc but got such a link error 'undefined reference to _mm512_mask_extloadunpacklo_epi32'. Note there are many AVX512 intrinsic in the code but just this one receives 'undefined reference to' error, while another mini test code that only contains '_mm512_mask_extloadunpacklo_epi32' doesn't get such error. Can anybody help see what wrong it is? Thanks in advance. Wei Jiang, PhD, Leadership Computing Facility Argonne National Laboratory 9700S, Cass Ave, Lemont, IL60439 630-252-8688 (office) 630-252-7121 (fax) _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Weird 'undefined reference to' error for an AVX512 intrinsic
by Jiang, Wei 21 Jun '16

21 Jun '16

Hi, I am compiling a test code with icpc but got such a link error 'undefined reference to _mm512_mask_extloadunpacklo_epi32'. Note there are many AVX512 intrinsic in the code but just this one receives 'undefined reference to' error, while another mini test code that only contains '_mm512_mask_extloadunpacklo_epi32' doesn't get such error. Can anybody help see what wrong it is? Thanks in advance. Wei Jiang, PhD, Leadership Computing Facility Argonne National Laboratory 9700S, Cass Ave, Lemont, IL60439 630-252-8688 (office) 630-252-7121 (fax)

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by Steven Pieper 21 Jun '16

21 Jun '16

Scott: That seems to be it! The code is now working (8 threads, -O2, no AVX yet). I had been using -s 400000 which works on blues but I guess more is needed here. I will keep pushing the optimizations and threads up. Thanks, Steve Steven C. Pieper: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/21/2016 03:08 PM, Scott Parker wrote: > > Steve- > > Have you tried setting 'ulimit -s unlimited' ? By default the system has a max stack > size set to 8MB and I've found I need to increase it in some cases or else the program > crashes at startup. > > Scott > > > On 6/20/16 11:46 AM, Steven C. Pieper wrote: >> OK, some progress... >> >> 1) I didn't realize that mcmodel=large (medium?) needs to be specified >> on compile commands in addition to the link command. With that and using >> no OpenMP it starts, even with the AVX commands. It eventually segfaults. >> >> 2) Using OpenMP it still instantly segfaults, even with just 1 thread allowed. >> >> Steve >> Steven C. Pieper, Senior Physicist SO:[email protected] >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 >> Secretary, Debra Beres, -4100 >> On 06/20/16 10:59, James Osborn wrote: >>> I guess you can try older versions of the compiler to see if they work. >>> You may be able to run the same binary compiled on blues at jlse. >>> >>> Cheers, >>> James >>> >>> >>> On 06/20/2016 10:56 AM, Steven C. Pieper wrote: >>>> With the source line given below I get the same relocation errors >>>> on thing01. >>>> >>>> Steve >>>> >>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>> Phone: 630-252-4232 Fax -6008 >>>> Secretary, Debra Beres, -4100 >>>> >>>> On 06/20/16 10:53, Steven C. Pieper wrote: >>>>> should I continue to use >>>>> >>>>> source >>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>> intel64 >>>>> >>>>> on thing01? >>>>> >>>>> Steve >>>>> >>>>> >>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 >>>>> Secretary, Debra Beres, -4100 >>>>> On 06/20/16 10:47, James Osborn wrote: >>>>>> Hi Steve, >>>>>> >>>>>> My only suggestion is to try using the same compilers and code on the >>>>>> jlse thing* nodes (assuming you have access) to see if it works >>>>>> there. The same executable you compile for those should also work on >>>>>> KNL. >>>>>> >>>>>> Cheers, >>>>>> James >>>>>> >>>>>> On 06/20/2016 10:35 AM, Steven C. Pieper wrote: >>>>>>> I tried running with just one OMP thread and it failed in the same way. >>>>>>> Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can >>>>>>> again make the simple case but the big case now fails in the link step: >>>>>>> >>>>>>> ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ >>>>>>> -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>> -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>> -g testwf.f >>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -o >>>>>>> x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o >>>>>>> adlb_fake.o \ >>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml >>>>>>> testwf.o: In function `_unnamed_main$$': >>>>>>> /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON >>>>>>> section in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>>>> section in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>>>> section in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_32 against symbol `parallel_' defined in COMMON >>>>>>> section in >>>>>>> testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON >>>>>>> section in >>>>>>> testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation >>>>>>> overflows omitted from the output >>>>>>> make: *** [testwf] Error 1 >>>>>>> [ >>>>>>> >>>>>>> The common blocks that it is complaining about are all very short >>>>>>> simple >>>>>>> ones. >>>>>>> >>>>>>> Help! I will be happy to come over to Bldg 240. >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>> IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> >>>>>>> On 06/18/16 14:17, Steven Pieper wrote: >>>>>>>> I will try various things when I can next get on. I don't believe AVX >>>>>>>> can be it: >>>>>>>> >>>>>>>> 1) The smaller case was compiled with the same options and works. >>>>>>>> 2) It hasn't gotten started at all, so I don't see how any AVX >>>>>>>> instructions could >>>>>>>> have been encountered. >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>>>>> 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>> >>>>>>>> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>>>>>>>> Steve, >>>>>>>>> >>>>>>>>> I have no further good ideas. You could try simplifying your >>>>>>>>> compile options to see if that will help produce a runnable binary. >>>>>>>>> The KNL should support AVX but maybe the first thing to try is >>>>>>>>> removing all the AVX flags. If that works, you can try adding >>>>>>>>> -xMIC-AVX512. >>>>>>>>> >>>>>>>>> kevin >>>>>>>>> >>>>>>>>>> Well further tests have to wait until Monday now, but >>>>>>>>>> >>>>>>>>>> 1) I used both mcmodel=medium and large -- both fail the same way >>>>>>>>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>>>>>>>> mcmodel=medium >>>>>>>>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>>>>>>>> 4) The size command does not show a huge executable (the big stuff >>>>>>>>>> will come >>>>>>>>>> later from allocates): >>>>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>>>>>>>> >>>>>>>>>> Steve >>>>>>>>>> >>>>>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>>>> IL 60439 >>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>> >>>>>>>>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>>>>>>>> >>>>>>>>>>> I would say it has something to do with the memory model and the >>>>>>>>>>> limitations that the large segments of the code cannot be correctly >>>>>>>>>>> located by the OS. The problems shows up at the first executable >>>>>>>>>>> statement. >>>>>>>>>>> >>>>>>>>>>> Two suggestions: try to set the number of OMP threads to some small >>>>>>>>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>>>>>>>> >>>>>>>>>>> Vitali >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>>>>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>>>>>>>> fortran 90. >>>>>>>>>>>> >>>>>>>>>>>> However the error doesn't seem to have anything to do with >>>>>>>>>>>> random2.f per se. >>>>>>>>>>>> I commented that line out and now the traceback points to the next >>>>>>>>>>>> non-comment line. So I think the segfault is happening at load >>>>>>>>>>>> time >>>>>>>>>>>> and the traceback just points to the first non comment line. >>>>>>>>>>>> >>>>>>>>>>>> I have to go now; I guess this will have to wait until next week. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Steve >>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>>>>>> IL 60439 >>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>>>>>>>> Steve, >>>>>>>>>>>>> >>>>>>>>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>>>>>>>> fortran or your code? >>>>>>>>>>>>> >>>>>>>>>>>>> kevin >>>>>>>>>>>>> >>>>>>>>>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>>>>>>>>> Here are the results from gdb: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>>>> This is free software: you are free to change and >>>>>>>>>>>>>> redistribute it. >>>>>>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type"show >>>>>>>>>>>>>> copying" >>>>>>>>>>>>>> and"show warranty" for details. >>>>>>>>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>>>>>>>> For bug reporting instructions, please see: >>>>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>>>>>>>> Reading symbols from >>>>>>>>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>>>>>>>> (gdb) run >>>>>>>>>>>>>> Starting program: >>>>>>>>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>>>> 13 use random2 >>>>>>>>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>> >>>>>>>>>>>>>> (gdb) where >>>>>>>>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>>>>>>>> /lib64/libc.so.6 >>>>>>>>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Try mpiifort. >>>>>>>>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>>>>>>>> Ye >>>>>>>>>>>>>> =================== >>>>>>>>>>>>>> Ye Luo, Ph.D. >>>>>>>>>>>>>> Leadership Computing Facility >>>>>>>>>>>>>> Argonne National Laboratory >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>> From: >>>>>>>>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>>>>>>>> To: Harms, Kevin; >>>>>>>>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>>>>>>>> >>>>>>>>>>>>>> Osborn, James C. >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Kevin, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>>>>>>>> However mpif90 >>>>>>>>>>>>>> is using gfortran: >>>>>>>>>>>>>> >>>>>>>>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>>>>>>> '-qopenmp-report' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>>>>>>>> [knl00 mc] which mpif90 >>>>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve >>>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has some >>>>>>>>>>>>>> useful notes on it: >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>>>>>>>> >>>>>>>>>>>>>> kevin >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve, >>>>>>>>>>>>>> >>>>>>>>>>>>>> First thing, try the latest compiler. >>>>>>>>>>>>>> >>>>>>>>>>>>>> source >>>>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>>>>>>>>> running it in gdb? >>>>>>>>>>>>>> >>>>>>>>>>>>>> kevin >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>>>>>>>>> computational >>>>>>>>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>>>>>>>> compiled >>>>>>>>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>>>>>>>> results. >>>>>>>>>>>>>> However when I recompile it for a realistic cases (the same >>>>>>>>>>>>>> source >>>>>>>>>>>>>> is used, only a bunch of parameter statements are different), it >>>>>>>>>>>>>> segfaults at line 13 of the main program: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in >>>>>>>>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>>>>>>>> Image PC Routine Line Source >>>>>>>>>>>>>> >>>>>>>>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>>>>>>>> 13 testwf.f >>>>>>>>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> Aborted (core dumped) >>>>>>>>>>>>>> [knl00 tests] >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Line 13 is the very beginning of declarations, not an executable >>>>>>>>>>>>>> statement. >>>>>>>>>>>>>> It is the first non-comment line and is: >>>>>>>>>>>>>> >>>>>>>>>>>>>> use random2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>>>>>>>> changed >>>>>>>>>>>>>> by the parameter statements that are different for the working >>>>>>>>>>>>>> and failing >>>>>>>>>>>>>> cases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here are the working and failing executable sizes: >>>>>>>>>>>>>> >>>>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>>>> x.testwf.4he-pn >>>>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>>>> x.testwf.12c-11_states-pn >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>>>>>>>> >>>>>>>>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>>>>>>>> -lsvml >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [knl00 mc] which ifort >>>>>>>>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>>>>>>>> >>>>>>>>>>>>>> This test case runs fine on blues. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any suggestions will be gratefully received. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Intel-nda mailing list >>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> >>>>> >>>> >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by Scott Parker 21 Jun '16

21 Jun '16

Steve- Have you tried setting 'ulimit -s unlimited' ? By default the system has a max stack size set to 8MB and I've found I need to increase it in some cases or else the program crashes at startup. Scott On 6/20/16 11:46 AM, Steven C. Pieper wrote: > OK, some progress... > > 1) I didn't realize that mcmodel=large (medium?) needs to be specified > on compile commands in addition to the link command. With that and using > no OpenMP it starts, even with the AVX commands. It eventually segfaults. > > 2) Using OpenMP it still instantly segfaults, even with just 1 thread > allowed. > > Steve > Steven C. Pieper, Senior Physicist SO:[email protected] > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 > Secretary, Debra Beres, -4100 > On 06/20/16 10:59, James Osborn wrote: >> I guess you can try older versions of the compiler to see if they work. >> You may be able to run the same binary compiled on blues at jlse. >> >> Cheers, >> James >> >> >> On 06/20/2016 10:56 AM, Steven C. Pieper wrote: >>> With the source line given below I get the same relocation errors >>> on thing01. >>> >>> Steve >>> >>> Steven C. Pieper, Senior Physicist SO:[email protected] >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>> IL 60439 >>> Phone: 630-252-4232 Fax -6008 >>> Secretary, Debra Beres, -4100 >>> >>> On 06/20/16 10:53, Steven C. Pieper wrote: >>>> should I continue to use >>>> >>>> source >>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>> >>>> intel64 >>>> >>>> on thing01? >>>> >>>> Steve >>>> >>>> >>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>> IL 60439 >>>> Phone: 630-252-4232 Fax -6008 >>>> Secretary, Debra Beres, -4100 >>>> On 06/20/16 10:47, James Osborn wrote: >>>>> Hi Steve, >>>>> >>>>> My only suggestion is to try using the same compilers and code on the >>>>> jlse thing* nodes (assuming you have access) to see if it works >>>>> there. The same executable you compile for those should also work on >>>>> KNL. >>>>> >>>>> Cheers, >>>>> James >>>>> >>>>> On 06/20/2016 10:35 AM, Steven C. Pieper wrote: >>>>>> I tried running with just one OMP thread and it failed in the >>>>>> same way. >>>>>> Now I'm trying to make a version with no OpenMP, no AVX, and -O0. >>>>>> I can >>>>>> again make the simple case but the big case now fails in the link >>>>>> step: >>>>>> >>>>>> ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 >>>>>> -I../common/ >>>>>> -I../mc/ -fpp -DSPIN_SUF= >>>>>> -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>> -g testwf.f >>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -o >>>>>> x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o >>>>>> adlb_fake.o \ >>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml >>>>>> testwf.o: In function `_unnamed_main$$': >>>>>> /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>>> section >>>>>> in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON >>>>>> section in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>>> section >>>>>> in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>>> section in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>>> section in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>>> section >>>>>> in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_32 against symbol `parallel_' defined in COMMON >>>>>> section in >>>>>> testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>>> section >>>>>> in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>>> section >>>>>> in testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation >>>>>> truncated to >>>>>> fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON >>>>>> section in >>>>>> testwf.o >>>>>> /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional >>>>>> relocation >>>>>> overflows omitted from the output >>>>>> make: *** [testwf] Error 1 >>>>>> [ >>>>>> >>>>>> The common blocks that it is complaining about are all very short >>>>>> simple >>>>>> ones. >>>>>> >>>>>> Help! I will be happy to come over to Bldg 240. >>>>>> >>>>>> Steve >>>>>> >>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>> IL 60439 >>>>>> Phone: 630-252-4232 Fax -6008 >>>>>> Secretary, Debra Beres, -4100 >>>>>> >>>>>> On 06/18/16 14:17, Steven Pieper wrote: >>>>>>> I will try various things when I can next get on. I don't >>>>>>> believe AVX >>>>>>> can be it: >>>>>>> >>>>>>> 1) The smaller case was compiled with the same options and works. >>>>>>> 2) It hasn't gotten started at all, so I don't see how any AVX >>>>>>> instructions could >>>>>>> have been encountered. >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>> Argonne, IL >>>>>>> 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> >>>>>>> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>>>>>>> Steve, >>>>>>>> >>>>>>>> I have no further good ideas. You could try simplifying your >>>>>>>> compile options to see if that will help produce a runnable >>>>>>>> binary. >>>>>>>> The KNL should support AVX but maybe the first thing to try is >>>>>>>> removing all the AVX flags. If that works, you can try adding >>>>>>>> -xMIC-AVX512. >>>>>>>> >>>>>>>> kevin >>>>>>>> >>>>>>>>> Well further tests have to wait until Monday now, but >>>>>>>>> >>>>>>>>> 1) I used both mcmodel=medium and large -- both fail the same >>>>>>>>> way >>>>>>>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>>>>>>> mcmodel=medium >>>>>>>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>>>>>>> 4) The size command does not show a huge executable (the big >>>>>>>>> stuff >>>>>>>>> will come >>>>>>>>> later from allocates): >>>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> >>>>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>> Argonne, >>>>>>>>> IL 60439 >>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>> >>>>>>>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>>>>>>> >>>>>>>>>> I would say it has something to do with the memory model and the >>>>>>>>>> limitations that the large segments of the code cannot be >>>>>>>>>> correctly >>>>>>>>>> located by the OS. The problems shows up at the first executable >>>>>>>>>> statement. >>>>>>>>>> >>>>>>>>>> Two suggestions: try to set the number of OMP threads to some >>>>>>>>>> small >>>>>>>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>>>>>>> >>>>>>>>>> Vitali >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>>>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>>>>>>> fortran 90. >>>>>>>>>>> >>>>>>>>>>> However the error doesn't seem to have anything to do with >>>>>>>>>>> random2.f per se. >>>>>>>>>>> I commented that line out and now the traceback points to >>>>>>>>>>> the next >>>>>>>>>>> non-comment line. So I think the segfault is happening at load >>>>>>>>>>> time >>>>>>>>>>> and the traceback just points to the first non comment line. >>>>>>>>>>> >>>>>>>>>>> I have to go now; I guess this will have to wait until next >>>>>>>>>>> week. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Steve >>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>> Argonne, >>>>>>>>>>> IL 60439 >>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>>>>>>> Steve, >>>>>>>>>>>> >>>>>>>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>>>>>>> fortran or your code? >>>>>>>>>>>> >>>>>>>>>>>> kevin >>>>>>>>>>>> >>>>>>>>>>>>> OK, with mpiifort it compiles but it still fails the same >>>>>>>>>>>>> way. >>>>>>>>>>>>> Here are the results from gdb: >>>>>>>>>>>>> >>>>>>>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>>> This is free software: you are free to change and >>>>>>>>>>>>> redistribute it. >>>>>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. >>>>>>>>>>>>> Type"show >>>>>>>>>>>>> copying" >>>>>>>>>>>>> and"show warranty" for details. >>>>>>>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>>>>>>> For bug reporting instructions, please see: >>>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>>>>>>> Reading symbols from >>>>>>>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>>>>>>> (gdb) run >>>>>>>>>>>>> Starting program: >>>>>>>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>>>>>>> >>>>>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>>> 13 use random2 >>>>>>>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>>>>>>> (gdb) >>>>>>>>>>>>> >>>>>>>>>>>>> (gdb) where >>>>>>>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>>>>>>> /lib64/libc.so.6 >>>>>>>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>>>>>>> (gdb) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>>>>>>> >>>>>>>>>>>>> Steve >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Try mpiifort. >>>>>>>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>>>>>>> Ye >>>>>>>>>>>>> =================== >>>>>>>>>>>>> Ye Luo, Ph.D. >>>>>>>>>>>>> Leadership Computing Facility >>>>>>>>>>>>> Argonne National Laboratory >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>> From: >>>>>>>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>>>>>>> To: Harms, Kevin; >>>>>>>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Osborn, James C. >>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Kevin, >>>>>>>>>>>>> >>>>>>>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>>>>>>> However mpif90 >>>>>>>>>>>>> is using gfortran: >>>>>>>>>>>>> >>>>>>>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ >>>>>>>>>>>>> -I../mc/ >>>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>>>>>> '-qopenmp-report' >>>>>>>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>>>>>> '-traceback' >>>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>>>>>>> [knl00 mc] which mpif90 >>>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Steve >>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki >>>>>>>>>>>>> has some >>>>>>>>>>>>> useful notes on it: >>>>>>>>>>>>> >>>>>>>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>>>>>>> >>>>>>>>>>>>> kevin >>>>>>>>>>>>> >>>>>>>>>>>>> Steve, >>>>>>>>>>>>> >>>>>>>>>>>>> First thing, try the latest compiler. >>>>>>>>>>>>> >>>>>>>>>>>>> source >>>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The 2017 beta has explicit KNL support. Second, have you >>>>>>>>>>>>> tried >>>>>>>>>>>>> running it in gdb? >>>>>>>>>>>>> >>>>>>>>>>>>> kevin >>>>>>>>>>>>> >>>>>>>>>>>>> I am trying to run a Fortran program that exercises some >>>>>>>>>>>>> of my >>>>>>>>>>>>> computational >>>>>>>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>>>>>>> compiled >>>>>>>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>>>>>>> results. >>>>>>>>>>>>> However when I recompile it for a realistic cases (the same >>>>>>>>>>>>> source >>>>>>>>>>>>> is used, only a bunch of parameter statements are >>>>>>>>>>>>> different), it >>>>>>>>>>>>> segfaults at line 13 of the main program: >>>>>>>>>>>>> >>>>>>>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>>> c12_11states-pn.in >>>>>>>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>>>>>>> Image PC Routine Line Source >>>>>>>>>>>>> >>>>>>>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>>>>>>> 13 testwf.f >>>>>>>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>> Aborted (core dumped) >>>>>>>>>>>>> [knl00 tests] >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Line 13 is the very beginning of declarations, not an >>>>>>>>>>>>> executable >>>>>>>>>>>>> statement. >>>>>>>>>>>>> It is the first non-comment line and is: >>>>>>>>>>>>> >>>>>>>>>>>>> use random2 >>>>>>>>>>>>> >>>>>>>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>>>>>>> changed >>>>>>>>>>>>> by the parameter statements that are different for the >>>>>>>>>>>>> working >>>>>>>>>>>>> and failing >>>>>>>>>>>>> cases. >>>>>>>>>>>>> >>>>>>>>>>>>> Here are the working and failing executable sizes: >>>>>>>>>>>>> >>>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>>> x.testwf.4he-pn >>>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>>> x.testwf.12c-11_states-pn >>>>>>>>>>>>> >>>>>>>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>>>>>>> >>>>>>>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ >>>>>>>>>>>>> -I../mc/ >>>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>>>>>>> >>>>>>>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>>>>>>> -lsvml >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [knl00 mc] which ifort >>>>>>>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>>>>>>> >>>>>>>>>>>>> This test case runs fine on blues. >>>>>>>>>>>>> >>>>>>>>>>>>> Any suggestions will be gratefully received. >>>>>>>>>>>>> >>>>>>>>>>>>> Steve >>>>>>>>>>>>> -- >>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>> >>>> >>> > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by Steven C. Pieper 20 Jun '16

20 Jun '16

OK, some progress... 1) I didn't realize that mcmodel=large (medium?) needs to be specified on compile commands in addition to the link command. With that and using no OpenMP it starts, even with the AVX commands. It eventually segfaults. 2) Using OpenMP it still instantly segfaults, even with just 1 thread allowed. Steve Steven C. Pieper, Senior Physicist SO: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/20/16 10:59, James Osborn wrote: > I guess you can try older versions of the compiler to see if they work. > You may be able to run the same binary compiled on blues at jlse. > > Cheers, > James > > > On 06/20/2016 10:56 AM, Steven C. Pieper wrote: >> With the source line given below I get the same relocation errors >> on thing01. >> >> Steve >> >> Steven C. Pieper, Senior Physicist SO:[email protected] >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >> 60439 >> Phone: 630-252-4232 Fax -6008 >> Secretary, Debra Beres, -4100 >> >> On 06/20/16 10:53, Steven C. Pieper wrote: >>> should I continue to use >>> >>> source >>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>> >>> intel64 >>> >>> on thing01? >>> >>> Steve >>> >>> >>> Steven C. Pieper, Senior Physicist SO:[email protected] >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>> IL 60439 >>> Phone: 630-252-4232 Fax -6008 >>> Secretary, Debra Beres, -4100 >>> On 06/20/16 10:47, James Osborn wrote: >>>> Hi Steve, >>>> >>>> My only suggestion is to try using the same compilers and code on the >>>> jlse thing* nodes (assuming you have access) to see if it works >>>> there. The same executable you compile for those should also work on >>>> KNL. >>>> >>>> Cheers, >>>> James >>>> >>>> On 06/20/2016 10:35 AM, Steven C. Pieper wrote: >>>>> I tried running with just one OMP thread and it failed in the same >>>>> way. >>>>> Now I'm trying to make a version with no OpenMP, no AVX, and -O0. >>>>> I can >>>>> again make the simple case but the big case now fails in the link >>>>> step: >>>>> >>>>> ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 >>>>> -I../common/ >>>>> -I../mc/ -fpp -DSPIN_SUF= >>>>> -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>> -DM0_DIV=1 -DDOCASE=0 >>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>> -g testwf.f >>>>> ifort -g -mcmodel=medium -traceback -fpe3 -o >>>>> x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o >>>>> adlb_fake.o \ >>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml >>>>> testwf.o: In function `_unnamed_main$$': >>>>> /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>> section >>>>> in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON >>>>> section in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>> section >>>>> in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>> section in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>> section in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>> section >>>>> in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation >>>>> truncated to >>>>> fit: R_X86_64_32 against symbol `parallel_' defined in COMMON >>>>> section in >>>>> testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>> section >>>>> in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON >>>>> section >>>>> in testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation >>>>> truncated to >>>>> fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON >>>>> section in >>>>> testwf.o >>>>> /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional >>>>> relocation >>>>> overflows omitted from the output >>>>> make: *** [testwf] Error 1 >>>>> [ >>>>> >>>>> The common blocks that it is complaining about are all very short >>>>> simple >>>>> ones. >>>>> >>>>> Help! I will be happy to come over to Bldg 240. >>>>> >>>>> Steve >>>>> >>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>> IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 >>>>> Secretary, Debra Beres, -4100 >>>>> >>>>> On 06/18/16 14:17, Steven Pieper wrote: >>>>>> I will try various things when I can next get on. I don't >>>>>> believe AVX >>>>>> can be it: >>>>>> >>>>>> 1) The smaller case was compiled with the same options and works. >>>>>> 2) It hasn't gotten started at all, so I don't see how any AVX >>>>>> instructions could >>>>>> have been encountered. >>>>>> >>>>>> Steve >>>>>> >>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>> Argonne, IL >>>>>> 60439 >>>>>> Phone: 630-252-4232 Fax -6008 >>>>>> Secretary, Debra Beres, -4100 >>>>>> >>>>>> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>>>>>> Steve, >>>>>>> >>>>>>> I have no further good ideas. You could try simplifying your >>>>>>> compile options to see if that will help produce a runnable binary. >>>>>>> The KNL should support AVX but maybe the first thing to try is >>>>>>> removing all the AVX flags. If that works, you can try adding >>>>>>> -xMIC-AVX512. >>>>>>> >>>>>>> kevin >>>>>>> >>>>>>>> Well further tests have to wait until Monday now, but >>>>>>>> >>>>>>>> 1) I used both mcmodel=medium and large -- both fail the same way >>>>>>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>>>>>> mcmodel=medium >>>>>>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>>>>>> 4) The size command does not show a huge executable (the big >>>>>>>> stuff >>>>>>>> will come >>>>>>>> later from allocates): >>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>> IL 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>> >>>>>>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>>>>>> >>>>>>>>> I would say it has something to do with the memory model and the >>>>>>>>> limitations that the large segments of the code cannot be >>>>>>>>> correctly >>>>>>>>> located by the OS. The problems shows up at the first executable >>>>>>>>> statement. >>>>>>>>> >>>>>>>>> Two suggestions: try to set the number of OMP threads to some >>>>>>>>> small >>>>>>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>>>>>> >>>>>>>>> Vitali >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>>>>>> fortran 90. >>>>>>>>>> >>>>>>>>>> However the error doesn't seem to have anything to do with >>>>>>>>>> random2.f per se. >>>>>>>>>> I commented that line out and now the traceback points to the >>>>>>>>>> next >>>>>>>>>> non-comment line. So I think the segfault is happening at load >>>>>>>>>> time >>>>>>>>>> and the traceback just points to the first non comment line. >>>>>>>>>> >>>>>>>>>> I have to go now; I guess this will have to wait until next >>>>>>>>>> week. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Steve >>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>> Argonne, >>>>>>>>>> IL 60439 >>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>>>>>> Steve, >>>>>>>>>>> >>>>>>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>>>>>> fortran or your code? >>>>>>>>>>> >>>>>>>>>>> kevin >>>>>>>>>>> >>>>>>>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>>>>>>> Here are the results from gdb: >>>>>>>>>>>> >>>>>>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>> This is free software: you are free to change and >>>>>>>>>>>> redistribute it. >>>>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. >>>>>>>>>>>> Type"show >>>>>>>>>>>> copying" >>>>>>>>>>>> and"show warranty" for details. >>>>>>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>>>>>> For bug reporting instructions, please see: >>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>>>>>> Reading symbols from >>>>>>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>>>>>> (gdb) run >>>>>>>>>>>> Starting program: >>>>>>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>>>>>> >>>>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>> 13 use random2 >>>>>>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>>>>>> (gdb) >>>>>>>>>>>> >>>>>>>>>>>> (gdb) where >>>>>>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>>>>>> /lib64/libc.so.6 >>>>>>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>>>>>> (gdb) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>>>>>> >>>>>>>>>>>> Steve >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Try mpiifort. >>>>>>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>>>>>> Ye >>>>>>>>>>>> =================== >>>>>>>>>>>> Ye Luo, Ph.D. >>>>>>>>>>>> Leadership Computing Facility >>>>>>>>>>>> Argonne National Laboratory >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ________________________________________ >>>>>>>>>>>> From: >>>>>>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>>>>>> To: Harms, Kevin; >>>>>>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Osborn, James C. >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Kevin, >>>>>>>>>>>> >>>>>>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>>>>>> However mpif90 >>>>>>>>>>>> is using gfortran: >>>>>>>>>>>> >>>>>>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ >>>>>>>>>>>> -I../mc/ >>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>>>>> '-qopenmp-report' >>>>>>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>>>>>> [knl00 mc] which mpif90 >>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Steve >>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has >>>>>>>>>>>> some >>>>>>>>>>>> useful notes on it: >>>>>>>>>>>> >>>>>>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>>>>>> >>>>>>>>>>>> kevin >>>>>>>>>>>> >>>>>>>>>>>> Steve, >>>>>>>>>>>> >>>>>>>>>>>> First thing, try the latest compiler. >>>>>>>>>>>> >>>>>>>>>>>> source >>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>>>>>>> running it in gdb? >>>>>>>>>>>> >>>>>>>>>>>> kevin >>>>>>>>>>>> >>>>>>>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>>>>>>> computational >>>>>>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>>>>>> compiled >>>>>>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>>>>>> results. >>>>>>>>>>>> However when I recompile it for a realistic cases (the same >>>>>>>>>>>> source >>>>>>>>>>>> is used, only a bunch of parameter statements are >>>>>>>>>>>> different), it >>>>>>>>>>>> segfaults at line 13 of the main program: >>>>>>>>>>>> >>>>>>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>> c12_11states-pn.in >>>>>>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>>>>>> Image PC Routine Line Source >>>>>>>>>>>> >>>>>>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>>>>>> 13 testwf.f >>>>>>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>> Aborted (core dumped) >>>>>>>>>>>> [knl00 tests] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Line 13 is the very beginning of declarations, not an >>>>>>>>>>>> executable >>>>>>>>>>>> statement. >>>>>>>>>>>> It is the first non-comment line and is: >>>>>>>>>>>> >>>>>>>>>>>> use random2 >>>>>>>>>>>> >>>>>>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>>>>>> changed >>>>>>>>>>>> by the parameter statements that are different for the working >>>>>>>>>>>> and failing >>>>>>>>>>>> cases. >>>>>>>>>>>> >>>>>>>>>>>> Here are the working and failing executable sizes: >>>>>>>>>>>> >>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>> x.testwf.4he-pn >>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>> x.testwf.12c-11_states-pn >>>>>>>>>>>> >>>>>>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>>>>>> >>>>>>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ >>>>>>>>>>>> -I../mc/ >>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>>>>>> >>>>>>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>>>>>> -lsvml >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [knl00 mc] which ifort >>>>>>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>>>>>> >>>>>>>>>>>> This test case runs fine on blues. >>>>>>>>>>>> >>>>>>>>>>>> Any suggestions will be gratefully received. >>>>>>>>>>>> >>>>>>>>>>>> Steve >>>>>>>>>>>> -- >>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Intel-nda mailing list >>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>> >>

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by James Osborn 20 Jun '16

20 Jun '16

I guess you can try older versions of the compiler to see if they work. You may be able to run the same binary compiled on blues at jlse. Cheers, James On 06/20/2016 10:56 AM, Steven C. Pieper wrote: > With the source line given below I get the same relocation errors > on thing01. > > Steve > > Steven C. Pieper, Senior Physicist SO:[email protected] > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 > Secretary, Debra Beres, -4100 > > On 06/20/16 10:53, Steven C. Pieper wrote: >> should I continue to use >> >> source >> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >> intel64 >> >> on thing01? >> >> Steve >> >> >> Steven C. Pieper, Senior Physicist SO:[email protected] >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 >> Secretary, Debra Beres, -4100 >> On 06/20/16 10:47, James Osborn wrote: >>> Hi Steve, >>> >>> My only suggestion is to try using the same compilers and code on the >>> jlse thing* nodes (assuming you have access) to see if it works >>> there. The same executable you compile for those should also work on >>> KNL. >>> >>> Cheers, >>> James >>> >>> On 06/20/2016 10:35 AM, Steven C. Pieper wrote: >>>> I tried running with just one OMP thread and it failed in the same way. >>>> Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can >>>> again make the simple case but the big case now fails in the link step: >>>> >>>> ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ >>>> -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>> -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>> -g testwf.f >>>> ifort -g -mcmodel=medium -traceback -fpe3 -o >>>> x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o >>>> adlb_fake.o \ >>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml >>>> testwf.o: In function `_unnamed_main$$': >>>> /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to >>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>> in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to >>>> fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON >>>> section in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to >>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>> in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation >>>> truncated to >>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>> section in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation >>>> truncated to >>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>> section in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation >>>> truncated to >>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>> in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation >>>> truncated to >>>> fit: R_X86_64_32 against symbol `parallel_' defined in COMMON >>>> section in >>>> testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation >>>> truncated to >>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>> in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation >>>> truncated to >>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>> in testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation >>>> truncated to >>>> fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON >>>> section in >>>> testwf.o >>>> /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation >>>> overflows omitted from the output >>>> make: *** [testwf] Error 1 >>>> [ >>>> >>>> The common blocks that it is complaining about are all very short >>>> simple >>>> ones. >>>> >>>> Help! I will be happy to come over to Bldg 240. >>>> >>>> Steve >>>> >>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>> IL 60439 >>>> Phone: 630-252-4232 Fax -6008 >>>> Secretary, Debra Beres, -4100 >>>> >>>> On 06/18/16 14:17, Steven Pieper wrote: >>>>> I will try various things when I can next get on. I don't believe AVX >>>>> can be it: >>>>> >>>>> 1) The smaller case was compiled with the same options and works. >>>>> 2) It hasn't gotten started at all, so I don't see how any AVX >>>>> instructions could >>>>> have been encountered. >>>>> >>>>> Steve >>>>> >>>>> Steven C. Pieper: spieper(a)anl.gov >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>> 60439 >>>>> Phone: 630-252-4232 Fax -6008 >>>>> Secretary, Debra Beres, -4100 >>>>> >>>>> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>>>>> Steve, >>>>>> >>>>>> I have no further good ideas. You could try simplifying your >>>>>> compile options to see if that will help produce a runnable binary. >>>>>> The KNL should support AVX but maybe the first thing to try is >>>>>> removing all the AVX flags. If that works, you can try adding >>>>>> -xMIC-AVX512. >>>>>> >>>>>> kevin >>>>>> >>>>>>> Well further tests have to wait until Monday now, but >>>>>>> >>>>>>> 1) I used both mcmodel=medium and large -- both fail the same way >>>>>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>>>>> mcmodel=medium >>>>>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>>>>> 4) The size command does not show a huge executable (the big stuff >>>>>>> will come >>>>>>> later from allocates): >>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>> IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> >>>>>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>>>>> >>>>>>>> I would say it has something to do with the memory model and the >>>>>>>> limitations that the large segments of the code cannot be correctly >>>>>>>> located by the OS. The problems shows up at the first executable >>>>>>>> statement. >>>>>>>> >>>>>>>> Two suggestions: try to set the number of OMP threads to some small >>>>>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>>>>> >>>>>>>> Vitali >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>>>>> fortran 90. >>>>>>>>> >>>>>>>>> However the error doesn't seem to have anything to do with >>>>>>>>> random2.f per se. >>>>>>>>> I commented that line out and now the traceback points to the next >>>>>>>>> non-comment line. So I think the segfault is happening at load >>>>>>>>> time >>>>>>>>> and the traceback just points to the first non comment line. >>>>>>>>> >>>>>>>>> I have to go now; I guess this will have to wait until next week. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Steve >>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>>> IL 60439 >>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>>>>> Steve, >>>>>>>>>> >>>>>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>>>>> fortran or your code? >>>>>>>>>> >>>>>>>>>> kevin >>>>>>>>>> >>>>>>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>>>>>> Here are the results from gdb: >>>>>>>>>>> >>>>>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>> This is free software: you are free to change and >>>>>>>>>>> redistribute it. >>>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type"show >>>>>>>>>>> copying" >>>>>>>>>>> and"show warranty" for details. >>>>>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>>>>> For bug reporting instructions, please see: >>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>>>>> Reading symbols from >>>>>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>>>>> (gdb) run >>>>>>>>>>> Starting program: >>>>>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>>>>> >>>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>> 13 use random2 >>>>>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>>>>> (gdb) >>>>>>>>>>> >>>>>>>>>>> (gdb) where >>>>>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>>>>> /lib64/libc.so.6 >>>>>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>>>>> (gdb) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>>>>> >>>>>>>>>>> Steve >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Try mpiifort. >>>>>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>>>>> Ye >>>>>>>>>>> =================== >>>>>>>>>>> Ye Luo, Ph.D. >>>>>>>>>>> Leadership Computing Facility >>>>>>>>>>> Argonne National Laboratory >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ________________________________________ >>>>>>>>>>> From: >>>>>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>>>>> >>>>>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>>>>> To: Harms, Kevin; >>>>>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>>>>> >>>>>>>>>>> Osborn, James C. >>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Kevin, >>>>>>>>>>> >>>>>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>>>>> However mpif90 >>>>>>>>>>> is using gfortran: >>>>>>>>>>> >>>>>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>>>> '-qopenmp-report' >>>>>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>>>>> [knl00 mc] which mpif90 >>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Steve >>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has some >>>>>>>>>>> useful notes on it: >>>>>>>>>>> >>>>>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>>>>> >>>>>>>>>>> kevin >>>>>>>>>>> >>>>>>>>>>> Steve, >>>>>>>>>>> >>>>>>>>>>> First thing, try the latest compiler. >>>>>>>>>>> >>>>>>>>>>> source >>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>>>>>> running it in gdb? >>>>>>>>>>> >>>>>>>>>>> kevin >>>>>>>>>>> >>>>>>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>>>>>> computational >>>>>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>>>>> compiled >>>>>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>>>>> results. >>>>>>>>>>> However when I recompile it for a realistic cases (the same >>>>>>>>>>> source >>>>>>>>>>> is used, only a bunch of parameter statements are different), it >>>>>>>>>>> segfaults at line 13 of the main program: >>>>>>>>>>> >>>>>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in >>>>>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>>>>> Image PC Routine Line Source >>>>>>>>>>> >>>>>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>>>>> 13 testwf.f >>>>>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>>>>> Unknown Unknown >>>>>>>>>>> Aborted (core dumped) >>>>>>>>>>> [knl00 tests] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Line 13 is the very beginning of declarations, not an executable >>>>>>>>>>> statement. >>>>>>>>>>> It is the first non-comment line and is: >>>>>>>>>>> >>>>>>>>>>> use random2 >>>>>>>>>>> >>>>>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>>>>> changed >>>>>>>>>>> by the parameter statements that are different for the working >>>>>>>>>>> and failing >>>>>>>>>>> cases. >>>>>>>>>>> >>>>>>>>>>> Here are the working and failing executable sizes: >>>>>>>>>>> >>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>> x.testwf.4he-pn >>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>> x.testwf.12c-11_states-pn >>>>>>>>>>> >>>>>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>>>>> >>>>>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>>>>> >>>>>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>>>>> -lsvml >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [knl00 mc] which ifort >>>>>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>>>>> >>>>>>>>>>> This test case runs fine on blues. >>>>>>>>>>> >>>>>>>>>>> Any suggestions will be gratefully received. >>>>>>>>>>> >>>>>>>>>>> Steve >>>>>>>>>>> -- >>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Intel-nda mailing list >>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >>>> >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >> >

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by Steven C. Pieper 20 Jun '16

20 Jun '16

With the source line given below I get the same relocation errors on thing01. Steve Steven C. Pieper, Senior Physicist SO: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/20/16 10:53, Steven C. Pieper wrote: > should I continue to use > > source > /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh > intel64 > > on thing01? > > Steve > > > Steven C. Pieper, Senior Physicist SO:[email protected] > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 > Secretary, Debra Beres, -4100 > On 06/20/16 10:47, James Osborn wrote: >> Hi Steve, >> >> My only suggestion is to try using the same compilers and code on the >> jlse thing* nodes (assuming you have access) to see if it works >> there. The same executable you compile for those should also work on >> KNL. >> >> Cheers, >> James >> >> On 06/20/2016 10:35 AM, Steven C. Pieper wrote: >>> I tried running with just one OMP thread and it failed in the same way. >>> Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can >>> again make the simple case but the big case now fails in the link step: >>> >>> ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ >>> -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>> -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>> -g testwf.f >>> ifort -g -mcmodel=medium -traceback -fpe3 -o >>> x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o >>> adlb_fake.o \ >>> ../common/wall_second-nompi.o ../common/stopnompi.o >>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml >>> testwf.o: In function `_unnamed_main$$': >>> /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to >>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>> in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to >>> fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON >>> section in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to >>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>> in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation >>> truncated to >>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>> section in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation >>> truncated to >>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>> section in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation >>> truncated to >>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>> in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation >>> truncated to >>> fit: R_X86_64_32 against symbol `parallel_' defined in COMMON >>> section in >>> testwf.o >>> /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation >>> truncated to >>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>> in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation >>> truncated to >>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>> in testwf.o >>> /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation >>> truncated to >>> fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON >>> section in >>> testwf.o >>> /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation >>> overflows omitted from the output >>> make: *** [testwf] Error 1 >>> [ >>> >>> The common blocks that it is complaining about are all very short >>> simple >>> ones. >>> >>> Help! I will be happy to come over to Bldg 240. >>> >>> Steve >>> >>> Steven C. Pieper, Senior Physicist SO:[email protected] >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>> IL 60439 >>> Phone: 630-252-4232 Fax -6008 >>> Secretary, Debra Beres, -4100 >>> >>> On 06/18/16 14:17, Steven Pieper wrote: >>>> I will try various things when I can next get on. I don't believe AVX >>>> can be it: >>>> >>>> 1) The smaller case was compiled with the same options and works. >>>> 2) It hasn't gotten started at all, so I don't see how any AVX >>>> instructions could >>>> have been encountered. >>>> >>>> Steve >>>> >>>> Steven C. Pieper: spieper(a)anl.gov >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>> 60439 >>>> Phone: 630-252-4232 Fax -6008 >>>> Secretary, Debra Beres, -4100 >>>> >>>> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>>>> Steve, >>>>> >>>>> I have no further good ideas. You could try simplifying your >>>>> compile options to see if that will help produce a runnable binary. >>>>> The KNL should support AVX but maybe the first thing to try is >>>>> removing all the AVX flags. If that works, you can try adding >>>>> -xMIC-AVX512. >>>>> >>>>> kevin >>>>> >>>>>> Well further tests have to wait until Monday now, but >>>>>> >>>>>> 1) I used both mcmodel=medium and large -- both fail the same way >>>>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>>>> mcmodel=medium >>>>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>>>> 4) The size command does not show a huge executable (the big stuff >>>>>> will come >>>>>> later from allocates): >>>>>>>>>> text data bss dec hex filename >>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>>>> >>>>>> Steve >>>>>> >>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>> IL 60439 >>>>>> Phone: 630-252-4232 Fax -6008 >>>>>> Secretary, Debra Beres, -4100 >>>>>> >>>>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>>>> >>>>>>> I would say it has something to do with the memory model and the >>>>>>> limitations that the large segments of the code cannot be correctly >>>>>>> located by the OS. The problems shows up at the first executable >>>>>>> statement. >>>>>>> >>>>>>> Two suggestions: try to set the number of OMP threads to some small >>>>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>>>> >>>>>>> Vitali >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>>>> fortran 90. >>>>>>>> >>>>>>>> However the error doesn't seem to have anything to do with >>>>>>>> random2.f per se. >>>>>>>> I commented that line out and now the traceback points to the next >>>>>>>> non-comment line. So I think the segfault is happening at load >>>>>>>> time >>>>>>>> and the traceback just points to the first non comment line. >>>>>>>> >>>>>>>> I have to go now; I guess this will have to wait until next week. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Steve >>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>> IL 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>>>> Steve, >>>>>>>>> >>>>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>>>> fortran or your code? >>>>>>>>> >>>>>>>>> kevin >>>>>>>>> >>>>>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>>>>> Here are the results from gdb: >>>>>>>>>> >>>>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>> This is free software: you are free to change and >>>>>>>>>> redistribute it. >>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type"show >>>>>>>>>> copying" >>>>>>>>>> and"show warranty" for details. >>>>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>>>> For bug reporting instructions, please see: >>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>>>> Reading symbols from >>>>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>>>> (gdb) run >>>>>>>>>> Starting program: >>>>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>>>> >>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>> 13 use random2 >>>>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>>>> (gdb) >>>>>>>>>> >>>>>>>>>> (gdb) where >>>>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>>>> /lib64/libc.so.6 >>>>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>>>> (gdb) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>>>> >>>>>>>>>> Steve >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>> Argonne, IL 60439 >>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Try mpiifort. >>>>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>>>> Ye >>>>>>>>>> =================== >>>>>>>>>> Ye Luo, Ph.D. >>>>>>>>>> Leadership Computing Facility >>>>>>>>>> Argonne National Laboratory >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ________________________________________ >>>>>>>>>> From: >>>>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>>>> >>>>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>>>> To: Harms, Kevin; >>>>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>>>> >>>>>>>>>> Osborn, James C. >>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Kevin, >>>>>>>>>> >>>>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>>>> However mpif90 >>>>>>>>>> is using gfortran: >>>>>>>>>> >>>>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>>> '-qopenmp-report' >>>>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>>>> [knl00 mc] which mpif90 >>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Steve >>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>> Argonne, IL 60439 >>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has some >>>>>>>>>> useful notes on it: >>>>>>>>>> >>>>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>>>> >>>>>>>>>> kevin >>>>>>>>>> >>>>>>>>>> Steve, >>>>>>>>>> >>>>>>>>>> First thing, try the latest compiler. >>>>>>>>>> >>>>>>>>>> source >>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>>>>> running it in gdb? >>>>>>>>>> >>>>>>>>>> kevin >>>>>>>>>> >>>>>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>>>>> computational >>>>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>>>> compiled >>>>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>>>> results. >>>>>>>>>> However when I recompile it for a realistic cases (the same >>>>>>>>>> source >>>>>>>>>> is used, only a bunch of parameter statements are different), it >>>>>>>>>> segfaults at line 13 of the main program: >>>>>>>>>> >>>>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in >>>>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>>>> Image PC Routine Line Source >>>>>>>>>> >>>>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>>>> 13 testwf.f >>>>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>>>> Unknown Unknown >>>>>>>>>> Aborted (core dumped) >>>>>>>>>> [knl00 tests] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Line 13 is the very beginning of declarations, not an executable >>>>>>>>>> statement. >>>>>>>>>> It is the first non-comment line and is: >>>>>>>>>> >>>>>>>>>> use random2 >>>>>>>>>> >>>>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>>>> changed >>>>>>>>>> by the parameter statements that are different for the working >>>>>>>>>> and failing >>>>>>>>>> cases. >>>>>>>>>> >>>>>>>>>> Here are the working and failing executable sizes: >>>>>>>>>> >>>>>>>>>> text data bss dec hex filename >>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>> x.testwf.4he-pn >>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>> x.testwf.12c-11_states-pn >>>>>>>>>> >>>>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>>>> >>>>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>>>> >>>>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>>>> -lsvml >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [knl00 mc] which ifort >>>>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>>>> >>>>>>>>>> This test case runs fine on blues. >>>>>>>>>> >>>>>>>>>> Any suggestions will be gratefully received. >>>>>>>>>> >>>>>>>>>> Steve >>>>>>>>>> -- >>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>> Argonne, IL 60439 >>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Intel-nda mailing list >>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >>> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by Steven C. Pieper 20 Jun '16

20 Jun '16

should I continue to use source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 on thing01? Steve Steven C. Pieper, Senior Physicist SO: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/20/16 10:47, James Osborn wrote: > Hi Steve, > > My only suggestion is to try using the same compilers and code on the > jlse thing* nodes (assuming you have access) to see if it works there. > The same executable you compile for those should also work on KNL. > > Cheers, > James > > On 06/20/2016 10:35 AM, Steven C. Pieper wrote: >> I tried running with just one OMP thread and it failed in the same way. >> Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can >> again make the simple case but the big case now fails in the link step: >> >> ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ >> -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >> -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >> -g testwf.f >> ifort -g -mcmodel=medium -traceback -fpe3 -o >> x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o >> adlb_fake.o \ >> ../common/wall_second-nompi.o ../common/stopnompi.o >> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml >> testwf.o: In function `_unnamed_main$$': >> /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to >> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >> in testwf.o >> /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to >> fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON >> section in testwf.o >> /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to >> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >> in testwf.o >> /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation truncated to >> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >> section in testwf.o >> /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation truncated to >> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >> section in testwf.o >> /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation truncated to >> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >> in testwf.o >> /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation truncated to >> fit: R_X86_64_32 against symbol `parallel_' defined in COMMON section in >> testwf.o >> /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation truncated to >> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >> in testwf.o >> /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation truncated to >> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >> in testwf.o >> /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation truncated to >> fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON section in >> testwf.o >> /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation >> overflows omitted from the output >> make: *** [testwf] Error 1 >> [ >> >> The common blocks that it is complaining about are all very short simple >> ones. >> >> Help! I will be happy to come over to Bldg 240. >> >> Steve >> >> Steven C. Pieper, Senior Physicist SO:[email protected] >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >> 60439 >> Phone: 630-252-4232 Fax -6008 >> Secretary, Debra Beres, -4100 >> >> On 06/18/16 14:17, Steven Pieper wrote: >>> I will try various things when I can next get on. I don't believe AVX >>> can be it: >>> >>> 1) The smaller case was compiled with the same options and works. >>> 2) It hasn't gotten started at all, so I don't see how any AVX >>> instructions could >>> have been encountered. >>> >>> Steve >>> >>> Steven C. Pieper: spieper(a)anl.gov >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>> 60439 >>> Phone: 630-252-4232 Fax -6008 >>> Secretary, Debra Beres, -4100 >>> >>> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>>> Steve, >>>> >>>> I have no further good ideas. You could try simplifying your >>>> compile options to see if that will help produce a runnable binary. >>>> The KNL should support AVX but maybe the first thing to try is >>>> removing all the AVX flags. If that works, you can try adding >>>> -xMIC-AVX512. >>>> >>>> kevin >>>> >>>>> Well further tests have to wait until Monday now, but >>>>> >>>>> 1) I used both mcmodel=medium and large -- both fail the same way >>>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>>> mcmodel=medium >>>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>>> 4) The size command does not show a huge executable (the big stuff >>>>> will come >>>>> later from allocates): >>>>>>>>> text data bss dec hex filename >>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>>> >>>>> Steve >>>>> >>>>> Steven C. Pieper: spieper(a)anl.gov >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>> IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 >>>>> Secretary, Debra Beres, -4100 >>>>> >>>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>>> >>>>>> I would say it has something to do with the memory model and the >>>>>> limitations that the large segments of the code cannot be correctly >>>>>> located by the OS. The problems shows up at the first executable >>>>>> statement. >>>>>> >>>>>> Two suggestions: try to set the number of OMP threads to some small >>>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>>> >>>>>> Vitali >>>>>> >>>>>> >>>>>> >>>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>>> fortran 90. >>>>>>> >>>>>>> However the error doesn't seem to have anything to do with >>>>>>> random2.f per se. >>>>>>> I commented that line out and now the traceback points to the next >>>>>>> non-comment line. So I think the segfault is happening at load time >>>>>>> and the traceback just points to the first non comment line. >>>>>>> >>>>>>> I have to go now; I guess this will have to wait until next week. >>>>>>> >>>>>>> Thanks, >>>>>>> Steve >>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>> IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>>> Steve, >>>>>>>> >>>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>>> fortran or your code? >>>>>>>> >>>>>>>> kevin >>>>>>>> >>>>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>>>> Here are the results from gdb: >>>>>>>>> >>>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>> This is free software: you are free to change and redistribute >>>>>>>>> it. >>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type"show >>>>>>>>> copying" >>>>>>>>> and"show warranty" for details. >>>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>>> For bug reporting instructions, please see: >>>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>>> Reading symbols from >>>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>>> (gdb) run >>>>>>>>> Starting program: >>>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>>> >>>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>> 13 use random2 >>>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>>> (gdb) >>>>>>>>> >>>>>>>>> (gdb) where >>>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>>> /lib64/libc.so.6 >>>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>>> (gdb) >>>>>>>>> >>>>>>>>> >>>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> >>>>>>>>> >>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>> Argonne, IL 60439 >>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Try mpiifort. >>>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>>> Ye >>>>>>>>> =================== >>>>>>>>> Ye Luo, Ph.D. >>>>>>>>> Leadership Computing Facility >>>>>>>>> Argonne National Laboratory >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ________________________________________ >>>>>>>>> From: >>>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>>> >>>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>>> To: Harms, Kevin; >>>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>>> >>>>>>>>> Osborn, James C. >>>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Kevin, >>>>>>>>> >>>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>>> However mpif90 >>>>>>>>> is using gfortran: >>>>>>>>> >>>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>> '-qopenmp-report' >>>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>>> [knl00 mc] which mpif90 >>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>> Argonne, IL 60439 >>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has some >>>>>>>>> useful notes on it: >>>>>>>>> >>>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>>> >>>>>>>>> kevin >>>>>>>>> >>>>>>>>> Steve, >>>>>>>>> >>>>>>>>> First thing, try the latest compiler. >>>>>>>>> >>>>>>>>> source >>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>>> >>>>>>>>> >>>>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>>>> running it in gdb? >>>>>>>>> >>>>>>>>> kevin >>>>>>>>> >>>>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>>>> computational >>>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>>> compiled >>>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>>> results. >>>>>>>>> However when I recompile it for a realistic cases (the same >>>>>>>>> source >>>>>>>>> is used, only a bunch of parameter statements are different), it >>>>>>>>> segfaults at line 13 of the main program: >>>>>>>>> >>>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in >>>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>>> Image PC Routine Line Source >>>>>>>>> >>>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>>> 13 testwf.f >>>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>>> Unknown Unknown >>>>>>>>> Aborted (core dumped) >>>>>>>>> [knl00 tests] >>>>>>>>> >>>>>>>>> >>>>>>>>> Line 13 is the very beginning of declarations, not an executable >>>>>>>>> statement. >>>>>>>>> It is the first non-comment line and is: >>>>>>>>> >>>>>>>>> use random2 >>>>>>>>> >>>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>>> changed >>>>>>>>> by the parameter statements that are different for the working >>>>>>>>> and failing >>>>>>>>> cases. >>>>>>>>> >>>>>>>>> Here are the working and failing executable sizes: >>>>>>>>> >>>>>>>>> text data bss dec hex filename >>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>> x.testwf.4he-pn >>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>> x.testwf.12c-11_states-pn >>>>>>>>> >>>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>>> >>>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>>> >>>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>>> -lsvml >>>>>>>>> >>>>>>>>> >>>>>>>>> [knl00 mc] which ifort >>>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>>> >>>>>>>>> >>>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>>> >>>>>>>>> This test case runs fine on blues. >>>>>>>>> >>>>>>>>> Any suggestions will be gratefully received. >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> -- >>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>> Argonne, IL 60439 >>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by James Osborn 20 Jun '16

20 Jun '16

Hi Steve, My only suggestion is to try using the same compilers and code on the jlse thing* nodes (assuming you have access) to see if it works there. The same executable you compile for those should also work on KNL. Cheers, James On 06/20/2016 10:35 AM, Steven C. Pieper wrote: > I tried running with just one OMP thread and it failed in the same way. > Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can > again make the simple case but the big case now fails in the link step: > > ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ > -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn > -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c > -g testwf.f > ifort -g -mcmodel=medium -traceback -fpe3 -o > x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o adlb_fake.o \ > ../common/wall_second-nompi.o ../common/stopnompi.o > ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen > -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml > testwf.o: In function `_unnamed_main$$': > /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to > fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section > in testwf.o > /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to > fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON > section in testwf.o > /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to > fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section > in testwf.o > /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation truncated to > fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON > section in testwf.o > /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation truncated to > fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON > section in testwf.o > /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation truncated to > fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section > in testwf.o > /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation truncated to > fit: R_X86_64_32 against symbol `parallel_' defined in COMMON section in > testwf.o > /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation truncated to > fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section > in testwf.o > /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation truncated to > fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section > in testwf.o > /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation truncated to > fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON section in > testwf.o > /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation > overflows omitted from the output > make: *** [testwf] Error 1 > [ > > The common blocks that it is complaining about are all very short simple > ones. > > Help! I will be happy to come over to Bldg 240. > > Steve > > Steven C. Pieper, Senior Physicist SO:[email protected] > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 > Secretary, Debra Beres, -4100 > > On 06/18/16 14:17, Steven Pieper wrote: >> I will try various things when I can next get on. I don't believe AVX >> can be it: >> >> 1) The smaller case was compiled with the same options and works. >> 2) It hasn't gotten started at all, so I don't see how any AVX >> instructions could >> have been encountered. >> >> Steve >> >> Steven C. Pieper: spieper(a)anl.gov >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >> 60439 >> Phone: 630-252-4232 Fax -6008 >> Secretary, Debra Beres, -4100 >> >> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>> Steve, >>> >>> I have no further good ideas. You could try simplifying your >>> compile options to see if that will help produce a runnable binary. >>> The KNL should support AVX but maybe the first thing to try is >>> removing all the AVX flags. If that works, you can try adding >>> -xMIC-AVX512. >>> >>> kevin >>> >>>> Well further tests have to wait until Monday now, but >>>> >>>> 1) I used both mcmodel=medium and large -- both fail the same way >>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>> mcmodel=medium >>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>> 4) The size command does not show a huge executable (the big stuff >>>> will come >>>> later from allocates): >>>>>>>> text data bss dec hex filename >>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>> >>>> Steve >>>> >>>> Steven C. Pieper: spieper(a)anl.gov >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>> IL 60439 >>>> Phone: 630-252-4232 Fax -6008 >>>> Secretary, Debra Beres, -4100 >>>> >>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>> >>>>> I would say it has something to do with the memory model and the >>>>> limitations that the large segments of the code cannot be correctly >>>>> located by the OS. The problems shows up at the first executable >>>>> statement. >>>>> >>>>> Two suggestions: try to set the number of OMP threads to some small >>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>> fortran 90. >>>>>> >>>>>> However the error doesn't seem to have anything to do with >>>>>> random2.f per se. >>>>>> I commented that line out and now the traceback points to the next >>>>>> non-comment line. So I think the segfault is happening at load time >>>>>> and the traceback just points to the first non comment line. >>>>>> >>>>>> I have to go now; I guess this will have to wait until next week. >>>>>> >>>>>> Thanks, >>>>>> Steve >>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>> IL 60439 >>>>>> Phone: 630-252-4232 Fax -6008 >>>>>> Secretary, Debra Beres, -4100 >>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>> Steve, >>>>>>> >>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>> fortran or your code? >>>>>>> >>>>>>> kevin >>>>>>> >>>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>>> Here are the results from gdb: >>>>>>>> >>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>> This is free software: you are free to change and redistribute it. >>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type"show >>>>>>>> copying" >>>>>>>> and"show warranty" for details. >>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>> For bug reporting instructions, please see: >>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>> Reading symbols from >>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>> (gdb) run >>>>>>>> Starting program: >>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>> >>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>> 13 use random2 >>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>> (gdb) >>>>>>>> >>>>>>>> (gdb) where >>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>> /lib64/libc.so.6 >>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>> (gdb) >>>>>>>> >>>>>>>> >>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> >>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>> Argonne, IL 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>> >>>>>>>> >>>>>>>> Try mpiifort. >>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>> Ye >>>>>>>> =================== >>>>>>>> Ye Luo, Ph.D. >>>>>>>> Leadership Computing Facility >>>>>>>> Argonne National Laboratory >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ________________________________________ >>>>>>>> From: >>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>> To: Harms, Kevin; >>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>> Osborn, James C. >>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>> >>>>>>>> >>>>>>>> Hi Kevin, >>>>>>>> >>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>> However mpif90 >>>>>>>> is using gfortran: >>>>>>>> >>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>> -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>> gfortran: error: unrecognized command line option '-qopenmp-report' >>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>> [knl00 mc] which mpif90 >>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Steve >>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>> Argonne, IL 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>> >>>>>>>> >>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has some >>>>>>>> useful notes on it: >>>>>>>> >>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>> >>>>>>>> kevin >>>>>>>> >>>>>>>> Steve, >>>>>>>> >>>>>>>> First thing, try the latest compiler. >>>>>>>> >>>>>>>> source >>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>> >>>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>>> running it in gdb? >>>>>>>> >>>>>>>> kevin >>>>>>>> >>>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>>> computational >>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>> compiled >>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>> results. >>>>>>>> However when I recompile it for a realistic cases (the same source >>>>>>>> is used, only a bunch of parameter statements are different), it >>>>>>>> segfaults at line 13 of the main program: >>>>>>>> >>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in >>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>> Image PC Routine Line Source >>>>>>>> >>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>> Unknown Unknown >>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>> Unknown Unknown >>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>> Unknown Unknown >>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>> Unknown Unknown >>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>> Unknown Unknown >>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>> Unknown Unknown >>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>> Unknown Unknown >>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>> 13 testwf.f >>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>> Unknown Unknown >>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>> Unknown Unknown >>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>> Unknown Unknown >>>>>>>> Aborted (core dumped) >>>>>>>> [knl00 tests] >>>>>>>> >>>>>>>> >>>>>>>> Line 13 is the very beginning of declarations, not an executable >>>>>>>> statement. >>>>>>>> It is the first non-comment line and is: >>>>>>>> >>>>>>>> use random2 >>>>>>>> >>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>> changed >>>>>>>> by the parameter statements that are different for the working >>>>>>>> and failing >>>>>>>> cases. >>>>>>>> >>>>>>>> Here are the working and failing executable sizes: >>>>>>>> >>>>>>>> text data bss dec hex filename >>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>> x.testwf.4he-pn >>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>> x.testwf.12c-11_states-pn >>>>>>>> >>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>> >>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>> >>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>> -lsvml >>>>>>>> >>>>>>>> >>>>>>>> [knl00 mc] which ifort >>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>> >>>>>>>> >>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>> >>>>>>>> This test case runs fine on blues. >>>>>>>> >>>>>>>> Any suggestions will be gratefully received. >>>>>>>> >>>>>>>> Steve >>>>>>>> -- >>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>> Argonne, IL 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by Steven C. Pieper 20 Jun '16

20 Jun '16

I tried running with just one OMP thread and it failed in the same way. Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can again make the simple case but the big case now fails in the link step: ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f ifort -g -mcmodel=medium -traceback -fpe3 -o x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o adlb_fake.o \ ../common/wall_second-nompi.o ../common/stopnompi.o ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml testwf.o: In function `_unnamed_main$$': /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation truncated to fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation truncated to fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation truncated to fit: R_X86_64_32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation truncated to fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation overflows omitted from the output make: *** [testwf] Error 1 [ The common blocks that it is complaining about are all very short simple ones. Help! I will be happy to come over to Bldg 240. Steve Steven C. Pieper, Senior Physicist SO: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/18/16 14:17, Steven Pieper wrote: > I will try various things when I can next get on. I don't believe AVX > can be it: > > 1) The smaller case was compiled with the same options and works. > 2) It hasn't gotten started at all, so I don't see how any AVX > instructions could > have been encountered. > > Steve > > Steven C. Pieper: spieper(a)anl.gov > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL > 60439 > Phone: 630-252-4232 Fax -6008 > Secretary, Debra Beres, -4100 > > On 06/17/2016 06:09 PM, Harms, Kevin wrote: >> Steve, >> >> I have no further good ideas. You could try simplifying your >> compile options to see if that will help produce a runnable binary. >> The KNL should support AVX but maybe the first thing to try is >> removing all the AVX flags. If that works, you can try adding >> -xMIC-AVX512. >> >> kevin >> >>> Well further tests have to wait until Monday now, but >>> >>> 1) I used both mcmodel=medium and large -- both fail the same way >>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>> mcmodel=medium >>> 3) I forget the number of OMP threads I was using -- I think 32. >>> 4) The size command does not show a huge executable (the big stuff >>> will come >>> later from allocates): >>>>>>> text data bss dec hex filename >>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>> x.testwf.4he-pn<<<< this works >>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>> x.testwf.12c-11_states-pn<<< this fails >>> >>> Steve >>> >>> Steven C. Pieper: spieper(a)anl.gov >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>> IL 60439 >>> Phone: 630-252-4232 Fax -6008 >>> Secretary, Debra Beres, -4100 >>> >>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>> >>>> I would say it has something to do with the memory model and the >>>> limitations that the large segments of the code cannot be correctly >>>> located by the OS. The problems shows up at the first executable >>>> statement. >>>> >>>> Two suggestions: try to set the number of OMP threads to some small >>>> number (4 or 8 to start with), and try -mcmodel large. >>>> >>>> Vitali >>>> >>>> >>>> >>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>> I'm not sure what you mean. random2.f is written in standard >>>>> fortran 90. >>>>> >>>>> However the error doesn't seem to have anything to do with >>>>> random2.f per se. >>>>> I commented that line out and now the traceback points to the next >>>>> non-comment line. So I think the segfault is happening at load time >>>>> and the traceback just points to the first non comment line. >>>>> >>>>> I have to go now; I guess this will have to wait until next week. >>>>> >>>>> Thanks, >>>>> Steve >>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>> IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 >>>>> Secretary, Debra Beres, -4100 >>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>> Steve, >>>>>> >>>>>> I don�t know fortran well, is random2 part of the standard >>>>>> fortran or your code? >>>>>> >>>>>> kevin >>>>>> >>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>> Here are the results from gdb: >>>>>>> >>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>> This is free software: you are free to change and redistribute it. >>>>>>> There is NO WARRANTY, to the extent permitted by law. Type"show >>>>>>> copying" >>>>>>> and"show warranty" for details. >>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>> For bug reporting instructions, please see: >>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>> Reading symbols from >>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>> (gdb) run >>>>>>> Starting program: >>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>> [Thread debugging using libthread_db enabled] >>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>> >>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>> 13 use random2 >>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>> (gdb) >>>>>>> >>>>>>> (gdb) where >>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>> #1 0x0000000000402c0e in main () >>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>> /lib64/libc.so.6 >>>>>>> #3 0x0000000000402b19 in _start () >>>>>>> (gdb) >>>>>>> >>>>>>> >>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> >>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>> Argonne, IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>> >>>>>>> >>>>>>> Try mpiifort. >>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>> Ye >>>>>>> =================== >>>>>>> Ye Luo, Ph.D. >>>>>>> Leadership Computing Facility >>>>>>> Argonne National Laboratory >>>>>>> >>>>>>> >>>>>>> >>>>>>> ________________________________________ >>>>>>> From: >>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>> Pieper [spieper(a)anl.gov] >>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>> To: Harms, Kevin; >>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>> Osborn, James C. >>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>> >>>>>>> >>>>>>> Hi Kevin, >>>>>>> >>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>> However mpif90 >>>>>>> is using gfortran: >>>>>>> >>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>> -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>> gfortran: error: unrecognized command line option '-qopenmp-report' >>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>> [knl00 mc] which mpif90 >>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>> >>>>>>> >>>>>>> >>>>>>> Steve >>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>> Argonne, IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>> >>>>>>> >>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has some >>>>>>> useful notes on it: >>>>>>> >>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>> >>>>>>> kevin >>>>>>> >>>>>>> Steve, >>>>>>> >>>>>>> First thing, try the latest compiler. >>>>>>> >>>>>>> source >>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>> >>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>> running it in gdb? >>>>>>> >>>>>>> kevin >>>>>>> >>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>> computational >>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>> compiled >>>>>>> it for a small case and that runs fine and gives the correct >>>>>>> results. >>>>>>> However when I recompile it for a realistic cases (the same source >>>>>>> is used, only a bunch of parameter statements are different), it >>>>>>> segfaults at line 13 of the main program: >>>>>>> >>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in >>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>> Image PC Routine Line Source >>>>>>> >>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>> Unknown Unknown >>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>> Unknown Unknown >>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>> Unknown Unknown >>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>> Unknown Unknown >>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>> Unknown Unknown >>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>> Unknown Unknown >>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>> Unknown Unknown >>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>> 13 testwf.f >>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>> Unknown Unknown >>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>> Unknown Unknown >>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>> Unknown Unknown >>>>>>> Aborted (core dumped) >>>>>>> [knl00 tests] >>>>>>> >>>>>>> >>>>>>> Line 13 is the very beginning of declarations, not an executable >>>>>>> statement. >>>>>>> It is the first non-comment line and is: >>>>>>> >>>>>>> use random2 >>>>>>> >>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>> changed >>>>>>> by the parameter statements that are different for the working >>>>>>> and failing >>>>>>> cases. >>>>>>> >>>>>>> Here are the working and failing executable sizes: >>>>>>> >>>>>>> text data bss dec hex filename >>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>> x.testwf.4he-pn >>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>> x.testwf.12c-11_states-pn >>>>>>> >>>>>>> Here is how the main program is compiled and module linked: >>>>>>> >>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>> >>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>> -lsvml >>>>>>> >>>>>>> >>>>>>> [knl00 mc] which ifort >>>>>>> /soft/compilers/intel/bin/ifort >>>>>>> >>>>>>> >>>>>>> This is in ~b21541/gfmc/mc . >>>>>>> >>>>>>> This test case runs fine on blues. >>>>>>> >>>>>>> Any suggestions will be gratefully received. >>>>>>> >>>>>>> Steve >>>>>>> -- >>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>> Argonne, IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0