Re: [JLSE-Intel-NDA] Help on knl00 segfault
Scott: That seems to be it! The code is now working (8 threads, -O2, no AVX yet). I had been using -s 400000 which works on blues but I guess more is needed here. I will keep pushing the optimizations and threads up. Thanks, Steve Steven C. Pieper: [email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/21/2016 03:08 PM, Scott Parker wrote:
Steve-
Have you tried setting 'ulimit -s unlimited' ? By default the system has a max stack size set to 8MB and I've found I need to increase it in some cases or else the program crashes at startup.
Scott
On 6/20/16 11:46 AM, Steven C. Pieper wrote:
OK, some progress...
1) I didn't realize that mcmodel=large (medium?) needs to be specified on compile commands in addition to the link command. With that and using no OpenMP it starts, even with the AVX commands. It eventually segfaults.
2) Using OpenMP it still instantly segfaults, even with just 1 thread allowed.
Steve Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/20/16 10:59, James Osborn wrote:
I guess you can try older versions of the compiler to see if they work. You may be able to run the same binary compiled on blues at jlse.
Cheers, James
On 06/20/2016 10:56 AM, Steven C. Pieper wrote:
With the source line given below I get the same relocation errors on thing01.
Steve
Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100
On 06/20/16 10:53, Steven C. Pieper wrote:
should I continue to use
source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64
on thing01?
Steve
Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/20/16 10:47, James Osborn wrote:
Hi Steve,
My only suggestion is to try using the same compilers and code on the jlse thing* nodes (assuming you have access) to see if it works there. The same executable you compile for those should also work on KNL.
Cheers, James
On 06/20/2016 10:35 AM, Steven C. Pieper wrote:
I tried running with just one OMP thread and it failed in the same way. Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can again make the simple case but the big case now fails in the link step:
ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f ifort -g -mcmodel=medium -traceback -fpe3 -o x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o adlb_fake.o \ ../common/wall_second-nompi.o ../common/stopnompi.o ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml testwf.o: In function `_unnamed_main$$': /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation truncated to fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation truncated to fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation truncated to fit: R_X86_64_32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation truncated to fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation truncated to fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON section in testwf.o /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation overflows omitted from the output make: *** [testwf] Error 1 [
The common blocks that it is complaining about are all very short simple ones.
Help! I will be happy to come over to Bldg 240.
Steve
Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100
On 06/18/16 14:17, Steven Pieper wrote:
I will try various things when I can next get on. I don't believe AVX can be it:
1) The smaller case was compiled with the same options and works. 2) It hasn't gotten started at all, so I don't see how any AVX instructions could have been encountered.
Steve
Steven C. Pieper: [email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100
On 06/17/2016 06:09 PM, Harms, Kevin wrote:
Steve,
I have no further good ideas. You could try simplifying your compile options to see if that will help produce a runnable binary. The KNL should support AVX but maybe the first thing to try is removing all the AVX flags. If that works, you can try adding -xMIC-AVX512.
kevin
Well further tests have to wait until Monday now, but
1) I used both mcmodel=medium and large -- both fail the same way 2) It runs on blues (haswell nodes and Intell 11 compiler) with mcmodel=medium 3) I forget the number of OMP threads I was using -- I think 32. 4) The size command does not show a huge executable (the big stuff will come later from allocates):
text data bss dec hex filename 2459352 124772 24979752 27563876 1a49764 x.testwf.4he-pn<<<< this works 2192689 92004 623277384 625562077 254951dd x.testwf.12c-11_states-pn<<< this fails
Steve
Steven C. Pieper: [email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100
On 06/16/2016 04:41 PM, Vitali A. Morozov wrote:
I would say it has something to do with the memory model and the limitations that the large segments of the code cannot be correctly located by the OS. The problems shows up at the first executable statement.
Two suggestions: try to set the number of OMP threads to some small number (4 or 8 to start with), and try -mcmodel large.
Vitali
On 06/16/2016 04:10 PM, Steven C. Pieper wrote:
I'm not sure what you mean. random2.f is written in standard fortran 90.
However the error doesn't seem to have anything to do with random2.f per se. I commented that line out and now the traceback points to the next non-comment line. So I think the segfault is happening at load time and the traceback just points to the first non comment line.
I have to go now; I guess this will have to wait until next week.
Thanks, Steve Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/16/16 16:02, Harms, Kevin wrote:
Steve,
I don�t know fortran well, is random2 part of the standard fortran or your code?
kevin
OK, with mpiifort it compiles but it still fails the same way. Here are the results from gdb:
[knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type"show copying" and"show warranty" for details. This GDB was configured as"x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/> <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. (gdb) run Starting program: /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn [Thread debugging using libthread_db enabled] Using host libthread_db library"/lib64/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault. 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 13 use random2 Missing separate debuginfos, use: debuginfo-install glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 (gdb)
(gdb) where #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 #1 0x0000000000402c0e in main () #2 0x00007ffff53b7b15 in __libc_start_main () from /lib64/libc.so.6 #3 0x0000000000402b19 in _start () (gdb)
I can come over to Bldg 240 if that seems helpful.
Steve
Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/16/16 15:08, Luo, Ye wrote:
Try mpiifort. You can verify the use of ifort by"mpiifort -show". Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory
________________________________________ From: [email protected]<mailto:[email protected]>
[[email protected]] on behalf of Steven C. Pieper [[email protected]] Sent: Thursday, June 16, 2016 3:04 PM To: Harms, Kevin; [email protected]<mailto:[email protected]>;
Osborn, James C. Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault
Hi Kevin,
I did the source command and ifort gets the beta compiler OK. However mpif90 is using gfortran:
mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -I../adlb/ -I../dmem/ -c kinetic.f gfortran: error: unrecognized command line option '-axAVX' gfortran: error: unrecognized command line option '-qopenmp' gfortran: error: unrecognized command line option '-qopenmp-report' gfortran: error: unrecognized command line option '-r8' gfortran: error: unrecognized command line option '-WB' gfortran: error: unrecognized command line option '-traceback' gfortran: error: unrecognized command line option '-fpe3' gfortran: error: unrecognized command line option '-fpp' make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 [knl00 mc] which mpif90 /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90
Steve Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/16/16 14:53, Harms, Kevin wrote:
Oops. I forgot to put �intel64� at the end. The wiki has some useful notes on it:
https://wiki.jlse.anl.gov/display/knl/Intel+KNL
kevin
Steve,
First thing, try the latest compiler.
source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh
The 2017 beta has explicit KNL support. Second, have you tried running it in gdb?
kevin
I am trying to run a Fortran program that exercises some of my computational kernels. The program does not use MPI, just OpenMP. I have compiled it for a small case and that runs fine and gives the correct results. However when I recompile it for a realistic cases (the same source is used, only a bunch of parameter statements are different), it segfaults at line 13 of the main program:
[knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source
libintlc.so.5 00007F569D060C05 Unknown Unknown Unknown libintlc.so.5 00007F569D05E827 Unknown Unknown Unknown libifcoremt.so.5 00007F569DEE1962 Unknown Unknown Unknown libifcoremt.so.5 00007F569DEE17B6 Unknown Unknown Unknown libifcoremt.so.5 00007F569DE36CAD Unknown Unknown Unknown libifcoremt.so.5 00007F569DE48FAD Unknown Unknown Unknown libpthread.so.0 00007F569F2D7100 Unknown Unknown Unknown x.testwf.12c-11_s 00000000004030F0 MAIN__ 13 testwf.f x.testwf.12c-11_s 0000000000402C0E Unknown Unknown Unknown libc.so.6 00007F569CAA3B15 Unknown Unknown Unknown x.testwf.12c-11_s 0000000000402B19 Unknown Unknown Unknown Aborted (core dumped) [knl00 tests]
Line 13 is the very beginning of declarations, not an executable statement. It is the first non-comment line and is:
use random2
random2 is a module that defines some subroutines. It is not changed by the parameter statements that are different for the working and failing cases.
Here are the working and failing executable sizes:
text data bss dec hex filename 2459352 124772 24979752 27563876 1a49764 x.testwf.4he-pn 2192689 92004 623277384 625562077 254951dd x.testwf.12c-11_states-pn
Here is how the main program is compiled and module linked:
ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f
ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o adlb_fake.o \ ../common/wall_second-nompi.o ../common/stopnompi.o ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml
[knl00 mc] which ifort /soft/compilers/intel/bin/ifort
This is in ~b21541/gfmc/mc .
This test case runs fine on blues.
Any suggestions will be gratefully received.
Steve -- Steven C. Pieper, Senior Physicist SO:[email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100
_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda
Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda
_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda
_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda
participants (1)
-
Steven Pieper