June 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Hal Finkel 28 Jun '16

28 Jun '16

Thanks! I'll upgrade. -Hal ----- Original Message ----- > From: "Kevin Harms" <harms(a)alcf.anl.gov> > To: "Hal J. Finkel" <hfinkel(a)anl.gov>, "Vitali A. Morozov" <morozov(a)alcf.anl.gov> > Cc: intel-nda(a)lists.jlse.anl.gov > Sent: Tuesday, June 28, 2016 11:59:20 AM > Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > > I updated this page to indicate how to upgrade to the mpsp 1.3.2 > kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > > No idea, if that will fix this problem though :) > > kevin > > >FYI: James and I have been experimenting with our box. We had to > >reset it after we hit some kernel bug: > > > >Message from syslogd@kingly at Jun 27 14:29:44 ... > > kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] > > > >Message from syslogd@kingly at Jun 27 14:30:43 ... > > kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] > > > >Message from syslogd@kingly at Jun 27 14:30:44 ... > > kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] > > > >Message from syslogd@kingly at Jun 27 14:30:46 ... > > kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] > > > >Message from syslogd@kingly at Jun 27 14:30:47 ... > > kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] > > > >Message from syslogd@kingly at Jun 27 14:30:50 ... > > kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] > > > >Message from syslogd@kingly at Jun 27 14:30:52 ... > > kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] > > > >Message from syslogd@kingly at Jun 27 14:30:54 ... > > kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] > > > >Message from syslogd@kingly at Jun 27 14:30:55 ... > > kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] > > > >Message from syslogd@kingly at Jun 27 14:30:55 ... > > kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] > > > >Message from syslogd@kingly at Jun 27 14:30:58 ... > > kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] > > > >Message from syslogd@kingly at Jun 27 14:31:00 ... > > kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] > > > >... (this went on for a while) > > > >The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems like > >some special build? > > > > -Hal > > > >----- Original Message ----- > >> From: "Vitali A. Morozov" <morozov(a)anl.gov> > >> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov>, > >> intel-nda(a)lists.jlse.anl.gov > >> Sent: Thursday, June 23, 2016 10:47:37 AM > >> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > >> > >> > >> > >> I have updated the Ethernet port issue that I have encountered. > >> > >> Vitali > >> > >> On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: > >> > >> > >> For folks setting these up please see instructions at: > >> > >> > >> https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > >> > >> > >> > >> Kevin, Scott and Vitali will be adding information as they set the > >> machines up. Others are welcome to add stuff. > >> > >> > >> -Kumar > >> > >> _______________________________________________ > >> Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov > >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >> > >> _______________________________________________ > >> Intel-nda mailing list > >> Intel-nda(a)lists.jlse.anl.gov > >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >> > > > >-- > >Hal Finkel > >Assistant Computational Scientist > >Leadership Computing Facility > >Argonne National Laboratory > >_______________________________________________ > >Intel-nda mailing list > >Intel-nda(a)lists.jlse.anl.gov > >https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Harms, Kevin 28 Jun '16

28 Jun '16

I updated this page to indicate how to upgrade to the mpsp 1.3.2 kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup No idea, if that will fix this problem though :) kevin >FYI: James and I have been experimenting with our box. We had to reset it after we hit some kernel bug: > >Message from syslogd@kingly at Jun 27 14:29:44 ... > kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] > >Message from syslogd@kingly at Jun 27 14:30:43 ... > kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] > >Message from syslogd@kingly at Jun 27 14:30:44 ... > kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] > >Message from syslogd@kingly at Jun 27 14:30:46 ... > kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] > >Message from syslogd@kingly at Jun 27 14:30:47 ... > kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] > >Message from syslogd@kingly at Jun 27 14:30:50 ... > kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] > >Message from syslogd@kingly at Jun 27 14:30:52 ... > kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] > >Message from syslogd@kingly at Jun 27 14:30:54 ... > kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] > >Message from syslogd@kingly at Jun 27 14:30:55 ... > kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] > >Message from syslogd@kingly at Jun 27 14:30:55 ... > kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] > >Message from syslogd@kingly at Jun 27 14:30:58 ... > kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] > >Message from syslogd@kingly at Jun 27 14:31:00 ... > kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] > >... (this went on for a while) > >The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems like some special build? > > -Hal > >----- Original Message ----- >> From: "Vitali A. Morozov" <morozov(a)anl.gov> >> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov>, intel-nda(a)lists.jlse.anl.gov >> Sent: Thursday, June 23, 2016 10:47:37 AM >> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up >> >> >> >> I have updated the Ethernet port issue that I have encountered. >> >> Vitali >> >> On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: >> >> >> For folks setting these up please see instructions at: >> >> >> https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup >> >> >> >> Kevin, Scott and Vitali will be adding information as they set the >> machines up. Others are welcome to add stuff. >> >> >> -Kumar >> >> _______________________________________________ >> Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> > >-- >Hal Finkel >Assistant Computational Scientist >Leadership Computing Facility >Argonne National Laboratory >_______________________________________________ >Intel-nda mailing list >Intel-nda(a)lists.jlse.anl.gov >https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Hal Finkel 27 Jun '16

27 Jun '16

FYI: James and I have been experimenting with our box. We had to reset it after we hit some kernel bug: Message from syslogd@kingly at Jun 27 14:29:44 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:43 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] Message from syslogd@kingly at Jun 27 14:30:44 ... kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] Message from syslogd@kingly at Jun 27 14:30:46 ... kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] Message from syslogd@kingly at Jun 27 14:30:47 ... kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] Message from syslogd@kingly at Jun 27 14:30:50 ... kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] Message from syslogd@kingly at Jun 27 14:30:52 ... kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] Message from syslogd@kingly at Jun 27 14:30:54 ... kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] Message from syslogd@kingly at Jun 27 14:30:58 ... kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] Message from syslogd@kingly at Jun 27 14:31:00 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] ... (this went on for a while) The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems like some special build? -Hal ----- Original Message ----- > From: "Vitali A. Morozov" <morozov(a)anl.gov> > To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov>, intel-nda(a)lists.jlse.anl.gov > Sent: Thursday, June 23, 2016 10:47:37 AM > Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > > > I have updated the Ethernet port issue that I have encountered. > > Vitali > > On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: > > > For folks setting these up please see instructions at: > > > https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > > > > Kevin, Scott and Vitali will be adding information as they set the > machines up. Others are welcome to add stuff. > > > -Kumar > > _______________________________________________ > Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

[JLSE-Intel-NDA] Add hwloc-dump-hwdata to sudoers
by Allen, Benjamin S. 23 Jun '16

23 Jun '16

I added sudo permissions for '/usr/sbin/hwloc-dump-hwdata'. This queries entries in /sys/firmware/dmi/* and shows currently running MCDRAM memory mode and cluster mode. I believe if the MCDRAM is in cached mode it will also dump that cache. Also anyone in the intel unix group now will get the needed sudo permissions as referenced on the wiki. No need to request that access from now on. You still will need to request the IPMI username and password to power the machine back on mode or cluster mode change. Ben

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Vitali A Morozov 23 Jun '16

23 Jun '16

So I went ahead with the Intel Parallel Studio installation, that is included with the box. The license is given for 1 year (until Aug 14, 2017, to be exact) and the current version of this product is Intel 16 Update 3. We know that this version has "partial" KNL support. We have been told yesterday that Update 4 with full KNL support will show up any day soon and Intel 17 - in August. Anyway, during the installation it warns me about ================ Unsupported OS Detected operating system is not supported. Supported operating systems for this release are: CentOS* 6.5 (Intel(R) 64) Debian* 7, 8, 8 Fedora* 21, 22 Red Hat Enterprise Linux* 5 (deprecated), 6, 7, 5 (deprecated) SUSE Linux Enterprise Server* 11, 12 Ubuntu* 12.04 (Intel(R) 64), 13.10, 14.04, 15.04, 15.04 The system does not use a supported Intel(R) processor. The system does not use a supported Intel(R) processor. The drivers for Hardware Event-based Sampling (EBS) data collection will be disabled. To enable this functionality, install the product on a system with a compliant processor architecture. ================= I find it weird - shipping a product with an unsupported OS. At the minimum, they should ship with the right OS. Also, it seems do not support KNL at all, it does not even read the id string. Let us see what Update 4 would bring us. Vitali On 6/23/16 10:42 AM, Kumaran, Kalyan wrote: > For folks setting these up please see instructions at: > > https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > > Kevin, Scott and Vitali will be adding information as they set the > machines up. Others are welcome to add stuff. > > -Kumar > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Vitali A. Morozov 23 Jun '16

23 Jun '16

I have updated the Ethernet port issue that I have encountered. Vitali On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: > For folks setting these up please see instructions at: > > https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > > Kevin, Scott and Vitali will be adding information as they set the > machines up. Others are welcome to add stuff. > > -Kumar > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Intel DAP boxes set up
by Kumaran, Kalyan 23 Jun '16

23 Jun '16

For folks setting these up please see instructions at: https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup Kevin, Scott and Vitali will be adding information as they set the machines up. Others are welcome to add stuff. -Kumar

1 0

Re: [JLSE-Intel-NDA] Weird 'undefined reference to' error for an AVX512 intrinsic
by Luo, Ye 21 Jun '16

21 Jun '16

Hi Jiang, According to https://software.intel.com/sites/landingpage/IntrinsicsGuide/# _mm512_mask_extloadunpacklo_epi32 belongs to the ISA of KNC but it is not part of AVX512. So it seems not supported on KNL. Intel only says KNL is binary compatible with Xeon but never said KNC. I'm sorry that I can't help you more. An expert on intrinsic might know how to make a workaround. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Jiang, Wei [wjiang(a)alcf.anl.gov] Sent: Tuesday, June 21, 2016 5:20 PM To: intel-nda(a)lists.jlse.anl.gov Subject: [JLSE-Intel-NDA] Weird 'undefined reference to' error for an AVX512 intrinsic Hi, I am compiling a test code with icpc but got such a link error 'undefined reference to _mm512_mask_extloadunpacklo_epi32'. Note there are many AVX512 intrinsic in the code but just this one receives 'undefined reference to' error, while another mini test code that only contains '_mm512_mask_extloadunpacklo_epi32' doesn't get such error. Can anybody help see what wrong it is? Thanks in advance. Wei Jiang, PhD, Leadership Computing Facility Argonne National Laboratory 9700S, Cass Ave, Lemont, IL60439 630-252-8688 (office) 630-252-7121 (fax) _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Weird 'undefined reference to' error for an AVX512 intrinsic
by Jiang, Wei 21 Jun '16

21 Jun '16

Hi, I am compiling a test code with icpc but got such a link error 'undefined reference to _mm512_mask_extloadunpacklo_epi32'. Note there are many AVX512 intrinsic in the code but just this one receives 'undefined reference to' error, while another mini test code that only contains '_mm512_mask_extloadunpacklo_epi32' doesn't get such error. Can anybody help see what wrong it is? Thanks in advance. Wei Jiang, PhD, Leadership Computing Facility Argonne National Laboratory 9700S, Cass Ave, Lemont, IL60439 630-252-8688 (office) 630-252-7121 (fax)

1 0

Re: [JLSE-Intel-NDA] Help on knl00 segfault
by Steven Pieper 21 Jun '16

21 Jun '16

Scott: That seems to be it! The code is now working (8 threads, -O2, no AVX yet). I had been using -s 400000 which works on blues but I guess more is needed here. I will keep pushing the optimizations and threads up. Thanks, Steve Steven C. Pieper: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 On 06/21/2016 03:08 PM, Scott Parker wrote: > > Steve- > > Have you tried setting 'ulimit -s unlimited' ? By default the system has a max stack > size set to 8MB and I've found I need to increase it in some cases or else the program > crashes at startup. > > Scott > > > On 6/20/16 11:46 AM, Steven C. Pieper wrote: >> OK, some progress... >> >> 1) I didn't realize that mcmodel=large (medium?) needs to be specified >> on compile commands in addition to the link command. With that and using >> no OpenMP it starts, even with the AVX commands. It eventually segfaults. >> >> 2) Using OpenMP it still instantly segfaults, even with just 1 thread allowed. >> >> Steve >> Steven C. Pieper, Senior Physicist SO:[email protected] >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 >> Secretary, Debra Beres, -4100 >> On 06/20/16 10:59, James Osborn wrote: >>> I guess you can try older versions of the compiler to see if they work. >>> You may be able to run the same binary compiled on blues at jlse. >>> >>> Cheers, >>> James >>> >>> >>> On 06/20/2016 10:56 AM, Steven C. Pieper wrote: >>>> With the source line given below I get the same relocation errors >>>> on thing01. >>>> >>>> Steve >>>> >>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>> Phone: 630-252-4232 Fax -6008 >>>> Secretary, Debra Beres, -4100 >>>> >>>> On 06/20/16 10:53, Steven C. Pieper wrote: >>>>> should I continue to use >>>>> >>>>> source >>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>> intel64 >>>>> >>>>> on thing01? >>>>> >>>>> Steve >>>>> >>>>> >>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 >>>>> Secretary, Debra Beres, -4100 >>>>> On 06/20/16 10:47, James Osborn wrote: >>>>>> Hi Steve, >>>>>> >>>>>> My only suggestion is to try using the same compilers and code on the >>>>>> jlse thing* nodes (assuming you have access) to see if it works >>>>>> there. The same executable you compile for those should also work on >>>>>> KNL. >>>>>> >>>>>> Cheers, >>>>>> James >>>>>> >>>>>> On 06/20/2016 10:35 AM, Steven C. Pieper wrote: >>>>>>> I tried running with just one OMP thread and it failed in the same way. >>>>>>> Now I'm trying to make a version with no OpenMP, no AVX, and -O0. I can >>>>>>> again make the simple case but the big case now fails in the link step: >>>>>>> >>>>>>> ifort -Imods.libgen -FR -g -O0 -r8 -WB -traceback -fpe3 -I../common/ >>>>>>> -I../mc/ -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>> -DM0_DIV=1 -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>> -g testwf.f >>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -o >>>>>>> x.testwf.12c-11_states-pn testwf.o do_adlb_no.o dmem_fake.o >>>>>>> adlb_fake.o \ >>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o -lsvml >>>>>>> testwf.o: In function `_unnamed_main$$': >>>>>>> /home/b21541/gfmc/mc/testwf.f:132:(.text+0x89): relocation truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:133:(.text+0xb7): relocation truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `wall_time_com_' defined in COMMON >>>>>>> section in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:135:(.text+0xbd): relocation truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:144:(.text+0x137): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>>>> section in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:145:(.text+0x141): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `alloc_stat_' defined in COMMON >>>>>>> section in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:147:(.text+0x14b): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:148:(.text+0x154): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_32 against symbol `parallel_' defined in COMMON >>>>>>> section in >>>>>>> testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:149:(.text+0x194): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:150:(.text+0x19e): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `parallel_' defined in COMMON section >>>>>>> in testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:152:(.text+0x1a8): relocation >>>>>>> truncated to >>>>>>> fit: R_X86_64_PC32 against symbol `nparam_' defined in COMMON >>>>>>> section in >>>>>>> testwf.o >>>>>>> /home/b21541/gfmc/mc/testwf.f:153:(.text+0x1b2): additional relocation >>>>>>> overflows omitted from the output >>>>>>> make: *** [testwf] Error 1 >>>>>>> [ >>>>>>> >>>>>>> The common blocks that it is complaining about are all very short >>>>>>> simple >>>>>>> ones. >>>>>>> >>>>>>> Help! I will be happy to come over to Bldg 240. >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>> IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>> Secretary, Debra Beres, -4100 >>>>>>> >>>>>>> On 06/18/16 14:17, Steven Pieper wrote: >>>>>>>> I will try various things when I can next get on. I don't believe AVX >>>>>>>> can be it: >>>>>>>> >>>>>>>> 1) The smaller case was compiled with the same options and works. >>>>>>>> 2) It hasn't gotten started at all, so I don't see how any AVX >>>>>>>> instructions could >>>>>>>> have been encountered. >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>>>>> 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>> >>>>>>>> On 06/17/2016 06:09 PM, Harms, Kevin wrote: >>>>>>>>> Steve, >>>>>>>>> >>>>>>>>> I have no further good ideas. You could try simplifying your >>>>>>>>> compile options to see if that will help produce a runnable binary. >>>>>>>>> The KNL should support AVX but maybe the first thing to try is >>>>>>>>> removing all the AVX flags. If that works, you can try adding >>>>>>>>> -xMIC-AVX512. >>>>>>>>> >>>>>>>>> kevin >>>>>>>>> >>>>>>>>>> Well further tests have to wait until Monday now, but >>>>>>>>>> >>>>>>>>>> 1) I used both mcmodel=medium and large -- both fail the same way >>>>>>>>>> 2) It runs on blues (haswell nodes and Intell 11 compiler) with >>>>>>>>>> mcmodel=medium >>>>>>>>>> 3) I forget the number of OMP threads I was using -- I think 32. >>>>>>>>>> 4) The size command does not show a huge executable (the big stuff >>>>>>>>>> will come >>>>>>>>>> later from allocates): >>>>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>>>> x.testwf.4he-pn<<<< this works >>>>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>>>> x.testwf.12c-11_states-pn<<< this fails >>>>>>>>>> >>>>>>>>>> Steve >>>>>>>>>> >>>>>>>>>> Steven C. Pieper: spieper(a)anl.gov >>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>>>> IL 60439 >>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>> >>>>>>>>>> On 06/16/2016 04:41 PM, Vitali A. Morozov wrote: >>>>>>>>>>> >>>>>>>>>>> I would say it has something to do with the memory model and the >>>>>>>>>>> limitations that the large segments of the code cannot be correctly >>>>>>>>>>> located by the OS. The problems shows up at the first executable >>>>>>>>>>> statement. >>>>>>>>>>> >>>>>>>>>>> Two suggestions: try to set the number of OMP threads to some small >>>>>>>>>>> number (4 or 8 to start with), and try -mcmodel large. >>>>>>>>>>> >>>>>>>>>>> Vitali >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 06/16/2016 04:10 PM, Steven C. Pieper wrote: >>>>>>>>>>>> I'm not sure what you mean. random2.f is written in standard >>>>>>>>>>>> fortran 90. >>>>>>>>>>>> >>>>>>>>>>>> However the error doesn't seem to have anything to do with >>>>>>>>>>>> random2.f per se. >>>>>>>>>>>> I commented that line out and now the traceback points to the next >>>>>>>>>>>> non-comment line. So I think the segfault is happening at load >>>>>>>>>>>> time >>>>>>>>>>>> and the traceback just points to the first non comment line. >>>>>>>>>>>> >>>>>>>>>>>> I have to go now; I guess this will have to wait until next week. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Steve >>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>>>>>> IL 60439 >>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>> On 06/16/16 16:02, Harms, Kevin wrote: >>>>>>>>>>>>> Steve, >>>>>>>>>>>>> >>>>>>>>>>>>> I don�t know fortran well, is random2 part of the standard >>>>>>>>>>>>> fortran or your code? >>>>>>>>>>>>> >>>>>>>>>>>>> kevin >>>>>>>>>>>>> >>>>>>>>>>>>>> OK, with mpiifort it compiles but it still fails the same way. >>>>>>>>>>>>>> Here are the results from gdb: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [knl00 tests] gdb ../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 >>>>>>>>>>>>>> Copyright (C) 2013 Free Software Foundation, Inc. >>>>>>>>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>>>>>>>> This is free software: you are free to change and >>>>>>>>>>>>>> redistribute it. >>>>>>>>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type"show >>>>>>>>>>>>>> copying" >>>>>>>>>>>>>> and"show warranty" for details. >>>>>>>>>>>>>> This GDB was configured as"x86_64-redhat-linux-gnu". >>>>>>>>>>>>>> For bug reporting instructions, please see: >>>>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/> >>>>>>>>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>>>>>>>> Reading symbols from >>>>>>>>>>>>>> /home/b21541/gfmc/mc/x.testwf.12c-11_states-pn...done. >>>>>>>>>>>>>> (gdb) run >>>>>>>>>>>>>> Starting program: >>>>>>>>>>>>>> /home/b21541/gfmc/tests/../mc/x.testwf.12c-11_states-pn >>>>>>>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>>>>>>> Using host libthread_db library"/lib64/libthread_db.so.1". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>>>>>>>>> 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>>>> 13 use random2 >>>>>>>>>>>>>> Missing separate debuginfos, use: debuginfo-install >>>>>>>>>>>>>> glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 >>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>> >>>>>>>>>>>>>> (gdb) where >>>>>>>>>>>>>> #0 0x00000000004030f0 in _unnamed_main$$ () at testwf.f:13 >>>>>>>>>>>>>> #1 0x0000000000402c0e in main () >>>>>>>>>>>>>> #2 0x00007ffff53b7b15 in __libc_start_main () from >>>>>>>>>>>>>> /lib64/libc.so.6 >>>>>>>>>>>>>> #3 0x0000000000402b19 in _start () >>>>>>>>>>>>>> (gdb) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I can come over to Bldg 240 if that seems helpful. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>>> On 06/16/16 15:08, Luo, Ye wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Try mpiifort. >>>>>>>>>>>>>> You can verify the use of ifort by"mpiifort -show". >>>>>>>>>>>>>> Ye >>>>>>>>>>>>>> =================== >>>>>>>>>>>>>> Ye Luo, Ph.D. >>>>>>>>>>>>>> Leadership Computing Facility >>>>>>>>>>>>>> Argonne National Laboratory >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>> From: >>>>>>>>>>>>>> intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Steven C. >>>>>>>>>>>>>> Pieper [spieper(a)anl.gov] >>>>>>>>>>>>>> Sent: Thursday, June 16, 2016 3:04 PM >>>>>>>>>>>>>> To: Harms, Kevin; >>>>>>>>>>>>>> intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>; >>>>>>>>>>>>>> >>>>>>>>>>>>>> Osborn, James C. >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] Help on knl00 segfault >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Kevin, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I did the source command and ifort gets the beta compiler OK. >>>>>>>>>>>>>> However mpif90 >>>>>>>>>>>>>> is using gfortran: >>>>>>>>>>>>>> >>>>>>>>>>>>>> mpif90 -FR -g -O4 -mavx -xAVX -axAVX -qopenmp -lpthread >>>>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn >>>>>>>>>>>>>> -I../adlb/ -I../dmem/ -c kinetic.f >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-axAVX' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-qopenmp' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option >>>>>>>>>>>>>> '-qopenmp-report' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-r8' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-WB' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-traceback' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpe3' >>>>>>>>>>>>>> gfortran: error: unrecognized command line option '-fpp' >>>>>>>>>>>>>> make: *** [lib12c-11_states-pn.a(kinetic.o)] Error 1 >>>>>>>>>>>>>> [knl00 mc] which mpif90 >>>>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/mpi/intel64/bin/mpif90 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve >>>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>>> On 06/16/16 14:53, Harms, Kevin wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Oops. I forgot to put �intel64� at the end. The wiki has some >>>>>>>>>>>>>> useful notes on it: >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>>>>>>>>>>>> >>>>>>>>>>>>>> kevin >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve, >>>>>>>>>>>>>> >>>>>>>>>>>>>> First thing, try the latest compiler. >>>>>>>>>>>>>> >>>>>>>>>>>>>> source >>>>>>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The 2017 beta has explicit KNL support. Second, have you tried >>>>>>>>>>>>>> running it in gdb? >>>>>>>>>>>>>> >>>>>>>>>>>>>> kevin >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am trying to run a Fortran program that exercises some of my >>>>>>>>>>>>>> computational >>>>>>>>>>>>>> kernels. The program does not use MPI, just OpenMP. I have >>>>>>>>>>>>>> compiled >>>>>>>>>>>>>> it for a small case and that runs fine and gives the correct >>>>>>>>>>>>>> results. >>>>>>>>>>>>>> However when I recompile it for a realistic cases (the same >>>>>>>>>>>>>> source >>>>>>>>>>>>>> is used, only a bunch of parameter statements are different), it >>>>>>>>>>>>>> segfaults at line 13 of the main program: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [knl00 tests] ../mc/x.testwf.12c-11_states-pn c12_11states-pn.in >>>>>>>>>>>>>> forrtl: severe (174): SIGSEGV, segmentation fault occurred >>>>>>>>>>>>>> Image PC Routine Line Source >>>>>>>>>>>>>> >>>>>>>>>>>>>> libintlc.so.5 00007F569D060C05 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libintlc.so.5 00007F569D05E827 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DEE1962 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DEE17B6 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DE36CAD Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libifcoremt.so.5 00007F569DE48FAD Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libpthread.so.0 00007F569F2D7100 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> x.testwf.12c-11_s 00000000004030F0 MAIN__ >>>>>>>>>>>>>> 13 testwf.f >>>>>>>>>>>>>> x.testwf.12c-11_s 0000000000402C0E Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> libc.so.6 00007F569CAA3B15 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> x.testwf.12c-11_s 0000000000402B19 Unknown >>>>>>>>>>>>>> Unknown Unknown >>>>>>>>>>>>>> Aborted (core dumped) >>>>>>>>>>>>>> [knl00 tests] >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Line 13 is the very beginning of declarations, not an executable >>>>>>>>>>>>>> statement. >>>>>>>>>>>>>> It is the first non-comment line and is: >>>>>>>>>>>>>> >>>>>>>>>>>>>> use random2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> random2 is a module that defines some subroutines. It is not >>>>>>>>>>>>>> changed >>>>>>>>>>>>>> by the parameter statements that are different for the working >>>>>>>>>>>>>> and failing >>>>>>>>>>>>>> cases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here are the working and failing executable sizes: >>>>>>>>>>>>>> >>>>>>>>>>>>>> text data bss dec hex filename >>>>>>>>>>>>>> 2459352 124772 24979752 27563876 1a49764 >>>>>>>>>>>>>> x.testwf.4he-pn >>>>>>>>>>>>>> 2192689 92004 623277384 625562077 254951dd >>>>>>>>>>>>>> x.testwf.12c-11_states-pn >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is how the main program is compiled and module linked: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ifort -Imods.libgen -FR -g -O0 -qopenmp -lpthread >>>>>>>>>>>>>> -qopenmp-report -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>>>>>> -fpp -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>>>>>> -DM0_DIV=1 -DDOCASE=0 >>>>>>>>>>>>>> -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c -g testwf.f >>>>>>>>>>>>>> >>>>>>>>>>>>>> ifort -g -mcmodel=medium -traceback -fpe3 -qopenmp -lpthread >>>>>>>>>>>>>> -qopenmp-report -o x.testwf.12c-11_states-pn testwf.o >>>>>>>>>>>>>> do_adlb_no.o dmem_fake.o adlb_fake.o \ >>>>>>>>>>>>>> ../common/wall_second-nompi.o ../common/stopnompi.o >>>>>>>>>>>>>> ../common/mpi_wtime_fake.o -L . -l12c-11_states-pn -lgen >>>>>>>>>>>>>> -l12c-11_states-pn -L ../common/ -lcomgen enable_abort_fake.o >>>>>>>>>>>>>> -lsvml >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [knl00 mc] which ifort >>>>>>>>>>>>>> /soft/compilers/intel/bin/ifort >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is in ~b21541/gfmc/mc . >>>>>>>>>>>>>> >>>>>>>>>>>>>> This test case runs fine on blues. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any suggestions will be gratefully received. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steve >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Steven C. Pieper, Senior Physicist SO:[email protected] >>>>>>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>>>>>> Argonne, IL 60439 >>>>>>>>>>>>>> Phone: 630-252-4232 Fax -6008 >>>>>>>>>>>>>> Secretary, Debra Beres, -4100 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Intel-nda mailing list >>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> >>>>> >>>> >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0