October 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Fortran and shared libraries
by Steven Pieper 18 Oct '16

18 Oct '16

Are you running interactively or by a script? I have your source command and an mpirun command (without -F) in a script and it works fine. I also have the same source command in my .bash_profile but I don't think that is relevant. Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 10/18/2016 09:01 AM, Tom LeCompte wrote: > Hi everyone- > > I already have a workaround, but I am puzzled as to why I need one. > > If I compile with: > > mpiifort -O3 -xMIC_AVX512 -mcmodel medium -shared-intel -o toys mt19937.o toys.f > > and run with > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > mpirun -f $COBALT_NODEFILE -ppn 2 ./toys > > I get the error (twice) > > ./toys: error while loading shared libraries: libifport.so.5: cannot open shared object file: No such file or directory > > This persists when I put /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/compiler/lib/intel64_lin_mic in my LD_LIBRARY_PATH, and even when I copy libifport.so.5 into the run area and put the run area in my LD_LIBRARY_PATH. Echoing $LD_LIBRARY_PATH shows that this "took". > > If I instead compile with: > > mpiifort -O3 -xMIC_AVX512 -static-intel -o toys mt19937.o toys.f > > It runs perfectly. > > Any ideas? > > Cheers, > > Tom > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Fortran and shared libraries
by Tom LeCompte 18 Oct '16

18 Oct '16

Hi everyone- I already have a workaround, but I am puzzled as to why I need one. If I compile with: mpiifort -O3 -xMIC_AVX512 -mcmodel medium -shared-intel -o toys mt19937.o toys.f and run with source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh mpirun -f $COBALT_NODEFILE -ppn 2 ./toys I get the error (twice) ./toys: error while loading shared libraries: libifport.so.5: cannot open shared object file: No such file or directory This persists when I put /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/compiler/lib/intel64_lin_mic in my LD_LIBRARY_PATH, and even when I copy libifport.so.5 into the run area and put the run area in my LD_LIBRARY_PATH. Echoing $LD_LIBRARY_PATH shows that this "took". If I instead compile with: mpiifort -O3 -xMIC_AVX512 -static-intel -o toys mt19937.o toys.f It runs perfectly. Any ideas? Cheers, Tom

1 0

Re: [JLSE-Intel-NDA] Space Request
by Allen, Benjamin S. 11 Oct '16

11 Oct '16

Hi Taylor, The KNL nodes have local 200GB SSDs there are formatted on each boot. On the short list of project we're adding a replacement NetApp filer for /home and /soft use, but Theta work has delayed this project. Ben > On Oct 10, 2016, at 4:49 PM, Childers, Taylor <jchilders(a)anl.gov> wrote: > > Hello, > I'd like to install our experimental framework and run some tests on the KNL, however this framework is between 250-500GB. Given our recent emails about space on the jlse:/home area, is there a better place for me to install this software? > Thanks, > Taylor > > > J. Taylor Childers, Assistant Physicist > Argonne National Laboratory > HEP Division Bldg. 360, Rm. C-217 > Office: (630) 252-4212 Mobile: (331) 302-4647 > jchilders(a)anl.gov > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Space Request
by Childers, Taylor 10 Oct '16

10 Oct '16

Hello, I'd like to install our experimental framework and run some tests on the KNL, however this framework is between 250-500GB. Given our recent emails about space on the jlse:/home area, is there a better place for me to install this software? Thanks, Taylor J. Taylor Childers, Assistant Physicist Argonne National Laboratory HEP Division Bldg. 360, Rm. C-217 Office: (630) 252-4212 Mobile: (331) 302-4647 jchilders(a)anl.gov<mailto:[email protected]>

1 0

Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults
by Steven C. Pieper 07 Oct '16

07 Oct '16

Kalyan, At least one of my problems has been fixed in a newer version of 17.0: This version of the compiler (version 17.0 Build 20161005 [I think he pasted the wrong thing here]) crashes. However, I also checked the latest compiler build and it looks like the problem is fixed there: $ ./build.sh $ ifort -V Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0 Build 20161005 Copyright (C) 1985-2016 Intel Corporation. All rights reserved. So, please expect the fix available in the upcoming update package. I will escalate this to the enigneering team anyway to double check that. I will investigate the problem with wrong results next week and update you as well. Any idea of when we will get this? Thanks, Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 10/06/16 22:01, Allen, Benjamin S. wrote: > License has been updated: > > [bsallen@jlselogin2 ~]$ ifort -V > Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 > Copyright (C) 1985-2016 Intel Corporation. All rights reserved. > > ifort: NOTE: The Beta evaluation period for this product ends on 6-nov-2016 UTC. > > Ben > >> On Oct 6, 2016, at 2:30 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: >> >> Hi Jeff, >> >> I put !DIR$ NOVECTOR in front of every do loop (including all the nested do's). >> It still gives the wrong result. The optimization report acknowledges some >> of my !DIRS but only a few. Many of the other loops were unrolled, which >> I guess is not vectorization. But I don't know why -no-vector on the compile >> line has more effect. >> >> Steve >> Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> On 10/06/16 13:13, Hammond, Jeff R wrote: >>> Thanks, Steve. >>> >>> I don’t know how many loops are in the relevant source file, but if you remove the -no-vec flag and bisect the code with “!DIR$ NOVECTOR” (https://software.intel.com/en-us/node/680017) on loops, you can localize the problem. >>> >>> Martyn noted that one regression from beta to 17.0 release has already been fixed and will be in the update. It is possible that this will fix your problem. I have no ETA for 17.0 update 1 but my guess is that it will be available soon, based upon two important bug fixes I know about. >>> >>> The license issue was addressed by Martyn already. Kumar and Ben have details. I assume they will update the license file, but in the event that your beta compiler stops working tomorrow, you can use the file-specific -no-vec workaround, or a loop-specific NOVECTOR if you can find where it is necessary. >>> >>> Best, >>> >>> Jeff >>> >>> From: "Steven C. Pieper" <spieper(a)anl.gov> >>> Organization: Physics, Argonne National Laboratory >>> Reply-To: "spieper(a)anl.gov" <spieper(a)anl.gov> >>> Date: Thursday, October 6, 2016 at 11:07 AM >>> To: Jeff R Hammond <jeff.r.hammond(a)intel.com>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov> >>> Cc: "intel-nda(a)jlse.anl.gov" <intel-nda(a)jlse.anl.gov> >>> Subject: Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults >>> >>> Using -no-vec does fix the problem. >>> >>> I have just submitted this and the compiler segfault bugs to the Intel site: >>> >>> reference number is : 6000163313. >>> >>> I included a 7 MB tar file which has a readme. >>> >>> The jlse beta version still says: >>> >>> [jlselogin1 ~] ifort -V >>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>> >>> ifort: NOTE: The Beta evaluation period for this product ends on 7-oct-2016 UTC. >>> >>> >>> Steve >>> >>> Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov >>> >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>> >>> http://www.phy.anl.gov/theory/staff/scp.html >>> On 10/05/16 18:35, Hammond, Jeff R wrote: >>>> Martyn suggests you try with -no-vec. If that also fixes the issue, then >>>> you can narrow it down to the specific loop that breaks with novector >>>> directives. >>>> >>>> Martyn said the beta license situation was fixed. I don’t yet know if any >>>> action is required on ALCF’s part. >>>> >>>> Jeff >>>> >>>> On 10/5/16, 1:04 PM, "Steven C. Pieper" >>>> <spieper(a)anl.gov> >>>> wrote: >>>> >>>> >>>>> Jeff and/or Kalyan: >>>>> >>>>> I have isolated the wrong numerical result to one (large) subroutine. >>>>> It is not the same one that causes the compiler to segfault; that >>>>> subroutine is not involved in this wrong numerical result. >>>>> I can compile everything with the production v17 using -O2 and openmp >>>>> (I haven't tried more optimization yet). Then if I compile this >>>>> one subroutine with -O0 and no openmp, it works. But if I compile >>>>> it with -O2 and no openmp, it gives the wrong numbers. So this is >>>>> not an OpenMP bug; it is a -O2 optimization error. This is >>>>> a bit of a mess to make a demo for -- the subroutine is large and >>>>> has a big table of input indices, etc., and it needs lots of other >>>>> subroutines to have run before it is called. >>>>> >>>>> As I said, all of this works fine with -O3, AVX512 with the v17-beta >>>>> compiler. >>>>> >>>>> I don't know what to do next. I can make an ugly package to give >>>>> to you for Intel, but making a simple demo is probably too difficult. >>>>> Is there any further update to the production v17 that could be >>>>> installed for me to try? >>>>> >>>>> Steve >>>>> >>>>> >>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>> >>>>> spieper(a)anl.gov >>>>> >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>> 60439 >>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>>>> -4100 >>>>> >>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>> >>>>> >>>>> On 10/05/16 13:40, Hammond, Jeff R wrote: >>>>> >>>>>> On 10/5/16, 11:04 AM, >>>>>> "intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>>>> Kumaran, Kalyan" >>>>>> >>>>>> <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>>>> kumaran(a)alcf.anl.gov> >>>>>> wrote: >>>>>> >>>>>> >>>>>>> Steve, are you saying the Beta compilers have been removed, or will be >>>>>>> removed in 2 days, from the JLSE cluster? If this is the case I can >>>>>>> request Ben to keep it though I have not see any message about removing >>>>>>> it. >>>>>>> >>>>>> I asked Martyn about this just now. Will post the substance of his >>>>>> response as soon as I get it. >>>>>> >>>>>> >>>>>>> Please do report bugs on the GA compilers. Without these I fear it will >>>>>>> never get fixed. >>>>>>> >>>>>> Please report every bug to IPS. It¹s possible that the same bug will be >>>>>> identified by a different codes, but it is not good to rely upon this. >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>>> -Kumar >>>>>>> >>>>>>> >>>>>>>> On Oct 5, 2016, at 12:51 PM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Tim, >>>>>>>> I didn't know where to send it. I just want it to be worked on. >>>>>>>> The non-beta V17 compiler is on jlse -- that is what is failing for >>>>>>>> me. >>>>>>>> >>>>>>>> Some more recent info: >>>>>>>> Only this one subroutine file causes the compiler to segfault. >>>>>>>> >>>>>>>> However much more seriously, I am getting the wrong numerical >>>>>>>> results >>>>>>>> using the non-beta compiler. They are wrong in that they differ >>>>>>>> from both the v17 beta and xlf on cetus. v17 beta and cetus give >>>>>>>> the same results. I'm trying now to narrow down where the wrong >>>>>>>> results are coming from. >>>>>>>> >>>>>>>> Can we keep the beta compiler going for more than 2 more days???? >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>>> >>>>>>>> spieper(a)anl.gov >>>>>>>> >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>>>>> 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>>>>>>> -4100 >>>>>>>> >>>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>>> >>>>>>>> >>>>>>>> On 10/05/16 12:45, Williams, Timothy J. wrote: >>>>>>>> >>>>>>>>> Hi Steve, >>>>>>>>> >>>>>>>>> Did you mean to send this to early-users-discuss or intel-nda? >>>>>>>>> >>>>>>>>> I think we should now have the non-beta V17 compilers on the white >>>>>>>>> boxes. Will try to find out. >>>>>>>>> >>>>>>>>> Tim >>>>>>>>> >>>>>>>>> ------- Timothy J. Williams, Ph.D. --- Argonne Leadership Computing >>>>>>>>> Facility --- >>>>>>>>> tjwilliams(a)anl.gov >>>>>>>>> --- 630-252-1154 ------- >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Oct 5, 2016, at 11:25 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> I am trying to recompile everything using the production intel >>>>>>>>>> compiler >>>>>>>>>> rather than the beta version which I previously used. >>>>>>>>>> >>>>>>>>>> With the beta version, it works: >>>>>>>>>> >>>>>>>>>> [jlselogin2 mc] source >>>>>>>>>> >>>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compiler >>>>>>>>>> va >>>>>>>>>> rs.sh intel64 >>>>>>>>>> [jlselogin2 mc] ifort -V >>>>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>>>> Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>>>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>>>> >>>>>>>>>> ifort: NOTE: The Beta evaluation period for this product ends on >>>>>>>>>> 7-oct-2016 UTC. >>>>>>>>>> >>>>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>>>> make_3pi.f >>>>>>>>>> [jlselogin2 mc] <<compile is OK> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The production verson segfaults: >>>>>>>>>> >>>>>>>>>> [jlselogin2 mc] source >>>>>>>>>> >>>>>>>>>> /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars. >>>>>>>>>> sh >>>>>>>>>> intel64[jlselogin2 mc] ifort -V >>>>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>>>> Intel(R) 64, Version 17.0.0.098 Build 20160721 >>>>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>>>> >>>>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>>>> make_3pi.f >>>>>>>>>> catastrophic error: **Internal compiler error: segmentation >>>>>>>>>> violation >>>>>>>>>> signal raised** Please report this error along with the >>>>>>>>>> circumstances >>>>>>>>>> in which it occurred in a Software Problem Report. Note: File and >>>>>>>>>> line >>>>>>>>>> given may not be explicit cause of this error. >>>>>>>>>> compilation aborted for make_3pi.f (code 1) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If I remove the -xMIC-AVX512 then the production version >>>>>>>>>> compiles without error. >>>>>>>>>> >>>>>>>>>> Lots of other subroutines are compiling OK. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The files are all in ~b21541/gfmc/mc/ >>>>>>>>>> >>>>>>>>>> I see the beta version is due to expire in 2 days. Can this be >>>>>>>>>> extended? >>>>>>>>>> >>>>>>>>>> Steve >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>>>>> >>>>>>>>>> spieper(a)anl.gov >>>>>>>>>> >>>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>>>> IL >>>>>>>>>> 60439 >>>>>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra >>>>>>>>>> Beres, >>>>>>>>>> -4100 >>>>>>>>>> >>>>>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> EarlyScience mailing list >>>>>>>>>> >>>>>>>>>> [email protected]://lists.alcf.anl.gov/mailman/listinfo/earlyscience >>>>>>>> _______________________________________________ >>>>>>>> Intel-nda mailing list >>>>>>>> >>>>>>>> [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> >>>>>>> [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults
by Allen, Benjamin S. 06 Oct '16

06 Oct '16

License has been updated: [bsallen@jlselogin2 ~]$ ifort -V Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 Copyright (C) 1985-2016 Intel Corporation. All rights reserved. ifort: NOTE: The Beta evaluation period for this product ends on 6-nov-2016 UTC. Ben > On Oct 6, 2016, at 2:30 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > Hi Jeff, > > I put !DIR$ NOVECTOR in front of every do loop (including all the nested do's). > It still gives the wrong result. The optimization report acknowledges some > of my !DIRS but only a few. Many of the other loops were unrolled, which > I guess is not vectorization. But I don't know why -no-vector on the compile > line has more effect. > > Steve > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > On 10/06/16 13:13, Hammond, Jeff R wrote: >> Thanks, Steve. >> >> I don’t know how many loops are in the relevant source file, but if you remove the -no-vec flag and bisect the code with “!DIR$ NOVECTOR” (https://software.intel.com/en-us/node/680017) on loops, you can localize the problem. >> >> Martyn noted that one regression from beta to 17.0 release has already been fixed and will be in the update. It is possible that this will fix your problem. I have no ETA for 17.0 update 1 but my guess is that it will be available soon, based upon two important bug fixes I know about. >> >> The license issue was addressed by Martyn already. Kumar and Ben have details. I assume they will update the license file, but in the event that your beta compiler stops working tomorrow, you can use the file-specific -no-vec workaround, or a loop-specific NOVECTOR if you can find where it is necessary. >> >> Best, >> >> Jeff >> >> From: "Steven C. Pieper" <spieper(a)anl.gov> >> Organization: Physics, Argonne National Laboratory >> Reply-To: "spieper(a)anl.gov" <spieper(a)anl.gov> >> Date: Thursday, October 6, 2016 at 11:07 AM >> To: Jeff R Hammond <jeff.r.hammond(a)intel.com>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov> >> Cc: "intel-nda(a)jlse.anl.gov" <intel-nda(a)jlse.anl.gov> >> Subject: Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults >> >> Using -no-vec does fix the problem. >> >> I have just submitted this and the compiler segfault bugs to the Intel site: >> >> reference number is : 6000163313. >> >> I included a 7 MB tar file which has a readme. >> >> The jlse beta version still says: >> >> [jlselogin1 ~] ifort -V >> Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >> >> ifort: NOTE: The Beta evaluation period for this product ends on 7-oct-2016 UTC. >> >> >> Steve >> >> Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> On 10/05/16 18:35, Hammond, Jeff R wrote: >>> Martyn suggests you try with -no-vec. If that also fixes the issue, then >>> you can narrow it down to the specific loop that breaks with novector >>> directives. >>> >>> Martyn said the beta license situation was fixed. I don’t yet know if any >>> action is required on ALCF’s part. >>> >>> Jeff >>> >>> On 10/5/16, 1:04 PM, "Steven C. Pieper" >>> <spieper(a)anl.gov> >>> wrote: >>> >>> >>>> Jeff and/or Kalyan: >>>> >>>> I have isolated the wrong numerical result to one (large) subroutine. >>>> It is not the same one that causes the compiler to segfault; that >>>> subroutine is not involved in this wrong numerical result. >>>> I can compile everything with the production v17 using -O2 and openmp >>>> (I haven't tried more optimization yet). Then if I compile this >>>> one subroutine with -O0 and no openmp, it works. But if I compile >>>> it with -O2 and no openmp, it gives the wrong numbers. So this is >>>> not an OpenMP bug; it is a -O2 optimization error. This is >>>> a bit of a mess to make a demo for -- the subroutine is large and >>>> has a big table of input indices, etc., and it needs lots of other >>>> subroutines to have run before it is called. >>>> >>>> As I said, all of this works fine with -O3, AVX512 with the v17-beta >>>> compiler. >>>> >>>> I don't know what to do next. I can make an ugly package to give >>>> to you for Intel, but making a simple demo is probably too difficult. >>>> Is there any further update to the production v17 that could be >>>> installed for me to try? >>>> >>>> Steve >>>> >>>> >>>> Steven C. Pieper, Senior Physicist Emeritus: >>>> >>>> spieper(a)anl.gov >>>> >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>> 60439 >>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>>> -4100 >>>> >>>> http://www.phy.anl.gov/theory/staff/scp.html >>>> >>>> >>>> On 10/05/16 13:40, Hammond, Jeff R wrote: >>>> >>>>> On 10/5/16, 11:04 AM, >>>>> "intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>>> Kumaran, Kalyan" >>>>> >>>>> <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>>> kumaran(a)alcf.anl.gov> >>>>> wrote: >>>>> >>>>> >>>>>> Steve, are you saying the Beta compilers have been removed, or will be >>>>>> removed in 2 days, from the JLSE cluster? If this is the case I can >>>>>> request Ben to keep it though I have not see any message about removing >>>>>> it. >>>>>> >>>>> I asked Martyn about this just now. Will post the substance of his >>>>> response as soon as I get it. >>>>> >>>>> >>>>>> Please do report bugs on the GA compilers. Without these I fear it will >>>>>> never get fixed. >>>>>> >>>>> Please report every bug to IPS. It¹s possible that the same bug will be >>>>> identified by a different codes, but it is not good to rely upon this. >>>>> >>>>> Jeff >>>>> >>>>> >>>>>> -Kumar >>>>>> >>>>>> >>>>>>> On Oct 5, 2016, at 12:51 PM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>> Tim, >>>>>>> I didn't know where to send it. I just want it to be worked on. >>>>>>> The non-beta V17 compiler is on jlse -- that is what is failing for >>>>>>> me. >>>>>>> >>>>>>> Some more recent info: >>>>>>> Only this one subroutine file causes the compiler to segfault. >>>>>>> >>>>>>> However much more seriously, I am getting the wrong numerical >>>>>>> results >>>>>>> using the non-beta compiler. They are wrong in that they differ >>>>>>> from both the v17 beta and xlf on cetus. v17 beta and cetus give >>>>>>> the same results. I'm trying now to narrow down where the wrong >>>>>>> results are coming from. >>>>>>> >>>>>>> Can we keep the beta compiler going for more than 2 more days???? >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>> >>>>>>> spieper(a)anl.gov >>>>>>> >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>>>> 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>>>>>> -4100 >>>>>>> >>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>> >>>>>>> >>>>>>> On 10/05/16 12:45, Williams, Timothy J. wrote: >>>>>>> >>>>>>>> Hi Steve, >>>>>>>> >>>>>>>> Did you mean to send this to early-users-discuss or intel-nda? >>>>>>>> >>>>>>>> I think we should now have the non-beta V17 compilers on the white >>>>>>>> boxes. Will try to find out. >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> ------- Timothy J. Williams, Ph.D. --- Argonne Leadership Computing >>>>>>>> Facility --- >>>>>>>> tjwilliams(a)anl.gov >>>>>>>> --- 630-252-1154 ------- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 5, 2016, at 11:25 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I am trying to recompile everything using the production intel >>>>>>>>> compiler >>>>>>>>> rather than the beta version which I previously used. >>>>>>>>> >>>>>>>>> With the beta version, it works: >>>>>>>>> >>>>>>>>> [jlselogin2 mc] source >>>>>>>>> >>>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compiler >>>>>>>>> va >>>>>>>>> rs.sh intel64 >>>>>>>>> [jlselogin2 mc] ifort -V >>>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>>> Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>>> >>>>>>>>> ifort: NOTE: The Beta evaluation period for this product ends on >>>>>>>>> 7-oct-2016 UTC. >>>>>>>>> >>>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>>> make_3pi.f >>>>>>>>> [jlselogin2 mc] <<compile is OK> >>>>>>>>> >>>>>>>>> >>>>>>>>> The production verson segfaults: >>>>>>>>> >>>>>>>>> [jlselogin2 mc] source >>>>>>>>> >>>>>>>>> /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars. >>>>>>>>> sh >>>>>>>>> intel64[jlselogin2 mc] ifort -V >>>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>>> Intel(R) 64, Version 17.0.0.098 Build 20160721 >>>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>>> >>>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>>> make_3pi.f >>>>>>>>> catastrophic error: **Internal compiler error: segmentation >>>>>>>>> violation >>>>>>>>> signal raised** Please report this error along with the >>>>>>>>> circumstances >>>>>>>>> in which it occurred in a Software Problem Report. Note: File and >>>>>>>>> line >>>>>>>>> given may not be explicit cause of this error. >>>>>>>>> compilation aborted for make_3pi.f (code 1) >>>>>>>>> >>>>>>>>> >>>>>>>>> If I remove the -xMIC-AVX512 then the production version >>>>>>>>> compiles without error. >>>>>>>>> >>>>>>>>> Lots of other subroutines are compiling OK. >>>>>>>>> >>>>>>>>> >>>>>>>>> The files are all in ~b21541/gfmc/mc/ >>>>>>>>> >>>>>>>>> I see the beta version is due to expire in 2 days. Can this be >>>>>>>>> extended? >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>>>> >>>>>>>>> spieper(a)anl.gov >>>>>>>>> >>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>>> IL >>>>>>>>> 60439 >>>>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra >>>>>>>>> Beres, >>>>>>>>> -4100 >>>>>>>>> >>>>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> EarlyScience mailing list >>>>>>>>> >>>>>>>>> [email protected]://lists.alcf.anl.gov/mailman/listinfo/earlyscience >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> >>>>>>> [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> >>>>>> [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults
by Steven C. Pieper 06 Oct '16

06 Oct '16

Hi Jeff, I put !DIR$ NOVECTOR in front of every do loop (including all the nested do's). It still gives the wrong result. The optimization report acknowledges some of my !DIRS but only a few. Many of the other loops were unrolled, which I guess is not vectorization. But I don't know why -no-vector on the compile line has more effect. Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 10/06/16 13:13, Hammond, Jeff R wrote: > Thanks, Steve. > > I don’t know how many loops are in the relevant source file, but if > you remove the -no-vec flag and bisect the code with “!DIR$ NOVECTOR” > (https://software.intel.com/en-us/node/680017 > <https://software.intel.com/en-us/node/680017?language=en>) on loops, > you can localize the problem. > > Martyn noted that one regression from beta to 17.0 release has already > been fixed and will be in the update. It is possible that this will > fix your problem. I have no ETA for 17.0 update 1 but my guess is > that it will be available soon, based upon two important bug fixes I > know about. > > The license issue was addressed by Martyn already. Kumar and Ben have > details. I assume they will update the license file, but in the event > that your beta compiler stops working tomorrow, you can use the > file-specific -no-vec workaround, or a loop-specific NOVECTOR if you > can find where it is necessary. > > Best, > > Jeff > > From: "Steven C. Pieper" <spieper(a)anl.gov <mailto:[email protected]>> > Organization: Physics, Argonne National Laboratory > Reply-To: "spieper(a)anl.gov <mailto:[email protected]>" <spieper(a)anl.gov > <mailto:[email protected]>> > Date: Thursday, October 6, 2016 at 11:07 AM > To: Jeff R Hammond <jeff.r.hammond(a)intel.com > <mailto:[email protected]>>, "Kumaran, Kalyan" > <kumaran(a)alcf.anl.gov <mailto:[email protected]>> > Cc: "intel-nda(a)jlse.anl.gov <mailto:[email protected]>" > <intel-nda(a)jlse.anl.gov <mailto:[email protected]>> > Subject: Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler > segfaults > > Using -no-vec does fix the problem. > > I have just submitted this and the compiler segfault bugs to the > Intel site: > > reference number is : 6000163313. > > I included a 7 MB tar file which has a readme. > > The jlse beta version still says: > > [jlselogin1 ~] ifort -V > Intel(R) Fortran Intel(R) 64 Compiler for applications running on > Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 > Copyright (C) 1985-2016 Intel Corporation. All rights reserved. > > ifort: NOTE: The Beta evaluation period for this product ends on > 7-oct-2016 UTC. > > > Steve > > Steven C. Pieper, Senior Physicist Emeritus:[email protected] > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > http://www.phy.anl.gov/theory/staff/scp.html > > On 10/05/16 18:35, Hammond, Jeff R wrote: >> Martyn suggests you try with -no-vec. If that also fixes the issue, then >> you can narrow it down to the specific loop that breaks with novector >> directives. >> >> Martyn said the beta license situation was fixed. I don’t yet know if any >> action is required on ALCF’s part. >> >> Jeff >> >> On 10/5/16, 1:04 PM, "Steven C. Pieper"<spieper(a)anl.gov> wrote: >> >>> Jeff and/or Kalyan: >>> >>> I have isolated the wrong numerical result to one (large) subroutine. >>> It is not the same one that causes the compiler to segfault; that >>> subroutine is not involved in this wrong numerical result. >>> I can compile everything with the production v17 using -O2 and openmp >>> (I haven't tried more optimization yet). Then if I compile this >>> one subroutine with -O0 and no openmp, it works. But if I compile >>> it with -O2 and no openmp, it gives the wrong numbers. So this is >>> not an OpenMP bug; it is a -O2 optimization error. This is >>> a bit of a mess to make a demo for -- the subroutine is large and >>> has a big table of input indices, etc., and it needs lots of other >>> subroutines to have run before it is called. >>> >>> As I said, all of this works fine with -O3, AVX512 with the v17-beta >>> compiler. >>> >>> I don't know what to do next. I can make an ugly package to give >>> to you for Intel, but making a simple demo is probably too difficult. >>> Is there any further update to the production v17 that could be >>> installed for me to try? >>> >>> Steve >>> >>> >>> Steven C. Pieper, Senior Physicist Emeritus: >>> spieper(a)anl.gov >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>> 60439 >>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>> -4100 >>> http://www.phy.anl.gov/theory/staff/scp.html >>> >>> On 10/05/16 13:40, Hammond, Jeff R wrote: >>>> On 10/5/16, 11:04 AM,"intel-nda-bounces(a)lists.jlse.anl.gov on behalf of Kumaran, Kalyan" <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>> kumaran(a)alcf.anl.gov> wrote: >>>> >>>>> Steve, are you saying the Beta compilers have been removed, or will be >>>>> removed in 2 days, from the JLSE cluster? If this is the case I can >>>>> request Ben to keep it though I have not see any message about removing >>>>> it. >>>> I asked Martyn about this just now. Will post the substance of his >>>> response as soon as I get it. >>>> >>>>> Please do report bugs on the GA compilers. Without these I fear it will >>>>> never get fixed. >>>> Please report every bug to IPS. It¹s possible that the same bug will be >>>> identified by a different codes, but it is not good to rely upon this. >>>> >>>> Jeff >>>> >>>>> -Kumar >>>>> >>>>>> On Oct 5, 2016, at 12:51 PM, Steven C. Pieper<spieper(a)anl.gov> wrote: >>>>>> >>>>>> Tim, >>>>>> I didn't know where to send it. I just want it to be worked on. >>>>>> The non-beta V17 compiler is on jlse -- that is what is failing for >>>>>> me. >>>>>> >>>>>> Some more recent info: >>>>>> Only this one subroutine file causes the compiler to segfault. >>>>>> >>>>>> However much more seriously, I am getting the wrong numerical >>>>>> results >>>>>> using the non-beta compiler. They are wrong in that they differ >>>>>> from both the v17 beta and xlf on cetus. v17 beta and cetus give >>>>>> the same results. I'm trying now to narrow down where the wrong >>>>>> results are coming from. >>>>>> >>>>>> Can we keep the beta compiler going for more than 2 more days???? >>>>>> >>>>>> Steve >>>>>> >>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>> spieper(a)anl.gov >>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>>> 60439 >>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>>>>> -4100 >>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>> >>>>>> On 10/05/16 12:45, Williams, Timothy J. wrote: >>>>>>> Hi Steve, >>>>>>> >>>>>>> Did you mean to send this to early-users-discuss or intel-nda? >>>>>>> >>>>>>> I think we should now have the non-beta V17 compilers on the white >>>>>>> boxes. Will try to find out. >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> ------- Timothy J. Williams, Ph.D. --- Argonne Leadership Computing >>>>>>> Facility ---tjwilliams(a)anl.gov --- 630-252-1154 ------- >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Oct 5, 2016, at 11:25 AM, Steven C. Pieper<spieper(a)anl.gov> >>>>>>>> wrote: >>>>>>>> >>>>>>>> I am trying to recompile everything using the production intel >>>>>>>> compiler >>>>>>>> rather than the beta version which I previously used. >>>>>>>> >>>>>>>> With the beta version, it works: >>>>>>>> >>>>>>>> [jlselogin2 mc] source >>>>>>>> >>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compiler >>>>>>>> va >>>>>>>> rs.sh intel64 >>>>>>>> [jlselogin2 mc] ifort -V >>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>> Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>> >>>>>>>> ifort: NOTE: The Beta evaluation period for this product ends on >>>>>>>> 7-oct-2016 UTC. >>>>>>>> >>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>> make_3pi.f >>>>>>>> [jlselogin2 mc] <<compile is OK> >>>>>>>> >>>>>>>> >>>>>>>> The production verson segfaults: >>>>>>>> >>>>>>>> [jlselogin2 mc] source >>>>>>>> >>>>>>>> /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars. >>>>>>>> sh >>>>>>>> intel64[jlselogin2 mc] ifort -V >>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>> Intel(R) 64, Version 17.0.0.098 Build 20160721 >>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>> >>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>> make_3pi.f >>>>>>>> catastrophic error: **Internal compiler error: segmentation >>>>>>>> violation >>>>>>>> signal raised** Please report this error along with the >>>>>>>> circumstances >>>>>>>> in which it occurred in a Software Problem Report. Note: File and >>>>>>>> line >>>>>>>> given may not be explicit cause of this error. >>>>>>>> compilation aborted for make_3pi.f (code 1) >>>>>>>> >>>>>>>> >>>>>>>> If I remove the -xMIC-AVX512 then the production version >>>>>>>> compiles without error. >>>>>>>> >>>>>>>> Lots of other subroutines are compiling OK. >>>>>>>> >>>>>>>> >>>>>>>> The files are all in ~b21541/gfmc/mc/ >>>>>>>> >>>>>>>> I see the beta version is due to expire in 2 days. Can this be >>>>>>>> extended? >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>>> spieper(a)anl.gov >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>> IL >>>>>>>> 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra >>>>>>>> Beres, >>>>>>>> -4100 >>>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> EarlyScience mailing list >>>>>>>> [email protected]://lists.alcf.anl.gov/mailman/listinfo/earlyscience >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults
by Hammond, Jeff R 06 Oct '16

06 Oct '16

Can you try with -no-vec? Don’t try to isolate. If that flag fixes it, it is likely it’s the same bug. Jeff On 10/6/16, 11:19 AM, "Kenneth E Jansen" <kenneth.jansen(a)colorado.edu> wrote: >Sorry if this question has already been resolved but I do hope we can >keep the beta license alive. > >I have also not had the time to get back to doing Intel’s requested >trouble shooting of the crashes on multinode with 17 Initial Release >that do not happen with the beta. I did submit a ticket but the >response was to do a lot of unit testing and this has just not made it to >the top of my to do list. Like Steve’s case, my code is rather complex >and not so easy to isolate what could be causing multinode crashes with >the released verstion that just don’t happen with the beta version. > >I will try to get back to that testing when I can. > >Best regards, > >Ken > >---------------------------------------------------------------- >Kenneth E. Jansen, Professor of Aerospace Engineering Sciences >ECAE 161 OFFICE (303) 492-4359 >429 UCB FAX (303) 492-4990 >University of Colorado at Boulder jansenke(a)colorado.edu >Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace >---------------------------------------------------------------- > > > >> On Oct 6, 2016, at 12:07 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: >> >> Using -no-vec does fix the problem. >> >> I have just submitted this and the compiler segfault bugs to the Intel >>site: >> >> reference number is : 6000163313. >> >> I included a 7 MB tar file which has a readme. >> >> The jlse beta version still says: >> >> [jlselogin1 ~] ifort -V >> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >> >> ifort: NOTE: The Beta evaluation period for this product ends on >>7-oct-2016 UTC. >> >> >> Steve >> >> Steven C. Pieper, Senior Physicist Emeritus: >>spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>-4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> On 10/05/16 18:35, Hammond, Jeff R wrote: >>> Martyn suggests you try with -no-vec. If that also fixes the issue, >>>then >>> you can narrow it down to the specific loop that breaks with novector >>> directives. >>> >>> Martyn said the beta license situation was fixed. I don’t yet know if >>>any >>> action is required on ALCF’s part. >>> >>> Jeff >>> >>> On 10/5/16, 1:04 PM, "Steven C. Pieper" >>> <spieper(a)anl.gov> >>> wrote: >>> >>> >>>> Jeff and/or Kalyan: >>>> >>>> I have isolated the wrong numerical result to one (large) subroutine. >>>> It is not the same one that causes the compiler to segfault; that >>>> subroutine is not involved in this wrong numerical result. >>>> I can compile everything with the production v17 using -O2 and openmp >>>> (I haven't tried more optimization yet). Then if I compile this >>>> one subroutine with -O0 and no openmp, it works. But if I compile >>>> it with -O2 and no openmp, it gives the wrong numbers. So this is >>>> not an OpenMP bug; it is a -O2 optimization error. This is >>>> a bit of a mess to make a demo for -- the subroutine is large and >>>> has a big table of input indices, etc., and it needs lots of other >>>> subroutines to have run before it is called. >>>> >>>> As I said, all of this works fine with -O3, AVX512 with the v17-beta >>>> compiler. >>>> >>>> I don't know what to do next. I can make an ugly package to give >>>> to you for Intel, but making a simple demo is probably too difficult. >>>> Is there any further update to the production v17 that could be >>>> installed for me to try? >>>> >>>> Steve >>>> >>>> >>>> Steven C. Pieper, Senior Physicist Emeritus: >>>> >>>> spieper(a)anl.gov >>>> >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>> 60439 >>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>>> -4100 >>>> >>>> http://www.phy.anl.gov/theory/staff/scp.html >>>> >>>> >>>> On 10/05/16 13:40, Hammond, Jeff R wrote: >>>> >>>>> On 10/5/16, 11:04 AM, >>>>> "intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>>> Kumaran, Kalyan" >>>>> >>>>> <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>>> kumaran(a)alcf.anl.gov> >>>>> wrote: >>>>> >>>>> >>>>>> Steve, are you saying the Beta compilers have been removed, or will >>>>>>be >>>>>> removed in 2 days, from the JLSE cluster? If this is the case I can >>>>>> request Ben to keep it though I have not see any message about >>>>>>removing >>>>>> it. >>>>>> >>>>> I asked Martyn about this just now. Will post the substance of his >>>>> response as soon as I get it. >>>>> >>>>> >>>>>> Please do report bugs on the GA compilers. Without these I fear it >>>>>>will >>>>>> never get fixed. >>>>>> >>>>> Please report every bug to IPS. It¹s possible that the same bug >>>>>will be >>>>> identified by a different codes, but it is not good to rely upon >>>>>this. >>>>> >>>>> Jeff >>>>> >>>>> >>>>>> -Kumar >>>>>> >>>>>> >>>>>>> On Oct 5, 2016, at 12:51 PM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>> Tim, >>>>>>> I didn't know where to send it. I just want it to be worked on. >>>>>>> The non-beta V17 compiler is on jlse -- that is what is failing >>>>>>>for >>>>>>> me. >>>>>>> >>>>>>> Some more recent info: >>>>>>> Only this one subroutine file causes the compiler to segfault. >>>>>>> >>>>>>> However much more seriously, I am getting the wrong numerical >>>>>>> results >>>>>>> using the non-beta compiler. They are wrong in that they differ >>>>>>> from both the v17 beta and xlf on cetus. v17 beta and cetus give >>>>>>> the same results. I'm trying now to narrow down where the wrong >>>>>>> results are coming from. >>>>>>> >>>>>>> Can we keep the beta compiler going for more than 2 more >>>>>>>days???? >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>> >>>>>>> spieper(a)anl.gov >>>>>>> >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>IL >>>>>>> 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra >>>>>>>Beres, >>>>>>> -4100 >>>>>>> >>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>> >>>>>>> >>>>>>> On 10/05/16 12:45, Williams, Timothy J. wrote: >>>>>>> >>>>>>>> Hi Steve, >>>>>>>> >>>>>>>> Did you mean to send this to early-users-discuss or intel-nda? >>>>>>>> >>>>>>>> I think we should now have the non-beta V17 compilers on the white >>>>>>>> boxes. Will try to find out. >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> ------- Timothy J. Williams, Ph.D. --- Argonne Leadership >>>>>>>>Computing >>>>>>>> Facility --- >>>>>>>> tjwilliams(a)anl.gov >>>>>>>> --- 630-252-1154 ------- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 5, 2016, at 11:25 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I am trying to recompile everything using the production intel >>>>>>>>> compiler >>>>>>>>> rather than the beta version which I previously used. >>>>>>>>> >>>>>>>>> With the beta version, it works: >>>>>>>>> >>>>>>>>> [jlselogin2 mc] source >>>>>>>>> >>>>>>>>> >>>>>>>>>/soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compi >>>>>>>>>ler >>>>>>>>> va >>>>>>>>> rs.sh intel64 >>>>>>>>> [jlselogin2 mc] ifort -V >>>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>>> Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>>> >>>>>>>>> ifort: NOTE: The Beta evaluation period for this product ends on >>>>>>>>> 7-oct-2016 UTC. >>>>>>>>> >>>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>-fpp >>>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>-DM0_DIV=1 >>>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>>> make_3pi.f >>>>>>>>> [jlselogin2 mc] <<compile is OK> >>>>>>>>> >>>>>>>>> >>>>>>>>> The production verson segfaults: >>>>>>>>> >>>>>>>>> [jlselogin2 mc] source >>>>>>>>> >>>>>>>>> >>>>>>>>>/soft/compilers/intel/compilers_and_libraries/linux/bin/compilerva >>>>>>>>>rs. >>>>>>>>> sh >>>>>>>>> intel64[jlselogin2 mc] ifort -V >>>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>>> Intel(R) 64, Version 17.0.0.098 Build 20160721 >>>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>>> >>>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ >>>>>>>>>-fpp >>>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn >>>>>>>>>-DM0_DIV=1 >>>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>>> make_3pi.f >>>>>>>>> catastrophic error: **Internal compiler error: segmentation >>>>>>>>> violation >>>>>>>>> signal raised** Please report this error along with the >>>>>>>>> circumstances >>>>>>>>> in which it occurred in a Software Problem Report. Note: File and >>>>>>>>> line >>>>>>>>> given may not be explicit cause of this error. >>>>>>>>> compilation aborted for make_3pi.f (code 1) >>>>>>>>> >>>>>>>>> >>>>>>>>> If I remove the -xMIC-AVX512 then the production version >>>>>>>>> compiles without error. >>>>>>>>> >>>>>>>>> Lots of other subroutines are compiling OK. >>>>>>>>> >>>>>>>>> >>>>>>>>> The files are all in ~b21541/gfmc/mc/ >>>>>>>>> >>>>>>>>> I see the beta version is due to expire in 2 days. Can this be >>>>>>>>> extended? >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>>>> >>>>>>>>> spieper(a)anl.gov >>>>>>>>> >>>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, >>>>>>>>>Argonne, >>>>>>>>> IL >>>>>>>>> 60439 >>>>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra >>>>>>>>> Beres, >>>>>>>>> -4100 >>>>>>>>> >>>>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> EarlyScience mailing list >>>>>>>>> >>>>>>>>> EarlyScience(a)lists.alcf.anl.gov >>>>>>>>> https://lists.alcf.anl.gov/mailman/listinfo/earlyscience >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults
by Kumaran, Kalyan 06 Oct '16

06 Oct '16

> On Oct 6, 2016, at 1:19 PM, Kenneth E Jansen <kenneth.jansen(a)colorado.edu> wrote: > > Sorry if this question has already been resolved but I do hope we can keep the beta license alive. We have an updated license that extends it by a month. Ben should install it sometime today. He has been very busy with Theta. -Kumar

1 0

Re: [JLSE-Intel-NDA] [EarlyScience] Production Intel compiler segfaults
by Kenneth E Jansen 06 Oct '16

06 Oct '16

Sorry if this question has already been resolved but I do hope we can keep the beta license alive. I have also not had the time to get back to doing Intel’s requested trouble shooting of the crashes on multinode with 17 Initial Release that do not happen with the beta. I did submit a ticket but the response was to do a lot of unit testing and this has just not made it to the top of my to do list. Like Steve’s case, my code is rather complex and not so easy to isolate what could be causing multinode crashes with the released verstion that just don’t happen with the beta version. I will try to get back to that testing when I can. Best regards, Ken ---------------------------------------------------------------- Kenneth E. Jansen, Professor of Aerospace Engineering Sciences ECAE 161 OFFICE (303) 492-4359 429 UCB FAX (303) 492-4990 University of Colorado at Boulder jansenke(a)colorado.edu Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace ---------------------------------------------------------------- > On Oct 6, 2016, at 12:07 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > Using -no-vec does fix the problem. > > I have just submitted this and the compiler segfault bugs to the Intel site: > > reference number is : 6000163313. > > I included a 7 MB tar file which has a readme. > > The jlse beta version still says: > > [jlselogin1 ~] ifort -V > Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 > Copyright (C) 1985-2016 Intel Corporation. All rights reserved. > > ifort: NOTE: The Beta evaluation period for this product ends on 7-oct-2016 UTC. > > > Steve > > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > On 10/05/16 18:35, Hammond, Jeff R wrote: >> Martyn suggests you try with -no-vec. If that also fixes the issue, then >> you can narrow it down to the specific loop that breaks with novector >> directives. >> >> Martyn said the beta license situation was fixed. I don’t yet know if any >> action is required on ALCF’s part. >> >> Jeff >> >> On 10/5/16, 1:04 PM, "Steven C. Pieper" >> <spieper(a)anl.gov> >> wrote: >> >> >>> Jeff and/or Kalyan: >>> >>> I have isolated the wrong numerical result to one (large) subroutine. >>> It is not the same one that causes the compiler to segfault; that >>> subroutine is not involved in this wrong numerical result. >>> I can compile everything with the production v17 using -O2 and openmp >>> (I haven't tried more optimization yet). Then if I compile this >>> one subroutine with -O0 and no openmp, it works. But if I compile >>> it with -O2 and no openmp, it gives the wrong numbers. So this is >>> not an OpenMP bug; it is a -O2 optimization error. This is >>> a bit of a mess to make a demo for -- the subroutine is large and >>> has a big table of input indices, etc., and it needs lots of other >>> subroutines to have run before it is called. >>> >>> As I said, all of this works fine with -O3, AVX512 with the v17-beta >>> compiler. >>> >>> I don't know what to do next. I can make an ugly package to give >>> to you for Intel, but making a simple demo is probably too difficult. >>> Is there any further update to the production v17 that could be >>> installed for me to try? >>> >>> Steve >>> >>> >>> Steven C. Pieper, Senior Physicist Emeritus: >>> >>> spieper(a)anl.gov >>> >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>> 60439 >>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>> -4100 >>> >>> http://www.phy.anl.gov/theory/staff/scp.html >>> >>> >>> On 10/05/16 13:40, Hammond, Jeff R wrote: >>> >>>> On 10/5/16, 11:04 AM, >>>> "intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>> Kumaran, Kalyan" >>>> >>>> <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of >>>> kumaran(a)alcf.anl.gov> >>>> wrote: >>>> >>>> >>>>> Steve, are you saying the Beta compilers have been removed, or will be >>>>> removed in 2 days, from the JLSE cluster? If this is the case I can >>>>> request Ben to keep it though I have not see any message about removing >>>>> it. >>>>> >>>> I asked Martyn about this just now. Will post the substance of his >>>> response as soon as I get it. >>>> >>>> >>>>> Please do report bugs on the GA compilers. Without these I fear it will >>>>> never get fixed. >>>>> >>>> Please report every bug to IPS. It¹s possible that the same bug will be >>>> identified by a different codes, but it is not good to rely upon this. >>>> >>>> Jeff >>>> >>>> >>>>> -Kumar >>>>> >>>>> >>>>>> On Oct 5, 2016, at 12:51 PM, Steven C. Pieper <spieper(a)anl.gov> >>>>>> wrote: >>>>>> >>>>>> Tim, >>>>>> I didn't know where to send it. I just want it to be worked on. >>>>>> The non-beta V17 compiler is on jlse -- that is what is failing for >>>>>> me. >>>>>> >>>>>> Some more recent info: >>>>>> Only this one subroutine file causes the compiler to segfault. >>>>>> >>>>>> However much more seriously, I am getting the wrong numerical >>>>>> results >>>>>> using the non-beta compiler. They are wrong in that they differ >>>>>> from both the v17 beta and xlf on cetus. v17 beta and cetus give >>>>>> the same results. I'm trying now to narrow down where the wrong >>>>>> results are coming from. >>>>>> >>>>>> Can we keep the beta compiler going for more than 2 more days???? >>>>>> >>>>>> Steve >>>>>> >>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>> >>>>>> spieper(a)anl.gov >>>>>> >>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL >>>>>> 60439 >>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, >>>>>> -4100 >>>>>> >>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>> >>>>>> >>>>>> On 10/05/16 12:45, Williams, Timothy J. wrote: >>>>>> >>>>>>> Hi Steve, >>>>>>> >>>>>>> Did you mean to send this to early-users-discuss or intel-nda? >>>>>>> >>>>>>> I think we should now have the non-beta V17 compilers on the white >>>>>>> boxes. Will try to find out. >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> ------- Timothy J. Williams, Ph.D. --- Argonne Leadership Computing >>>>>>> Facility --- >>>>>>> tjwilliams(a)anl.gov >>>>>>> --- 630-252-1154 ------- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Oct 5, 2016, at 11:25 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> I am trying to recompile everything using the production intel >>>>>>>> compiler >>>>>>>> rather than the beta version which I previously used. >>>>>>>> >>>>>>>> With the beta version, it works: >>>>>>>> >>>>>>>> [jlselogin2 mc] source >>>>>>>> >>>>>>>> /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compiler >>>>>>>> va >>>>>>>> rs.sh intel64 >>>>>>>> [jlselogin2 mc] ifort -V >>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>> Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>> >>>>>>>> ifort: NOTE: The Beta evaluation period for this product ends on >>>>>>>> 7-oct-2016 UTC. >>>>>>>> >>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>> make_3pi.f >>>>>>>> [jlselogin2 mc] <<compile is OK> >>>>>>>> >>>>>>>> >>>>>>>> The production verson segfaults: >>>>>>>> >>>>>>>> [jlselogin2 mc] source >>>>>>>> >>>>>>>> /soft/compilers/intel/compilers_and_libraries/linux/bin/compilervars. >>>>>>>> sh >>>>>>>> intel64[jlselogin2 mc] ifort -V >>>>>>>> Intel(R) Fortran Intel(R) 64 Compiler for applications running on >>>>>>>> Intel(R) 64, Version 17.0.0.098 Build 20160721 >>>>>>>> Copyright (C) 1985-2016 Intel Corporation. All rights reserved. >>>>>>>> >>>>>>>> [jlselogin2 mc] ifort -FR -g -O2 -xMIC-AVX512 -qopenmp -lpthread >>>>>>>> -mcmodel=medium -r8 -WB -traceback -fpe3 -I../common/ -I../mc/ -fpp >>>>>>>> -DSPIN_SUF= -DPARAMNUC=nclude/paramnuc.12c-11_states-pn -DM0_DIV=1 >>>>>>>> -DDOCASE=0 -DPARAMCASE=nclude/paramcase.12c-11_states-pn -c >>>>>>>> make_3pi.f >>>>>>>> catastrophic error: **Internal compiler error: segmentation >>>>>>>> violation >>>>>>>> signal raised** Please report this error along with the >>>>>>>> circumstances >>>>>>>> in which it occurred in a Software Problem Report. Note: File and >>>>>>>> line >>>>>>>> given may not be explicit cause of this error. >>>>>>>> compilation aborted for make_3pi.f (code 1) >>>>>>>> >>>>>>>> >>>>>>>> If I remove the -xMIC-AVX512 then the production version >>>>>>>> compiles without error. >>>>>>>> >>>>>>>> Lots of other subroutines are compiling OK. >>>>>>>> >>>>>>>> >>>>>>>> The files are all in ~b21541/gfmc/mc/ >>>>>>>> >>>>>>>> I see the beta version is due to expire in 2 days. Can this be >>>>>>>> extended? >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>>> >>>>>>>> spieper(a)anl.gov >>>>>>>> >>>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, >>>>>>>> IL >>>>>>>> 60439 >>>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra >>>>>>>> Beres, >>>>>>>> -4100 >>>>>>>> >>>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> EarlyScience mailing list >>>>>>>> >>>>>>>> EarlyScience(a)lists.alcf.anl.gov >>>>>>>> https://lists.alcf.anl.gov/mailman/listinfo/earlyscience >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0