- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Allen, Benjamin S. 25 Jul '16

25 Jul '16

Applied to all but knl01 and knl02 since they're in use. Ben > On Jul 25, 2016, at 4:38 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Great ! > > Was able to succesfully run Vtune with memory statistics collection on several nodes using multiple threads. > Note that I have not looked at the performance results yet. The data collection takes quite a lot of time. I wonder whether it's a Vtune issue or it can be related to Vtune reading/writing files. > > Can you apply the fix to all knl nodes ? (or at least knl00 and knl05 which are the ones I am working on :-)). > > Best, > > Fabien > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Monday, July 25, 2016 11:08 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > Suppose it would help if I actually start the drivers after installing them... > > Give it another try. > > Ben > >> On Jul 25, 2016, at 3:59 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> I did a couple of tests. It looks like the issue with perf_event_paranoid is gone. Thanks Ben ! >> The last step seems to be related to the drivers reported earlier: >> >> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >> >> Let me know when you have an image to be tested. >> Best, >> >> Fabien >> ________________________________________ >> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >> Sent: Monday, July 25, 2016 9:03 PM >> To: Delalondre Fabien Jonathan >> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >> >> Reinstalled the kernel modules, and fixed kernel.perf_event_paranoid again on knl00. >> >> Looks like /proc/sys/kernel/perf_event_mlock_kb is 516 currently. >> >> Possibly related ulimit -l is also only 64k currently. >> >> Ben >> >>> On Jul 25, 2016, at 1:41 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>> >>> Hi Ben, >>> >>> So I tried the following test on KNL00 then rebooted (and forgot the image would be wiped out...). Sorry about that Ben, can you please re-apply the image with fix on KNL00 so that I can test the quadrant/cache configuration because I believe the issue I have been running into (related to heap memory in SNC4/cache mode) over the week end is a different one... >>> >>> I am using the following compilers and vtune versions >>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >>> >>> ** Test 1 - SNC4/Cache >>> I had configured the system in SNC4-Cache last Friday. Did some testing over the week end but ran into issues related to heap allocation. I d like to possibly discuss those after I am done with reading the Intel KNL book. >>> >>> Nevertheless, executed the following command on this configuration (which is a test I could execute fine last May): >>> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NUM_NODES ... <EXE> <EXE_PARAM> >>> >>> I get the following error is: >>> amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/delalond/test_knl_cn/Result4.knl00.ftm.alcf.anl.gov -command stop. >>> amplxe: Error: Permission error mapping pages. >>> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >>> or try again with a smaller value of -m/--mmap_pages. >>> (current value: 4294967295) >>> >>> amplxe: Error: Permission error mapping pages. >>> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >>> or try again with a smaller value of -m/--mmap_pages. >>> (current value: 4294967295) >>> >>> amplxe: Collection failed. >>> amplxe: Collection failed. >>> amplxe: Internal Error >>> amplxe: Internal Error >>> amplxe: Error: Permission error mapping pages. >>> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >>> or try again with a smaller value of -m/--mmap_pages. >>> (current value: 4294967295) >>> >>> amplxe: Collection failed. >>> amplxe: Internal Error >>> >>> >>> >>> ________________________________________ >>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> Sent: Monday, July 25, 2016 7:04 PM >>> To: Delalondre Fabien Jonathan >>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>> >>> Hi Fabien, >>> >>> Could you test this on knl00? I installed the sampling driver and set perf_event_paranoid to 0 on the running OS image (i.e. it will go away if rebooted). I'd like to see if everything is working before adding it the the OS image. >>> >>> Thanks, >>> >>> Ben >>> >>>> On Jul 22, 2016, at 4:44 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>> >>>> Cool ! Thanks ben. >>>> Let me know when I can give it a try again. >>>> >>>> Best, >>>> >>>> Fabien >>>> ________________________________________ >>>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>> Sent: Friday, July 22, 2016 11:39 PM >>>> To: Delalondre Fabien Jonathan >>>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>>> >>>> Hi Fabien, >>>> >>>> Both these items are on my todo list. >>>> >>>> 1. Loading the sep4_0 driver >>>> 2. Setting perf_event_paranoid to 0 >>>> >>>> Should get to them sometime next week. >>>> >>>> Ben >>>> >>>>> On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>>> >>>>> Thanks All. >>>>> >>>>> So I face a couple of issues. Please let me know if this is coming from me or something else: >>>>> >>>>> On knl00 (configured, SNC4-Cache) >>>>> I use: >>>>> compiler (2017): >>>>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>>>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>>>> vtune (2016 v4): >>>>> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >>>>> >>>>> When I execute the an MPI+OpenMP application >>>>> 1) It works (running basic vtune hot spot analysis) >>>>> amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> >>>>> >>>>> 2) It used to work in May (with Vtune patched NDA - now it returns an error) >>>>> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> >>>>> >>>>> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >>>>> >>>>> 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) >>>>> >>>>> mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> >>>>> >>>>> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. >>>>> >>>>> Any hint ? Can we change the permissions to run uncore events collection on all KNL? >>>>> Let me know if I am doing something wrong. >>>>> >>>>> Thanks, >>>>> >>>>> Fabien >>>>> >>>>> ________________________________________ >>>>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>> Sent: Friday, July 22, 2016 3:15 PM >>>>> To: Delalondre Fabien Jonathan >>>>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>>>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>>>> >>>>> VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. >>>>> >>>>> Ben >>>>> >>>>> >>>>>> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>>>> >>>>>> You can check, but I think Release 16 is a sure bet and had full KNL support. >>>>>> >>>>>> -Kumar >>>>>> >>>>>> >>>>>>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>>>>>> >>>>>>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>>>>>> >>>>>>> Thanks ! >>>>>>> >>>>>>> Fabien >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>>> >>> >> >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Delalondre Fabien Jonathan 25 Jul '16

25 Jul '16

Great ! Was able to succesfully run Vtune with memory statistics collection on several nodes using multiple threads. Note that I have not looked at the performance results yet. The data collection takes quite a lot of time. I wonder whether it's a Vtune issue or it can be related to Vtune reading/writing files. Can you apply the fix to all knl nodes ? (or at least knl00 and knl05 which are the ones I am working on :-)). Best, Fabien ________________________________________ From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Monday, July 25, 2016 11:08 PM To: Delalondre Fabien Jonathan Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis Suppose it would help if I actually start the drivers after installing them... Give it another try. Ben > On Jul 25, 2016, at 3:59 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > I did a couple of tests. It looks like the issue with perf_event_paranoid is gone. Thanks Ben ! > The last step seems to be related to the drivers reported earlier: > > amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. > > Let me know when you have an image to be tested. > Best, > > Fabien > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Monday, July 25, 2016 9:03 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > Reinstalled the kernel modules, and fixed kernel.perf_event_paranoid again on knl00. > > Looks like /proc/sys/kernel/perf_event_mlock_kb is 516 currently. > > Possibly related ulimit -l is also only 64k currently. > > Ben > >> On Jul 25, 2016, at 1:41 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Hi Ben, >> >> So I tried the following test on KNL00 then rebooted (and forgot the image would be wiped out...). Sorry about that Ben, can you please re-apply the image with fix on KNL00 so that I can test the quadrant/cache configuration because I believe the issue I have been running into (related to heap memory in SNC4/cache mode) over the week end is a different one... >> >> I am using the following compilers and vtune versions >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >> >> ** Test 1 - SNC4/Cache >> I had configured the system in SNC4-Cache last Friday. Did some testing over the week end but ran into issues related to heap allocation. I d like to possibly discuss those after I am done with reading the Intel KNL book. >> >> Nevertheless, executed the following command on this configuration (which is a test I could execute fine last May): >> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NUM_NODES ... <EXE> <EXE_PARAM> >> >> I get the following error is: >> amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/delalond/test_knl_cn/Result4.knl00.ftm.alcf.anl.gov -command stop. >> amplxe: Error: Permission error mapping pages. >> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >> or try again with a smaller value of -m/--mmap_pages. >> (current value: 4294967295) >> >> amplxe: Error: Permission error mapping pages. >> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >> or try again with a smaller value of -m/--mmap_pages. >> (current value: 4294967295) >> >> amplxe: Collection failed. >> amplxe: Collection failed. >> amplxe: Internal Error >> amplxe: Internal Error >> amplxe: Error: Permission error mapping pages. >> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >> or try again with a smaller value of -m/--mmap_pages. >> (current value: 4294967295) >> >> amplxe: Collection failed. >> amplxe: Internal Error >> >> >> >> ________________________________________ >> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >> Sent: Monday, July 25, 2016 7:04 PM >> To: Delalondre Fabien Jonathan >> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >> >> Hi Fabien, >> >> Could you test this on knl00? I installed the sampling driver and set perf_event_paranoid to 0 on the running OS image (i.e. it will go away if rebooted). I'd like to see if everything is working before adding it the the OS image. >> >> Thanks, >> >> Ben >> >>> On Jul 22, 2016, at 4:44 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>> >>> Cool ! Thanks ben. >>> Let me know when I can give it a try again. >>> >>> Best, >>> >>> Fabien >>> ________________________________________ >>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> Sent: Friday, July 22, 2016 11:39 PM >>> To: Delalondre Fabien Jonathan >>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>> >>> Hi Fabien, >>> >>> Both these items are on my todo list. >>> >>> 1. Loading the sep4_0 driver >>> 2. Setting perf_event_paranoid to 0 >>> >>> Should get to them sometime next week. >>> >>> Ben >>> >>>> On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>> >>>> Thanks All. >>>> >>>> So I face a couple of issues. Please let me know if this is coming from me or something else: >>>> >>>> On knl00 (configured, SNC4-Cache) >>>> I use: >>>> compiler (2017): >>>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>>> vtune (2016 v4): >>>> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >>>> >>>> When I execute the an MPI+OpenMP application >>>> 1) It works (running basic vtune hot spot analysis) >>>> amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> >>>> >>>> 2) It used to work in May (with Vtune patched NDA - now it returns an error) >>>> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> >>>> >>>> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >>>> >>>> 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) >>>> >>>> mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> >>>> >>>> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. >>>> >>>> Any hint ? Can we change the permissions to run uncore events collection on all KNL? >>>> Let me know if I am doing something wrong. >>>> >>>> Thanks, >>>> >>>> Fabien >>>> >>>> ________________________________________ >>>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>> Sent: Friday, July 22, 2016 3:15 PM >>>> To: Delalondre Fabien Jonathan >>>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>>> >>>> VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. >>>> >>>> Ben >>>> >>>> >>>>> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>>> >>>>> You can check, but I think Release 16 is a sure bet and had full KNL support. >>>>> >>>>> -Kumar >>>>> >>>>> >>>>>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>>>>> >>>>>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>>>>> >>>>>> Thanks ! >>>>>> >>>>>> Fabien >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >>> >> >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Allen, Benjamin S. 25 Jul '16

25 Jul '16

Suppose it would help if I actually start the drivers after installing them... Give it another try. Ben > On Jul 25, 2016, at 3:59 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > I did a couple of tests. It looks like the issue with perf_event_paranoid is gone. Thanks Ben ! > The last step seems to be related to the drivers reported earlier: > > amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. > > Let me know when you have an image to be tested. > Best, > > Fabien > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Monday, July 25, 2016 9:03 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > Reinstalled the kernel modules, and fixed kernel.perf_event_paranoid again on knl00. > > Looks like /proc/sys/kernel/perf_event_mlock_kb is 516 currently. > > Possibly related ulimit -l is also only 64k currently. > > Ben > >> On Jul 25, 2016, at 1:41 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Hi Ben, >> >> So I tried the following test on KNL00 then rebooted (and forgot the image would be wiped out...). Sorry about that Ben, can you please re-apply the image with fix on KNL00 so that I can test the quadrant/cache configuration because I believe the issue I have been running into (related to heap memory in SNC4/cache mode) over the week end is a different one... >> >> I am using the following compilers and vtune versions >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >> >> ** Test 1 - SNC4/Cache >> I had configured the system in SNC4-Cache last Friday. Did some testing over the week end but ran into issues related to heap allocation. I d like to possibly discuss those after I am done with reading the Intel KNL book. >> >> Nevertheless, executed the following command on this configuration (which is a test I could execute fine last May): >> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NUM_NODES ... <EXE> <EXE_PARAM> >> >> I get the following error is: >> amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/delalond/test_knl_cn/Result4.knl00.ftm.alcf.anl.gov -command stop. >> amplxe: Error: Permission error mapping pages. >> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >> or try again with a smaller value of -m/--mmap_pages. >> (current value: 4294967295) >> >> amplxe: Error: Permission error mapping pages. >> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >> or try again with a smaller value of -m/--mmap_pages. >> (current value: 4294967295) >> >> amplxe: Collection failed. >> amplxe: Collection failed. >> amplxe: Internal Error >> amplxe: Internal Error >> amplxe: Error: Permission error mapping pages. >> Consider increasing /proc/sys/kernel/perf_event_mlock_kb, >> or try again with a smaller value of -m/--mmap_pages. >> (current value: 4294967295) >> >> amplxe: Collection failed. >> amplxe: Internal Error >> >> >> >> ________________________________________ >> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >> Sent: Monday, July 25, 2016 7:04 PM >> To: Delalondre Fabien Jonathan >> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >> >> Hi Fabien, >> >> Could you test this on knl00? I installed the sampling driver and set perf_event_paranoid to 0 on the running OS image (i.e. it will go away if rebooted). I'd like to see if everything is working before adding it the the OS image. >> >> Thanks, >> >> Ben >> >>> On Jul 22, 2016, at 4:44 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>> >>> Cool ! Thanks ben. >>> Let me know when I can give it a try again. >>> >>> Best, >>> >>> Fabien >>> ________________________________________ >>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> Sent: Friday, July 22, 2016 11:39 PM >>> To: Delalondre Fabien Jonathan >>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>> >>> Hi Fabien, >>> >>> Both these items are on my todo list. >>> >>> 1. Loading the sep4_0 driver >>> 2. Setting perf_event_paranoid to 0 >>> >>> Should get to them sometime next week. >>> >>> Ben >>> >>>> On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>> >>>> Thanks All. >>>> >>>> So I face a couple of issues. Please let me know if this is coming from me or something else: >>>> >>>> On knl00 (configured, SNC4-Cache) >>>> I use: >>>> compiler (2017): >>>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>>> vtune (2016 v4): >>>> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >>>> >>>> When I execute the an MPI+OpenMP application >>>> 1) It works (running basic vtune hot spot analysis) >>>> amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> >>>> >>>> 2) It used to work in May (with Vtune patched NDA - now it returns an error) >>>> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> >>>> >>>> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >>>> >>>> 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) >>>> >>>> mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> >>>> >>>> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. >>>> >>>> Any hint ? Can we change the permissions to run uncore events collection on all KNL? >>>> Let me know if I am doing something wrong. >>>> >>>> Thanks, >>>> >>>> Fabien >>>> >>>> ________________________________________ >>>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>> Sent: Friday, July 22, 2016 3:15 PM >>>> To: Delalondre Fabien Jonathan >>>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>>> >>>> VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. >>>> >>>> Ben >>>> >>>> >>>>> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>>> >>>>> You can check, but I think Release 16 is a sure bet and had full KNL support. >>>>> >>>>> -Kumar >>>>> >>>>> >>>>>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>>>>> >>>>>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>>>>> >>>>>> Thanks ! >>>>>> >>>>>> Fabien >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >>> >> >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Delalondre Fabien Jonathan 25 Jul '16

25 Jul '16

I did a couple of tests. It looks like the issue with perf_event_paranoid is gone. Thanks Ben ! The last step seems to be related to the drivers reported earlier: amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. Let me know when you have an image to be tested. Best, Fabien ________________________________________ From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Monday, July 25, 2016 9:03 PM To: Delalondre Fabien Jonathan Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis Reinstalled the kernel modules, and fixed kernel.perf_event_paranoid again on knl00. Looks like /proc/sys/kernel/perf_event_mlock_kb is 516 currently. Possibly related ulimit -l is also only 64k currently. Ben > On Jul 25, 2016, at 1:41 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Hi Ben, > > So I tried the following test on KNL00 then rebooted (and forgot the image would be wiped out...). Sorry about that Ben, can you please re-apply the image with fix on KNL00 so that I can test the quadrant/cache configuration because I believe the issue I have been running into (related to heap memory in SNC4/cache mode) over the week end is a different one... > > I am using the following compilers and vtune versions > source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 > source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh > > ** Test 1 - SNC4/Cache > I had configured the system in SNC4-Cache last Friday. Did some testing over the week end but ran into issues related to heap allocation. I d like to possibly discuss those after I am done with reading the Intel KNL book. > > Nevertheless, executed the following command on this configuration (which is a test I could execute fine last May): > amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NUM_NODES ... <EXE> <EXE_PARAM> > > I get the following error is: > amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/delalond/test_knl_cn/Result4.knl00.ftm.alcf.anl.gov -command stop. > amplxe: Error: Permission error mapping pages. > Consider increasing /proc/sys/kernel/perf_event_mlock_kb, > or try again with a smaller value of -m/--mmap_pages. > (current value: 4294967295) > > amplxe: Error: Permission error mapping pages. > Consider increasing /proc/sys/kernel/perf_event_mlock_kb, > or try again with a smaller value of -m/--mmap_pages. > (current value: 4294967295) > > amplxe: Collection failed. > amplxe: Collection failed. > amplxe: Internal Error > amplxe: Internal Error > amplxe: Error: Permission error mapping pages. > Consider increasing /proc/sys/kernel/perf_event_mlock_kb, > or try again with a smaller value of -m/--mmap_pages. > (current value: 4294967295) > > amplxe: Collection failed. > amplxe: Internal Error > > > > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Monday, July 25, 2016 7:04 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > Hi Fabien, > > Could you test this on knl00? I installed the sampling driver and set perf_event_paranoid to 0 on the running OS image (i.e. it will go away if rebooted). I'd like to see if everything is working before adding it the the OS image. > > Thanks, > > Ben > >> On Jul 22, 2016, at 4:44 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Cool ! Thanks ben. >> Let me know when I can give it a try again. >> >> Best, >> >> Fabien >> ________________________________________ >> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >> Sent: Friday, July 22, 2016 11:39 PM >> To: Delalondre Fabien Jonathan >> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >> >> Hi Fabien, >> >> Both these items are on my todo list. >> >> 1. Loading the sep4_0 driver >> 2. Setting perf_event_paranoid to 0 >> >> Should get to them sometime next week. >> >> Ben >> >>> On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>> >>> Thanks All. >>> >>> So I face a couple of issues. Please let me know if this is coming from me or something else: >>> >>> On knl00 (configured, SNC4-Cache) >>> I use: >>> compiler (2017): >>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>> vtune (2016 v4): >>> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >>> >>> When I execute the an MPI+OpenMP application >>> 1) It works (running basic vtune hot spot analysis) >>> amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> >>> >>> 2) It used to work in May (with Vtune patched NDA - now it returns an error) >>> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> >>> >>> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >>> >>> 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) >>> >>> mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> >>> >>> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. >>> >>> Any hint ? Can we change the permissions to run uncore events collection on all KNL? >>> Let me know if I am doing something wrong. >>> >>> Thanks, >>> >>> Fabien >>> >>> ________________________________________ >>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> Sent: Friday, July 22, 2016 3:15 PM >>> To: Delalondre Fabien Jonathan >>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>> >>> VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. >>> >>> Ben >>> >>> >>>> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>> >>>> You can check, but I think Release 16 is a sure bet and had full KNL support. >>>> >>>> -Kumar >>>> >>>> >>>>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>>>> >>>>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>>>> >>>>> Thanks ! >>>>> >>>>> Fabien >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >> >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Allen, Benjamin S. 25 Jul '16

25 Jul '16

Reinstalled the kernel modules, and fixed kernel.perf_event_paranoid again on knl00. Looks like /proc/sys/kernel/perf_event_mlock_kb is 516 currently. Possibly related ulimit -l is also only 64k currently. Ben > On Jul 25, 2016, at 1:41 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Hi Ben, > > So I tried the following test on KNL00 then rebooted (and forgot the image would be wiped out...). Sorry about that Ben, can you please re-apply the image with fix on KNL00 so that I can test the quadrant/cache configuration because I believe the issue I have been running into (related to heap memory in SNC4/cache mode) over the week end is a different one... > > I am using the following compilers and vtune versions > source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 > source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh > > ** Test 1 - SNC4/Cache > I had configured the system in SNC4-Cache last Friday. Did some testing over the week end but ran into issues related to heap allocation. I d like to possibly discuss those after I am done with reading the Intel KNL book. > > Nevertheless, executed the following command on this configuration (which is a test I could execute fine last May): > amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NUM_NODES ... <EXE> <EXE_PARAM> > > I get the following error is: > amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/delalond/test_knl_cn/Result4.knl00.ftm.alcf.anl.gov -command stop. > amplxe: Error: Permission error mapping pages. > Consider increasing /proc/sys/kernel/perf_event_mlock_kb, > or try again with a smaller value of -m/--mmap_pages. > (current value: 4294967295) > > amplxe: Error: Permission error mapping pages. > Consider increasing /proc/sys/kernel/perf_event_mlock_kb, > or try again with a smaller value of -m/--mmap_pages. > (current value: 4294967295) > > amplxe: Collection failed. > amplxe: Collection failed. > amplxe: Internal Error > amplxe: Internal Error > amplxe: Error: Permission error mapping pages. > Consider increasing /proc/sys/kernel/perf_event_mlock_kb, > or try again with a smaller value of -m/--mmap_pages. > (current value: 4294967295) > > amplxe: Collection failed. > amplxe: Internal Error > > > > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Monday, July 25, 2016 7:04 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > Hi Fabien, > > Could you test this on knl00? I installed the sampling driver and set perf_event_paranoid to 0 on the running OS image (i.e. it will go away if rebooted). I'd like to see if everything is working before adding it the the OS image. > > Thanks, > > Ben > >> On Jul 22, 2016, at 4:44 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Cool ! Thanks ben. >> Let me know when I can give it a try again. >> >> Best, >> >> Fabien >> ________________________________________ >> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >> Sent: Friday, July 22, 2016 11:39 PM >> To: Delalondre Fabien Jonathan >> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >> >> Hi Fabien, >> >> Both these items are on my todo list. >> >> 1. Loading the sep4_0 driver >> 2. Setting perf_event_paranoid to 0 >> >> Should get to them sometime next week. >> >> Ben >> >>> On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>> >>> Thanks All. >>> >>> So I face a couple of issues. Please let me know if this is coming from me or something else: >>> >>> On knl00 (configured, SNC4-Cache) >>> I use: >>> compiler (2017): >>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>> vtune (2016 v4): >>> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >>> >>> When I execute the an MPI+OpenMP application >>> 1) It works (running basic vtune hot spot analysis) >>> amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> >>> >>> 2) It used to work in May (with Vtune patched NDA - now it returns an error) >>> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> >>> >>> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >>> >>> 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) >>> >>> mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> >>> >>> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. >>> >>> Any hint ? Can we change the permissions to run uncore events collection on all KNL? >>> Let me know if I am doing something wrong. >>> >>> Thanks, >>> >>> Fabien >>> >>> ________________________________________ >>> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> Sent: Friday, July 22, 2016 3:15 PM >>> To: Delalondre Fabien Jonathan >>> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >>> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >>> >>> VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. >>> >>> Ben >>> >>> >>>> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>> >>>> You can check, but I think Release 16 is a sure bet and had full KNL support. >>>> >>>> -Kumar >>>> >>>> >>>>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>>>> >>>>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>>>> >>>>> Thanks ! >>>>> >>>>> Fabien >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >> >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Delalondre Fabien Jonathan 25 Jul '16

25 Jul '16

Hi Ben, So I tried the following test on KNL00 then rebooted (and forgot the image would be wiped out...). Sorry about that Ben, can you please re-apply the image with fix on KNL00 so that I can test the quadrant/cache configuration because I believe the issue I have been running into (related to heap memory in SNC4/cache mode) over the week end is a different one... I am using the following compilers and vtune versions source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh ** Test 1 - SNC4/Cache I had configured the system in SNC4-Cache last Friday. Did some testing over the week end but ran into issues related to heap allocation. I d like to possibly discuss those after I am done with reading the Intel KNL book. Nevertheless, executed the following command on this configuration (which is a test I could execute fine last May): amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NUM_NODES ... <EXE> <EXE_PARAM> I get the following error is: amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/delalond/test_knl_cn/Result4.knl00.ftm.alcf.anl.gov -command stop. amplxe: Error: Permission error mapping pages. Consider increasing /proc/sys/kernel/perf_event_mlock_kb, or try again with a smaller value of -m/--mmap_pages. (current value: 4294967295) amplxe: Error: Permission error mapping pages. Consider increasing /proc/sys/kernel/perf_event_mlock_kb, or try again with a smaller value of -m/--mmap_pages. (current value: 4294967295) amplxe: Collection failed. amplxe: Collection failed. amplxe: Internal Error amplxe: Internal Error amplxe: Error: Permission error mapping pages. Consider increasing /proc/sys/kernel/perf_event_mlock_kb, or try again with a smaller value of -m/--mmap_pages. (current value: 4294967295) amplxe: Collection failed. amplxe: Internal Error ________________________________________ From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Monday, July 25, 2016 7:04 PM To: Delalondre Fabien Jonathan Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis Hi Fabien, Could you test this on knl00? I installed the sampling driver and set perf_event_paranoid to 0 on the running OS image (i.e. it will go away if rebooted). I'd like to see if everything is working before adding it the the OS image. Thanks, Ben > On Jul 22, 2016, at 4:44 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Cool ! Thanks ben. > Let me know when I can give it a try again. > > Best, > > Fabien > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Friday, July 22, 2016 11:39 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > Hi Fabien, > > Both these items are on my todo list. > > 1. Loading the sep4_0 driver > 2. Setting perf_event_paranoid to 0 > > Should get to them sometime next week. > > Ben > >> On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Thanks All. >> >> So I face a couple of issues. Please let me know if this is coming from me or something else: >> >> On knl00 (configured, SNC4-Cache) >> I use: >> compiler (2017): >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >> vtune (2016 v4): >> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >> >> When I execute the an MPI+OpenMP application >> 1) It works (running basic vtune hot spot analysis) >> amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> >> >> 2) It used to work in May (with Vtune patched NDA - now it returns an error) >> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> >> >> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >> >> 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) >> >> mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> >> >> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. >> >> Any hint ? Can we change the permissions to run uncore events collection on all KNL? >> Let me know if I am doing something wrong. >> >> Thanks, >> >> Fabien >> >> ________________________________________ >> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >> Sent: Friday, July 22, 2016 3:15 PM >> To: Delalondre Fabien Jonathan >> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >> >> VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. >> >> Ben >> >> >>> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>> >>> You can check, but I think Release 16 is a sure bet and had full KNL support. >>> >>> -Kumar >>> >>> >>>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>> >>>> Hi All, >>>> >>>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>>> >>>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>>> >>>> Thanks ! >>>> >>>> Fabien >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Allen, Benjamin S. 25 Jul '16

25 Jul '16

Hi Fabien, Could you test this on knl00? I installed the sampling driver and set perf_event_paranoid to 0 on the running OS image (i.e. it will go away if rebooted). I'd like to see if everything is working before adding it the the OS image. Thanks, Ben > On Jul 22, 2016, at 4:44 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Cool ! Thanks ben. > Let me know when I can give it a try again. > > Best, > > Fabien > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Friday, July 22, 2016 11:39 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > Hi Fabien, > > Both these items are on my todo list. > > 1. Loading the sep4_0 driver > 2. Setting perf_event_paranoid to 0 > > Should get to them sometime next week. > > Ben > >> On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Thanks All. >> >> So I face a couple of issues. Please let me know if this is coming from me or something else: >> >> On knl00 (configured, SNC4-Cache) >> I use: >> compiler (2017): >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >> vtune (2016 v4): >> source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh >> >> When I execute the an MPI+OpenMP application >> 1) It works (running basic vtune hot spot analysis) >> amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> >> >> 2) It used to work in May (with Vtune patched NDA - now it returns an error) >> amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> >> >> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. >> >> 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) >> >> mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> >> >> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. >> >> Any hint ? Can we change the permissions to run uncore events collection on all KNL? >> Let me know if I am doing something wrong. >> >> Thanks, >> >> Fabien >> >> ________________________________________ >> From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >> Sent: Friday, July 22, 2016 3:15 PM >> To: Delalondre Fabien Jonathan >> Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan >> Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis >> >> VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. >> >> Ben >> >> >>> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>> >>> You can check, but I think Release 16 is a sure bet and had full KNL support. >>> >>> -Kumar >>> >>> >>>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>>> >>>> Hi All, >>>> >>>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>>> >>>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>>> >>>> Thanks ! >>>> >>>> Fabien >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Delalondre Fabien Jonathan 22 Jul '16

22 Jul '16

Cool ! Thanks ben. Let me know when I can give it a try again. Best, Fabien ________________________________________ From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Friday, July 22, 2016 11:39 PM To: Delalondre Fabien Jonathan Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis Hi Fabien, Both these items are on my todo list. 1. Loading the sep4_0 driver 2. Setting perf_event_paranoid to 0 Should get to them sometime next week. Ben > On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Thanks All. > > So I face a couple of issues. Please let me know if this is coming from me or something else: > > On knl00 (configured, SNC4-Cache) > I use: > compiler (2017): > source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 > source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > vtune (2016 v4): > source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh > > When I execute the an MPI+OpenMP application > 1) It works (running basic vtune hot spot analysis) > amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> > > 2) It used to work in May (with Vtune patched NDA - now it returns an error) > amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> > > amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. > > 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) > > mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> > > Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. > > Any hint ? Can we change the permissions to run uncore events collection on all KNL? > Let me know if I am doing something wrong. > > Thanks, > > Fabien > > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Friday, July 22, 2016 3:15 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. > > Ben > > >> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >> >> You can check, but I think Release 16 is a sure bet and had full KNL support. >> >> -Kumar >> >> >>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>> >>> Hi All, >>> >>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>> >>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>> >>> Thanks ! >>> >>> Fabien >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Allen, Benjamin S. 22 Jul '16

22 Jul '16

Hi Fabien, Both these items are on my todo list. 1. Loading the sep4_0 driver 2. Setting perf_event_paranoid to 0 Should get to them sometime next week. Ben > On Jul 22, 2016, at 3:55 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Thanks All. > > So I face a couple of issues. Please let me know if this is coming from me or something else: > > On knl00 (configured, SNC4-Cache) > I use: > compiler (2017): > source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 > source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > vtune (2016 v4): > source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh > > When I execute the an MPI+OpenMP application > 1) It works (running basic vtune hot spot analysis) > amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> > > 2) It used to work in May (with Vtune patched NDA - now it returns an error) > amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> > > amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. > > 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) > > mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> > > Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. > > Any hint ? Can we change the permissions to run uncore events collection on all KNL? > Let me know if I am doing something wrong. > > Thanks, > > Fabien > > ________________________________________ > From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Friday, July 22, 2016 3:15 PM > To: Delalondre Fabien Jonathan > Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan > Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis > > VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. > > Ben > > >> On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >> >> You can check, but I think Release 16 is a sure bet and had full KNL support. >> >> -Kumar >> >> >>> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >>> >>> Hi All, >>> >>> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >>> >>> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >>> >>> Thanks ! >>> >>> Fabien >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis
by Delalondre Fabien Jonathan 22 Jul '16

22 Jul '16

Thanks All. So I face a couple of issues. Please let me know if this is coming from me or something else: On knl00 (configured, SNC4-Cache) I use: compiler (2017): source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 source /soft/compilers/intel_beta/compilers_and_libraries/linux/mpi/bin64/mpivars.sh vtune (2016 v4): source /soft/compilers/intel/vtune_amplifier_xe_2016.4.0.470476/amplxe-vars.sh When I execute the an MPI+OpenMP application 1) It works (running basic vtune hot spot analysis) amplxe-cl -collect hotspots -- mpirun -n $NODES <EXEC> <ParamEXEC> 2) It used to work in May (with Vtune patched NDA - now it returns an error) amplxe-cl -c memory-access -knob analyze-mem-objects=true -knob sampling-interval=10 -data-limit=0 -d 60 -- mpirun -n $NODES <EXEC> <ParamEXEC> amplxe: Error: Cannot enable Hardware Event-based Sampling: problem with the driver (sep*/sepdrv*). Check that the driver is running and the driver group is in the current user group list. See "Building and Managing the Sampling Driver" help topic for further details. 3) Instructions provided on wiki by Vitali do not work for me due to lack of permission (set to 1 in X file) mpirun -n $NODE_COUNT -env KMP_AFFINITY=balanced,granularity=fine -gtool "/soft/restricted/intel-vtune-nda/vtune_amplifier_xe/bin64/amplxe-cl -collect memory-access -knob sampling-interval=10 -knob analyze-mem-objects=true -data-limit=1500 -mrte-mode=native -result-dir=Result4:0-3" <EXEC> <EXEC_PARAM> Error message: amplxe: Error: Not enough permissions to run uncore events collection. Consider setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less. Any hint ? Can we change the permissions to run uncore events collection on all KNL? Let me know if I am doing something wrong. Thanks, Fabien ________________________________________ From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Friday, July 22, 2016 3:15 PM To: Delalondre Fabien Jonathan Cc: intel-nda(a)lists.jlse.anl.gov; Kumaran, Kalyan Subject: Re: [JLSE-Intel-NDA] Last version of compiler & Vtune working with memory analysis VTune 2016 Update 4 added KNL support, so the NDA'ed add-on bits are no longer needed. The compilers in 2017 and 2016 haven't been updated recently, so they should be the same as when you last worked with them. Ben > On Jul 22, 2016, at 7:15 AM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: > > You can check, but I think Release 16 is a sure bet and had full KNL support. > > -Kumar > > >> On Jul 21, 2016, at 6:00 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Hi All, >> >> Sorry to ask but ... Can someone confirm which version of the compiler and Vtune work fine together for the data memory analysis. >> >> I double check Vitali's comments on the wiki but I have been told everything should work fine with the latest 2017 versions. >> >> Thanks ! >> >> Fabien >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0