- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] ESP - KNL00 questions
by Allen, Benjamin S. 04 May '16

04 May '16

Yea it is, https://lists.jlse.anl.gov/mailman/private/intel-nda/. Ben > On May 4, 2016, at 3:57 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: > > I hope the list is being archived and folks can search the archives! We should include the link to the archives in the introductory email. > > -Kumar > > >> On May 4, 2016, at 3:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Hi Fabien, >> >> Answered in-line below: >> >>> For those of you who dont know me, I am Fabien Delalondre from EPFL working on brain simulation in Switzerland. I have just got access to knl00 as part of the ALCF Early Science Program. I have a few questions related to KNL00. I apologize already if they sound like silly questions ... Appreciate all the help I can get ... >>> >>> 0) Is there any wiki accessible where users would add interesting facts/experience/general documentation etc ? >> >> Nothing setup, but we could if there's interest from the group. >> >>> 1) What is the recommended placeholder for the files to be written to ? Should I use home ? What is the file system/expected bandwidth on this file system >> >> Use the local SSD for any temp files. This is just a Intel DC S3710 SATA 200GB. /home is fine, but don't expect it to be fast, nor rely on it for any I/O sensitive work. >> >>> 2) Is there a possibility to change the card configuration from user space ? Or should I use intel-nda(a)lists.jlse.anl.gov mailing list every time I want to ask a different configuration ? >> >> Assume when you say card you're referring to the KNL system itself, and I'm assuming when you say configuration you're asking about MCDRAM's mode and model. If so yes this is possible if needed. It requires BIOS changes, but we have a utility to change those settings in Linux. Changes are applied on reboot. >> >>> 3) When the /soft/scheduling/knl/knl.txt text says knl00 node status: in use by William Scullin until 11:30 am CT 4 May 2016 - e-mail to to preempt running benchmark. Can I still use the system or is there a policy defined ? How can I "reserve" the KNL ? Should I send an email to the mailing list ? With the time difference between the US and Chicago I could use it every night in Chicago :-) >> >> No you cannot use the system if someone has listed that they're using it in that file. You can of course email this list asking to preempt whoever using it if you think you need to. For reservations just add text to the bottom of the same file stating the reservation date, start and end times. >> >>> 4) I understand there is no scheduler on this system up to now. Please let me know if this is not correct. >> >> Correct no scheduler. >> >>> 5) Can you point me to someone who has lots of experience with VTune on KNL ? Vitali was mentioning to me it does not produce the results he expected. I d like to know more about this. Any other profiling tool you would recommend ? Any documentation available on performance counters available ? >> >> We still don't have access to Vtune 2016's special KNL add-on, so we're a bit out of luck so far. As Kevin said, we're at the forefront of KNL knowledge. Unsure if Advisor is any better. >> >>> 6) Is there a debugger available on this node ? If the answer is yes, which one ? Can I use run a vnc server to connect and improve response time of the graphical interfaces ? >> >> No VNC. Use X11 forwarding. >> >>> 7) Is memkind/autoHBM installed on the system (http://www.nersc.gov/users/computational-systems/cori/application-porting-a… >>> Any recommendation regarding using or not using them ? >> >> Yea they're installed. >> >> >> >> >> Ben Allen >> >> HPC Systems Administrator >> ALCF Operations - Joint Laboratory for Systems Evaluation >> Argonne National Laboratory >> Email: bsallen(a)alcf.anl.gov >> Office: (630) 252-0554 >> >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] ESP - KNL00 questions
by Kumaran, Kalyan 04 May '16

04 May '16

I hope the list is being archived and folks can search the archives! We should include the link to the archives in the introductory email. -Kumar > On May 4, 2016, at 3:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Hi Fabien, > > Answered in-line below: > >> For those of you who dont know me, I am Fabien Delalondre from EPFL working on brain simulation in Switzerland. I have just got access to knl00 as part of the ALCF Early Science Program. I have a few questions related to KNL00. I apologize already if they sound like silly questions ... Appreciate all the help I can get ... >> >> 0) Is there any wiki accessible where users would add interesting facts/experience/general documentation etc ? > > Nothing setup, but we could if there's interest from the group. > >> 1) What is the recommended placeholder for the files to be written to ? Should I use home ? What is the file system/expected bandwidth on this file system > > Use the local SSD for any temp files. This is just a Intel DC S3710 SATA 200GB. /home is fine, but don't expect it to be fast, nor rely on it for any I/O sensitive work. > >> 2) Is there a possibility to change the card configuration from user space ? Or should I use intel-nda(a)lists.jlse.anl.gov mailing list every time I want to ask a different configuration ? > > Assume when you say card you're referring to the KNL system itself, and I'm assuming when you say configuration you're asking about MCDRAM's mode and model. If so yes this is possible if needed. It requires BIOS changes, but we have a utility to change those settings in Linux. Changes are applied on reboot. > >> 3) When the /soft/scheduling/knl/knl.txt text says knl00 node status: in use by William Scullin until 11:30 am CT 4 May 2016 - e-mail to to preempt running benchmark. Can I still use the system or is there a policy defined ? How can I "reserve" the KNL ? Should I send an email to the mailing list ? With the time difference between the US and Chicago I could use it every night in Chicago :-) > > No you cannot use the system if someone has listed that they're using it in that file. You can of course email this list asking to preempt whoever using it if you think you need to. For reservations just add text to the bottom of the same file stating the reservation date, start and end times. > >> 4) I understand there is no scheduler on this system up to now. Please let me know if this is not correct. > > Correct no scheduler. > >> 5) Can you point me to someone who has lots of experience with VTune on KNL ? Vitali was mentioning to me it does not produce the results he expected. I d like to know more about this. Any other profiling tool you would recommend ? Any documentation available on performance counters available ? > > We still don't have access to Vtune 2016's special KNL add-on, so we're a bit out of luck so far. As Kevin said, we're at the forefront of KNL knowledge. Unsure if Advisor is any better. > >> 6) Is there a debugger available on this node ? If the answer is yes, which one ? Can I use run a vnc server to connect and improve response time of the graphical interfaces ? > > No VNC. Use X11 forwarding. > >> 7) Is memkind/autoHBM installed on the system (http://www.nersc.gov/users/computational-systems/cori/application-porting-a… >> Any recommendation regarding using or not using them ? > > Yea they're installed. > > > > > Ben Allen > > HPC Systems Administrator > ALCF Operations - Joint Laboratory for Systems Evaluation > Argonne National Laboratory > Email: bsallen(a)alcf.anl.gov > Office: (630) 252-0554 > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] ESP - KNL00 questions
by Allen, Benjamin S. 04 May '16

04 May '16

Hi Fabien, Answered in-line below: > For those of you who dont know me, I am Fabien Delalondre from EPFL working on brain simulation in Switzerland. I have just got access to knl00 as part of the ALCF Early Science Program. I have a few questions related to KNL00. I apologize already if they sound like silly questions ... Appreciate all the help I can get ... > > 0) Is there any wiki accessible where users would add interesting facts/experience/general documentation etc ? Nothing setup, but we could if there's interest from the group. > 1) What is the recommended placeholder for the files to be written to ? Should I use home ? What is the file system/expected bandwidth on this file system Use the local SSD for any temp files. This is just a Intel DC S3710 SATA 200GB. /home is fine, but don't expect it to be fast, nor rely on it for any I/O sensitive work. > 2) Is there a possibility to change the card configuration from user space ? Or should I use intel-nda(a)lists.jlse.anl.gov mailing list every time I want to ask a different configuration ? Assume when you say card you're referring to the KNL system itself, and I'm assuming when you say configuration you're asking about MCDRAM's mode and model. If so yes this is possible if needed. It requires BIOS changes, but we have a utility to change those settings in Linux. Changes are applied on reboot. > 3) When the /soft/scheduling/knl/knl.txt text says knl00 node status: in use by William Scullin until 11:30 am CT 4 May 2016 - e-mail to to preempt running benchmark. Can I still use the system or is there a policy defined ? How can I "reserve" the KNL ? Should I send an email to the mailing list ? With the time difference between the US and Chicago I could use it every night in Chicago :-) No you cannot use the system if someone has listed that they're using it in that file. You can of course email this list asking to preempt whoever using it if you think you need to. For reservations just add text to the bottom of the same file stating the reservation date, start and end times. > 4) I understand there is no scheduler on this system up to now. Please let me know if this is not correct. Correct no scheduler. > 5) Can you point me to someone who has lots of experience with VTune on KNL ? Vitali was mentioning to me it does not produce the results he expected. I d like to know more about this. Any other profiling tool you would recommend ? Any documentation available on performance counters available ? We still don't have access to Vtune 2016's special KNL add-on, so we're a bit out of luck so far. As Kevin said, we're at the forefront of KNL knowledge. Unsure if Advisor is any better. > 6) Is there a debugger available on this node ? If the answer is yes, which one ? Can I use run a vnc server to connect and improve response time of the graphical interfaces ? No VNC. Use X11 forwarding. > 7) Is memkind/autoHBM installed on the system (http://www.nersc.gov/users/computational-systems/cori/application-porting-a… > Any recommendation regarding using or not using them ? Yea they're installed. Ben Allen HPC Systems Administrator ALCF Operations - Joint Laboratory for Systems Evaluation Argonne National Laboratory Email: bsallen(a)alcf.anl.gov Office: (630) 252-0554

1 0

Re: [JLSE-Intel-NDA] ESP - KNL00 questions
by Harms, Kevin 04 May '16

04 May '16

I can answer a few of these. >3) When the /soft/scheduling/knl/knl.txt text says knl00 node status: in use by William Scullin until 11:30 am CT 4 May 2016 - e-mail to to preempt running benchmark. Can I still use the system or is there a policy > defined ? How can I "reserve" the KNL ? Should I send an email to the mailing list ? With the time difference between the US and Chicago I could use it every night in Chicago :-) You look in the /soft/scheduling/knl/knl.txt file and if it says "free", you update it to put your name in and time period your using it. It would be nice if you put the time in CT but the node's time is set to UTC so that would work as well. >5) Can you point me to someone who has lots of experience with VTune on KNL ? As far as I can tell, no such person exists. Part of the issue is the Intel tools do not fully support (or support at all) KNL yet. If you want to try vtune, you should use intel compiler version 16 with the latest update. If you want the best KNL support you should use intel beta 17 compilers, but 17 doesn't have KNL vtune support. >7) Is memkind/autoHBM installed on the system (http://www.nersc.gov/users/computational-systems/cori/application-porting-a…) <http://www.nersc.gov/users/computational-systems/cori/application-porting-a…>? Any recommendation regarding > using or not using them ? Yes hbw_malloc.h and memkind are installed. kevin

1 0

[JLSE-Intel-NDA] ESP - KNL00 questions
by Delalondre Fabien Jonathan 04 May '16

04 May '16

Hi All, For those of you who dont know me, I am Fabien Delalondre from EPFL working on brain simulation in Switzerland. I have just got access to knl00 as part of the ALCF Early Science Program. I have a few questions related to KNL00. I apologize already if they sound like silly questions ... Appreciate all the help I can get ... 0) Is there any wiki accessible where users would add interesting facts/experience/general documentation etc ? 1) What is the recommended placeholder for the files to be written to ? Should I use home ? What is the file system/expected bandwidth on this file system 2) Is there a possibility to change the card configuration from user space ? Or should I use intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> mailing list every time I want to ask a different configuration ? 3) When the /soft/scheduling/knl/knl.txt? text says knl00 node status: in use by William Scullin until 11:30 am CT 4 May 2016 - e-mail to to preempt running benchmark?. Can I still use the system or is there a policy defined ? How can I "reserve" the KNL ? Should I send an email to the mailing list ? With the time difference between the US and Chicago I could use it every night in Chicago :-) 4) I understand there is no scheduler on this system up to now. Please let me know if this is not correct. 5) Can you point me to someone who has lots of experience with VTune on KNL ? Vitali was mentioning to me it does not produce the results he expected. I d like to know more about this. Any other profiling tool you would recommend ? Any documentation available on performance counters available ? 6) Is there a debugger available on this node ? If the answer is yes, which one ? Can I use run a vnc server to connect and improve response time of the graphical interfaces ? 7) Is memkind/autoHBM installed on the system (http://www.nersc.gov/users/computational-systems/cori/application-porting-a…)<http://www.nersc.gov/users/computational-systems/cori/application-porting-a…>? Any recommendation regarding using or not using them ? Thanks again for your help ! Best, Fabien

1 0

Re: [JLSE-Intel-NDA] Fwd: KNL Working Group Meeting - slides by Ron Green & John Levesque
by Harms, Kevin 04 May '16

04 May '16

not much to add. There was general agreement that SNC-4 mode isn't performing right so it's just not us. For the Cray results, these are bin1 parts and in cases where he compares "flat" mode, he means DDR. kevin >I was on vacation today and could not forward this before the call. Maybe someone who attended from perf engr can share any important things covered. > > >-Kumar > > >Begin forwarded message: > >From: "Nam, Hai Ah" <hnam(a)lanl.gov> > >Subject: KNL Working Group Meeting - slides by Ron Green & John Levesque > >Date: May 3, 2016 at 1:15:11 PM CDT > >To: "knlchatter(a)lanl.gov" <knlchatter(a)lanl.gov> > > >Hi Everyone, > >Here are the slides presented today at the KNL Working Group meeting by >Ron Green and John Levesque. Please note these are NDA and should not be >shared externally. > >If you have any follow-up questions/thoughts/suggestions, feel free to >email this list to chatter. Everyone here is interested in sharing KNL >information and covered under Intel NDA. > >If you have results you¹d like to share for the next meeting, please let >me know to get on the agenda. > >Thanks, > >Hai Ah > > > >--------------------------------------- >Hai Ah Nam, Ph.D. >Scientist, CCS-2 >Los Alamos National Laboratory >Phone: (505) 667-3768 >Email: hnam(a)lanl.gov > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

[JLSE-Intel-NDA] FYI - folks on this mailing list
by Kumaran, Kalyan 04 May '16

04 May '16

Just FYI that the folks on this mailing list so far have been from ALCF and MCS divisions. Now ALCF Theta ESP users, external to Argonne, are being provided access and added to this email list. Just FYI. -Kumar

1 0

[JLSE-Intel-NDA] Fwd: KNL Working Group Meeting - slides by Ron Green & John Levesque
by Kumaran, Kalyan 03 May '16

03 May '16

I was on vacation today and could not forward this before the call. Maybe someone who attended from perf engr can share any important things covered. -Kumar Begin forwarded message: From: "Nam, Hai Ah" <hnam(a)lanl.gov<mailto:[email protected]>> Subject: KNL Working Group Meeting - slides by Ron Green & John Levesque Date: May 3, 2016 at 1:15:11 PM CDT To: "knlchatter(a)lanl.gov<mailto:[email protected]>" <knlchatter(a)lanl.gov<mailto:[email protected]>> Hi Everyone, Here are the slides presented today at the KNL Working Group meeting by Ron Green and John Levesque. Please note these are NDA and should not be shared externally. If you have any follow-up questions/thoughts/suggestions, feel free to email this list to chatter. Everyone here is interested in sharing KNL information and covered under Intel NDA. If you have results you¹d like to share for the next meeting, please let me know to get on the agenda. Thanks, Hai Ah --------------------------------------- Hai Ah Nam, Ph.D. Scientist, CCS-2 Los Alamos National Laboratory Phone: (505) 667-3768 Email: hnam(a)lanl.gov<mailto:[email protected]>

1 0

Re: [JLSE-Intel-NDA] !DIR$ ATTRIBUTES FASTMEM in cache mode
by Vitali A. Morozov 03 May '16

03 May '16

Thanks for reporting it, Marta - I have been distracted yesterday and failed sending a message on this issue. We are facing with tho issues. First of, we see a non-intuitive behaviour of the default policy HBW_POLICY_PREFERRED. As documented "If insufficient memory is available from the high bandwidth NUMA node closest at allocation time, fall back to standard memory (default) with the smallest NUMA distance." This brings two questions. What happens if there is no HBW memory on the system? The expected answer - hbw_malloc (and its Fortran equivalent ATTRIBUTES FASTMEM) should behave equivalently to malloc. What happens on KNL in cache mode, where the is no memory other then DDR memory? The expected answer - hbw_malloc (and its Fortran equivalent) should behave equivalently to malloc. Marta and I have found that on both Xeon and KNL in cache mode both Fortran and C codes failed to allocate a small chunk of memory with hbw_malloc interface. We think this is a bug and it need to be reported and fixed. We found a workaround. Additionally to the mentioned conditional compilation (which is not an appropriate way to fix it), the user may define MEMKIND_HBW_NODE=0 This seems to direct all memory requests to numa node 0, and this exactly what hbw_malloc is expected to do if it runs out of memory on other numa nodes. Secondly, we seems to fail setting a policy from Fortran (not that the policy works as follows from the first report, but still). In C, the function "hbw_set_policy" changes the policy and the following "hbw_get_policy" returns a new policy. In Fortran the equivalent ierr = hbw_set_policy( %VAL(1) ) returns an error and the following ierr = hbw_get_policy() returns the default policy as if the previous function is ignored. The correspondent codes are in JLSE servers in /home/morozov/KNL/HSCD-intel/test5 for those of you who want to verify the reports. Best, Vitali On 05/02/2016 10:19 PM, Garcia Martinez, Marta wrote: > Hi, > > Vitali and I are using FASTMEM in a subroutine and it works well in flat mode. However, trying to run the same test in cache mode is causing problems (issues with not enough memory) at runtime (not compilation). > > !DIR$ ATTRIBUTES FASTMEM :: var > > A turnaround for the moment is to enclose that between an if defined declaration like this: > > !DEC$ IF DEFINED(name) > !DIR$ ATTRIBUTES FASTMEM :: var > !DEC$ ENDIF > > and then compile with the variable ‘name'. > > Marta > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] !DIR$ ATTRIBUTES FASTMEM in cache mode
by Garcia Martinez, Marta 02 May '16

02 May '16

Hi, Vitali and I are using FASTMEM in a subroutine and it works well in flat mode. However, trying to run the same test in cache mode is causing problems (issues with not enough memory) at runtime (not compilation). !DIR$ ATTRIBUTES FASTMEM :: var A turnaround for the moment is to enclose that between an if defined declaration like this: !DEC$ IF DEFINED(name) !DIR$ ATTRIBUTES FASTMEM :: var !DEC$ ENDIF and then compile with the variable ‘name'. Marta

1 0