January 2015 - Admins - mailman3.cels.anl.gov

Re: [JLSE-Admins] network to tinkerlab
by Allen, Benjamin S. 13 Jan '15

13 Jan '15

Hi Linda and Corby, Where are we at getting the network cabinet replaced in the Tinkerlab? Thanks, Ben > On Jan 13, 2015, at 12:16 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Hi Ben, > > Could you remind me who is working on the network to tinkerlab? > > Tinkerlab intel compiler's license will expire soon. > One of the solution is to use the JLSE license server but > I guess that requires the network to be configured. > > - kaz > > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

[JLSE-Admins] network to tinkerlab
by Kazutomo Yoshii 13 Jan '15

13 Jan '15

Hi Ben, Could you remind me who is working on the network to tinkerlab? Tinkerlab intel compiler's license will expire soon. One of the solution is to use the JLSE license server but I guess that requires the network to be configured. - kaz

1 0

Re: [JLSE-Admins] The progress of stencil-
by Kazutomo Yoshii 12 Jan '15

12 Jan '15

Hi Ben, Thank you! 4096 would be enough for now. If you want to set a larger value, of course, no problem :-) - kaz On 01/12/2015 04:30 PM, Allen, Benjamin S. wrote: > Hi Kaz, > > Started working on this. I'm going to set the two sysctl values by default on the JLSE RHEL7 image, and add a noauto,user entry in fstab. This will let users mount and unmount it as will. > >> sysctl vm.nr_hugepages=Y > This takes a numerical value not "Y": > > /proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge > pages in the kernel's huge page pool. "Persistent" huge pages will be > returned to the huge page pool when freed by a task. A user with root > privileges can dynamically allocate more or free some persistent huge pages > by increasing or decreasing the value of 'nr_hugepages'. > > Please advise on what it should be set to. > > Thanks, > > Ben > > >> On Jan 9, 2015, at 3:38 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Hi Ben, >> >> Are you able to give a sudo access on Haswell nodes to Xiamin, Junchao (Pavan's group), >> or create a separate hugetlbfs mount for them (see the command below)? >> >> They may need an isolated environment for performance benchmark. >> At this point, it seems very few people uses haswell, probably we don't need any action >> for them. >> >> - kaz >> >> -------- Forwarded Message -------- >> Subject: Re: The progress of stencil- >> Date: Fri, 9 Jan 2015 14:17:00 -0600 >> From: woodslong <woodslong(a)gmail.com> >> To: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> >> CC: Pavan <balaji(a)anl.gov>, Junchao Zhang <junchao.zhang(a)gmail.com>, Shigang Li <shigangli.cs(a)gmail.com>, Yunquan Zhang <yunquan.cas(a)gmail.com> >> >> Hi, Kaz >> The necessary operations to use huge page are shown below(most copy from your email): >> >> >> $ sysctl vm.hugetlb_shm_group=X # X >> is the group id for sharing, be sure that my account is the >> this group >> >> $ sysctl vm.nr_hugepages=Y >> $ echo always > /sys/kernel/mm/transparent_hugepage/enabled (maybe not needed) >> $ mkdir -p /mnt/hugetlbfs >> >> $ mount -t hugetlbfs none >> /mnt/hugetlbfs -o mode=1770,gid=X >> >> >> Thanks! >> Xiaomin Zhu >> >> >> >> >> Original Message >> >> >> >> Sender: woodslong<woodslong(a)gmail.com> >> Recipient: Pavan<balaji(a)anl.gov> >> Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >> Date: Friday, Jan 9, 2015 10:03 >> Subject: Re: The progress of stencil >> >> Hi, all >> Just now, I discussed with Junchao. >> We think Cache miss and page fault are the two factors. I will try to verify that. >> The less floating computation for MPI3 is not accurate. Please just ignore it. >> >> Thanks, >> >> Xiaomin Zhu >> >> >> >> >> Original Message >> >> >> >> Sender: woodslong<woodslong(a)gmail.com> >> Recipient: Pavan<balaji(a)anl.gov> >> Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >> Date: Friday, Jan 9, 2015 09:35 >> Subject: Re: The progress of stencil >> >> OK! Pavan >> I suddenly remembered another issue which affects the performance, that is the cache miss rate. >> >> As the MPI3 version removes the ghost area then changes the data structure of the matrix, as well as the data layout. According with the data access pattern of a grid update, this cause more cache miss if the matrix size happens to be power of 2. In the former profiling, I used 4096*4096 matrix, so the MPI3 version has more cache miss. >> >> Just now, I did another profiling with matrix 5034*5034, and MPI3 shows a better performance (only between MPI1 without huge page and MPI3 without huge page, noncontig), as MPI3 version has less floating point computations and the cache miss should be nearly the same with this random matrix size. >> >> I am sorry I forgot that. I will add more profiling which contains all the 6 versions. If all the mpi3 version showed a better performance, it will be good. >> >> Xiaomin Zhu >> >> >> >> >> Original Message >> >> >> >> Sender: Pavan" <balaji(a)anl.gov> >> Recipient: woodslong<woodslong(a)gmail.com> >> Cc: Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >> Date: Friday, Jan 9, 2015 08:09 >> Subject: Re: The progress of stencil >> >> >> We should write the paper as-is. We can say that we identified some issues, but not all. That's OK. But we can't say that it's impossible to fix, unless we know what the reason is and can demonstrate that it's impossible to fix. >> >> Sent from my iPad >> >> On Jan 9, 2015, at 6:56 PM, woodslong <woodslong(a)gmail.com> wrote: >> >>> OK, Pavan >>> >>> Then how do you think about the future work? >>> >>> >>> I think if the reason is shared memory mechanism itself, it will be difficult to fix. Do you have any suggestions on the future work? >>> >>> >>> Xiaomin Zhu >>> >>> Original Message >>> >>> >>> >>> Sender: Pavan" <balaji(a)anl.gov> >>> Recipient: Xiaomin Zhu<woodslong(a)gmail.com> >>> Cc: Pavan" <balaji(a)anl.gov>; Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>> Date: Friday, Jan 9, 2015 00:55 >>> Subject: Re: The progress of stencil >>> >>>> On Jan 9, 2015, at 7:45 AM, woodslong <woodslong(a)gmail.com >>>> wrote: >>>> >>>> (3-4) Our aim is to make the shared memory have the same performance with malloc, but it seems impossible as the related overhead is inevitable. >>> I'm not sure how you came to this conclusion. >>> >>> So far, we identified two problems: (1) compiler aliasing, and (2) huge pages. You are saying that there is at least one more reason that's causing performance degradation that we haven't identified yet. But that doesn't mean it's *impossible* to fix. >>> >>> -- Pavan >>> >>> -- >>> Pavan Balaji ✉️ >>> >>> http://www.mcs.anl.gov/~balaji >>> >>> >>> >> >> _______________________________________________ >> Admins mailing list >> Admins(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] The progress of stencil-
by Allen, Benjamin S. 12 Jan '15

12 Jan '15

Hi Kaz, Started working on this. I'm going to set the two sysctl values by default on the JLSE RHEL7 image, and add a noauto,user entry in fstab. This will let users mount and unmount it as will. > sysctl vm.nr_hugepages=Y This takes a numerical value not "Y": /proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge pages in the kernel's huge page pool. "Persistent" huge pages will be returned to the huge page pool when freed by a task. A user with root privileges can dynamically allocate more or free some persistent huge pages by increasing or decreasing the value of 'nr_hugepages'. Please advise on what it should be set to. Thanks, Ben > On Jan 9, 2015, at 3:38 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Hi Ben, > > Are you able to give a sudo access on Haswell nodes to Xiamin, Junchao (Pavan's group), > or create a separate hugetlbfs mount for them (see the command below)? > > They may need an isolated environment for performance benchmark. > At this point, it seems very few people uses haswell, probably we don't need any action > for them. > > - kaz > > -------- Forwarded Message -------- > Subject: Re: The progress of stencil- > Date: Fri, 9 Jan 2015 14:17:00 -0600 > From: woodslong <woodslong(a)gmail.com> > To: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> > CC: Pavan <balaji(a)anl.gov>, Junchao Zhang <junchao.zhang(a)gmail.com>, Shigang Li <shigangli.cs(a)gmail.com>, Yunquan Zhang <yunquan.cas(a)gmail.com> > > Hi, Kaz > The necessary operations to use huge page are shown below(most copy from your email): > > > $ sysctl vm.hugetlb_shm_group=X # X > is the group id for sharing, be sure that my account is the > this group > > $ sysctl vm.nr_hugepages=Y > $ echo always > /sys/kernel/mm/transparent_hugepage/enabled (maybe not needed) > $ mkdir -p /mnt/hugetlbfs > > $ mount -t hugetlbfs none > /mnt/hugetlbfs -o mode=1770,gid=X > > > Thanks! > Xiaomin Zhu > > > > > Original Message > > > > Sender: woodslong<woodslong(a)gmail.com> > Recipient: Pavan<balaji(a)anl.gov> > Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> > Date: Friday, Jan 9, 2015 10:03 > Subject: Re: The progress of stencil > > Hi, all > Just now, I discussed with Junchao. > We think Cache miss and page fault are the two factors. I will try to verify that. > The less floating computation for MPI3 is not accurate. Please just ignore it. > > Thanks, > > Xiaomin Zhu > > > > > Original Message > > > > Sender: woodslong<woodslong(a)gmail.com> > Recipient: Pavan<balaji(a)anl.gov> > Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> > Date: Friday, Jan 9, 2015 09:35 > Subject: Re: The progress of stencil > > OK! Pavan > I suddenly remembered another issue which affects the performance, that is the cache miss rate. > > As the MPI3 version removes the ghost area then changes the data structure of the matrix, as well as the data layout. According with the data access pattern of a grid update, this cause more cache miss if the matrix size happens to be power of 2. In the former profiling, I used 4096*4096 matrix, so the MPI3 version has more cache miss. > > Just now, I did another profiling with matrix 5034*5034, and MPI3 shows a better performance (only between MPI1 without huge page and MPI3 without huge page, noncontig), as MPI3 version has less floating point computations and the cache miss should be nearly the same with this random matrix size. > > I am sorry I forgot that. I will add more profiling which contains all the 6 versions. If all the mpi3 version showed a better performance, it will be good. > > Xiaomin Zhu > > > > > Original Message > > > > Sender: Pavan" <balaji(a)anl.gov> > Recipient: woodslong<woodslong(a)gmail.com> > Cc: Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> > Date: Friday, Jan 9, 2015 08:09 > Subject: Re: The progress of stencil > > > We should write the paper as-is. We can say that we identified some issues, but not all. That's OK. But we can't say that it's impossible to fix, unless we know what the reason is and can demonstrate that it's impossible to fix. > > Sent from my iPad > > On Jan 9, 2015, at 6:56 PM, woodslong <woodslong(a)gmail.com> wrote: > >> OK, Pavan >> >> Then how do you think about the future work? >> >> >> I think if the reason is shared memory mechanism itself, it will be difficult to fix. Do you have any suggestions on the future work? >> >> >> Xiaomin Zhu >> >> Original Message >> >> >> >> Sender: Pavan" <balaji(a)anl.gov> >> Recipient: Xiaomin Zhu<woodslong(a)gmail.com> >> Cc: Pavan" <balaji(a)anl.gov>; Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >> Date: Friday, Jan 9, 2015 00:55 >> Subject: Re: The progress of stencil >> >> > On Jan 9, 2015, at 7:45 AM, woodslong <woodslong(a)gmail.com >> > wrote: >> > >> > (3-4) Our aim is to make the shared memory have the same performance with malloc, but it seems impossible as the related overhead is inevitable. >> >> I'm not sure how you came to this conclusion. >> >> So far, we identified two problems: (1) compiler aliasing, and (2) huge pages. You are saying that there is at least one more reason that's causing performance degradation that we haven't identified yet. But that doesn't mean it's *impossible* to fix. >> >> -- Pavan >> >> -- >> Pavan Balaji ✉️ >> >> http://www.mcs.anl.gov/~balaji >> >> >> > > > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

[JLSE-Admins] cluster-on-die on haswell
by Kazutomo Yoshii 12 Jan '15

12 Jan '15

Hi, I finally found the option for 'cluster-on-die' in BIOS, which sets two NUMA domains per CPU. On JLSE haswell nodes, it's going to be four NUMA domains. node distances: node 0 1 2 3 0: 10 21 31 31 1: 21 10 31 31 2: 31 31 10 21 3: 31 31 21 10 Ron and Andrew in the CESAR group are interested in comparing the cluster-on die performance with regular performance on their codes I guess possibly other folks are interested in running in this mode. - kaz

1 0

Re: [JLSE-Admins] Project Request Form
by Kazutomo Yoshii 12 Jan '15

12 Jan '15

Ben, Thank you for your help! I don't care much about the speed of hard drive. Any drive would be fine as long as it works. - kaz On 01/09/2015 02:43 PM, Allen, Benjamin S. wrote: > Hi Kaz, > > Yea its possible. I still need to move my infrastructure off the JLSE IPMI network before I give the login nodes access to it. This is to isolate production JLSE IPMI interfaces from user access. I hope to figure that out next week. > > The Haswell nodes currently don't have any storage installed, so we'll either need to order something or borrow. > > Ben > > >> On Jan 9, 2015, at 1:58 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Ben: >> >> Adam needs the followings. >> >> - IPMI access to change BIOS and reboot the machine >> - a dedicated disk partition to boot a custom image ( e.g., centoos plus >> a custom Linux kernel) >> >> Is this kind of stuff possible in the JLSE machine space? >> >> - kaz >> >> On 01/09/2015 12:44 PM, Thakur, Rajeev wrote: >>>> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. >>> Adam, >>> You should be able to do what you want on the Haswell nodes we have in JLSE. Kaz can help you with that. >>> >>> Rajeev >>> >>> >>> On Dec 23, 2014, at 12:59 PM, Adam Hammouda <ahammouda(a)cs.uchicago.edu> wrote: >>> >>>> From: Adam Hammouda <ahammouda(a)cs.uchicago.edu> >>>> Energy Savings and Performance Optimizations With Pliable Algorithms on Intelï¿½s E5-2699v3 (Haswell) Nodes >>>> Adam Hammouda, Andrew Siegel, Pete Beckman >>>> MCS >>>> The goal of our research is to study operating system design decisions for next generation HPC architectures. In particular, we aim to study the impact which those decisions have on application performance characteristics. Further, we aim to understand how these decisions and performance characteristics relate to the thermal stability and energy effeciency of the architecture in question. Broad questions of interest to our research include the following: >>>> >>>> ï¿½ Is there a layer of the HPC runtime environment best equipped for mitigating thermal load imbalances (i.e. OS, runtime system, or application)? >>>> ï¿½ If it is more optimal to share these responsibilities between layers, what is the optimal division of labor? >>>> ï¿½ What are the best metrics for the above referenced optimality? >>>> ï¿½ With such metrics, what are acceptable tradeoffs between performance and energy? >>>> ï¿½ What are generalizable strategies for the application layer to handle thermal load imbalances? What is a generalizable strategy for the application layer to receive and share information with the runtime system? >>>> >>>> In an earlier work, we reformulated an algorithm class of bulk-synchronous computation to account for nonuniformities in process execution rates. The challenge of energy and thermal efficiency presented by next generation machines is precisely what motivated our algorithmic developments in this previous work. The COOLR project presents an opportunity to pursue this line of inquiry even further by exploring the precise characteristics of these anticipated nonuniformities. >>>> To this end we have already begun our explorations of the Intel Xeon E5-2670 machine available to us. We are exploring the effects of per-socket controls over cpu frequency, and pstate setings, and we have developed metrics to quantify the tradeoffs in performance and thermal load balance. However, our algorithmic work in Hammouda and Siegel 2013 was designed to exploit per-process nonuniformities in execution rates, and we therefore could benefit greatly from a machine which provides access to these controls. >>>> COOLR, CESAR >>>> Northwestern >>>> Adam Hammouda, Kaicheng Zhang (Northwestern) >>>> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. >>>> [Schedule (when access is needed and for how long):] >>>> >>>> >>>> -- >>>> This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) >>>> >>>> _______________________________________________ >>>> Admins mailing list >>>> Admins(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/admins >>> _______________________________________________ >>> Admins mailing list >>> Admins(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/admins >> _______________________________________________ >> Admins mailing list >> Admins(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

[JLSE-Admins] Fwd: Re: The progress of stencil-
by Kazutomo Yoshii 09 Jan '15

09 Jan '15

Hi Ben, Are you able to give a sudo access on Haswell nodes to Xiamin, Junchao (Pavan's group), or create a separate hugetlbfs mount for them (see the command below)? They may need an isolated environment for performance benchmark. At this point, it seems very few people uses haswell, probably we don't need any action for them. - kaz -------- Forwarded Message -------- Subject: Re: The progress of stencil- Date: Fri, 9 Jan 2015 14:17:00 -0600 From: woodslong <woodslong(a)gmail.com> To: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> CC: Pavan <balaji(a)anl.gov>, Junchao Zhang <junchao.zhang(a)gmail.com>, Shigang Li <shigangli.cs(a)gmail.com>, Yunquan Zhang <yunquan.cas(a)gmail.com> Hi, Kaz The necessary operations to use huge page are shown below(most copy from your email): $ sysctl vm.hugetlb_shm_group=X # X is the group id for sharing, be sure that my account is the this group $ sysctl vm.nr_hugepages=Y $ echo always > /sys/kernel/mm/transparent_hugepage/enabled (maybe not needed) $ mkdir -p /mnt/hugetlbfs $ mount -t hugetlbfs none /mnt/hugetlbfs -o mode=1770,gid=X Thanks! Xiaomin Zhu Original Message *Sender:* woodslong<woodslong(a)gmail.com> *Recipient:* Pavan<balaji(a)anl.gov> *Cc:* Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> *Date:* Friday, Jan 9, 2015 10:03 *Subject:* Re: The progress of stencil Hi, all Just now, I discussed with Junchao. We think Cache miss and page fault are the two factors. I will try to verify that. The less floating computation for MPI3 is not accurate. Please just ignore it. Thanks, Xiaomin Zhu Original Message *Sender:* woodslong<woodslong(a)gmail.com <mailto:[email protected]>> *Recipient:* Pavan<balaji(a)anl.gov <mailto:[email protected]>> *Cc:* Kazutomo Yoshii<kazutomo(a)mcs.anl.gov <mailto:[email protected]>>; Junchao Zhang<junchao.zhang(a)gmail.com <mailto:[email protected]>>; Shigang Li<shigangli.cs(a)gmail.com <mailto:[email protected]>>; Yunquan Zhang<yunquan.cas(a)gmail.com <mailto:[email protected]>> *Date:* Friday, Jan 9, 2015 09:35 *Subject:* Re: The progress of stencil OK! Pavan I suddenly remembered another issue which affects the performance, that is the cache miss rate. As the MPI3 version removes the ghost area then changes the data structure of the matrix, as well as the data layout. According with the data access pattern of a grid update, this cause more cache miss if the matrix size happens to be power of 2. In the former profiling, I used 4096*4096 matrix, so the MPI3 version has more cache miss. Just now, I did another profiling with matrix 5034*5034, and MPI3 shows a better performance (only between MPI1 without huge page and MPI3 without huge page, noncontig), as MPI3 version has less floating point computations and the cache miss should be nearly the same with this random matrix size. I am sorry I forgot that. I will add more profiling which contains all the 6 versions. If all the mpi3 version showed a better performance, it will be good. Xiaomin Zhu Original Message *Sender:* Pavan" <balaji(a)anl.gov <mailto:[email protected]>> *Recipient:* woodslong<woodslong(a)gmail.com <mailto:[email protected]>> *Cc:* Kazutomo" <kazutomo(a)mcs.anl.gov <mailto:[email protected]>>; Junchao Zhang<junchao.zhang(a)gmail.com <mailto:[email protected]>>; Shigang Li<shigangli.cs(a)gmail.com <mailto:[email protected]>>; Yunquan Zhang<yunquan.cas(a)gmail.com <mailto:[email protected]>> *Date:* Friday, Jan 9, 2015 08:09 *Subject:* Re: The progress of stencil We should write the paper as-is. We can say that we identified some issues, but not all. That's OK. But we can't say that it's impossible to fix, unless we know what the reason is and can demonstrate that it's impossible to fix. Sent from my iPad On Jan 9, 2015, at 6:56 PM, woodslong <woodslong(a)gmail.com <mailto:[email protected]>> wrote: > OK, Pavan > > Then how do you think about the future work? > > I think if the reason is shared memory mechanism itself, it will be > difficult to fix. Do you have any suggestions on the future work? > > > Xiaomin Zhu > > Original Message > *Sender:* Pavan" <balaji(a)anl.gov <mailto:[email protected]>> > *Recipient:* Xiaomin Zhu<woodslong(a)gmail.com <mailto:[email protected]>> > *Cc:* Pavan" <balaji(a)anl.gov <mailto:[email protected]>>; Kazutomo" > <kazutomo(a)mcs.anl.gov <mailto:[email protected]>>; Junchao > Zhang<junchao.zhang(a)gmail.com <mailto:[email protected]>>; > Shigang Li<shigangli.cs(a)gmail.com <mailto:[email protected]>>; > Yunquan Zhang<yunquan.cas(a)gmail.com <mailto:[email protected]>> > *Date:* Friday, Jan 9, 2015 00:55 > *Subject:* Re: The progress of stencil > > > On Jan 9, 2015, at 7:45 AM, woodslong <woodslong(a)gmail.com <mailto:[email protected]>> wrote: > > > > (3-4) Our aim is to make the shared memory have the same performance with malloc, but it seems impossible as the related overhead is inevitable. > > I'm not sure how you came to this conclusion. > > So far, we identified two problems: (1) compiler aliasing, and (2) huge pages. You are saying that there is at least one more reason that's causing performance degradation that we haven't identified yet. But that doesn't mean it's *impossible* to fix. > > -- Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji <http://www.mcs.anl.gov/%7Ebalaji> >

1 0

Re: [JLSE-Admins] Project Request Form
by Allen, Benjamin S. 09 Jan '15

09 Jan '15

Hi Kaz, Yea its possible. I still need to move my infrastructure off the JLSE IPMI network before I give the login nodes access to it. This is to isolate production JLSE IPMI interfaces from user access. I hope to figure that out next week. The Haswell nodes currently don't have any storage installed, so we'll either need to order something or borrow. Ben > On Jan 9, 2015, at 1:58 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Ben: > > Adam needs the followings. > > - IPMI access to change BIOS and reboot the machine > - a dedicated disk partition to boot a custom image ( e.g., centoos plus > a custom Linux kernel) > > Is this kind of stuff possible in the JLSE machine space? > > - kaz > > On 01/09/2015 12:44 PM, Thakur, Rajeev wrote: >>> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. >> Adam, >> You should be able to do what you want on the Haswell nodes we have in JLSE. Kaz can help you with that. >> >> Rajeev >> >> >> On Dec 23, 2014, at 12:59 PM, Adam Hammouda <ahammouda(a)cs.uchicago.edu> wrote: >> >>> From: Adam Hammouda <ahammouda(a)cs.uchicago.edu> >>> Energy Savings and Performance Optimizations With Pliable Algorithms on Intel’s E5-2699v3 (Haswell) Nodes >>> Adam Hammouda, Andrew Siegel, Pete Beckman >>> MCS >>> The goal of our research is to study operating system design decisions for next generation HPC architectures. In particular, we aim to study the impact which those decisions have on application performance characteristics. Further, we aim to understand how these decisions and performance characteristics relate to the thermal stability and energy effeciency of the architecture in question. Broad questions of interest to our research include the following: >>> >>> • Is there a layer of the HPC runtime environment best equipped for mitigating thermal load imbalances (i.e. OS, runtime system, or application)? >>> • If it is more optimal to share these responsibilities between layers, what is the optimal division of labor? >>> • What are the best metrics for the above referenced optimality? >>> • With such metrics, what are acceptable tradeoffs between performance and energy? >>> • What are generalizable strategies for the application layer to handle thermal load imbalances? What is a generalizable strategy for the application layer to receive and share information with the runtime system? >>> >>> In an earlier work, we reformulated an algorithm class of bulk-synchronous computation to account for nonuniformities in process execution rates. The challenge of energy and thermal efficiency presented by next generation machines is precisely what motivated our algorithmic developments in this previous work. The COOLR project presents an opportunity to pursue this line of inquiry even further by exploring the precise characteristics of these anticipated nonuniformities. >>> To this end we have already begun our explorations of the Intel Xeon E5-2670 machine available to us. We are exploring the effects of per-socket controls over cpu frequency, and pstate setings, and we have developed metrics to quantify the tradeoffs in performance and thermal load balance. However, our algorithmic work in Hammouda and Siegel 2013 was designed to exploit per-process nonuniformities in execution rates, and we therefore could benefit greatly from a machine which provides access to these controls. >>> COOLR, CESAR >>> Northwestern >>> Adam Hammouda, Kaicheng Zhang (Northwestern) >>> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. >>> [Schedule (when access is needed and for how long):] >>> >>> >>> -- >>> This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) >>> >>> _______________________________________________ >>> Admins mailing list >>> Admins(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/admins >> _______________________________________________ >> Admins mailing list >> Admins(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/admins > > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] Project Request Form
by Kazutomo Yoshii 09 Jan '15

09 Jan '15

Ben: Adam needs the followings. - IPMI access to change BIOS and reboot the machine - a dedicated disk partition to boot a custom image ( e.g., centoos plus a custom Linux kernel) Is this kind of stuff possible in the JLSE machine space? - kaz On 01/09/2015 12:44 PM, Thakur, Rajeev wrote: >> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. > Adam, > You should be able to do what you want on the Haswell nodes we have in JLSE. Kaz can help you with that. > > Rajeev > > > On Dec 23, 2014, at 12:59 PM, Adam Hammouda <ahammouda(a)cs.uchicago.edu> wrote: > >> From: Adam Hammouda <ahammouda(a)cs.uchicago.edu> >> Energy Savings and Performance Optimizations With Pliable Algorithms on Intelï¿½s E5-2699v3 (Haswell) Nodes >> Adam Hammouda, Andrew Siegel, Pete Beckman >> MCS >> The goal of our research is to study operating system design decisions for next generation HPC architectures. In particular, we aim to study the impact which those decisions have on application performance characteristics. Further, we aim to understand how these decisions and performance characteristics relate to the thermal stability and energy effeciency of the architecture in question. Broad questions of interest to our research include the following: >> >> ï¿½ Is there a layer of the HPC runtime environment best equipped for mitigating thermal load imbalances (i.e. OS, runtime system, or application)? >> ï¿½ If it is more optimal to share these responsibilities between layers, what is the optimal division of labor? >> ï¿½ What are the best metrics for the above referenced optimality? >> ï¿½ With such metrics, what are acceptable tradeoffs between performance and energy? >> ï¿½ What are generalizable strategies for the application layer to handle thermal load imbalances? What is a generalizable strategy for the application layer to receive and share information with the runtime system? >> >> In an earlier work, we reformulated an algorithm class of bulk-synchronous computation to account for nonuniformities in process execution rates. The challenge of energy and thermal efficiency presented by next generation machines is precisely what motivated our algorithmic developments in this previous work. The COOLR project presents an opportunity to pursue this line of inquiry even further by exploring the precise characteristics of these anticipated nonuniformities. >> To this end we have already begun our explorations of the Intel Xeon E5-2670 machine available to us. We are exploring the effects of per-socket controls over cpu frequency, and pstate setings, and we have developed metrics to quantify the tradeoffs in performance and thermal load balance. However, our algorithmic work in Hammouda and Siegel 2013 was designed to exploit per-process nonuniformities in execution rates, and we therefore could benefit greatly from a machine which provides access to these controls. >> COOLR, CESAR >> Northwestern >> Adam Hammouda, Kaicheng Zhang (Northwestern) >> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. >> [Schedule (when access is needed and for how long):] >> >> >> -- >> This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) >> >> _______________________________________________ >> Admins mailing list >> Admins(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/admins > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] Project Request Form
by Thakur, Rajeev 09 Jan '15

09 Jan '15

> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. Adam, You should be able to do what you want on the Haswell nodes we have in JLSE. Kaz can help you with that. Rajeev On Dec 23, 2014, at 12:59 PM, Adam Hammouda <ahammouda(a)cs.uchicago.edu> wrote: > From: Adam Hammouda <ahammouda(a)cs.uchicago.edu> > Energy Savings and Performance Optimizations With Pliable Algorithms on Intel’s E5-2699v3 (Haswell) Nodes > Adam Hammouda, Andrew Siegel, Pete Beckman > MCS > The goal of our research is to study operating system design decisions for next generation HPC architectures. In particular, we aim to study the impact which those decisions have on application performance characteristics. Further, we aim to understand how these decisions and performance characteristics relate to the thermal stability and energy effeciency of the architecture in question. Broad questions of interest to our research include the following: > > • Is there a layer of the HPC runtime environment best equipped for mitigating thermal load imbalances (i.e. OS, runtime system, or application)? > • If it is more optimal to share these responsibilities between layers, what is the optimal division of labor? > • What are the best metrics for the above referenced optimality? > • With such metrics, what are acceptable tradeoffs between performance and energy? > • What are generalizable strategies for the application layer to handle thermal load imbalances? What is a generalizable strategy for the application layer to receive and share information with the runtime system? > > In an earlier work, we reformulated an algorithm class of bulk-synchronous computation to account for nonuniformities in process execution rates. The challenge of energy and thermal efficiency presented by next generation machines is precisely what motivated our algorithmic developments in this previous work. The COOLR project presents an opportunity to pursue this line of inquiry even further by exploring the precise characteristics of these anticipated nonuniformities. > To this end we have already begun our explorations of the Intel Xeon E5-2670 machine available to us. We are exploring the effects of per-socket controls over cpu frequency, and pstate setings, and we have developed metrics to quantify the tradeoffs in performance and thermal load balance. However, our algorithmic work in Hammouda and Siegel 2013 was designed to exploit per-process nonuniformities in execution rates, and we therefore could benefit greatly from a machine which provides access to these controls. > COOLR, CESAR > Northwestern > Adam Hammouda, Kaicheng Zhang (Northwestern) > E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. > [Schedule (when access is needed and for how long):] > > > -- > This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) > > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0