March 2015 - Admins - mailman3.cels.anl.gov

[JLSE-Admins] Project Request Form
by Kazutomo Yoshii 31 Mar '15

31 Mar '15

From: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> a purchase request for Haswell nodes Kazutomo Yoshii I'm writing on behalf of MCS related projects. Since the "Things" nodes will be dedicated to LCF purposes, we need to purchase two additional Haswell node that have similar hardware configuration as the "Things" node because several MCS users depend on such node. Currently two project are listed in JLSE require haswell nodes: 1. Energy Savings and Performance Optimizations With Pliable Algorithms on Inte\ l’s E5-2699v3 (Haswell) Nodes (PIs: Adam Hammouda, Andrew Siegel and Pete Beckm\ an) 2. Exploring the Potential of Modern Processors (PI: Kazutomo Yoshii) Beside the above projects, I think Ron Rahaman and other CESAR folks are exploring cluster-on-die. I have an on-going work with Will Boyd @ MIT for application energy study. The members of the COOLR project are interested in. I'd like my student to have an access to the Haswell node this summer for low-level OS work. Aaron Knoll (currently at U. of Utah) is interested in many-core with a big memory configuration. Northwestern University, University of Chicago, University of Illinois, MIT, University of Utah, PNNL, LLNL CESAR, ARGO, COOLR and other pilot MCS projects two Haswell nodes with the "Things" hardware configuration with 512 GB or more memory. the quote is $14K each. [Schedule (when access is needed and for how long):] -- This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse)

1 0

Re: [JLSE-Admins] Haswell-EP Node
by Kazutomo Yoshii 31 Mar '15

31 Mar '15

Yes, $14k makes sense. I thought it's a real bargain:-) Are a couple of the Things nodes still available for MCS projects? or should I make a purchase request for additional Haswell nodes on behalf of MCS folks? (maybe this is a question to Kumar) I'll ask individual to write a project description if they have submitted it yet. The current version of sniper simulator does not scale well. It's barely scale to 10 cores. NUMA does not help. NUMA just adds additional complexity. A higher clock, single socket node may be a better choice, btw. I'll return the node I'm currently using on Fri. I'll be out of town anyway. - kaz On 03/30/2015 11:31 PM, Allen, Benjamin S. wrote: > Thanks for compiling all of the projects. > > I misread our recent quote. Thing nodes cost $14,135.45 each (double what I mentioned before). The E5-2699v3 goes for about $4200 per socket, so that cost shouldn't be all that surprising. > > It may make more sense to purchase hardware for file system benchmark clients to offload the ALCF Lustre and DDN IME projects from the Thing nodes versus purchasing more E5-2699v3 servers. > > Ben > >> On Mar 30, 2015, at 10:17 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> >> Currently two project are listed in JLSE require haswell nodes. >> >> - Energy Savings and Performance Optimizations With Pliable Algorithms on Intelï¿½s E5-2699v3 (Haswell) Nodes (PIs: Adam Hammouda, Andrew Siegel and Pete Beckman) >> >> - Exploring the Potential of Modern Processors (PI: Kazutomo Yoshii) >> >> >> Beside the above projects, I think Ron Rahaman and other CESAR folks are exploring cluster-on-die. >> >> I have an on-going work with a MIT guy for application energy study. >> >> The members of the COOLR project are interested in. I'd like my student to have >> an access to the haswell node this summer for low-level OS work. >> >> Aaron Knoll (currently at U. of Utah) is interested in many-core with a big memory configuration. >> He is visiting ANL this summer. >> >> I guess several people are interested in or may depend on haswell nodes. >> >> - kaz >> >> On 03/30/2015 09:53 PM, Allen, Benjamin S. wrote: >>> Yea lets wait till Haswell-EX (4 & 8 socket Haswell) ships before looking at a 4 socket machine. >>> >>> We're purchasing two more "Thing" machines to dedicate to the Intel Simulator project at minimum. What other projects do you have in mind? >>> >>> Ben >>> >>>> On Mar 30, 2015, at 9:49 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>> >>>> >>>> Thank you for the info! >>>> >>>> I think two or three projects depend on haswell nodes, >>>> so it would be nice to have additional "Things" nodes, at least two. >>>> (one for regular sharing and another one for time-sharing; isolated performance test or OS devel) >>>> >>>> If we can add more memory to the node, we can accommodate one more project. >>>> >>>> I was thinking to request a 4-way node but E7 SKUs may not be available yet >>>> or very expensive, so I'll postpone it. >>>> >>>> - kaz >>>> >>>> On 03/30/2015 06:54 PM, Allen, Benjamin S. wrote: >>>>> We can keep the HD installed, however I'd highly suggest you purchase a dedicated node, since LCF will be using these nodes for Lustre and DDN IME testing for the next few months. These projects are high priority. >>>>> >>>>> Cost for one of the "Thing" Haswell-EP nodes is $7,025.225 as of today. >>>>> >>>>> Ben >>>>> >>>>>> On Mar 30, 2015, at 5:50 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>>>> >>>>>> Ben, >>>>>> >>>>>> Thank you for letting me playing with the node! >>>>>> I'll unplug the node on this Fri. >>>>>> >>>>>> The machine helped my work a lot! >>>>>> One paper was accepted, which I haven't reported to JLSE yet. >>>>>> I have done a couple of other works, including a 4.0 Linux kernel bug fix. >>>>>> >>>>>> Are you able to keep the hard drive I installed to the node? >>>>>> I'd like to reboot the node with my disk image later when >>>>>> the machine is not busy. I have a couple of unfinished works related to Haswell. >>>>>> >>>>>> - kaz >>>>>> >>>>>> On 03/30/2015 04:51 PM, Allen, Benjamin S. wrote: >>>>>>> Hi Kaz, >>>>>>> >>>>>>> I need the Haswell-EP node back by next Monday. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Ben

1 0

Re: [JLSE-Admins] Haswell-EP Node
by Allen, Benjamin S. 30 Mar '15

30 Mar '15

Thanks for compiling all of the projects. I misread our recent quote. Thing nodes cost $14,135.45 each (double what I mentioned before). The E5-2699v3 goes for about $4200 per socket, so that cost shouldn't be all that surprising. It may make more sense to purchase hardware for file system benchmark clients to offload the ALCF Lustre and DDN IME projects from the Thing nodes versus purchasing more E5-2699v3 servers. Ben > On Mar 30, 2015, at 10:17 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > > Currently two project are listed in JLSE require haswell nodes. > > - Energy Savings and Performance Optimizations With Pliable Algorithms on Intel’s E5-2699v3 (Haswell) Nodes (PIs: Adam Hammouda, Andrew Siegel and Pete Beckman) > > - Exploring the Potential of Modern Processors (PI: Kazutomo Yoshii) > > > Beside the above projects, I think Ron Rahaman and other CESAR folks are exploring cluster-on-die. > > I have an on-going work with a MIT guy for application energy study. > > The members of the COOLR project are interested in. I'd like my student to have > an access to the haswell node this summer for low-level OS work. > > Aaron Knoll (currently at U. of Utah) is interested in many-core with a big memory configuration. > He is visiting ANL this summer. > > I guess several people are interested in or may depend on haswell nodes. > > - kaz > > On 03/30/2015 09:53 PM, Allen, Benjamin S. wrote: >> Yea lets wait till Haswell-EX (4 & 8 socket Haswell) ships before looking at a 4 socket machine. >> >> We're purchasing two more "Thing" machines to dedicate to the Intel Simulator project at minimum. What other projects do you have in mind? >> >> Ben >> >>> On Mar 30, 2015, at 9:49 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>> >>> >>> Thank you for the info! >>> >>> I think two or three projects depend on haswell nodes, >>> so it would be nice to have additional "Things" nodes, at least two. >>> (one for regular sharing and another one for time-sharing; isolated performance test or OS devel) >>> >>> If we can add more memory to the node, we can accommodate one more project. >>> >>> I was thinking to request a 4-way node but E7 SKUs may not be available yet >>> or very expensive, so I'll postpone it. >>> >>> - kaz >>> >>> On 03/30/2015 06:54 PM, Allen, Benjamin S. wrote: >>>> We can keep the HD installed, however I'd highly suggest you purchase a dedicated node, since LCF will be using these nodes for Lustre and DDN IME testing for the next few months. These projects are high priority. >>>> >>>> Cost for one of the "Thing" Haswell-EP nodes is $7,025.225 as of today. >>>> >>>> Ben >>>> >>>>> On Mar 30, 2015, at 5:50 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>>> >>>>> Ben, >>>>> >>>>> Thank you for letting me playing with the node! >>>>> I'll unplug the node on this Fri. >>>>> >>>>> The machine helped my work a lot! >>>>> One paper was accepted, which I haven't reported to JLSE yet. >>>>> I have done a couple of other works, including a 4.0 Linux kernel bug fix. >>>>> >>>>> Are you able to keep the hard drive I installed to the node? >>>>> I'd like to reboot the node with my disk image later when >>>>> the machine is not busy. I have a couple of unfinished works related to Haswell. >>>>> >>>>> - kaz >>>>> >>>>> On 03/30/2015 04:51 PM, Allen, Benjamin S. wrote: >>>>>> Hi Kaz, >>>>>> >>>>>> I need the Haswell-EP node back by next Monday. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ben >

1 0

Re: [JLSE-Admins] Haswell-EP Node
by Kazutomo Yoshii 30 Mar '15

30 Mar '15

Currently two project are listed in JLSE require haswell nodes. - Energy Savings and Performance Optimizations With Pliable Algorithms on Intelï¿½s E5-2699v3 (Haswell) Nodes (PIs: Adam Hammouda, Andrew Siegel and Pete Beckman) - Exploring the Potential of Modern Processors (PI: Kazutomo Yoshii) Beside the above projects, I think Ron Rahaman and other CESAR folks are exploring cluster-on-die. I have an on-going work with a MIT guy for application energy study. The members of the COOLR project are interested in. I'd like my student to have an access to the haswell node this summer for low-level OS work. Aaron Knoll (currently at U. of Utah) is interested in many-core with a big memory configuration. He is visiting ANL this summer. I guess several people are interested in or may depend on haswell nodes. - kaz On 03/30/2015 09:53 PM, Allen, Benjamin S. wrote: > Yea lets wait till Haswell-EX (4 & 8 socket Haswell) ships before looking at a 4 socket machine. > > We're purchasing two more "Thing" machines to dedicate to the Intel Simulator project at minimum. What other projects do you have in mind? > > Ben > >> On Mar 30, 2015, at 9:49 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> >> Thank you for the info! >> >> I think two or three projects depend on haswell nodes, >> so it would be nice to have additional "Things" nodes, at least two. >> (one for regular sharing and another one for time-sharing; isolated performance test or OS devel) >> >> If we can add more memory to the node, we can accommodate one more project. >> >> I was thinking to request a 4-way node but E7 SKUs may not be available yet >> or very expensive, so I'll postpone it. >> >> - kaz >> >> On 03/30/2015 06:54 PM, Allen, Benjamin S. wrote: >>> We can keep the HD installed, however I'd highly suggest you purchase a dedicated node, since LCF will be using these nodes for Lustre and DDN IME testing for the next few months. These projects are high priority. >>> >>> Cost for one of the "Thing" Haswell-EP nodes is $7,025.225 as of today. >>> >>> Ben >>> >>>> On Mar 30, 2015, at 5:50 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>> >>>> Ben, >>>> >>>> Thank you for letting me playing with the node! >>>> I'll unplug the node on this Fri. >>>> >>>> The machine helped my work a lot! >>>> One paper was accepted, which I haven't reported to JLSE yet. >>>> I have done a couple of other works, including a 4.0 Linux kernel bug fix. >>>> >>>> Are you able to keep the hard drive I installed to the node? >>>> I'd like to reboot the node with my disk image later when >>>> the machine is not busy. I have a couple of unfinished works related to Haswell. >>>> >>>> - kaz >>>> >>>> On 03/30/2015 04:51 PM, Allen, Benjamin S. wrote: >>>>> Hi Kaz, >>>>> >>>>> I need the Haswell-EP node back by next Monday. >>>>> >>>>> Thanks, >>>>> >>>>> Ben

1 0

Re: [JLSE-Admins] Haswell-EP Node
by Allen, Benjamin S. 30 Mar '15

30 Mar '15

Yea lets wait till Haswell-EX (4 & 8 socket Haswell) ships before looking at a 4 socket machine. We're purchasing two more "Thing" machines to dedicate to the Intel Simulator project at minimum. What other projects do you have in mind? Ben > On Mar 30, 2015, at 9:49 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > > Thank you for the info! > > I think two or three projects depend on haswell nodes, > so it would be nice to have additional "Things" nodes, at least two. > (one for regular sharing and another one for time-sharing; isolated performance test or OS devel) > > If we can add more memory to the node, we can accommodate one more project. > > I was thinking to request a 4-way node but E7 SKUs may not be available yet > or very expensive, so I'll postpone it. > > - kaz > > On 03/30/2015 06:54 PM, Allen, Benjamin S. wrote: >> We can keep the HD installed, however I'd highly suggest you purchase a dedicated node, since LCF will be using these nodes for Lustre and DDN IME testing for the next few months. These projects are high priority. >> >> Cost for one of the "Thing" Haswell-EP nodes is $7,025.225 as of today. >> >> Ben >> >>> On Mar 30, 2015, at 5:50 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>> >>> Ben, >>> >>> Thank you for letting me playing with the node! >>> I'll unplug the node on this Fri. >>> >>> The machine helped my work a lot! >>> One paper was accepted, which I haven't reported to JLSE yet. >>> I have done a couple of other works, including a 4.0 Linux kernel bug fix. >>> >>> Are you able to keep the hard drive I installed to the node? >>> I'd like to reboot the node with my disk image later when >>> the machine is not busy. I have a couple of unfinished works related to Haswell. >>> >>> - kaz >>> >>> On 03/30/2015 04:51 PM, Allen, Benjamin S. wrote: >>>> Hi Kaz, >>>> >>>> I need the Haswell-EP node back by next Monday. >>>> >>>> Thanks, >>>> >>>> Ben >

1 0

Re: [JLSE-Admins] Haswell-EP Node
by Kazutomo Yoshii 30 Mar '15

30 Mar '15

Thank you for the info! I think two or three projects depend on haswell nodes, so it would be nice to have additional "Things" nodes, at least two. (one for regular sharing and another one for time-sharing; isolated performance test or OS devel) If we can add more memory to the node, we can accommodate one more project. I was thinking to request a 4-way node but E7 SKUs may not be available yet or very expensive, so I'll postpone it. - kaz On 03/30/2015 06:54 PM, Allen, Benjamin S. wrote: > We can keep the HD installed, however I'd highly suggest you purchase a dedicated node, since LCF will be using these nodes for Lustre and DDN IME testing for the next few months. These projects are high priority. > > Cost for one of the "Thing" Haswell-EP nodes is $7,025.225 as of today. > > Ben > >> On Mar 30, 2015, at 5:50 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Ben, >> >> Thank you for letting me playing with the node! >> I'll unplug the node on this Fri. >> >> The machine helped my work a lot! >> One paper was accepted, which I haven't reported to JLSE yet. >> I have done a couple of other works, including a 4.0 Linux kernel bug fix. >> >> Are you able to keep the hard drive I installed to the node? >> I'd like to reboot the node with my disk image later when >> the machine is not busy. I have a couple of unfinished works related to Haswell. >> >> - kaz >> >> On 03/30/2015 04:51 PM, Allen, Benjamin S. wrote: >>> Hi Kaz, >>> >>> I need the Haswell-EP node back by next Monday. >>> >>> Thanks, >>> >>> Ben

1 0

Re: [JLSE-Admins] Haswell-EP Node
by Allen, Benjamin S. 30 Mar '15

30 Mar '15

We can keep the HD installed, however I'd highly suggest you purchase a dedicated node, since LCF will be using these nodes for Lustre and DDN IME testing for the next few months. These projects are high priority. Cost for one of the "Thing" Haswell-EP nodes is $7,025.225 as of today. Ben > On Mar 30, 2015, at 5:50 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Ben, > > Thank you for letting me playing with the node! > I'll unplug the node on this Fri. > > The machine helped my work a lot! > One paper was accepted, which I haven't reported to JLSE yet. > I have done a couple of other works, including a 4.0 Linux kernel bug fix. > > Are you able to keep the hard drive I installed to the node? > I'd like to reboot the node with my disk image later when > the machine is not busy. I have a couple of unfinished works related to Haswell. > > - kaz > > On 03/30/2015 04:51 PM, Allen, Benjamin S. wrote: >> Hi Kaz, >> >> I need the Haswell-EP node back by next Monday. >> >> Thanks, >> >> Ben >

1 0

Re: [JLSE-Admins] Haswell-EP Node
by Kazutomo Yoshii 30 Mar '15

30 Mar '15

Ben, Thank you for letting me playing with the node! I'll unplug the node on this Fri. The machine helped my work a lot! One paper was accepted, which I haven't reported to JLSE yet. I have done a couple of other works, including a 4.0 Linux kernel bug fix. Are you able to keep the hard drive I installed to the node? I'd like to reboot the node with my disk image later when the machine is not busy. I have a couple of unfinished works related to Haswell. - kaz On 03/30/2015 04:51 PM, Allen, Benjamin S. wrote: > Hi Kaz, > > I need the Haswell-EP node back by next Monday. > > Thanks, > > Ben

1 0

[JLSE-Admins] Project Request Form
by Ken Raffenetti 30 Mar '15

30 Mar '15

From: Ken Raffenetti <raffenet(a)mcs.anl.gov> MPICH/OFI Testing and Validation Ken Raffenetti MCS The OpenFabrics Interfaces Working Group is actively defining a new, hardware independent API for HPC networks. A reference implementation of the API, usable over various network configurations, is available at https://github.com/ofiwg/libfabric. Intel has contributed support for libfabric in the form of an MPICH netmod. In order to better test and support this netmod, Intel also donated some TrueScale InfiniBand hardware (4 HCAs, 1 switch), with the expectation that we will integrate it into our Jenkins testing framework. ASCR Intel Ken Raffenetti I am prepared to contribute the Intel supplied hardware to JLSE. To start, a single node + HCA is sufficient. It can run the default Linux build (plus any additional packages needed to support the PSM network API on TrueScale hardware). The MPICH Jenkins server lives in MCS IP space. We will need access to the greater ANL network, either via proxy or public interface in order to complete the setup. [Schedule (when access is needed and for how long):] -- This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse)

1 0

[JLSE-Admins] Haswell-EP Node
by Allen, Benjamin S. 30 Mar '15

30 Mar '15

Hi Kaz, I need the Haswell-EP node back by next Monday. Thanks, Ben

1 0