March 2015 - Admins - mailman3.cels.anl.gov

[JLSE-Admins] multi-hop SSH to JLSE machines?
by Ron Rahaman 27 Mar '15

27 Mar '15

Hi all, I'd like to configure a multi-hop ssh connection to some of the JLSE machines, but I"m having issues. On my local machine, I put the following in my ~/.ssh/config Host jlseLogin > Hostname login.jlse.anl.gov > User rahaman > Host maud > ProxyCommand ssh -q jlseLogin nc -q0 %h 22 > User rahaman But the version of netcat on the JLSE login node doesn't recognize many options I'm used to, and I can't get it to establish a connection. Similar commands work fine on the MCS login node, where the version of netcat is different. For example, this works fine: Host mcsLogin > Hostname login.mcs.anl.gov > User rahaman > Host thrash > ProxyCommand ssh -q mcsLogin nc -q0 %h 22 > User rahaman Thanks! Ron Argonne National Laboratory, Math and Computer Science Division University of Chicago, Department of Computer Science

1 0

Re: [JLSE-Admins] account for Pablo Llopis
by Kazutomo Yoshii 19 Mar '15

19 Mar '15

Ben, I'm using Google Authenticator for my google accounts, which is nice. I think only requirement is to have a iPhone or an android device. I think it's reasonable. Yubikey is also nice but costs us a bit? how about durability? I probably attach Yubikey to a key chain and put it into my pocket. I understand that he can not sudo in the default JLSE image unless he request his ALCF account. It would be nice if he can sudo on things. Can we boot one of the Things nodes from a local drive when he wants to use it? - kaz On 03/17/2015 02:49 PM, Allen, Benjamin S. wrote: >> Does he need to request ALCF account (in order to obtain his crypto card)? >> or can JLSE provide a crypto card directly to him? > We cannot use JLSE as a justification for creating him an ALCF account, so unless he has other ALCF project duties we'll have to use another second factor authentication method. > > I assume he requires this for privileged access within JLSE? If not then he has no need for a cryptocard. > > I've been investigating Google Authenticator style one time passwords as a second factor. We could also start supporting Yubikey with relative ease. I tend to like the idea of a soft token as with Google Authenticator, as there's no cost for physical tokens. On a per host basis, implementation is fairly easy, if we wanted a centralized, i.e. single token for all of JLSE per user, some implementation time would be needed. It would also start out as, "come see Ben for your special QR code or shared key", as there's no integration into Userbase (accounts.jlse.anl.gov) for this type of credential. > > Ben > >> On Mar 17, 2015, at 1:56 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Hi, >> >> Pablo Llopis, a student who works with me this summer, has requested his >> JLSE account, >> selecting Rajeev as a sponsor. >> >> Does he need to request ALCF account (in order to obtain his crypto card)? >> or can JLSE provide a crypto card directly to him? >> >> - kaz >> >> _______________________________________________ >> Admins mailing list >> Admins(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov
by Kazutomo Yoshii 19 Mar '15

19 Mar '15

Hi Ben, Yes, I can sudo. Thanks! I'm mounting an ext4 on /dev/tdh, to check the vfs performance It is a low-hanging-fruits solution to provide malloc()-like function, which stresses the teradimm driver a lot, so the performance is terrible for a larger size (4 or 8 GB allocation). - kaz On 03/17/2015 03:55 PM, Allen, Benjamin S. wrote: > Sudo access has been added for you and John on wally. Please document any changes you make. > > Ben > >> On Mar 17, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Hi Ben, >> >> Thank you for sudo access! >> >> John, >> Let's fill the request form together if exFlash SSD still sounds good >> for you. >> I guess other CESAR folks are interested in this device. If so, we can >> add this to a request. >> >> Here is a list of the current JLSE projects. >> >> http://jlse.anl.gov/projects/ >> >> - kaz >> >> On 03/17/2015 01:04 PM, Allen, Benjamin S. wrote: >>> Hi Kaz, >>> >>> Sudo access is fine. I'll add it later today. >>> >>> Would you or John please fill out a project request: http://press3.mcs.anl.gov/jlse/projects/project-request-form/, so we can list the project description? >>> >>> We don't have a start date for the student yet. As far as exclusivity it will depend on how quick the student gets spun up, but more or less I would assume yes. >>> >>> Ben >>> >>>> On Mar 16, 2015, at 1:32 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>> >>>> Ben, >>>> >>>> Thanks! >>>> >>>> John and I would like to have a sudo access, so that >>>> we can try something before the student starts working here. >>>> >>>> Do you know when the student starts? >>>> I assume that the student use this machine exclusively when he starts here. >>>> >>>> - kaz >>>> >>>> On 03/16/2015 01:25 PM, Allen, Benjamin S. wrote: >>>>>> On Mar 16, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>>>> >>>>>> Hi Ben, >>>>>> >>>>>> Can we have an access to one of the teradimm block devices? >>>>>> or can we move some of exFlash DIMMs to other system? >>>>> We only have one system. We have a summer student this year under Kevin Harms thats slated todo research on the box, so it can be used till then. I'll need to be able to put back to normal before the student starts. >>>>> >>>>> Hostname is "wally.ftm.alcf.anl.gov". Let me know if you need sudo access. >>>>> >>>>>> John is interested in trying out teradimm. >>>>>> I'm also interested in hacking the kernel module. >>>>> Feel free to mess with the kernel module. No editing the BIOS settings on the box however. The BIOS appears to be vary fragile. It's very easy to get the machine into a state that doesn't boot. >>>>> >>>>>> BTW, do you know the cost of each DIMM? >>>>>> do we have any other NVRAM devices in JLSE? >>>>> We ordered the box last April. As a whole the server costs $29,611.00 with 8 exFlash DIMMs installed. The quote shows the list price as $4,949.00 for the 400GB exFlash DIMM. Our overall discounted price for the server was 44% off list price. >>>>> >>>>> No we don't have any other. I don't know of any other nvram product actually shipping yet. >>>>> >>>>> Ben

1 0

Re: [JLSE-Admins] Project Request Form
by Kazutomo Yoshii 19 Mar '15

19 Mar '15

Kumar, There is one project related to exFlash: "Viability of Storage Class Memory for use as Application Extended-Memory" I noticed this project listed twice, btw. Both projects looks similar, expect the list of PIs and Fusion I/O. I guess someone forgot to remove it. or the title of one of the project needs to be updated. ------------- Viability of Storage Class Memory for use as Application Extended-Memory PIs: Kevin Harms and Ben Allen Objective: We will investigate the viability of using storage class memory technologies, such as Flash, for use in HPC computing as extended memory beyond standard DRAM. The testbed, wally, will have IBM exFlash DIMMs which provide much lower latency than normal Flash PCI devices and will allow prototyping this type of system. Testbed description: Wally ï¿½ IBM 3650 M4 with (8) IBM exFlash 400GB DIMMs ------------ Viability of Storage Class Memory for use as Application Extended-Memory PIs: Kevin Harms, Ben Allen and Venkatram Vishwanath Objective: We will investigate the viability of using storage class memory technologies, such as Flash, for use in HPC computing as extended memory beyond standard DRAM. The testbed, wally, will have IBM exFlash DIMMs which provide much lower latency than normal Flash PCI devices and will allow prototyping this type of system. The FusionI/O cards will likely provide a PCI-e based testbed to better understand the efficacy of FLASH as an extended memory. Testbed description: Wally ï¿½ IBM 3650 M4 with (8) IBM exFlash 400GB DIMMs 3x FusionIO Duo 320GB cards repurposed from Intrepid-home file servers, installed into 3 Intel Westmere servers. Servers originally repurposed from Tukey. - kaz On 03/19/2015 04:20 PM, Kalyan Kumaran wrote: > John, there are a couple of projects on the project page in this space. Should we add your name to these or would you prefer a separate project? > > Thanks, > Kumar > > >> On Mar 18, 2015, at 5:59 PM, John Tramm <jtramm(a)anl.gov> wrote: >> >> From: John Tramm <jtramm(a)anl.gov> >> Exploring the Viability of NVRAM for use in Task-Based 3D Method of Characteristics (MOC) Style Nuclear Reactor Simulation Applications. >> John Tramm, Kazutomo Yoshii >> MCS >> Many new upcoming HPC architectures feature large NVRAM storage on-node, offering a huge boost to memory capacity provided that the application's algorithms can handle the high latency involved in NVRAM access while also accommodating the low durability of NVRAM DIMMs. >> >> In light of the potential benefits that NVRAM provides, we are investigating the usage of high capacity NVRAM modules for use in task-based 3D Method of Characteristics (MOC) full-core nuclear reactor simulation applications, such as SimpleMOC and OpenMOC. NVRAM offers large potential benefits to this style of application as each node requires a large data footprint, yet this data only needs to be accessed once per computational sweep. The amount of computation required between each read/write from NVRAM is predicted to be extremely high. This fact, combined with the task-based work model of SimpleMOC, means that masking NVRAM latency is feasible with only minor algorithmic improvements, and also that durability requirements can be met. >> >> Additionally, the large data footprint of 3D MOC applications is also required to be exchanged between node neighbors in a ghost cell exchange pattern following each computational sweep. In the case of SimpleMOC, the ghost cells compose nearly the entire data footprint of the application, which allows for vastly fewer nodes to be required by the computation if on node NVRAM modules can be utilized effectively. While ghost cell exchanges may be difficult to directly execute on a single node or few node testbed system, there are still useful metrics we can gather in order to model the altered costs of ghost cell exchanges given the use of NVRAM over traditional DRAM. >> >> The primary goal of this study is to implement NVRAM usage techniques into the SimpleMOC application in order that we may quantify the costs/rewards of NVRAM usage on 3D MOC reaction simulation algorithms. With these latency and performance parameters, we will be able to build a performance model for the application when run on future many-node HPC systems. This study is done to better prepare nuclear simulation applications for usage on exascale class supercomputers as part of the Center for Exascale Simulation of Advanced Reactors (CESAR) project. >> CESAR (MCS) >> MIT >> John Tramm, Kazutomo Yoshii >> Wally (for NVRAM exFlash DIMMs) >> [Schedule (when access is needed and for how long):] >> >> >> -- >> This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) >> >> _______________________________________________ >> Admins mailing list >> Admins(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/admins > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] Project Request Form
by Kalyan Kumaran 19 Mar '15

19 Mar '15

John, there are a couple of projects on the project page in this space. Should we add your name to these or would you prefer a separate project? Thanks, Kumar > On Mar 18, 2015, at 5:59 PM, John Tramm <jtramm(a)anl.gov> wrote: > > From: John Tramm <jtramm(a)anl.gov> > Exploring the Viability of NVRAM for use in Task-Based 3D Method of Characteristics (MOC) Style Nuclear Reactor Simulation Applications. > John Tramm, Kazutomo Yoshii > MCS > Many new upcoming HPC architectures feature large NVRAM storage on-node, offering a huge boost to memory capacity provided that the application's algorithms can handle the high latency involved in NVRAM access while also accommodating the low durability of NVRAM DIMMs. > > In light of the potential benefits that NVRAM provides, we are investigating the usage of high capacity NVRAM modules for use in task-based 3D Method of Characteristics (MOC) full-core nuclear reactor simulation applications, such as SimpleMOC and OpenMOC. NVRAM offers large potential benefits to this style of application as each node requires a large data footprint, yet this data only needs to be accessed once per computational sweep. The amount of computation required between each read/write from NVRAM is predicted to be extremely high. This fact, combined with the task-based work model of SimpleMOC, means that masking NVRAM latency is feasible with only minor algorithmic improvements, and also that durability requirements can be met. > > Additionally, the large data footprint of 3D MOC applications is also required to be exchanged between node neighbors in a ghost cell exchange pattern following each computational sweep. In the case of SimpleMOC, the ghost cells compose nearly the entire data footprint of the application, which allows for vastly fewer nodes to be required by the computation if on node NVRAM modules can be utilized effectively. While ghost cell exchanges may be difficult to directly execute on a single node or few node testbed system, there are still useful metrics we can gather in order to model the altered costs of ghost cell exchanges given the use of NVRAM over traditional DRAM. > > The primary goal of this study is to implement NVRAM usage techniques into the SimpleMOC application in order that we may quantify the costs/rewards of NVRAM usage on 3D MOC reaction simulation algorithms. With these latency and performance parameters, we will be able to build a performance model for the application when run on future many-node HPC systems. This study is done to better prepare nuclear simulation applications for usage on exascale class supercomputers as part of the Center for Exascale Simulation of Advanced Reactors (CESAR) project. > CESAR (MCS) > MIT > John Tramm, Kazutomo Yoshii > Wally (for NVRAM exFlash DIMMs) > [Schedule (when access is needed and for how long):] > > > -- > This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) > > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] Project Request Form
by Allen, Benjamin S. 18 Mar '15

18 Mar '15

Thanks John and Kaz for submitting this. Let me know if you need any help on Wally. I've updated the Teradimms to the latest IBM released firmware: 1.5.0.35, and kernel module: 1.5.0.41. Ben > On Mar 18, 2015, at 5:59 PM, John Tramm <jtramm(a)anl.gov> wrote: > > From: John Tramm <jtramm(a)anl.gov> > Exploring the Viability of NVRAM for use in Task-Based 3D Method of Characteristics (MOC) Style Nuclear Reactor Simulation Applications. > John Tramm, Kazutomo Yoshii > MCS > Many new upcoming HPC architectures feature large NVRAM storage on-node, offering a huge boost to memory capacity provided that the application's algorithms can handle the high latency involved in NVRAM access while also accommodating the low durability of NVRAM DIMMs. > > In light of the potential benefits that NVRAM provides, we are investigating the usage of high capacity NVRAM modules for use in task-based 3D Method of Characteristics (MOC) full-core nuclear reactor simulation applications, such as SimpleMOC and OpenMOC. NVRAM offers large potential benefits to this style of application as each node requires a large data footprint, yet this data only needs to be accessed once per computational sweep. The amount of computation required between each read/write from NVRAM is predicted to be extremely high. This fact, combined with the task-based work model of SimpleMOC, means that masking NVRAM latency is feasible with only minor algorithmic improvements, and also that durability requirements can be met. > > Additionally, the large data footprint of 3D MOC applications is also required to be exchanged between node neighbors in a ghost cell exchange pattern following each computational sweep. In the case of SimpleMOC, the ghost cells compose nearly the entire data footprint of the application, which allows for vastly fewer nodes to be required by the computation if on node NVRAM modules can be utilized effectively. While ghost cell exchanges may be difficult to directly execute on a single node or few node testbed system, there are still useful metrics we can gather in order to model the altered costs of ghost cell exchanges given the use of NVRAM over traditional DRAM. > > The primary goal of this study is to implement NVRAM usage techniques into the SimpleMOC application in order that we may quantify the costs/rewards of NVRAM usage on 3D MOC reaction simulation algorithms. With these latency and performance parameters, we will be able to build a performance model for the application when run on future many-node HPC systems. This study is done to better prepare nuclear simulation applications for usage on exascale class supercomputers as part of the Center for Exascale Simulation of Advanced Reactors (CESAR) project. > CESAR (MCS) > MIT > John Tramm, Kazutomo Yoshii > Wally (for NVRAM exFlash DIMMs) > [Schedule (when access is needed and for how long):] > > > -- > This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) > > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

[JLSE-Admins] Project Request Form
by John Tramm 18 Mar '15

18 Mar '15

From: John Tramm <jtramm(a)anl.gov> Exploring the Viability of NVRAM for use in Task-Based 3D Method of Characteristics (MOC) Style Nuclear Reactor Simulation Applications. John Tramm, Kazutomo Yoshii MCS Many new upcoming HPC architectures feature large NVRAM storage on-node, offering a huge boost to memory capacity provided that the application's algorithms can handle the high latency involved in NVRAM access while also accommodating the low durability of NVRAM DIMMs. In light of the potential benefits that NVRAM provides, we are investigating the usage of high capacity NVRAM modules for use in task-based 3D Method of Characteristics (MOC) full-core nuclear reactor simulation applications, such as SimpleMOC and OpenMOC. NVRAM offers large potential benefits to this style of application as each node requires a large data footprint, yet this data only needs to be accessed once per computational sweep. The amount of computation required between each read/write from NVRAM is predicted to be extremely high. This fact, combined with the task-based work model of SimpleMOC, means that masking NVRAM latency is feasible with only minor algorithmic improvements, and also that durability requirements can be met. Additionally, the large data footprint of 3D MOC applications is also required to be exchanged between node neighbors in a ghost cell exchange pattern following each computational sweep. In the case of SimpleMOC, the ghost cells compose nearly the entire data footprint of the application, which allows for vastly fewer nodes to be required by the computation if on node NVRAM modules can be utilized effectively. While ghost cell exchanges may be difficult to directly execute on a single node or few node testbed system, there are still useful metrics we can gather in order to model the altered costs of ghost cell exchanges given the use of NVRAM over traditional DRAM. The primary goal of this study is to implement NVRAM usage techniques into the SimpleMOC application in order that we may quantify the costs/rewards of NVRAM usage on 3D MOC reaction simulation algorithms. With these latency and performance parameters, we will be able to build a performance model for the application when run on future many-node HPC systems. This study is done to better prepare nuclear simulation applications for usage on exascale class supercomputers as part of the Center for Exascale Simulation of Advanced Reactors (CESAR) project. CESAR (MCS) MIT John Tramm, Kazutomo Yoshii Wally (for NVRAM exFlash DIMMs) [Schedule (when access is needed and for how long):] -- This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse)

1 0

Re: [JLSE-Admins] [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov
by Allen, Benjamin S. 17 Mar '15

17 Mar '15

Sudo access has been added for you and John on wally. Please document any changes you make. Ben > On Mar 17, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Hi Ben, > > Thank you for sudo access! > > John, > Let's fill the request form together if exFlash SSD still sounds good > for you. > I guess other CESAR folks are interested in this device. If so, we can > add this to a request. > > Here is a list of the current JLSE projects. > > http://jlse.anl.gov/projects/ > > - kaz > > On 03/17/2015 01:04 PM, Allen, Benjamin S. wrote: >> Hi Kaz, >> >> Sudo access is fine. I'll add it later today. >> >> Would you or John please fill out a project request: http://press3.mcs.anl.gov/jlse/projects/project-request-form/, so we can list the project description? >> >> We don't have a start date for the student yet. As far as exclusivity it will depend on how quick the student gets spun up, but more or less I would assume yes. >> >> Ben >> >>> On Mar 16, 2015, at 1:32 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>> >>> Ben, >>> >>> Thanks! >>> >>> John and I would like to have a sudo access, so that >>> we can try something before the student starts working here. >>> >>> Do you know when the student starts? >>> I assume that the student use this machine exclusively when he starts here. >>> >>> - kaz >>> >>> On 03/16/2015 01:25 PM, Allen, Benjamin S. wrote: >>>>> On Mar 16, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>>> >>>>> Hi Ben, >>>>> >>>>> Can we have an access to one of the teradimm block devices? >>>>> or can we move some of exFlash DIMMs to other system? >>>> We only have one system. We have a summer student this year under Kevin Harms thats slated todo research on the box, so it can be used till then. I'll need to be able to put back to normal before the student starts. >>>> >>>> Hostname is "wally.ftm.alcf.anl.gov". Let me know if you need sudo access. >>>> >>>>> John is interested in trying out teradimm. >>>>> I'm also interested in hacking the kernel module. >>>> Feel free to mess with the kernel module. No editing the BIOS settings on the box however. The BIOS appears to be vary fragile. It's very easy to get the machine into a state that doesn't boot. >>>> >>>>> BTW, do you know the cost of each DIMM? >>>>> do we have any other NVRAM devices in JLSE? >>>> We ordered the box last April. As a whole the server costs $29,611.00 with 8 exFlash DIMMs installed. The quote shows the list price as $4,949.00 for the 400GB exFlash DIMM. Our overall discounted price for the server was 44% off list price. >>>> >>>> No we don't have any other. I don't know of any other nvram product actually shipping yet. >>>> >>>> Ben >

1 0

Re: [JLSE-Admins] account for Pablo Llopis
by Allen, Benjamin S. 17 Mar '15

17 Mar '15

> Does he need to request ALCF account (in order to obtain his crypto card)? > or can JLSE provide a crypto card directly to him? We cannot use JLSE as a justification for creating him an ALCF account, so unless he has other ALCF project duties we'll have to use another second factor authentication method. I assume he requires this for privileged access within JLSE? If not then he has no need for a cryptocard. I've been investigating Google Authenticator style one time passwords as a second factor. We could also start supporting Yubikey with relative ease. I tend to like the idea of a soft token as with Google Authenticator, as there's no cost for physical tokens. On a per host basis, implementation is fairly easy, if we wanted a centralized, i.e. single token for all of JLSE per user, some implementation time would be needed. It would also start out as, "come see Ben for your special QR code or shared key", as there's no integration into Userbase (accounts.jlse.anl.gov) for this type of credential. Ben > On Mar 17, 2015, at 1:56 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Hi, > > Pablo Llopis, a student who works with me this summer, has requested his > JLSE account, > selecting Rajeev as a sponsor. > > Does he need to request ALCF account (in order to obtain his crypto card)? > or can JLSE provide a crypto card directly to him? > > - kaz > > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

[JLSE-Admins] account for Pablo Llopis
by Kazutomo Yoshii 17 Mar '15

17 Mar '15

Hi, Pablo Llopis, a student who works with me this summer, has requested his JLSE account, selecting Rajeev as a sponsor. Does he need to request ALCF account (in order to obtain his crypto card)? or can JLSE provide a crypto card directly to him? - kaz

1 0