July 2014 - Admins - mailman3.cels.anl.gov

[JLSE-Admins] Meeting this Friday?
by Kalyan Kumaran 17 Jul '14

17 Jul '14

I'm on vacation. So is Rajeev I think. Feel free to meet if you like. I will not cancel the room reservation. -Kumar

1 0

Re: [JLSE-Admins] Fwd: RE: evaluation systems
by Kalyan Kumaran 16 Jul '14

16 Jul '14

Kaz, David Martin can help with the NDA signed. Let him know. This would be the first NDA resource. It would be good to see how the process works. -Kumar On Jul 16, 2014, at 10:31 AM, Kazutomo Yoshii <kazutomo(a)anl.gov> wrote: > FYI, > > This is originally started from the CESAR context, > but I think this fits to JLSE. > > I'm going to arrange a meeting with AMD if needed. > The machines are likely to be in the data center, so need a help from Ben. > > - kaz > > -------- Original Message -------- > Subject: RE: evaluation systems > Date: Tue, 15 Jul 2014 17:44:10 +0000 > From: Gallmeier, Jonathan <Jonathan.Gallmeier(a)amd.com> > To: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> > CC: Schulte, Michael <Michael.Schulte(a)amd.com>, Brantley, Bill <bill.brantley(a)amd.com> > > Hi Kaz, > > Last December we sent over the attached AMD hardware loan agreement form for your legal team to review. We would like to see if your legal team has been able to review the document and if we can move forward with getting an agreement in place. > > I copied Bill Brantley and Mike Schulte from AMD. Bill would like to send early access hardware to ANL and is motivated to help this process along. Could you keep him and Mike on the email CC line? > > Best regards, > > Jonathan > From: Kazutomo Yoshii [kazutomo(a)mcs.anl.gov] > Sent: Wednesday, December 11, 2013 6:37 PM > To: Gallmeier, Jonathan > Subject: Re: evaluation systems > > This message has been archived. View the original item > Hi Jonathan, > > Thank you for sending the document, > > Our legal group is reading the agreement right now. > They said because this is a loan agreement, we might need to > go through a procurement process. We're figuring out it. > Could you please wait a little bit until those things get clear? > > - kaz > > > On 12/06/2013 10:50 AM, Gallmeier, Jonathan wrote: > > Hello Kaz, > > > > Please find attached AMD's hardware loan agreement form. Let me know if there are any concerns or issues with the content of this document. I can put you in touch with AMD's legal department if edits are required. > > > > Best regards, > > > > Jonathan > > ________________________________________ > > From: Kazutomo Yoshii [kazutomo(a)mcs.anl.gov] > > Sent: Tuesday, November 19, 2013 9:41 AM > > To: Gallmeier, Jonathan > > Subject: Re: evaluation systems > > > > Hi Jonathan, > > > > Thank you for you reply, > > > > Please send the NDA-like agreement to me. > > I'll forward it to our legal department. > > > > - kaz > > > > On 11/13/2013 10:34 AM, Gallmeier, Jonathan wrote: > >> Hi Kaz, > >> > >> Thanks for the email. > >> > >> We are setting up a remote system, but we are still defining exactly > >> what system will be used for remote access. > >> > >> We have early silicon for our next server part -- referred to as > >> Kaveri (sometimes as Berlin). Berlin is the server part code name. > >> Kaveri is essentially the same part, but targets the client markets. > >> Internally, we refer to the early silicon as Kaveri. We are planning > >> on providing a small number of these systems to various labs for the > >> purpose of co-design and early access HSA development. The timeframe > >> depends upon driver stability. > >> > >> To ship hardware to you we will need to get a signed hardware loan > >> agreement form signed. This is a reasonably painless AMD NDA-like > >> agreement. Who is the right person for me to send this to? > >> > >> Jonathan ________________________________________ From: Kazutomo > >> Yoshii [kazutomo(a)mcs.anl.gov] Sent: Tuesday, November 12, 2013 3:24 > >> PM To: Gallmeier, Jonathan Cc: John Tramm Subject: evaluation > >> systems > >> > >> Hi Jonathan, > >> > >> This is Kaz who joined the conf call last Fri. > >> > >> As I mentioned, we're interested in trying out the evaluation > >> systems. What systems are you able to provide us to access? or send > >> us evals boards? HSA would be really interesting arch for us. I'd > >> like to try NVRAM stuff if available. > >> > >> - kaz > >> > >> > >> > > > > > > > > > > > <Argonne%20National%20Lab.system%20loan%20agreement-2.docx>_______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

[JLSE-Admins] Fwd: RE: evaluation systems
by Kazutomo Yoshii 16 Jul '14

16 Jul '14

FYI, This is originally started from the CESAR context, but I think this fits to JLSE. I'm going to arrange a meeting with AMD if needed. The machines are likely to be in the data center, so need a help from Ben. - kaz -------- Original Message -------- Subject: RE: evaluation systems Date: Tue, 15 Jul 2014 17:44:10 +0000 From: Gallmeier, Jonathan <Jonathan.Gallmeier(a)amd.com> To: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> CC: Schulte, Michael <Michael.Schulte(a)amd.com>, Brantley, Bill <bill.brantley(a)amd.com> Hi Kaz, Last December we sent over the attached AMD hardware loan agreement form for your legal team to review. We would like to see if your legal team has been able to review the document and if we can move forward with getting an agreement in place. I copied Bill Brantley and Mike Schulte from AMD. Bill would like to send early access hardware to ANL and is motivated to help this process along. Could you keep him and Mike on the email CC line? Best regards, Jonathan ------------------------------------------------------------------------ *From:* Kazutomo Yoshii [kazutomo(a)mcs.anl.gov] *Sent:* Wednesday, December 11, 2013 6:37 PM *To:* Gallmeier, Jonathan *Subject:* Re: evaluation systems This message has been archived. View the original item <http://ausev4.amd.com/EnterpriseVault/ViewMessage.asp?VaultId=11B107DF8938C…> Hi Jonathan, Thank you for sending the document, Our legal group is reading the agreement right now. They said because this is a loan agreement, we might need to go through a procurement process. We're figuring out it. Could you please wait a little bit until those things get clear? - kaz On 12/06/2013 10:50 AM, Gallmeier, Jonathan wrote: > Hello Kaz, > > Please find attached AMD's hardware loan agreement form. Let me know if there are any concerns or issues with the content of this document. I can put you in touch with AMD's legal department if edits are required. > > Best regards, > > Jonathan > ________________________________________ > From: Kazutomo Yoshii [kazutomo(a)mcs.anl.gov] > Sent: Tuesday, November 19, 2013 9:41 AM > To: Gallmeier, Jonathan > Subject: Re: evaluation systems > > Hi Jonathan, > > Thank you for you reply, > > Please send the NDA-like agreement to me. > I'll forward it to our legal department. > > - kaz > > On 11/13/2013 10:34 AM, Gallmeier, Jonathan wrote: >> Hi Kaz, >> >> Thanks for the email. >> >> We are setting up a remote system, but we are still defining exactly >> what system will be used for remote access. >> >> We have early silicon for our next server part -- referred to as >> Kaveri (sometimes as Berlin). Berlin is the server part code name. >> Kaveri is essentially the same part, but targets the client markets. >> Internally, we refer to the early silicon as Kaveri. We are planning >> on providing a small number of these systems to various labs for the >> purpose of co-design and early access HSA development. The timeframe >> depends upon driver stability. >> >> To ship hardware to you we will need to get a signed hardware loan >> agreement form signed. This is a reasonably painless AMD NDA-like >> agreement. Who is the right person for me to send this to? >> >> Jonathan ________________________________________ From: Kazutomo >> Yoshii [kazutomo(a)mcs.anl.gov] Sent: Tuesday, November 12, 2013 3:24 >> PM To: Gallmeier, Jonathan Cc: John Tramm Subject: evaluation >> systems >> >> Hi Jonathan, >> >> This is Kaz who joined the conf call last Fri. >> >> As I mentioned, we're interested in trying out the evaluation >> systems. What systems are you able to provide us to access? or send >> us evals boards? HSA would be really interesting arch for us. I'd >> like to try NVRAM stuff if available. >> >> - kaz >> >> >> > > >

1 0

Re: [JLSE-Admins] Storage-aware Large-scale Data Transfers Testbed
by Raj Kettimuthu 14 Jul '14

14 Jul '14

Great, thanks. On Jul 14, 2014, at 3:33 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > Thanks Raj. I've initiated the ANL border conduit request based on this information. Ed and Ryan will implement further rules within LCF's network to allow access to NERSC' DTNs and Mira's DTNs. I've also put into place iptables rulesets to restrict inbound traffic to just GridFTP ports on the file servers' public interfaces. > > Quick status update, fs9-24 are up and provisioned. Eric Ross is working on getting two 8 server GPFS filesystems configured. We should be able to turn this testbed over to you by Wednesday. > > Ben > > > On Jul 14, 2014, at 3:12 PM, Raj Kettimuthu <kettimut(a)anl.gov> wrote: > >> I think access to/from NERSC DTNs (dtn{01-04}.nersc.gov) and Mira DTNs should be fine for now. >> >> Eunsung: >> If you want access to/from other nodes, please let Ben know. >> >> Raj >> >> On Jul 14, 2014, at 11:53 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >>> Raj, et al.: >>> >>> What firewall rules need to be put in place for external access to this testbed (both inbound and outbound)? >>> >>> Do you need just the GridFTP set of ports? Do you have specific external DTNs you’re testing with, so we don’t have to open this testbed’s GridFTP services up to the entire public internet? >>> >>> Thanks, >>> >>> Ben >>> >>> On Jul 8, 2014, at 10:59 AM, Jung, Eun Sung <esjung(a)mcs.anl.gov> wrote: >>> >>>> Ok, thank you. >>>> >>>> -Eunsung >>>> >>>> On 7/8/14, 10:54 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >>>> >>>>> Hi Eun Sung, >>>>> >>>>>> * Questions regarding the testbed for baseline testing. >>>>>> - Is there any additional node which can play a role of DTN except two 8 >>>>>> file-server nodes? >>>>> >>>>> In the proposed testbed no. Just to be clear I’m proposing 16 file >>>>> servers, split into two groups. The 16 file servers are backed by two DDN >>>>> S2A9900 racks of storage. The groups of 8 are to be defined by the file >>>>> servers connected to the same backing DDN S2A9900 >>>>> >>>>> The idea for the baseline testing would be to use one or more of the 8 >>>>> file servers from one group as clients across the network to the other >>>>> group of 8 file servers. Thus simulating DTNs accessing a shared >>>>> filesystem over a network. >>>>> >>>>>> - If so, all nodes including fs nodes and a DTN node share the same >>>>>> network? >>>>> >>>>> All 16 file servers will be on the same 10Gb Myrinet network. Each file >>>>> server will also have a 1GbE public interface, and a 1GbE internal >>>>> management interface. The same configuration as the Petrel testbed. >>>>> >>>>> The public interfaces will be heavily firewall’ed by default. We’ll >>>>> explicitly allow inbound and outbound traffic as requested. Ideally we >>>>> can get the list of firewall rules from Raj, Mike and you in by tomorrow. >>>>> If you only need standard GridFTP ports opened, please let us know that. >>>>> >>>>>> >>>>>> * Questions regarding the testbed for storage-aware large data transfers >>>>>> - Do you mean a single file server by a single file system? >>>>> >>>>> I mean a single shared filesystem (GPFS) across the 16 file servers. For >>>>> testing using GPFS’ data locality hints. >>>>> >>>>>> - Once a file system created, can we adjust the number of file servers >>>>>> later? For example, up to 32 servers, 2 servers per fs node. >>>>> >>>>> You will have 16 physical file servers. You’ll be able to reconfigure >>>>> them how you see fit within reason. This includes destroying and >>>>> recreating the GPFS if required. >>>>> >>>>> By default all 51 LUNs (8+2p RAID6 LUNs) per DDN S2A9900 rack are >>>>> presented to each of the 8 attached file servers via directly connected >>>>> DDR Infiniband. We can restrict or group which LUNs are presented to >>>>> which of the 8 file servers to force further data locality if you need to >>>>> simulate a less ideal storage configuration. >>>>> >>>>> Please let me know if you have further questions, >>>>> >>>>> Ben >>>>> >>>>> >>>>>> On 7/7/14, 11:44 PM, "Raj Kettimuthu" <kettimut(a)anl.gov> wrote: >>>>>> >>>>>>> Thanks Ben. This sounds good to me. >>>>>>> >>>>>>> Eunsung, >>>>>>> Do you have any concerns/questions? >>>>>>> >>>>>>> Raj >>>>>>> >>>>>>> Sent from my iPhone >>>>>>> >>>>>>>> On Jul 7, 2014, at 4:05 PM, "Allen, Benjamin S." >>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Raj, >>>>>>>> >>>>>>>> To summarize the requirements for everyone CC¹ed your two objectives >>>>>>>> are: >>>>>>>> >>>>>>>> 1. Baseline testing a traditional DTN which accesses the shared >>>>>>>> filesystem over a network fabric, compared to a DTN directly on a >>>>>>>> storage controller. The JLSE file servers provide reasonably >>>>>>>> facsimiles >>>>>>>> for embedded storage controllers as they have a point-to-point DDR IB >>>>>>>> connection to the D2A9900 controllers. They¹re also lower powered >>>>>>>> servers so should simulate the lower resources available on an >>>>>>>> embedded >>>>>>>> storage controller virtual machine. >>>>>>>> >>>>>>>> 2. Storage-aware Large Data Transfers: Using data locality information >>>>>>>> from GPFS to direct GridFTP to the DTN which has local block access to >>>>>>>> the data. Thus allowing GridFTP to be more efficient in its use of the >>>>>>>> storage network (Myrinet in this case). A good analogy is Hadoop¹s >>>>>>>> HDFS, >>>>>>>> but for data transfer instead of compute. >>>>>>>> >>>>>>>> Please correct or expand the above as needed. >>>>>>>> >>>>>>>> Due to the Petrel testbed being setup as a single file system, and >>>>>>>> being a pseudo production testbed for Globus Share work (i.e. we can't >>>>>>>> just destroy the existing globus-fs0 and rebuild it), I think the best >>>>>>>> course of action to support the above work is to setup two 8 >>>>>>>> file-server >>>>>>>> filesystems on a new Myrinet fabric. We¹ll initially create a file >>>>>>>> system on one (or optionally both) sets of 8 file servers. This will >>>>>>>> let >>>>>>>> you use opposing file servers for the baseline testing across a >>>>>>>> network >>>>>>>> fabric. This testbed would be similar in specs (same hardware, same >>>>>>>> networking) as the Petrel testbed. >>>>>>>> >>>>>>>> Once that testing is complete, we will recreate the GPFS as a single >>>>>>>> file system so you can experiment with data locality. Without further >>>>>>>> configuration this will give you two silos of data locality (two >>>>>>>> S2A9900 >>>>>>>> racks). If you want to further worsen the test cases, we can limit >>>>>>>> which >>>>>>>> LUNs are presented to which file server, lowering the number of >>>>>>>> file-servers with block access to a given set of data. >>>>>>>> >>>>>>>> It looks like we could have this testbed put together by early next >>>>>>>> week, as we have a near ready set of 16 file servers available. I¹m >>>>>>>> also >>>>>>>> suggesting we use this new testbed as a more general resource for >>>>>>>> those >>>>>>>> that need GPFS but can withstand the file system being destroyed. I >>>>>>>> believe this fits Bill¹s project needs, and he has similar goals in >>>>>>>> testing data locality with GPFS. >>>>>>>> >>>>>>>> All, please let me know if you have any comments or concerns. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Ben >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> >

1 0

Re: [JLSE-Admins] Storage-aware Large-scale Data Transfers Testbed
by Allen, Benjamin S. 14 Jul '14

14 Jul '14

Thanks Raj. I've initiated the ANL border conduit request based on this information. Ed and Ryan will implement further rules within LCF's network to allow access to NERSC' DTNs and Mira's DTNs. I've also put into place iptables rulesets to restrict inbound traffic to just GridFTP ports on the file servers' public interfaces. Quick status update, fs9-24 are up and provisioned. Eric Ross is working on getting two 8 server GPFS filesystems configured. We should be able to turn this testbed over to you by Wednesday. Ben On Jul 14, 2014, at 3:12 PM, Raj Kettimuthu <kettimut(a)anl.gov> wrote: > I think access to/from NERSC DTNs (dtn{01-04}.nersc.gov) and Mira DTNs should be fine for now. > > Eunsung: > If you want access to/from other nodes, please let Ben know. > > Raj > > On Jul 14, 2014, at 11:53 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > >> Raj, et al.: >> >> What firewall rules need to be put in place for external access to this testbed (both inbound and outbound)? >> >> Do you need just the GridFTP set of ports? Do you have specific external DTNs you’re testing with, so we don’t have to open this testbed’s GridFTP services up to the entire public internet? >> >> Thanks, >> >> Ben >> >> On Jul 8, 2014, at 10:59 AM, Jung, Eun Sung <esjung(a)mcs.anl.gov> wrote: >> >>> Ok, thank you. >>> >>> -Eunsung >>> >>> On 7/8/14, 10:54 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >>> >>>> Hi Eun Sung, >>>> >>>>> * Questions regarding the testbed for baseline testing. >>>>> - Is there any additional node which can play a role of DTN except two 8 >>>>> file-server nodes? >>>> >>>> In the proposed testbed no. Just to be clear I’m proposing 16 file >>>> servers, split into two groups. The 16 file servers are backed by two DDN >>>> S2A9900 racks of storage. The groups of 8 are to be defined by the file >>>> servers connected to the same backing DDN S2A9900 >>>> >>>> The idea for the baseline testing would be to use one or more of the 8 >>>> file servers from one group as clients across the network to the other >>>> group of 8 file servers. Thus simulating DTNs accessing a shared >>>> filesystem over a network. >>>> >>>>> - If so, all nodes including fs nodes and a DTN node share the same >>>>> network? >>>> >>>> All 16 file servers will be on the same 10Gb Myrinet network. Each file >>>> server will also have a 1GbE public interface, and a 1GbE internal >>>> management interface. The same configuration as the Petrel testbed. >>>> >>>> The public interfaces will be heavily firewall’ed by default. We’ll >>>> explicitly allow inbound and outbound traffic as requested. Ideally we >>>> can get the list of firewall rules from Raj, Mike and you in by tomorrow. >>>> If you only need standard GridFTP ports opened, please let us know that. >>>> >>>>> >>>>> * Questions regarding the testbed for storage-aware large data transfers >>>>> - Do you mean a single file server by a single file system? >>>> >>>> I mean a single shared filesystem (GPFS) across the 16 file servers. For >>>> testing using GPFS’ data locality hints. >>>> >>>>> - Once a file system created, can we adjust the number of file servers >>>>> later? For example, up to 32 servers, 2 servers per fs node. >>>> >>>> You will have 16 physical file servers. You’ll be able to reconfigure >>>> them how you see fit within reason. This includes destroying and >>>> recreating the GPFS if required. >>>> >>>> By default all 51 LUNs (8+2p RAID6 LUNs) per DDN S2A9900 rack are >>>> presented to each of the 8 attached file servers via directly connected >>>> DDR Infiniband. We can restrict or group which LUNs are presented to >>>> which of the 8 file servers to force further data locality if you need to >>>> simulate a less ideal storage configuration. >>>> >>>> Please let me know if you have further questions, >>>> >>>> Ben >>>> >>>> >>>>> On 7/7/14, 11:44 PM, "Raj Kettimuthu" <kettimut(a)anl.gov> wrote: >>>>> >>>>>> Thanks Ben. This sounds good to me. >>>>>> >>>>>> Eunsung, >>>>>> Do you have any concerns/questions? >>>>>> >>>>>> Raj >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On Jul 7, 2014, at 4:05 PM, "Allen, Benjamin S." >>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>> Hi Raj, >>>>>>> >>>>>>> To summarize the requirements for everyone CC¹ed your two objectives >>>>>>> are: >>>>>>> >>>>>>> 1. Baseline testing a traditional DTN which accesses the shared >>>>>>> filesystem over a network fabric, compared to a DTN directly on a >>>>>>> storage controller. The JLSE file servers provide reasonably >>>>>>> facsimiles >>>>>>> for embedded storage controllers as they have a point-to-point DDR IB >>>>>>> connection to the D2A9900 controllers. They¹re also lower powered >>>>>>> servers so should simulate the lower resources available on an >>>>>>> embedded >>>>>>> storage controller virtual machine. >>>>>>> >>>>>>> 2. Storage-aware Large Data Transfers: Using data locality information >>>>>>> from GPFS to direct GridFTP to the DTN which has local block access to >>>>>>> the data. Thus allowing GridFTP to be more efficient in its use of the >>>>>>> storage network (Myrinet in this case). A good analogy is Hadoop¹s >>>>>>> HDFS, >>>>>>> but for data transfer instead of compute. >>>>>>> >>>>>>> Please correct or expand the above as needed. >>>>>>> >>>>>>> Due to the Petrel testbed being setup as a single file system, and >>>>>>> being a pseudo production testbed for Globus Share work (i.e. we can't >>>>>>> just destroy the existing globus-fs0 and rebuild it), I think the best >>>>>>> course of action to support the above work is to setup two 8 >>>>>>> file-server >>>>>>> filesystems on a new Myrinet fabric. We¹ll initially create a file >>>>>>> system on one (or optionally both) sets of 8 file servers. This will >>>>>>> let >>>>>>> you use opposing file servers for the baseline testing across a >>>>>>> network >>>>>>> fabric. This testbed would be similar in specs (same hardware, same >>>>>>> networking) as the Petrel testbed. >>>>>>> >>>>>>> Once that testing is complete, we will recreate the GPFS as a single >>>>>>> file system so you can experiment with data locality. Without further >>>>>>> configuration this will give you two silos of data locality (two >>>>>>> S2A9900 >>>>>>> racks). If you want to further worsen the test cases, we can limit >>>>>>> which >>>>>>> LUNs are presented to which file server, lowering the number of >>>>>>> file-servers with block access to a given set of data. >>>>>>> >>>>>>> It looks like we could have this testbed put together by early next >>>>>>> week, as we have a near ready set of 16 file servers available. I¹m >>>>>>> also >>>>>>> suggesting we use this new testbed as a more general resource for >>>>>>> those >>>>>>> that need GPFS but can withstand the file system being destroyed. I >>>>>>> believe this fits Bill¹s project needs, and he has similar goals in >>>>>>> testing data locality with GPFS. >>>>>>> >>>>>>> All, please let me know if you have any comments or concerns. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>> >>>>> >>>> >>> >> >

1 0

Re: [JLSE-Admins] Add Daqing Yun and Qishi Wu to tubes
by Raj Kettimuthu 14 Jul '14

14 Jul '14

I was able to see it now. I just approved it. On Jul 14, 2014, at 11:00 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > Hi raj, > > Can you check again? > > Thanks, > > Ben > > On Jul 12, 2014, at 9:55 PM, Raj Kettimuthu <kettimut(a)anl.gov> wrote: > >> Hmm, I do not see any pending sponsored accounts. >> >> On Jul 11, 2014, at 10:49 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >>> Hi Raj, >>> >>> I’ve updated Daping and Chase Wu’s JLSE accounts to list you as the sponsor, and I’ve removed Kumar. I also had to force Userbase to pull the updated 593 status from FAVOR for Daping for some reason, so it would show that it was approved. >>> >>> You should now have Daping and Chase Wu listed in https://accounts.jlse.anl.gov -> Sponsorships -> Pending Sponsored Accounts. Please approve the accounts as you see fit. >>> >>> Also for new accounts, you’re in the list of possible sponsors, so going forward please have people associated with your projects select you as the sponsor. This also means you’ll be responsible for any 593’s requested for those accounts. >>> >>> Please let me know if you have any issues approving Daping and Chase Wu. >>> >>> Ben >>> >>> >>> On Jul 10, 2014, at 4:07 PM, Raj Kettimuthu <kettimut(a)anl.gov> wrote: >>> >>>> Daqing’s 593 has been approved already. It would be helpful if we can get them access by early next week. They want to run some tests for a report that is due end of next week. >>>> >>>> Thanks, >>>> Raj >>>> >>>> On Jul 10, 2014, at 1:54 PM, Kalyan Kumaran <kumaran(a)anl.gov> wrote: >>>> >>>>> We agreed that Raj needs to be the sponsor as Daqing's is waiting for 593 approval and a gate pass. I do not know and have the resources to do this. >>>>> >>>>> -Kumar >>>>> >>>>> >>>>> On Jul 10, 2014, at 1:38 PM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>>> Kumar, >>>>>> >>>>>> Daqing picked you and is awaiting approval. I don’t see an account for Qishi, let me know what account name was used. >>>>>> >>>>>> As a side note, you also have Chase Wu and Preeti Malakar awaiting your approval. I have no idea who they are however. >>>>>> >>>>>> I’ve asked Ti to show me how to add a user as a sponsor, as what I thought would work, appears not to. >>>>>> >>>>>> Ben >>>>>> >>>>>> >>>>>> On Jul 10, 2014, at 1:22 PM, Raj Kettimuthu <kettimut(a)anl.gov> wrote: >>>>>> >>>>>>> I think they picked you as sponsor but I will check again to make sure. >>>>>>> >>>>>>> Raj >>>>>>> >>>>>>> On Jul 10, 2014, at 1:20 PM, Kalyan Kumaran <kumaran(a)anl.gov> wrote: >>>>>>> >>>>>>>> I have not seen any requests for approval. Did they pick me as sponsor? Or was it Rajeev? Ben? Ken? or Kaz? >>>>>>>> >>>>>>>> -Kumar >>>>>>>> >>>>>>>> >>>>>>>> On Jul 10, 2014, at 1:16 PM, Raj Kettimuthu <kettimut(a)anl.gov> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Ok. They already applied for accounts. >>>>>>>>> >>>>>>>>> On Jul 10, 2014, at 1:13 PM, Kalyan Kumaran <kumaran(a)anl.gov> wrote: >>>>>>>>> >>>>>>>>>> Raj, they can apply for accounts. But you should be their sponsor. >>>>>>>>>> >>>>>>>>>> Ken and Ben were working on figuring out with Ti on how to add you as the account sponsor. They can let you know if they were able to accomplish this. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Kumar >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Jul 10, 2014, at 1:10 PM, Raj Kettimuthu <kettimut(a)anl.gov> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Ok, thanks. >>>>>>>>>>> >>>>>>>>>>> Kumar, >>>>>>>>>>> Can you please approve the accounts for Daqing and Qishi? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Raj >>>>>>>>>>> >>>>>>>>>>> On Jul 10, 2014, at 1:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Raj, >>>>>>>>>>>> >>>>>>>>>>>> Whenever they’re JLSE accounts are approved, they can access tubes. We don’t currently have any special restrictions for which JLSE users can access that host. >>>>>>>>>>>> >>>>>>>>>>>> Ben >>>>>>>>>>>> >>>>>>>>>>>> On Jul 10, 2014, at 1:06 PM, Raj Kettimuthu <kettimut(a)anl.gov> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Ben, >>>>>>>>>>>>> Daqing and Qishi applied for JLSE account a couple of weeks ago. Qishi is a US citizen and Daqing’s 593 got approved a few days ago. Can you please get them setup on tubes? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Raj >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >

1 0

Re: [JLSE-Admins] Storage-aware Large-scale Data Transfers Testbed
by Raj Kettimuthu 14 Jul '14

14 Jul '14

I think access to/from NERSC DTNs (dtn{01-04}.nersc.gov) and Mira DTNs should be fine for now. Eunsung: If you want access to/from other nodes, please let Ben know. Raj On Jul 14, 2014, at 11:53 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > Raj, et al.: > > What firewall rules need to be put in place for external access to this testbed (both inbound and outbound)? > > Do you need just the GridFTP set of ports? Do you have specific external DTNs you’re testing with, so we don’t have to open this testbed’s GridFTP services up to the entire public internet? > > Thanks, > > Ben > > On Jul 8, 2014, at 10:59 AM, Jung, Eun Sung <esjung(a)mcs.anl.gov> wrote: > >> Ok, thank you. >> >> -Eunsung >> >> On 7/8/14, 10:54 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >> >>> Hi Eun Sung, >>> >>>> * Questions regarding the testbed for baseline testing. >>>> - Is there any additional node which can play a role of DTN except two 8 >>>> file-server nodes? >>> >>> In the proposed testbed no. Just to be clear I’m proposing 16 file >>> servers, split into two groups. The 16 file servers are backed by two DDN >>> S2A9900 racks of storage. The groups of 8 are to be defined by the file >>> servers connected to the same backing DDN S2A9900 >>> >>> The idea for the baseline testing would be to use one or more of the 8 >>> file servers from one group as clients across the network to the other >>> group of 8 file servers. Thus simulating DTNs accessing a shared >>> filesystem over a network. >>> >>>> - If so, all nodes including fs nodes and a DTN node share the same >>>> network? >>> >>> All 16 file servers will be on the same 10Gb Myrinet network. Each file >>> server will also have a 1GbE public interface, and a 1GbE internal >>> management interface. The same configuration as the Petrel testbed. >>> >>> The public interfaces will be heavily firewall’ed by default. We’ll >>> explicitly allow inbound and outbound traffic as requested. Ideally we >>> can get the list of firewall rules from Raj, Mike and you in by tomorrow. >>> If you only need standard GridFTP ports opened, please let us know that. >>> >>>> >>>> * Questions regarding the testbed for storage-aware large data transfers >>>> - Do you mean a single file server by a single file system? >>> >>> I mean a single shared filesystem (GPFS) across the 16 file servers. For >>> testing using GPFS’ data locality hints. >>> >>>> - Once a file system created, can we adjust the number of file servers >>>> later? For example, up to 32 servers, 2 servers per fs node. >>> >>> You will have 16 physical file servers. You’ll be able to reconfigure >>> them how you see fit within reason. This includes destroying and >>> recreating the GPFS if required. >>> >>> By default all 51 LUNs (8+2p RAID6 LUNs) per DDN S2A9900 rack are >>> presented to each of the 8 attached file servers via directly connected >>> DDR Infiniband. We can restrict or group which LUNs are presented to >>> which of the 8 file servers to force further data locality if you need to >>> simulate a less ideal storage configuration. >>> >>> Please let me know if you have further questions, >>> >>> Ben >>> >>> >>>> On 7/7/14, 11:44 PM, "Raj Kettimuthu" <kettimut(a)anl.gov> wrote: >>>> >>>>> Thanks Ben. This sounds good to me. >>>>> >>>>> Eunsung, >>>>> Do you have any concerns/questions? >>>>> >>>>> Raj >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Jul 7, 2014, at 4:05 PM, "Allen, Benjamin S." >>>>>> <bsallen(a)alcf.anl.gov> >>>>>> wrote: >>>>>> >>>>>> Hi Raj, >>>>>> >>>>>> To summarize the requirements for everyone CC¹ed your two objectives >>>>>> are: >>>>>> >>>>>> 1. Baseline testing a traditional DTN which accesses the shared >>>>>> filesystem over a network fabric, compared to a DTN directly on a >>>>>> storage controller. The JLSE file servers provide reasonably >>>>>> facsimiles >>>>>> for embedded storage controllers as they have a point-to-point DDR IB >>>>>> connection to the D2A9900 controllers. They¹re also lower powered >>>>>> servers so should simulate the lower resources available on an >>>>>> embedded >>>>>> storage controller virtual machine. >>>>>> >>>>>> 2. Storage-aware Large Data Transfers: Using data locality information >>>>>> from GPFS to direct GridFTP to the DTN which has local block access to >>>>>> the data. Thus allowing GridFTP to be more efficient in its use of the >>>>>> storage network (Myrinet in this case). A good analogy is Hadoop¹s >>>>>> HDFS, >>>>>> but for data transfer instead of compute. >>>>>> >>>>>> Please correct or expand the above as needed. >>>>>> >>>>>> Due to the Petrel testbed being setup as a single file system, and >>>>>> being a pseudo production testbed for Globus Share work (i.e. we can't >>>>>> just destroy the existing globus-fs0 and rebuild it), I think the best >>>>>> course of action to support the above work is to setup two 8 >>>>>> file-server >>>>>> filesystems on a new Myrinet fabric. We¹ll initially create a file >>>>>> system on one (or optionally both) sets of 8 file servers. This will >>>>>> let >>>>>> you use opposing file servers for the baseline testing across a >>>>>> network >>>>>> fabric. This testbed would be similar in specs (same hardware, same >>>>>> networking) as the Petrel testbed. >>>>>> >>>>>> Once that testing is complete, we will recreate the GPFS as a single >>>>>> file system so you can experiment with data locality. Without further >>>>>> configuration this will give you two silos of data locality (two >>>>>> S2A9900 >>>>>> racks). If you want to further worsen the test cases, we can limit >>>>>> which >>>>>> LUNs are presented to which file server, lowering the number of >>>>>> file-servers with block access to a given set of data. >>>>>> >>>>>> It looks like we could have this testbed put together by early next >>>>>> week, as we have a near ready set of 16 file servers available. I¹m >>>>>> also >>>>>> suggesting we use this new testbed as a more general resource for >>>>>> those >>>>>> that need GPFS but can withstand the file system being destroyed. I >>>>>> believe this fits Bill¹s project needs, and he has similar goals in >>>>>> testing data locality with GPFS. >>>>>> >>>>>> All, please let me know if you have any comments or concerns. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ben >>>>>> >>>>>> >>>> >>> >> >

1 0

Re: [JLSE-Admins] Storage-aware Large-scale Data Transfers Testbed
by Allen, Benjamin S. 14 Jul '14

14 Jul '14

Eunsung, Could you generate a list of NERSC' GridFTP hostnames/IPs that you expect to work with? Thanks, Ben On Jul 14, 2014, at 2:51 PM, Jung, Eun Sung <esjung(a)mcs.anl.gov> wrote: > Hi Ben, > > As of now, we will plan to do only GridFTP experiments with DTNs at NERSC. > > Regards, > Eunsung > > On 7/14/14, 11:53 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: > >> Raj, et al.: >> >> What firewall rules need to be put in place for external access to this >> testbed (both inbound and outbound)? >> >> Do you need just the GridFTP set of ports? Do you have specific external >> DTNs you’re testing with, so we don’t have to open this testbed’s GridFTP >> services up to the entire public internet? >> >> Thanks, >> >> Ben >> >> On Jul 8, 2014, at 10:59 AM, Jung, Eun Sung <esjung(a)mcs.anl.gov> wrote: >> >>> Ok, thank you. >>> >>> -Eunsung >>> >>> On 7/8/14, 10:54 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >>> >>>> Hi Eun Sung, >>>> >>>>> * Questions regarding the testbed for baseline testing. >>>>> - Is there any additional node which can play a role of DTN except >>>>> two 8 >>>>> file-server nodes? >>>> >>>> In the proposed testbed no. Just to be clear I’m proposing 16 file >>>> servers, split into two groups. The 16 file servers are backed by two >>>> DDN >>>> S2A9900 racks of storage. The groups of 8 are to be defined by the file >>>> servers connected to the same backing DDN S2A9900 >>>> >>>> The idea for the baseline testing would be to use one or more of the 8 >>>> file servers from one group as clients across the network to the other >>>> group of 8 file servers. Thus simulating DTNs accessing a shared >>>> filesystem over a network. >>>> >>>>> - If so, all nodes including fs nodes and a DTN node share the same >>>>> network? >>>> >>>> All 16 file servers will be on the same 10Gb Myrinet network. Each file >>>> server will also have a 1GbE public interface, and a 1GbE internal >>>> management interface. The same configuration as the Petrel testbed. >>>> >>>> The public interfaces will be heavily firewall’ed by default. We’ll >>>> explicitly allow inbound and outbound traffic as requested. Ideally we >>>> can get the list of firewall rules from Raj, Mike and you in by >>>> tomorrow. >>>> If you only need standard GridFTP ports opened, please let us know >>>> that. >>>> >>>>> >>>>> * Questions regarding the testbed for storage-aware large data >>>>> transfers >>>>> - Do you mean a single file server by a single file system? >>>> >>>> I mean a single shared filesystem (GPFS) across the 16 file servers. >>>> For >>>> testing using GPFS’ data locality hints. >>>> >>>>> - Once a file system created, can we adjust the number of file servers >>>>> later? For example, up to 32 servers, 2 servers per fs node. >>>> >>>> You will have 16 physical file servers. You’ll be able to reconfigure >>>> them how you see fit within reason. This includes destroying and >>>> recreating the GPFS if required. >>>> >>>> By default all 51 LUNs (8+2p RAID6 LUNs) per DDN S2A9900 rack are >>>> presented to each of the 8 attached file servers via directly connected >>>> DDR Infiniband. We can restrict or group which LUNs are presented to >>>> which of the 8 file servers to force further data locality if you need >>>> to >>>> simulate a less ideal storage configuration. >>>> >>>> Please let me know if you have further questions, >>>> >>>> Ben >>>> >>>> >>>>> On 7/7/14, 11:44 PM, "Raj Kettimuthu" <kettimut(a)anl.gov> wrote: >>>>> >>>>>> Thanks Ben. This sounds good to me. >>>>>> >>>>>> Eunsung, >>>>>> Do you have any concerns/questions? >>>>>> >>>>>> Raj >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On Jul 7, 2014, at 4:05 PM, "Allen, Benjamin S." >>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>> Hi Raj, >>>>>>> >>>>>>> To summarize the requirements for everyone CC¹ed your two objectives >>>>>>> are: >>>>>>> >>>>>>> 1. Baseline testing a traditional DTN which accesses the shared >>>>>>> filesystem over a network fabric, compared to a DTN directly on a >>>>>>> storage controller. The JLSE file servers provide reasonably >>>>>>> facsimiles >>>>>>> for embedded storage controllers as they have a point-to-point DDR >>>>>>> IB >>>>>>> connection to the D2A9900 controllers. They¹re also lower powered >>>>>>> servers so should simulate the lower resources available on an >>>>>>> embedded >>>>>>> storage controller virtual machine. >>>>>>> >>>>>>> 2. Storage-aware Large Data Transfers: Using data locality >>>>>>> information >>>>>>> from GPFS to direct GridFTP to the DTN which has local block access >>>>>>> to >>>>>>> the data. Thus allowing GridFTP to be more efficient in its use of >>>>>>> the >>>>>>> storage network (Myrinet in this case). A good analogy is Hadoop¹s >>>>>>> HDFS, >>>>>>> but for data transfer instead of compute. >>>>>>> >>>>>>> Please correct or expand the above as needed. >>>>>>> >>>>>>> Due to the Petrel testbed being setup as a single file system, and >>>>>>> being a pseudo production testbed for Globus Share work (i.e. we >>>>>>> can't >>>>>>> just destroy the existing globus-fs0 and rebuild it), I think the >>>>>>> best >>>>>>> course of action to support the above work is to setup two 8 >>>>>>> file-server >>>>>>> filesystems on a new Myrinet fabric. We¹ll initially create a file >>>>>>> system on one (or optionally both) sets of 8 file servers. This will >>>>>>> let >>>>>>> you use opposing file servers for the baseline testing across a >>>>>>> network >>>>>>> fabric. This testbed would be similar in specs (same hardware, same >>>>>>> networking) as the Petrel testbed. >>>>>>> >>>>>>> Once that testing is complete, we will recreate the GPFS as a single >>>>>>> file system so you can experiment with data locality. Without >>>>>>> further >>>>>>> configuration this will give you two silos of data locality (two >>>>>>> S2A9900 >>>>>>> racks). If you want to further worsen the test cases, we can limit >>>>>>> which >>>>>>> LUNs are presented to which file server, lowering the number of >>>>>>> file-servers with block access to a given set of data. >>>>>>> >>>>>>> It looks like we could have this testbed put together by early next >>>>>>> week, as we have a near ready set of 16 file servers available. I¹m >>>>>>> also >>>>>>> suggesting we use this new testbed as a more general resource for >>>>>>> those >>>>>>> that need GPFS but can withstand the file system being destroyed. I >>>>>>> believe this fits Bill¹s project needs, and he has similar goals in >>>>>>> testing data locality with GPFS. >>>>>>> >>>>>>> All, please let me know if you have any comments or concerns. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>> >>>>> >>>> >>> >> >

1 0

Re: [JLSE-Admins] Storage-aware Large-scale Data Transfers Testbed
by Jung, Eun Sung 14 Jul '14

14 Jul '14

Hi Ben, As of now, we will plan to do only GridFTP experiments with DTNs at NERSC. Regards, Eunsung On 7/14/14, 11:53 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >Raj, et al.: > >What firewall rules need to be put in place for external access to this >testbed (both inbound and outbound)? > >Do you need just the GridFTP set of ports? Do you have specific external >DTNs you’re testing with, so we don’t have to open this testbed’s GridFTP >services up to the entire public internet? > >Thanks, > >Ben > >On Jul 8, 2014, at 10:59 AM, Jung, Eun Sung <esjung(a)mcs.anl.gov> wrote: > >> Ok, thank you. >> >> -Eunsung >> >> On 7/8/14, 10:54 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >> >>> Hi Eun Sung, >>> >>>> * Questions regarding the testbed for baseline testing. >>>> - Is there any additional node which can play a role of DTN except >>>>two 8 >>>> file-server nodes? >>> >>> In the proposed testbed no. Just to be clear I’m proposing 16 file >>> servers, split into two groups. The 16 file servers are backed by two >>>DDN >>> S2A9900 racks of storage. The groups of 8 are to be defined by the file >>> servers connected to the same backing DDN S2A9900 >>> >>> The idea for the baseline testing would be to use one or more of the 8 >>> file servers from one group as clients across the network to the other >>> group of 8 file servers. Thus simulating DTNs accessing a shared >>> filesystem over a network. >>> >>>> - If so, all nodes including fs nodes and a DTN node share the same >>>> network? >>> >>> All 16 file servers will be on the same 10Gb Myrinet network. Each file >>> server will also have a 1GbE public interface, and a 1GbE internal >>> management interface. The same configuration as the Petrel testbed. >>> >>> The public interfaces will be heavily firewall’ed by default. We’ll >>> explicitly allow inbound and outbound traffic as requested. Ideally we >>> can get the list of firewall rules from Raj, Mike and you in by >>>tomorrow. >>> If you only need standard GridFTP ports opened, please let us know >>>that. >>> >>>> >>>> * Questions regarding the testbed for storage-aware large data >>>>transfers >>>> - Do you mean a single file server by a single file system? >>> >>> I mean a single shared filesystem (GPFS) across the 16 file servers. >>>For >>> testing using GPFS’ data locality hints. >>> >>>> - Once a file system created, can we adjust the number of file servers >>>> later? For example, up to 32 servers, 2 servers per fs node. >>> >>> You will have 16 physical file servers. You’ll be able to reconfigure >>> them how you see fit within reason. This includes destroying and >>> recreating the GPFS if required. >>> >>> By default all 51 LUNs (8+2p RAID6 LUNs) per DDN S2A9900 rack are >>> presented to each of the 8 attached file servers via directly connected >>> DDR Infiniband. We can restrict or group which LUNs are presented to >>> which of the 8 file servers to force further data locality if you need >>>to >>> simulate a less ideal storage configuration. >>> >>> Please let me know if you have further questions, >>> >>> Ben >>> >>> >>>> On 7/7/14, 11:44 PM, "Raj Kettimuthu" <kettimut(a)anl.gov> wrote: >>>> >>>>> Thanks Ben. This sounds good to me. >>>>> >>>>> Eunsung, >>>>> Do you have any concerns/questions? >>>>> >>>>> Raj >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Jul 7, 2014, at 4:05 PM, "Allen, Benjamin S." >>>>>> <bsallen(a)alcf.anl.gov> >>>>>> wrote: >>>>>> >>>>>> Hi Raj, >>>>>> >>>>>> To summarize the requirements for everyone CC¹ed your two objectives >>>>>> are: >>>>>> >>>>>> 1. Baseline testing a traditional DTN which accesses the shared >>>>>> filesystem over a network fabric, compared to a DTN directly on a >>>>>> storage controller. The JLSE file servers provide reasonably >>>>>> facsimiles >>>>>> for embedded storage controllers as they have a point-to-point DDR >>>>>>IB >>>>>> connection to the D2A9900 controllers. They¹re also lower powered >>>>>> servers so should simulate the lower resources available on an >>>>>> embedded >>>>>> storage controller virtual machine. >>>>>> >>>>>> 2. Storage-aware Large Data Transfers: Using data locality >>>>>>information >>>>>> from GPFS to direct GridFTP to the DTN which has local block access >>>>>>to >>>>>> the data. Thus allowing GridFTP to be more efficient in its use of >>>>>>the >>>>>> storage network (Myrinet in this case). A good analogy is Hadoop¹s >>>>>> HDFS, >>>>>> but for data transfer instead of compute. >>>>>> >>>>>> Please correct or expand the above as needed. >>>>>> >>>>>> Due to the Petrel testbed being setup as a single file system, and >>>>>> being a pseudo production testbed for Globus Share work (i.e. we >>>>>>can't >>>>>> just destroy the existing globus-fs0 and rebuild it), I think the >>>>>>best >>>>>> course of action to support the above work is to setup two 8 >>>>>> file-server >>>>>> filesystems on a new Myrinet fabric. We¹ll initially create a file >>>>>> system on one (or optionally both) sets of 8 file servers. This will >>>>>> let >>>>>> you use opposing file servers for the baseline testing across a >>>>>> network >>>>>> fabric. This testbed would be similar in specs (same hardware, same >>>>>> networking) as the Petrel testbed. >>>>>> >>>>>> Once that testing is complete, we will recreate the GPFS as a single >>>>>> file system so you can experiment with data locality. Without >>>>>>further >>>>>> configuration this will give you two silos of data locality (two >>>>>> S2A9900 >>>>>> racks). If you want to further worsen the test cases, we can limit >>>>>> which >>>>>> LUNs are presented to which file server, lowering the number of >>>>>> file-servers with block access to a given set of data. >>>>>> >>>>>> It looks like we could have this testbed put together by early next >>>>>> week, as we have a near ready set of 16 file servers available. I¹m >>>>>> also >>>>>> suggesting we use this new testbed as a more general resource for >>>>>> those >>>>>> that need GPFS but can withstand the file system being destroyed. I >>>>>> believe this fits Bill¹s project needs, and he has similar goals in >>>>>> testing data locality with GPFS. >>>>>> >>>>>> All, please let me know if you have any comments or concerns. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ben >>>>>> >>>>>> >>>> >>> >> >

1 0

Re: [JLSE-Admins] Storage-aware Large-scale Data Transfers Testbed
by Allen, Benjamin S. 14 Jul '14

14 Jul '14

Raj, et al.: What firewall rules need to be put in place for external access to this testbed (both inbound and outbound)? Do you need just the GridFTP set of ports? Do you have specific external DTNs you’re testing with, so we don’t have to open this testbed’s GridFTP services up to the entire public internet? Thanks, Ben On Jul 8, 2014, at 10:59 AM, Jung, Eun Sung <esjung(a)mcs.anl.gov> wrote: > Ok, thank you. > > -Eunsung > > On 7/8/14, 10:54 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: > >> Hi Eun Sung, >> >>> * Questions regarding the testbed for baseline testing. >>> - Is there any additional node which can play a role of DTN except two 8 >>> file-server nodes? >> >> In the proposed testbed no. Just to be clear I’m proposing 16 file >> servers, split into two groups. The 16 file servers are backed by two DDN >> S2A9900 racks of storage. The groups of 8 are to be defined by the file >> servers connected to the same backing DDN S2A9900 >> >> The idea for the baseline testing would be to use one or more of the 8 >> file servers from one group as clients across the network to the other >> group of 8 file servers. Thus simulating DTNs accessing a shared >> filesystem over a network. >> >>> - If so, all nodes including fs nodes and a DTN node share the same >>> network? >> >> All 16 file servers will be on the same 10Gb Myrinet network. Each file >> server will also have a 1GbE public interface, and a 1GbE internal >> management interface. The same configuration as the Petrel testbed. >> >> The public interfaces will be heavily firewall’ed by default. We’ll >> explicitly allow inbound and outbound traffic as requested. Ideally we >> can get the list of firewall rules from Raj, Mike and you in by tomorrow. >> If you only need standard GridFTP ports opened, please let us know that. >> >>> >>> * Questions regarding the testbed for storage-aware large data transfers >>> - Do you mean a single file server by a single file system? >> >> I mean a single shared filesystem (GPFS) across the 16 file servers. For >> testing using GPFS’ data locality hints. >> >>> - Once a file system created, can we adjust the number of file servers >>> later? For example, up to 32 servers, 2 servers per fs node. >> >> You will have 16 physical file servers. You’ll be able to reconfigure >> them how you see fit within reason. This includes destroying and >> recreating the GPFS if required. >> >> By default all 51 LUNs (8+2p RAID6 LUNs) per DDN S2A9900 rack are >> presented to each of the 8 attached file servers via directly connected >> DDR Infiniband. We can restrict or group which LUNs are presented to >> which of the 8 file servers to force further data locality if you need to >> simulate a less ideal storage configuration. >> >> Please let me know if you have further questions, >> >> Ben >> >> >>> On 7/7/14, 11:44 PM, "Raj Kettimuthu" <kettimut(a)anl.gov> wrote: >>> >>>> Thanks Ben. This sounds good to me. >>>> >>>> Eunsung, >>>> Do you have any concerns/questions? >>>> >>>> Raj >>>> >>>> Sent from my iPhone >>>> >>>>> On Jul 7, 2014, at 4:05 PM, "Allen, Benjamin S." >>>>> <bsallen(a)alcf.anl.gov> >>>>> wrote: >>>>> >>>>> Hi Raj, >>>>> >>>>> To summarize the requirements for everyone CC¹ed your two objectives >>>>> are: >>>>> >>>>> 1. Baseline testing a traditional DTN which accesses the shared >>>>> filesystem over a network fabric, compared to a DTN directly on a >>>>> storage controller. The JLSE file servers provide reasonably >>>>> facsimiles >>>>> for embedded storage controllers as they have a point-to-point DDR IB >>>>> connection to the D2A9900 controllers. They¹re also lower powered >>>>> servers so should simulate the lower resources available on an >>>>> embedded >>>>> storage controller virtual machine. >>>>> >>>>> 2. Storage-aware Large Data Transfers: Using data locality information >>>>> from GPFS to direct GridFTP to the DTN which has local block access to >>>>> the data. Thus allowing GridFTP to be more efficient in its use of the >>>>> storage network (Myrinet in this case). A good analogy is Hadoop¹s >>>>> HDFS, >>>>> but for data transfer instead of compute. >>>>> >>>>> Please correct or expand the above as needed. >>>>> >>>>> Due to the Petrel testbed being setup as a single file system, and >>>>> being a pseudo production testbed for Globus Share work (i.e. we can't >>>>> just destroy the existing globus-fs0 and rebuild it), I think the best >>>>> course of action to support the above work is to setup two 8 >>>>> file-server >>>>> filesystems on a new Myrinet fabric. We¹ll initially create a file >>>>> system on one (or optionally both) sets of 8 file servers. This will >>>>> let >>>>> you use opposing file servers for the baseline testing across a >>>>> network >>>>> fabric. This testbed would be similar in specs (same hardware, same >>>>> networking) as the Petrel testbed. >>>>> >>>>> Once that testing is complete, we will recreate the GPFS as a single >>>>> file system so you can experiment with data locality. Without further >>>>> configuration this will give you two silos of data locality (two >>>>> S2A9900 >>>>> racks). If you want to further worsen the test cases, we can limit >>>>> which >>>>> LUNs are presented to which file server, lowering the number of >>>>> file-servers with block access to a given set of data. >>>>> >>>>> It looks like we could have this testbed put together by early next >>>>> week, as we have a near ready set of 16 file servers available. I¹m >>>>> also >>>>> suggesting we use this new testbed as a more general resource for >>>>> those >>>>> that need GPFS but can withstand the file system being destroyed. I >>>>> believe this fits Bill¹s project needs, and he has similar goals in >>>>> testing data locality with GPFS. >>>>> >>>>> All, please let me know if you have any comments or concerns. >>>>> >>>>> Thanks, >>>>> >>>>> Ben >>>>> >>>>> >>> >> >

1 0