April 2013 - Kbase-core - mailman3.cels.anl.gov

Re: [Kbase-core] shock
by Robert Olson 27 Apr '13

27 Apr '13

This is why I want to do the test. Right now we do this with the filesystem and seek indexes and it works great; folks keep proposing new tech thats better than anything currently done and I'd like to put it to the test. ----- Original Message ----- > Well, I would love to see the results of a field test. Can you > efficiently save down 1 million objects and then quickly query and > load up 4000 of them? > Of course… if the objects are going into the file system… consider > what happens when you try and read in 1000 files. If that's really > how it's implemented… forget about it. > Every file read will initiate a new disk access. But maybe I just > don't get the implementation. It's worth an experiment. > > I would personally love to see how mongodb and gridfs compares for > shock. So if we set up a test server somewhere, let me know. > > Chris > > On Apr 26, 2013, at 10:46 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > > > didn't you say you had 9TB? That data is just 160GB so should fit > > fine, right? > > > > I'm thinking of testing shock as an example of one of your object > > stores, since as I understand it they're putting it up as being an > > efficient data store for that sort of data. > > > > What I want to do with the data is retrieve arbitrary pieces of it > > efficiently. Can't ask for much more than that. > > > > ----- Original Message ----- > >> > >>> So say I wanted to store 40 million pieces data, 4K each. Is > >>> shock > >>> appropriate for that? > >> > >> Just storing the data, sure. I haven't tried this scenario, but I > >> haven't seen anything in shock that would prevent you from doing > >> this. But you could also store that data on thumbdrives, tapes, > >> or > >> cd-roms :) It really depends on what you want to do with the > >> data, > >> and what kind of availability and performance you need. > >> > >> Shock stores data on a filesystem at the end of the day, and the > >> last > >> time I checked it would only store it on one filesystem - on one > >> server. The last time I spoke with Jared Wilkening about this > >> last > >> year we were talking about sync methods for replicas and referrals > >> to other servers - so I know this was in the works. > >> > >> If you want to test shock at that scale, it's not difficult to > >> start > >> an instance following the readme. > >> I don't think we have anything with that much storage free for > >> trying > >> this out. > >> > >> ----- Original Message ----- > >>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >>> Cc: kbase-core(a)lists.kbase.us > >>> Sent: Friday, April 26, 2013 5:17:10 PM > >>> Subject: Re: [Kbase-core] shock > >>> > >>> Thanks. > >>> > >>> So say I wanted to store 40 million pieces data, 4K each. Is > >>> shock > >>> appropriate for that? > >>> > >>> ----- Original Message ----- > >>>> Yes, shock is running at kbase.us/services/shock and > >>>> kbase.us/services/shock-api > >>>> Globus creds should get you write access. There is 9.5T > >>>> available. > >>>> > >>>> Let me know if you need anything else to get things going. > >>>> > >>>> ----- Original Message ----- > >>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>>> To: kbase-core(a)lists.kbase.us > >>>>> Sent: Friday, April 26, 2013 3:07:37 PM > >>>>> Subject: [Kbase-core] shock > >>>>> > >>>>> Is there a production shock server running in KB land? Is there > >>>>> doc? > >>>>> I searched for "shock" on kbase.us and got no hits. > >>>>> > >>>>> I'd like to look into pushing all of the sequence data (contigs > >>>>> + > >>>>> features) into shock and using its range lookups for pulling > >>>>> sequence data chunks back out, the way we currently do with > >>>>> flat > >>>>> files in SEED and the file+offset+length indexes we build. > >>>>> > >>>>> --bob > >>>>> _______________________________________________ > >>>>> Kbase-core mailing list > >>>>> Kbase-core(a)lists.kbase.us > >>>>> https://lists.kbase.us/mailman/listinfo/kbase-core > >>>>> > >>>> > >>> > >> > > _______________________________________________ > > Kbase-core mailing list > > Kbase-core(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-core > >

1 0

Re: [Kbase-core] shock
by Dan Olson 27 Apr '13

27 Apr '13

About the size, yes. I was off by an order of magnitude. We have space for that. You should test it in a test environment first before Pushing that into the production instance. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Robert Olson <olson(a)mcs.anl.gov> wrote: > didn't you say you had 9TB? That data is just 160GB so should fit fine, right? > > I'm thinking of testing shock as an example of one of your object stores, since as I understand it they're putting it up as being an efficient data store for that sort of data. > > What I want to do with the data is retrieve arbitrary pieces of it efficiently. Can't ask for much more than that. > > ----- Original Message ----- > > > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > > appropriate for that? > > > > Just storing the data, sure. I haven't tried this scenario, but I > > haven't seen anything in shock that would prevent you from doing > > this. But you could also store that data on thumbdrives, tapes, or > > cd-roms :) It really depends on what you want to do with the data, > > and what kind of availability and performance you need. > > > > Shock stores data on a filesystem at the end of the day, and the last > > time I checked it would only store it on one filesystem - on one > > server. The last time I spoke with Jared Wilkening about this last > > year we were talking about sync methods for replicas and referrals > > to other servers - so I know this was in the works. > > > > If you want to test shock at that scale, it's not difficult to start > > an instance following the readme. > > I don't think we have anything with that much storage free for trying > > this out. > > > > ----- Original Message ----- > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > To: "Dan Olson" <dolson(a)mcs.anl.gov> > > > Cc: kbase-core(a)lists.kbase.us > > > Sent: Friday, April 26, 2013 5:17:10 PM > > > Subject: Re: [Kbase-core] shock > > > > > > Thanks. > > > > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > > appropriate for that? > > > > > > ----- Original Message ----- > > > > Yes, shock is running at kbase.us/services/shock and > > > > kbase.us/services/shock-api > > > > Globus creds should get you write access. There is 9.5T > > > > available. > > > > > > > > Let me know if you need anything else to get things going. > > > > > > > > ----- Original Message ----- > > > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > > > To: kbase-core(a)lists.kbase.us > > > > > Sent: Friday, April 26, 2013 3:07:37 PM > > > > > Subject: [Kbase-core] shock > > > > > > > > > > Is there a production shock server running in KB land? Is there > > > > > doc? > > > > > I searched for "shock" on kbase.us and got no hits. > > > > > > > > > > I'd like to look into pushing all of the sequence data (contigs > > > > > + > > > > > features) into shock and using its range lookups for pulling > > > > > sequence data chunks back out, the way we currently do with > > > > > flat > > > > > files in SEED and the file+offset+length indexes we build. > > > > > > > > > > --bob > > > > > _______________________________________________ > > > > > Kbase-core mailing list > > > > > Kbase-core(a)lists.kbase.us > > > > > https://lists.kbase.us/mailman/listinfo/kbase-core > > > > > > > > > > > > > >

1 0

Re: [Kbase-core] shock
by Christopher Henry 26 Apr '13

26 Apr '13

Well, I would love to see the results of a field test. Can you efficiently save down 1 million objects and then quickly query and load up 4000 of them? Of course… if the objects are going into the file system… consider what happens when you try and read in 1000 files. If that's really how it's implemented… forget about it. Every file read will initiate a new disk access. But maybe I just don't get the implementation. It's worth an experiment. I would personally love to see how mongodb and gridfs compares for shock. So if we set up a test server somewhere, let me know. Chris On Apr 26, 2013, at 10:46 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > didn't you say you had 9TB? That data is just 160GB so should fit fine, right? > > I'm thinking of testing shock as an example of one of your object stores, since as I understand it they're putting it up as being an efficient data store for that sort of data. > > What I want to do with the data is retrieve arbitrary pieces of it efficiently. Can't ask for much more than that. > > ----- Original Message ----- >> >>> So say I wanted to store 40 million pieces data, 4K each. Is shock >>> appropriate for that? >> >> Just storing the data, sure. I haven't tried this scenario, but I >> haven't seen anything in shock that would prevent you from doing >> this. But you could also store that data on thumbdrives, tapes, or >> cd-roms :) It really depends on what you want to do with the data, >> and what kind of availability and performance you need. >> >> Shock stores data on a filesystem at the end of the day, and the last >> time I checked it would only store it on one filesystem - on one >> server. The last time I spoke with Jared Wilkening about this last >> year we were talking about sync methods for replicas and referrals >> to other servers - so I know this was in the works. >> >> If you want to test shock at that scale, it's not difficult to start >> an instance following the readme. >> I don't think we have anything with that much storage free for trying >> this out. >> >> ----- Original Message ----- >>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>> To: "Dan Olson" <dolson(a)mcs.anl.gov> >>> Cc: kbase-core(a)lists.kbase.us >>> Sent: Friday, April 26, 2013 5:17:10 PM >>> Subject: Re: [Kbase-core] shock >>> >>> Thanks. >>> >>> So say I wanted to store 40 million pieces data, 4K each. Is shock >>> appropriate for that? >>> >>> ----- Original Message ----- >>>> Yes, shock is running at kbase.us/services/shock and >>>> kbase.us/services/shock-api >>>> Globus creds should get you write access. There is 9.5T >>>> available. >>>> >>>> Let me know if you need anything else to get things going. >>>> >>>> ----- Original Message ----- >>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>>> To: kbase-core(a)lists.kbase.us >>>>> Sent: Friday, April 26, 2013 3:07:37 PM >>>>> Subject: [Kbase-core] shock >>>>> >>>>> Is there a production shock server running in KB land? Is there >>>>> doc? >>>>> I searched for "shock" on kbase.us and got no hits. >>>>> >>>>> I'd like to look into pushing all of the sequence data (contigs >>>>> + >>>>> features) into shock and using its range lookups for pulling >>>>> sequence data chunks back out, the way we currently do with >>>>> flat >>>>> files in SEED and the file+offset+length indexes we build. >>>>> >>>>> --bob >>>>> _______________________________________________ >>>>> Kbase-core mailing list >>>>> Kbase-core(a)lists.kbase.us >>>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>>>> >>>> >>> >> > _______________________________________________ > Kbase-core mailing list > Kbase-core(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-core

1 0

Re: [Kbase-core] shock
by Robert Olson 26 Apr '13

26 Apr '13

didn't you say you had 9TB? That data is just 160GB so should fit fine, right? I'm thinking of testing shock as an example of one of your object stores, since as I understand it they're putting it up as being an efficient data store for that sort of data. What I want to do with the data is retrieve arbitrary pieces of it efficiently. Can't ask for much more than that. ----- Original Message ----- > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > appropriate for that? > > Just storing the data, sure. I haven't tried this scenario, but I > haven't seen anything in shock that would prevent you from doing > this. But you could also store that data on thumbdrives, tapes, or > cd-roms :) It really depends on what you want to do with the data, > and what kind of availability and performance you need. > > Shock stores data on a filesystem at the end of the day, and the last > time I checked it would only store it on one filesystem - on one > server. The last time I spoke with Jared Wilkening about this last > year we were talking about sync methods for replicas and referrals > to other servers - so I know this was in the works. > > If you want to test shock at that scale, it's not difficult to start > an instance following the readme. > I don't think we have anything with that much storage free for trying > this out. > > ----- Original Message ----- > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > To: "Dan Olson" <dolson(a)mcs.anl.gov> > > Cc: kbase-core(a)lists.kbase.us > > Sent: Friday, April 26, 2013 5:17:10 PM > > Subject: Re: [Kbase-core] shock > > > > Thanks. > > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > appropriate for that? > > > > ----- Original Message ----- > > > Yes, shock is running at kbase.us/services/shock and > > > kbase.us/services/shock-api > > > Globus creds should get you write access. There is 9.5T > > > available. > > > > > > Let me know if you need anything else to get things going. > > > > > > ----- Original Message ----- > > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > > To: kbase-core(a)lists.kbase.us > > > > Sent: Friday, April 26, 2013 3:07:37 PM > > > > Subject: [Kbase-core] shock > > > > > > > > Is there a production shock server running in KB land? Is there > > > > doc? > > > > I searched for "shock" on kbase.us and got no hits. > > > > > > > > I'd like to look into pushing all of the sequence data (contigs > > > > + > > > > features) into shock and using its range lookups for pulling > > > > sequence data chunks back out, the way we currently do with > > > > flat > > > > files in SEED and the file+offset+length indexes we build. > > > > > > > > --bob > > > > _______________________________________________ > > > > Kbase-core mailing list > > > > Kbase-core(a)lists.kbase.us > > > > https://lists.kbase.us/mailman/listinfo/kbase-core > > > > > > > > > >

1 0

Re: [Kbase-core] shock
by Dan Olson 26 Apr '13

26 Apr '13

> So say I wanted to store 40 million pieces data, 4K each. Is shock > appropriate for that? Just storing the data, sure. I haven't tried this scenario, but I haven't seen anything in shock that would prevent you from doing this. But you could also store that data on thumbdrives, tapes, or cd-roms :) It really depends on what you want to do with the data, and what kind of availability and performance you need. Shock stores data on a filesystem at the end of the day, and the last time I checked it would only store it on one filesystem - on one server. The last time I spoke with Jared Wilkening about this last year we were talking about sync methods for replicas and referrals to other servers - so I know this was in the works. If you want to test shock at that scale, it's not difficult to start an instance following the readme. I don't think we have anything with that much storage free for trying this out. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Friday, April 26, 2013 5:17:10 PM > Subject: Re: [Kbase-core] shock > > Thanks. > > So say I wanted to store 40 million pieces data, 4K each. Is shock > appropriate for that? > > ----- Original Message ----- > > Yes, shock is running at kbase.us/services/shock and > > kbase.us/services/shock-api > > Globus creds should get you write access. There is 9.5T available. > > > > Let me know if you need anything else to get things going. > > > > ----- Original Message ----- > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > To: kbase-core(a)lists.kbase.us > > > Sent: Friday, April 26, 2013 3:07:37 PM > > > Subject: [Kbase-core] shock > > > > > > Is there a production shock server running in KB land? Is there > > > doc? > > > I searched for "shock" on kbase.us and got no hits. > > > > > > I'd like to look into pushing all of the sequence data (contigs + > > > features) into shock and using its range lookups for pulling > > > sequence data chunks back out, the way we currently do with flat > > > files in SEED and the file+offset+length indexes we build. > > > > > > --bob > > > _______________________________________________ > > > Kbase-core mailing list > > > Kbase-core(a)lists.kbase.us > > > https://lists.kbase.us/mailman/listinfo/kbase-core > > > > > >

1 0

Re: [Kbase-core] shock
by Robert Olson 26 Apr '13

26 Apr '13

Thanks. So say I wanted to store 40 million pieces data, 4K each. Is shock appropriate for that? ----- Original Message ----- > Yes, shock is running at kbase.us/services/shock and > kbase.us/services/shock-api > Globus creds should get you write access. There is 9.5T available. > > Let me know if you need anything else to get things going. > > ----- Original Message ----- > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > To: kbase-core(a)lists.kbase.us > > Sent: Friday, April 26, 2013 3:07:37 PM > > Subject: [Kbase-core] shock > > > > Is there a production shock server running in KB land? Is there > > doc? > > I searched for "shock" on kbase.us and got no hits. > > > > I'd like to look into pushing all of the sequence data (contigs + > > features) into shock and using its range lookups for pulling > > sequence data chunks back out, the way we currently do with flat > > files in SEED and the file+offset+length indexes we build. > > > > --bob > > _______________________________________________ > > Kbase-core mailing list > > Kbase-core(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-core > > >

1 0

Re: [Kbase-core] shock
by Dan Olson 26 Apr '13

26 Apr '13

Oh.. The only doc I'm aware of is the readme, accesible https://github.com/MG-RAST/Shock ----- Original Message ----- > From: "Dan Olson" <dolson(a)mcs.anl.gov> > To: "Robert Olson" <olson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Friday, April 26, 2013 3:59:09 PM > Subject: Re: [Kbase-core] shock > > Yes, shock is running at kbase.us/services/shock and > kbase.us/services/shock-api > Globus creds should get you write access. There is 9.5T available. > > Let me know if you need anything else to get things going. > > ----- Original Message ----- > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > To: kbase-core(a)lists.kbase.us > > Sent: Friday, April 26, 2013 3:07:37 PM > > Subject: [Kbase-core] shock > > > > Is there a production shock server running in KB land? Is there > > doc? > > I searched for "shock" on kbase.us and got no hits. > > > > I'd like to look into pushing all of the sequence data (contigs + > > features) into shock and using its range lookups for pulling > > sequence data chunks back out, the way we currently do with flat > > files in SEED and the file+offset+length indexes we build. > > > > --bob > > _______________________________________________ > > Kbase-core mailing list > > Kbase-core(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-core > > >

1 0

Re: [Kbase-core] shock
by Dan Olson 26 Apr '13

26 Apr '13

Yes, shock is running at kbase.us/services/shock and kbase.us/services/shock-api Globus creds should get you write access. There is 9.5T available. Let me know if you need anything else to get things going. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: kbase-core(a)lists.kbase.us > Sent: Friday, April 26, 2013 3:07:37 PM > Subject: [Kbase-core] shock > > Is there a production shock server running in KB land? Is there doc? > I searched for "shock" on kbase.us and got no hits. > > I'd like to look into pushing all of the sequence data (contigs + > features) into shock and using its range lookups for pulling > sequence data chunks back out, the way we currently do with flat > files in SEED and the file+offset+length indexes we build. > > --bob > _______________________________________________ > Kbase-core mailing list > Kbase-core(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-core >

1 0

shock
by Robert Olson 26 Apr '13

26 Apr '13

Is there a production shock server running in KB land? Is there doc? I searched for "shock" on kbase.us and got no hits. I'd like to look into pushing all of the sequence data (contigs + features) into shock and using its range lookups for pulling sequence data chunks back out, the way we currently do with flat files in SEED and the file+offset+length indexes we build. --bob

1 0

Re: [Kbase-core] Globus online
by Dan Olson 25 Apr '13

25 Apr '13

This only uses local user accounts. I had set up an account: kbase / kbase4me123 According to Rachana, you need a local user account to actually configure a share - but after the initial folder is created you can share to globus nexus accounts. I'm still getting familiar with it, so I should know more soon. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Thursday, April 25, 2013 12:53:50 PM > Subject: Re: [Kbase-core] Globus online > > I tried to start a transfer to one but: > > > > [image/png:PastedGraphic-5.png] > > > > > Didn't recognize my GO or my KB password. > > On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > > > We do have the dtn's defined in GO. kbase-chicago, and > > kbase-knoxville are defined. > > > > In order to use shock directly we need to figure out the optimal > > way to display the > > shock data as a filesystem abstraction. I think this should be a > > priority for the > > shock developers. > > > > GCMU isn't required to use globus connect. You can fire up globus > > connect and transfer > > data to the kbase DTNs today. However, you have to use your > > developer credentials to > > auth. > > > > We are trying to get GCMU installed on some KBase endpoints to > > support richer user-user > > sharing of data. The globus connect to standard gridftp interface > > just puts the data > > down on the filesystem with no good way to link it to accounts > > outside of unix perms > > on the DTN. > > > > ----- Original Message ----- > >> From: "Robert Olson" <olson(a)mcs.anl.gov> > >> To: kbase-core(a)lists.kbase.us > >> Sent: Wednesday, April 24, 2013 2:55:13 PM > >> Subject: [Kbase-core] Globus online > >> > >> > >> Do we have any KBase Globus Online endpoints defined? It seems it > >> would make sense to have ADM storage visible via both Globus > >> Online > >> and Shock. Since Shock doesn't supply any real upload support (the > >> client application doesn't seem to do much from my read of it) it > >> would make sense to utilize GO to provide high performance upload. > >> > >> > >> With Globus Connect (a client-side application that appears to > >> integrate nicely with the Mac at least), we could enable high > >> performance up/downloads between user systems and KBase. > >> > >> > >> This seems related - globus connect multiuser > >> https://www.globusonline.org/gcmu/ and would be required on the KB > >> transfer node (it requires firewall be opened). > >> > >> > >> The context of this is enabling real-life large file upload for KB > >> users without forcing them to handle the huge details required > >> since > >> the Shock folks don't care to make things easy for the end user. > >> _______________________________________________ > >> Kbase-core mailing list > >> Kbase-core(a)lists.kbase.us > >> https://lists.kbase.us/mailman/listinfo/kbase-core > >> > >

1 0