didn't you say you had 9TB? That data is just 160GB so should fit fine, right? I'm thinking of testing shock as an example of one of your object stores, since as I understand it they're putting it up as being an efficient data store for that sort of data. What I want to do with the data is retrieve arbitrary pieces of it efficiently. Can't ask for much more than that. ----- Original Message -----
So say I wanted to store 40 million pieces data, 4K each. Is shock appropriate for that?
Just storing the data, sure. I haven't tried this scenario, but I haven't seen anything in shock that would prevent you from doing this. But you could also store that data on thumbdrives, tapes, or cd-roms :) It really depends on what you want to do with the data, and what kind of availability and performance you need.
Shock stores data on a filesystem at the end of the day, and the last time I checked it would only store it on one filesystem - on one server. The last time I spoke with Jared Wilkening about this last year we were talking about sync methods for replicas and referrals to other servers - so I know this was in the works.
If you want to test shock at that scale, it's not difficult to start an instance following the readme. I don't think we have anything with that much storage free for trying this out.
----- Original Message -----
From: "Robert Olson" <[email protected]> To: "Dan Olson" <[email protected]> Cc: [email protected] Sent: Friday, April 26, 2013 5:17:10 PM Subject: Re: [Kbase-core] shock
Thanks.
So say I wanted to store 40 million pieces data, 4K each. Is shock appropriate for that?
----- Original Message -----
Yes, shock is running at kbase.us/services/shock and kbase.us/services/shock-api Globus creds should get you write access. There is 9.5T available.
Let me know if you need anything else to get things going.
----- Original Message -----
From: "Robert Olson" <[email protected]> To: [email protected] Sent: Friday, April 26, 2013 3:07:37 PM Subject: [Kbase-core] shock
Is there a production shock server running in KB land? Is there doc? I searched for "shock" on kbase.us and got no hits.
I'd like to look into pushing all of the sequence data (contigs + features) into shock and using its range lookups for pulling sequence data chunks back out, the way we currently do with flat files in SEED and the file+offset+length indexes we build.
--bob _______________________________________________ Kbase-core mailing list [email protected] https://lists.kbase.us/mailman/listinfo/kbase-core