So say I wanted to store 40 million pieces data, 4K each. Is shock appropriate for that?
Just storing the data, sure. I haven't tried this scenario, but I haven't seen anything in shock that would prevent you from doing this. But you could also store that data on thumbdrives, tapes, or cd-roms :) It really depends on what you want to do with the data, and what kind of availability and performance you need. Shock stores data on a filesystem at the end of the day, and the last time I checked it would only store it on one filesystem - on one server. The last time I spoke with Jared Wilkening about this last year we were talking about sync methods for replicas and referrals to other servers - so I know this was in the works. If you want to test shock at that scale, it's not difficult to start an instance following the readme. I don't think we have anything with that much storage free for trying this out. ----- Original Message -----
From: "Robert Olson" <[email protected]> To: "Dan Olson" <[email protected]> Cc: [email protected] Sent: Friday, April 26, 2013 5:17:10 PM Subject: Re: [Kbase-core] shock
Thanks.
So say I wanted to store 40 million pieces data, 4K each. Is shock appropriate for that?
----- Original Message -----
Yes, shock is running at kbase.us/services/shock and kbase.us/services/shock-api Globus creds should get you write access. There is 9.5T available.
Let me know if you need anything else to get things going.
----- Original Message -----
From: "Robert Olson" <[email protected]> To: [email protected] Sent: Friday, April 26, 2013 3:07:37 PM Subject: [Kbase-core] shock
Is there a production shock server running in KB land? Is there doc? I searched for "shock" on kbase.us and got no hits.
I'd like to look into pushing all of the sequence data (contigs + features) into shock and using its range lookups for pulling sequence data chunks back out, the way we currently do with flat files in SEED and the file+offset+length indexes we build.
--bob _______________________________________________ Kbase-core mailing list [email protected] https://lists.kbase.us/mailman/listinfo/kbase-core