Chris, what's the timeframe for your manuscript? One big question that we need to answer at the policy level is, are we going to be a data repository? (I think we have to be for some data types) I'm not totally sure what it will mean to be a repository in terms of time and user support. There are probably good reasons NCBI makes you jump through so many hoops. We need to settle on the data models and ids. And distribute those in a way that avoids lock-in so we have the flexibility to change our tech & policies. I think your data link: http://permanent.kbase.us/linking.cgi?id=XYZ<http://permanent.url/linkin.cgi?id=XYZ> would probably do that. On Wed, Apr 17, 2013 at 9:50 AM, Christopher Henry <[email protected]>wrote:
Folker,
As far as I can tell, the current workspace implementation in KBase basically gives you all of these things: - The project notion is important as it generates one identifier for multiple data objects - Authorization needs to be inherited from the project (otherwise there will be users missing rights to some data sets for pre-pub sharing) - projects probably need to support multiple data types (the current MG-RAST implementation is a bit metagenome centric) - everything we share or publish requires permanent IDs and indeed permanent "objects", we cannot use any temporary structures - data provenance and metadata is very important (provenance= how was it computed, metadata= where is it from, where was it sampled) - we have used a specific permanent link (data set: http://permanent.url/linkin.cgi?id=XYZ or project: http://permanent.url/linkin.cgi?project=123) - we allow users to share with arbitrary email addresses, requiring recipients to create accounts ... - versioning, will we maintain the original published analysis for data or will we update the data once the pipeline(s) change?
Workspace = Project
For permanence, we would probably want to have a special class of workspace called "Published space"… but really we're just talking about setting permissions so the published workspace cannot be deleted… an easy thing to do.
We've got versioning and provenance, and the system handles different object types already. We've got ownership linked to KBase user accounts, which have emails attached to them.
The one thing we are perhaps missing is a permanent URL.
I understand you have a similar notion in MG-RAST, but I'm looking for a KBase solution. I understand you want to integrate your stuff wholesale into KBase and perhaps replace the workspace with it. I'd like to have assurances that this will work… will it perform the same way as the workspace? Is this what KBase leadership wants, because the MGRAST system strikes me as far far more complex and heavy weight in terms of infrastructure requirements than the workspace… and do we want to tie-up our workspace maintenance with permanent maintenance of a behemoth that contains over >20TB. And will it happen on a short time frame, and will the workspace API be preserved, because everything I have is linked to that. I'll just say that one thing I really like about the current workspace implementation is that it's trivial to set up a new workspace server. It's just mongodb and the perl API wrapper. I do understand that a key limitation of the current workspace is that you wouldn't want to put really big data objects into it.
So if this question really does boil down to "What are we going to do with the user store in KBase?", then I'll ask it. What are we going to do with the user store in KBase? This seems to be a decision that's rather essential to make in KBase right now.
Chris
On Apr 17, 2013, at 7:30 AM, Folker Meyer <[email protected]> wrote:
Hi Chris,
we have had a solution for this in MG-RAST for a couple of years now, actually there are two parts to the solutions:
- publishing/sharing of data (with required metadata) AND - projects combining data sets together
The publishing mechanism we currently have probably needs to be extended to cover reviewer access (so that owners can generate an anonymous reviewer token for their manuscripts), we have not had sufficient engineering time. Most data in MG-RAST (>95%, over 60,000 data sets) is actually organized in projects now.
The feature (as well as the pre-publication sharing) is quite popular with users, we have actually had requests from sociologists and network analysis people for the interaction graphs, with many thousand users sharing many thousand data sets. The emerging properties are quite interesting ==> science is very clique-ish.
here are some technical details that we have learned: - The project notion is important as it generates one identifier for multiple data objects - Authorization needs to be inherited from the project (otherwise there will be users missing rights to some data sets for pre-pub sharing) - projects probably need to support multiple data types (the current MG-RAST implementation is a bit metagenome centric) - everything we share or publish requires permanent IDs and indeed permanent "objects", we cannot use any temporary structures - data provenance and metadata is very important (provenance= how was it computed, metadata= where is it from, where was it sampled) - we have used a specific permanent link (data set: http://permanent.url/linkin.cgi?id=XYZ or project: http://permanent.url/linkin.cgi?project=123) - we allow users to share with arbitrary email addresses, requiring recipients to create accounts ... - versioning, will we maintain the original published analysis for data or will we update the data once the pipeline(s) change? SHOCK can allow us to maintain the original analysis results, however what happens if the API / UI changes dramatically? One option might be an archive of the web pages in addition to the data. The pipeline(s) we can archive via archiving the VMs (to some extent).
We currently have an encoding for metadata for genomes and metagenomes and some CV standards. The CVs (controlled vocabularies) are fortunately maintained by external parties. For provenance, this will be somewhat harder: how was it computed? how can I recompute the same information.)
Best, Folker
On Apr 17, 2013, at 7:04 AM, Christopher Henry <[email protected]> wrote:
Hi all,
I started this thread with some KBase leads, and Adam raised some good points, but I'd like to bring this discussion project wide to the devel list.
I want to publish a manuscript where we sequenced 124 environmental isolates, resulting in a total of 166 new genomes.
We sequenced the genomes, assembled them in KBase, annotated them, modeled them. We have 124 biolog arrays for all sample. I want KBase to be my supplementary material. I want to send people to KBase to get all this data and analysis, rather than pointlessly dumping them into NCBI where they won't be connected to any of the analysis we did (plus I don't really want to jump through the hoops that NCBI would probably make me jump through).
So how do we support this?
To get things started, here's a few things I think we need:
1.) Persistent KBase-wide IDs for the genomes and phenotype sets that will never change I'm suggesting "kb|g.###" for genomes, and "kb|g.###.phenos.###" for phenotype sets. Comments?
2.) A place people can go to look at the data, download data, access data Initially I'm suggesting I put the genomes and all derived data in one or more workspaces, and put links to the workspace in the workspace browser in the publication, then we make sure the workspace offers good mechanisms for viewing and searching all these phenotypes. Comments?
3.) We need to decide on what metadata we want to support for the release of genomes, models, and Biolog array data in KBase I don't have a strong proposal here. I'm hoping some of the metadata experts speak up.
4.) What are the things about KBase that we find to be the most embarrassing… that we would absolutely want to correct before any manuscript like the one I'm proposing gets published and starts driving people enmasse to KBase. I'm thinking documentation at the moment… and I'm the number one offender here. We have shining spots in the documentation, but we've got some real sore spots too.
This is something I think we want to address in short order. If we shift to campaigns, I think we should have a campaign specifically to address these issues. Because if a user of KBase cannot easily publish the work they do in KBase right now, then we're going to have trouble attracting users. The best possible advertisement is a good manuscript that people read and get excited about all the powerful things they can do in KBase. Then they click on our supplementary material and come to KBase and see the tools in action.
Chris _______________________________________________ Kbase-devel mailing list [email protected] https://lists.kbase.us/mailman/listinfo/kbase-devel
_______________________________________________ Microbe mailing list [email protected] https://lists.kbase.us/mailman/listinfo/microbe