April 2013 - Microbe - mailman3.cels.anl.gov

Re: [Microbe] Microbes Call
by Christopher Henry 29 Apr '13

29 Apr '13

Sorry for the lack of notification. We won't have a call this week. Continue working on solidifying your workflows. Let's plan on definitely having a call next week though. In general, we may consider moving to an "every other week" schedule, as we have less coordination to do at the moment. On Apr 29, 2013, at 1:01 PM, Gavin Price <gaprice(a)lbl.gov> wrote: > Is there a call today? > > On 4/8/2013 9:58 AM, Paramvir Dehal wrote: >> Call today at 11am P / 1pm C >> >> 866-740-1260 >> >> Code: 4958182 >> >> >> The call should be brief. >> We'll talk about last weeks all hands meeting and what it >> means for us. I'd like to focus on enumerating and >> prioritizing the campaigns for the first 3 month where we >> shore up our platform. I'll start a google doc and send it >> out before the call. >> >> >> >> >> >> Paramvir >> >> >> _______________________________________________ >> Microbe mailing list >> >> Microbe(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/microbe > _______________________________________________ > Microbe mailing list > Microbe(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/microbe

1 0

Re: [Microbe] Microbes Call
by John-Marc Chandonia 29 Apr '13

29 Apr '13

On Mon, Apr 29, 2013 at 11:01:34AM -0700, Gavin Price wrote: > Is there a call today? No chair yet on the call. If it is today, please email for those of us not in Emeryville. JMC -- John-Marc Chandonia (JMChandonia(a)lbl.gov) We're everywhere... Physical Biosciences Division, Berkeley National Lab for your convenience. http://www.strgen.org/~jmc/ -- Psi Corps <*>

1 0

Re: [Microbe] Microbes Call
by Gavin Price 29 Apr '13

29 Apr '13

Is there a call today? On 4/8/2013 9:58 AM, Paramvir Dehal wrote: > Call today at 11am P / 1pm C > > *866-740-1260Code: 4958182 * > > The call should be brief. We'll talk about last weeks all hands > meeting and what it means for us. I'd like to focus on enumerating and > prioritizing the campaigns for the first 3 month where we shore up our > platform. I'll start a google doc and send it out before the call. > > Paramvir > > > _______________________________________________ > Microbe mailing list > Microbe(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/microbe

1 0

no call today
by Paramvir Dehal 22 Apr '13

22 Apr '13

I'm not going to be able to make the call today. We haven't received updates from Adam and Rick about the new group structure. So for now, we'll continue the platform solidification we've been working on. Paramvir

1 0

Re: [Microbe] [Kbase-devel] How do we enable KBase to support publication of genome and phenotype data
by Christopher Henry 18 Apr '13

18 Apr '13

"If we are going to link in to workspaces then they need to be made immutable .. Which means there needs to be a transition from a user workspace to an archived reference workspace/data store. Once that transition happens its a reference bel object and "owned" by kbase not the user. It becomes a public dataset etc. Genomes if they are public should be in the CDM and treated the same as other public genomes. (Btw I'm discussing with jgi the issue of publishing genomes without going through NCBI. They wish to do this as well since they can no longer afford the overhead of iterating with NCBI on genome submission) .. Jim Bristow and I would like to do this in a coordinated way starting ona particular date." I agree with all of this. I like the idea of a special type of "Workspace" or "Project" that is a "Published Workspace" and is immutable. I would argue that the public genomes should be in the CDM AND the workspace/project space. I say this, because annotations and gene calls in the CDM may periodically change (I think that's the plan). So we still need the workspace version to serve as a permanent snapshot of the genomes as they stood when the analysis was first performed (I think). "You need something like the MIGS, MIM and MIBA (I made up the last two) standards. There are two aspects of this. The information we require for these datasets and the metadata mechanism. I strongly believe we need a single metadata mechanism for all of kbase that works for data in all of the stores (rather than one for each store or even worse a different mechanism for each type). Folker has proposed a mechanism that with some thought could be adopted as the scheme for all the stores. Each core data type needs to have a well defined set of items they want for each data set. I believe the mechanism proposed can support that." Yeah, I think Folker is way ahead of all of us on thinking about the metadata issue. I agree we should make every effort to adopt the system he's developed for this system "kbase-wide". I really like Tom's suggestion of a sprint team designated to the issue of a metadata and ID policy for KBase. I think his suggestion of people for the sprint team is also spot on: Chris, Folker, Paramvir, Annette, Shiran If someone else has an urgent burning desire to be on this team, they should volunteer. Then hopefully we can arrange some conference calls and prep a document with a firm KBase policy on the IDs and metadata plans, and identify the services that might be needed to support this? Does this seem amenable to everyone? Chris On Apr 18, 2013, at 8:12 AM, Rick stevens <stevens(a)anl.gov> wrote: > > > Sent from my iPad > > On Apr 17, 2013, at 12:04 AM, Christopher Henry <chrisshenry(a)gmail.com> wrote: > >> Hi all, >> >> I started this thread with some KBase leads, and Adam raised some good points, but I'd like to bring this discussion project wide to the devel list. >> >> I want to publish a manuscript where we sequenced 124 environmental isolates, resulting in a total of 166 new genomes. >> >> We sequenced the genomes, assembled them in KBase, annotated them, modeled them. We have 124 biolog arrays for all sample. I want KBase to be my supplementary material. I want to send people to KBase to get all this data and analysis, rather than pointlessly dumping them into NCBI where they won't be connected to any of the analysis we did (plus I don't really want to jump through the hoops that NCBI would probably make me jump through). >> >> So how do we support this? >> >> To get things started, here's a few things I think we need: >> >> 1.) Persistent KBase-wide IDs for the genomes and phenotype sets that will never change >> I'm suggesting "kb|g.###" for genomes, and "kb|g.###.phenos.###" for phenotype sets. Comments? >> >> 2.) A place people can go to look at the data, download data, access data >> Initially I'm suggesting I put the genomes and all derived data in one or more workspaces, and put links to the workspace in the workspace browser in the publication, then we make sure the workspace offers good mechanisms for viewing and searching all these phenotypes. Comments? > > If we are going to link in to workspaces then they need to be made immutable .. Which means there needs to be a transition from a user workspace to an archived reference workspace/data store. Once that transition happens its a reference bel object and "owned" by kbase not the user. It becomes a public dataset etc. > > Genomes if they are public should be in the CDM and treated the same as other public genomes. > > (Btw I'm discussing with jgi the issue of publishing genomes without going through NCBI. They wish to do this as well since they can no longer afford the overhead of iterating with NCBI on genome submission) .. > > Jim Bristow and I would like to do this in a coordinated way starting ona particular date. > >> >> 3.) We need to decide on what metadata we want to support for the release of genomes, models, and Biolog array data in KBase >> I don't have a strong proposal here. I'm hoping some of the metadata experts speak up. > > You need something like the MIGS, MIM and MIBA (I made up the last two) standards. There are two aspects of this. The information we require for these datasets and the metadata mechanism. > > I strongly believe we need a single metadata mechanism for all of kbase that works for data in all of the stores (rather than one for each store or even worse a different mechanism for each type). > > Folker has proposed a mechanism that with some thought could be adopted as the scheme for all the stores. Each core data type needs to have a well defined set of items they want for each data set. I believe the mechanism proposed can support that. > >> >> 4.) What are the things about KBase that we find to be the most embarrassing… that we would absolutely want to correct before any manuscript like the one I'm proposing gets published and starts driving people enmasse to KBase. >> I'm thinking documentation at the moment… and I'm the number one offender here. We have shining spots in the documentation, but we've got some real sore spots too. > > Documentation for sure... Needs to be updated and made consistent with the current releases of tools etc. > > >> >> This is something I think we want to address in short order. If we shift to campaigns, I think we should have a campaign specifically to address these issues. Because if a user of KBase cannot easily publish the work they do in KBase right now, then we're going to have trouble attracting users. The best possible advertisement is a good manuscript that people read and get excited about all the powerful things they can do in KBase. Then they click on our supplementary material and come to KBase and see the tools in action. >> >> Chris >> _______________________________________________ >> Kbase-devel mailing list >> Kbase-devel(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-devel

1 0

Re: [Microbe] [Kbase-devel] How do we enable KBase to support publication of genome and phenotype data
by Rick stevens 18 Apr '13

18 Apr '13

Sent from my iPad On Apr 17, 2013, at 12:04 AM, Christopher Henry <chrisshenry(a)gmail.com> wrote: > Hi all, > > I started this thread with some KBase leads, and Adam raised some good points, but I'd like to bring this discussion project wide to the devel list. > > I want to publish a manuscript where we sequenced 124 environmental isolates, resulting in a total of 166 new genomes. > > We sequenced the genomes, assembled them in KBase, annotated them, modeled them. We have 124 biolog arrays for all sample. I want KBase to be my supplementary material. I want to send people to KBase to get all this data and analysis, rather than pointlessly dumping them into NCBI where they won't be connected to any of the analysis we did (plus I don't really want to jump through the hoops that NCBI would probably make me jump through). > > So how do we support this? > > To get things started, here's a few things I think we need: > > 1.) Persistent KBase-wide IDs for the genomes and phenotype sets that will never change > I'm suggesting "kb|g.###" for genomes, and "kb|g.###.phenos.###" for phenotype sets. Comments? > > 2.) A place people can go to look at the data, download data, access data > Initially I'm suggesting I put the genomes and all derived data in one or more workspaces, and put links to the workspace in the workspace browser in the publication, then we make sure the workspace offers good mechanisms for viewing and searching all these phenotypes. Comments? If we are going to link in to workspaces then they need to be made immutable .. Which means there needs to be a transition from a user workspace to an archived reference workspace/data store. Once that transition happens its a reference bel object and "owned" by kbase not the user. It becomes a public dataset etc. Genomes if they are public should be in the CDM and treated the same as other public genomes. (Btw I'm discussing with jgi the issue of publishing genomes without going through NCBI. They wish to do this as well since they can no longer afford the overhead of iterating with NCBI on genome submission) .. Jim Bristow and I would like to do this in a coordinated way starting ona particular date. > > 3.) We need to decide on what metadata we want to support for the release of genomes, models, and Biolog array data in KBase > I don't have a strong proposal here. I'm hoping some of the metadata experts speak up. You need something like the MIGS, MIM and MIBA (I made up the last two) standards. There are two aspects of this. The information we require for these datasets and the metadata mechanism. I strongly believe we need a single metadata mechanism for all of kbase that works for data in all of the stores (rather than one for each store or even worse a different mechanism for each type). Folker has proposed a mechanism that with some thought could be adopted as the scheme for all the stores. Each core data type needs to have a well defined set of items they want for each data set. I believe the mechanism proposed can support that. > > 4.) What are the things about KBase that we find to be the most embarrassing… that we would absolutely want to correct before any manuscript like the one I'm proposing gets published and starts driving people enmasse to KBase. > I'm thinking documentation at the moment… and I'm the number one offender here. We have shining spots in the documentation, but we've got some real sore spots too. Documentation for sure... Needs to be updated and made consistent with the current releases of tools etc. > > This is something I think we want to address in short order. If we shift to campaigns, I think we should have a campaign specifically to address these issues. Because if a user of KBase cannot easily publish the work they do in KBase right now, then we're going to have trouble attracting users. The best possible advertisement is a good manuscript that people read and get excited about all the powerful things they can do in KBase. Then they click on our supplementary material and come to KBase and see the tools in action. > > Chris > _______________________________________________ > Kbase-devel mailing list > Kbase-devel(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-devel

1 0

Re: [Microbe] [Kbase-devel] How do we enable KBase to support publication of genome and phenotype data
by Rick stevens 18 Apr '13

18 Apr '13

Phenos --> p Sent from my iPad On Apr 17, 2013, at 2:50 PM, Christopher Henry <chrisshenry(a)gmail.com> wrote: > Well, my paper won't be submitted for a few months. But these issues must be dealt with sooner or later… it kind of comes with the territory of having users, which is the goal. I would say I don't mean to put undue urgency on the issue… but actually, I think I do mean to do it. Nothing focuses our current "solidification" effort better than actually trying to do something for real. > > "One big question that we need to answer at the policy level is, are we going to be a data repository?" > > Well…. let's contemplate this a moment. What I'm asking to do is publish a paper about an analysis I did in KBase. And you seem to be asking if we should even hold onto the data from the analysis. > If this is where we're at right now… then we are not only dead in the water, but we're at the bottom of the ocean. How is this not completely a "no-brainer"? What am I missing here? What's with all the talk about provenance, etc if this isn't the goal? What is so magical about NCBI? I guarantee you our annotations are more consistent (and better) than theirs… and we deliver genome sequences just as well as they do. > > "There are probably good reasons NCBI makes you jump through so many hoops." > > Well, I think the reason is mostly because "they can". Perhaps I'm being overly cynical... > > "We need to settle on the data models and ids." > > Yeah sure. I can go along with this. Let's settle the ID issue right now. > > Genome IDs: "kb|g.####". I'm afraid the ship sailed long ago on that one, and I don't like the "|" any more than you do. > Phenotypes: "kb|g.###.phenos.###". > Speak now, or forever hold your peace. > > Data models. Well, I just don't see us setting those in stone right this minute, or anytime in the next 1-2 years. How about we decide what is most critical to have in the data model and make sure it's there? > > On Apr 17, 2013, at 1:07 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: > >> One big question that we need to answer at the policy level is, are we going to be a data repository? > > _______________________________________________ > Kbase-devel mailing list > Kbase-devel(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-devel

1 0

Re: [Microbe] [Kbase-devel] How do we enable KBase to support publication of genome and phenotype data
by Christopher Henry 17 Apr '13

17 Apr '13

Well, my paper won't be submitted for a few months. But these issues must be dealt with sooner or later… it kind of comes with the territory of having users, which is the goal. I would say I don't mean to put undue urgency on the issue… but actually, I think I do mean to do it. Nothing focuses our current "solidification" effort better than actually trying to do something for real. "One big question that we need to answer at the policy level is, are we going to be a data repository?" Well…. let's contemplate this a moment. What I'm asking to do is publish a paper about an analysis I did in KBase. And you seem to be asking if we should even hold onto the data from the analysis. If this is where we're at right now… then we are not only dead in the water, but we're at the bottom of the ocean. How is this not completely a "no-brainer"? What am I missing here? What's with all the talk about provenance, etc if this isn't the goal? What is so magical about NCBI? I guarantee you our annotations are more consistent (and better) than theirs… and we deliver genome sequences just as well as they do. "There are probably good reasons NCBI makes you jump through so many hoops." Well, I think the reason is mostly because "they can". Perhaps I'm being overly cynical... "We need to settle on the data models and ids." Yeah sure. I can go along with this. Let's settle the ID issue right now. Genome IDs: "kb|g.####". I'm afraid the ship sailed long ago on that one, and I don't like the "|" any more than you do. Phenotypes: "kb|g.###.phenos.###". Speak now, or forever hold your peace. Data models. Well, I just don't see us setting those in stone right this minute, or anytime in the next 1-2 years. How about we decide what is most critical to have in the data model and make sure it's there? On Apr 17, 2013, at 1:07 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: > One big question that we need to answer at the policy level is, are we going to be a data repository?

1 0

Re: [Microbe] [Kbase-devel] How do we enable KBase to support publication of genome and phenotype data
by Paramvir Dehal 17 Apr '13

17 Apr '13

Chris, what's the timeframe for your manuscript? One big question that we need to answer at the policy level is, are we going to be a data repository? (I think we have to be for some data types) I'm not totally sure what it will mean to be a repository in terms of time and user support. There are probably good reasons NCBI makes you jump through so many hoops. We need to settle on the data models and ids. And distribute those in a way that avoids lock-in so we have the flexibility to change our tech & policies. I think your data link: http://permanent.kbase.us/linking.cgi?id=XYZ<http://permanent.url/linkin.cgi?id=XYZ> would probably do that. On Wed, Apr 17, 2013 at 9:50 AM, Christopher Henry <chrisshenry(a)gmail.com>wrote: > Folker, > > As far as I can tell, the current workspace implementation in KBase > basically gives you all of these things: > - The project notion is important as it generates one identifier for > multiple data objects > - Authorization needs to be inherited from the project (otherwise there > will be users missing rights to some data sets for pre-pub sharing) > - projects probably need to support multiple data types (the current > MG-RAST implementation is a bit metagenome centric) > - everything we share or publish requires permanent IDs and indeed > permanent "objects", we cannot use any temporary structures > - data provenance and metadata is very important (provenance= how was it > computed, metadata= where is it from, where was it sampled) > - we have used a specific permanent link (data set: > http://permanent.url/linkin.cgi?id=XYZ or project: > http://permanent.url/linkin.cgi?project=123) > - we allow users to share with arbitrary email addresses, requiring > recipients to create accounts ... > - versioning, will we maintain the original published analysis for data or > will we update the data once the pipeline(s) change? > > Workspace = Project > > For permanence, we would probably want to have a special class of > workspace called "Published space"… but really we're just talking about > setting permissions so the published workspace cannot be deleted… an easy > thing to do. > > We've got versioning and provenance, and the system handles different > object types already. > We've got ownership linked to KBase user accounts, which have emails > attached to them. > > The one thing we are perhaps missing is a permanent URL. > > I understand you have a similar notion in MG-RAST, but I'm looking for a > KBase solution. I understand you want to integrate your stuff wholesale > into KBase and perhaps replace the workspace with it. I'd like to have > assurances that this will work… will it perform the same way as the > workspace? Is this what KBase leadership wants, because the MGRAST system > strikes me as far far more complex and heavy weight in terms of > infrastructure requirements than the workspace… and do we want to tie-up > our workspace maintenance with permanent maintenance of a behemoth that > contains over >20TB. And will it happen on a short time frame, and will the > workspace API be preserved, because everything I have is linked to that. > I'll just say that one thing I really like about the current workspace > implementation is that it's trivial to set up a new workspace server. It's > just mongodb and the perl API wrapper. I do understand that a key > limitation of the current workspace is that you wouldn't want to put really > big data objects into it. > > So if this question really does boil down to "What are we going to do with > the user store in KBase?", then I'll ask it. What are we going to do with > the user store in KBase? This seems to be a decision that's rather > essential to make in KBase right now. > > Chris > > On Apr 17, 2013, at 7:30 AM, Folker Meyer <folker(a)anl.gov> wrote: > > > Hi Chris, > > > > we have had a solution for this in MG-RAST for a couple of years now, > actually there are two parts to the solutions: > > > > - publishing/sharing of data (with required metadata) > > AND > > - projects combining data sets together > > > > The publishing mechanism we currently have probably needs to be extended > to cover reviewer access (so that owners can generate an anonymous reviewer > token for their manuscripts), we have not had sufficient engineering time. > Most data in MG-RAST (>95%, over 60,000 data sets) is actually organized in > projects now. > > > > The feature (as well as the pre-publication sharing) is quite popular > with users, we have actually had requests from sociologists and network > analysis people for the interaction graphs, with many thousand users > sharing many thousand data sets. The emerging properties are quite > interesting ==> science is very clique-ish. > > > > here are some technical details that we have learned: > > - The project notion is important as it generates one identifier for > multiple data objects > > - Authorization needs to be inherited from the project (otherwise there > will be users missing rights to some data sets for pre-pub sharing) > > - projects probably need to support multiple data types (the current > MG-RAST implementation is a bit metagenome centric) > > - everything we share or publish requires permanent IDs and indeed > permanent "objects", we cannot use any temporary structures > > - data provenance and metadata is very important (provenance= how was it > computed, metadata= where is it from, where was it sampled) > > - we have used a specific permanent link (data set: > http://permanent.url/linkin.cgi?id=XYZ or project: > http://permanent.url/linkin.cgi?project=123) > > - we allow users to share with arbitrary email addresses, requiring > recipients to create accounts ... > > - versioning, will we maintain the original published analysis for data > or will we update the data once the pipeline(s) change? > > SHOCK can allow us to maintain the original analysis results, however > what happens if the API / UI changes dramatically? > > One option might be an archive of the web pages in addition to the > data. The pipeline(s) we can archive via archiving the VMs (to some extent). > > > > We currently have an encoding for metadata for genomes and metagenomes > and some CV standards. The CVs (controlled vocabularies) are fortunately > maintained by external parties. For provenance, this will be somewhat > harder: how was it computed? how can I recompute the same information.) > > > > Best, > > Folker > > > > > > > > > > On Apr 17, 2013, at 7:04 AM, Christopher Henry <chrisshenry(a)gmail.com> > wrote: > > > >> Hi all, > >> > >> I started this thread with some KBase leads, and Adam raised some good > points, but I'd like to bring this discussion project wide to the devel > list. > >> > >> I want to publish a manuscript where we sequenced 124 environmental > isolates, resulting in a total of 166 new genomes. > >> > >> We sequenced the genomes, assembled them in KBase, annotated them, > modeled them. We have 124 biolog arrays for all sample. I want KBase to be > my supplementary material. I want to send people to KBase to get all this > data and analysis, rather than pointlessly dumping them into NCBI where > they won't be connected to any of the analysis we did (plus I don't really > want to jump through the hoops that NCBI would probably make me jump > through). > >> > >> So how do we support this? > >> > >> To get things started, here's a few things I think we need: > >> > >> 1.) Persistent KBase-wide IDs for the genomes and phenotype sets that > will never change > >> I'm suggesting "kb|g.###" for genomes, and "kb|g.###.phenos.###" > for phenotype sets. Comments? > >> > >> 2.) A place people can go to look at the data, download data, access > data > >> Initially I'm suggesting I put the genomes and all derived data in > one or more workspaces, and put links to the workspace in the workspace > browser in the publication, then we make sure the workspace offers good > mechanisms for viewing and searching all these phenotypes. Comments? > >> > >> 3.) We need to decide on what metadata we want to support for the > release of genomes, models, and Biolog array data in KBase > >> I don't have a strong proposal here. I'm hoping some of the > metadata experts speak up. > >> > >> 4.) What are the things about KBase that we find to be the most > embarrassing… that we would absolutely want to correct before any > manuscript like the one I'm proposing gets published and starts driving > people enmasse to KBase. > >> I'm thinking documentation at the moment… and I'm the number one > offender here. We have shining spots in the documentation, but we've got > some real sore spots too. > >> > >> This is something I think we want to address in short order. If we > shift to campaigns, I think we should have a campaign specifically to > address these issues. Because if a user of KBase cannot easily publish the > work they do in KBase right now, then we're going to have trouble > attracting users. The best possible advertisement is a good manuscript that > people read and get excited about all the powerful things they can do in > KBase. Then they click on our supplementary material and come to KBase and > see the tools in action. > >> > >> Chris > >> _______________________________________________ > >> Kbase-devel mailing list > >> Kbase-devel(a)lists.kbase.us > >> https://lists.kbase.us/mailman/listinfo/kbase-devel > > > > _______________________________________________ > Microbe mailing list > Microbe(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/microbe >

1 0

Re: [Microbe] [Kbase-devel] How do we enable KBase to support publication of genome and phenotype data
by Christopher Henry 17 Apr '13

17 Apr '13

Folker, As far as I can tell, the current workspace implementation in KBase basically gives you all of these things: - The project notion is important as it generates one identifier for multiple data objects - Authorization needs to be inherited from the project (otherwise there will be users missing rights to some data sets for pre-pub sharing) - projects probably need to support multiple data types (the current MG-RAST implementation is a bit metagenome centric) - everything we share or publish requires permanent IDs and indeed permanent "objects", we cannot use any temporary structures - data provenance and metadata is very important (provenance= how was it computed, metadata= where is it from, where was it sampled) - we have used a specific permanent link (data set: http://permanent.url/linkin.cgi?id=XYZ or project: http://permanent.url/linkin.cgi?project=123) - we allow users to share with arbitrary email addresses, requiring recipients to create accounts ... - versioning, will we maintain the original published analysis for data or will we update the data once the pipeline(s) change? Workspace = Project For permanence, we would probably want to have a special class of workspace called "Published space"… but really we're just talking about setting permissions so the published workspace cannot be deleted… an easy thing to do. We've got versioning and provenance, and the system handles different object types already. We've got ownership linked to KBase user accounts, which have emails attached to them. The one thing we are perhaps missing is a permanent URL. I understand you have a similar notion in MG-RAST, but I'm looking for a KBase solution. I understand you want to integrate your stuff wholesale into KBase and perhaps replace the workspace with it. I'd like to have assurances that this will work… will it perform the same way as the workspace? Is this what KBase leadership wants, because the MGRAST system strikes me as far far more complex and heavy weight in terms of infrastructure requirements than the workspace… and do we want to tie-up our workspace maintenance with permanent maintenance of a behemoth that contains over >20TB. And will it happen on a short time frame, and will the workspace API be preserved, because everything I have is linked to that. I'll just say that one thing I really like about the current workspace implementation is that it's trivial to set up a new workspace server. It's just mongodb and the perl API wrapper. I do understand that a key limitation of the current workspace is that you wouldn't want to put really big data objects into it. So if this question really does boil down to "What are we going to do with the user store in KBase?", then I'll ask it. What are we going to do with the user store in KBase? This seems to be a decision that's rather essential to make in KBase right now. Chris On Apr 17, 2013, at 7:30 AM, Folker Meyer <folker(a)anl.gov> wrote: > Hi Chris, > > we have had a solution for this in MG-RAST for a couple of years now, actually there are two parts to the solutions: > > - publishing/sharing of data (with required metadata) > AND > - projects combining data sets together > > The publishing mechanism we currently have probably needs to be extended to cover reviewer access (so that owners can generate an anonymous reviewer token for their manuscripts), we have not had sufficient engineering time. Most data in MG-RAST (>95%, over 60,000 data sets) is actually organized in projects now. > > The feature (as well as the pre-publication sharing) is quite popular with users, we have actually had requests from sociologists and network analysis people for the interaction graphs, with many thousand users sharing many thousand data sets. The emerging properties are quite interesting ==> science is very clique-ish. > > here are some technical details that we have learned: > - The project notion is important as it generates one identifier for multiple data objects > - Authorization needs to be inherited from the project (otherwise there will be users missing rights to some data sets for pre-pub sharing) > - projects probably need to support multiple data types (the current MG-RAST implementation is a bit metagenome centric) > - everything we share or publish requires permanent IDs and indeed permanent "objects", we cannot use any temporary structures > - data provenance and metadata is very important (provenance= how was it computed, metadata= where is it from, where was it sampled) > - we have used a specific permanent link (data set: http://permanent.url/linkin.cgi?id=XYZ or project: http://permanent.url/linkin.cgi?project=123) > - we allow users to share with arbitrary email addresses, requiring recipients to create accounts ... > - versioning, will we maintain the original published analysis for data or will we update the data once the pipeline(s) change? > SHOCK can allow us to maintain the original analysis results, however what happens if the API / UI changes dramatically? > One option might be an archive of the web pages in addition to the data. The pipeline(s) we can archive via archiving the VMs (to some extent). > > We currently have an encoding for metadata for genomes and metagenomes and some CV standards. The CVs (controlled vocabularies) are fortunately maintained by external parties. For provenance, this will be somewhat harder: how was it computed? how can I recompute the same information.) > > Best, > Folker > > > > > On Apr 17, 2013, at 7:04 AM, Christopher Henry <chrisshenry(a)gmail.com> wrote: > >> Hi all, >> >> I started this thread with some KBase leads, and Adam raised some good points, but I'd like to bring this discussion project wide to the devel list. >> >> I want to publish a manuscript where we sequenced 124 environmental isolates, resulting in a total of 166 new genomes. >> >> We sequenced the genomes, assembled them in KBase, annotated them, modeled them. We have 124 biolog arrays for all sample. I want KBase to be my supplementary material. I want to send people to KBase to get all this data and analysis, rather than pointlessly dumping them into NCBI where they won't be connected to any of the analysis we did (plus I don't really want to jump through the hoops that NCBI would probably make me jump through). >> >> So how do we support this? >> >> To get things started, here's a few things I think we need: >> >> 1.) Persistent KBase-wide IDs for the genomes and phenotype sets that will never change >> I'm suggesting "kb|g.###" for genomes, and "kb|g.###.phenos.###" for phenotype sets. Comments? >> >> 2.) A place people can go to look at the data, download data, access data >> Initially I'm suggesting I put the genomes and all derived data in one or more workspaces, and put links to the workspace in the workspace browser in the publication, then we make sure the workspace offers good mechanisms for viewing and searching all these phenotypes. Comments? >> >> 3.) We need to decide on what metadata we want to support for the release of genomes, models, and Biolog array data in KBase >> I don't have a strong proposal here. I'm hoping some of the metadata experts speak up. >> >> 4.) What are the things about KBase that we find to be the most embarrassing… that we would absolutely want to correct before any manuscript like the one I'm proposing gets published and starts driving people enmasse to KBase. >> I'm thinking documentation at the moment… and I'm the number one offender here. We have shining spots in the documentation, but we've got some real sore spots too. >> >> This is something I think we want to address in short order. If we shift to campaigns, I think we should have a campaign specifically to address these issues. Because if a user of KBase cannot easily publish the work they do in KBase right now, then we're going to have trouble attracting users. The best possible advertisement is a good manuscript that people read and get excited about all the powerful things they can do in KBase. Then they click on our supplementary material and come to KBase and see the tools in action. >> >> Chris >> _______________________________________________ >> Kbase-devel mailing list >> Kbase-devel(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-devel >

1 0