- Kbase-microcomm - mailman3.cels.anl.gov

Cancel conf call today
by Andreas Wilke 23 May '13

23 May '13

Hi, I am sorry but I have to cancel the conference call today. I have to interview someone and another conflicting event. Andreas Andreas Wilke Argonne National Laboratory Mathematics and Computer Science Division Bldg. 240 , 4F8 9700 South Cass Avenue Argonne, IL 60439 (630) 252-3190 (phone) (630) 252-5986.(fax)

1 0

Re: [Kbase-MicroComm] repurpose call?
by Folker Meyer 17 May '13

17 May '13

On May 16, 2013, at 8:35 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > As Rick points out, the problem is one then needs to check whether the transitive similarity is still within threshold (i.e. KB_16S - m5_16S >= 97% and m5_16S - MG_16S >= 97% does not necessarily mean KB_16S - MG_16S >= 97%) and also you could miss stuff in the other direction where the KB_16S is closer to the MG_16S. I think you just draw a 2x radius circle around the KB_16S and then realign the KB_16S against the MG_16S and check identity. So if you want to be within 97%, then the KB_16S - m5_16S >= 94% (i.e. within 6%), collect the set of MG_16S that satisfy m5_16S - MG_16S >= 97% (that should be a Communities_API service), and then check KB_16S - MG_16S >= 97% (i.e. within 3%). Is my math right? > > Is there a query to return all sequences (or maybe just the abundance counts) within thresholding of identity, e-value, length to a given m5 reference sequence? There currently is not a way (unless you want to download all SIMs). Once we are converted to SHOCK fully and have adapted the API, you can compose this query. Not today. This is however part of the plan. best, Folker > > Thanks, > - Dylan > > > > On Thu, May 16, 2013 at 5:26 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: > Yes using the pre computed metagenome analysis as an index into the reads is the right idea IMO > > Sent from mobile device > > On May 16, 2013, at 19:08, "Rick @ Argonne" <stevens(a)anl.gov> wrote: > >> One approach (im sure there ate variations and better) to the clever case could be resolved by finding a set of more loosely related genomes gathering the superset then a sequence search in that smaller set. >> >> Ideally you would be searching contigs or representatives rather than raw MG reads. So it would be tractable. But I bet we would almost never be in this case. >> >> Sent from my iPhone >> >> On May 16, 2013, at 18:52, "Rick @ Argonne" <stevens(a)anl.gov> wrote: >> >>> I would build tables that map metagenome IDs to the IDs for isolate genomes in the CDM and vice versa. >>> >>> If one derefrenced via the M5nr, the M5rna and silva >>> >>> Then the vast majority would be captured by the normal annotation process of each domain. >>> >>> That leaves the case where something is too new to have been captured in the M5s >>> >>> If the new genome is a close relative then a simple transitive search works. >>> >>> If its really different then one would need to do something more clever. >>> >>> I would focus on the non clever cases first which will cover 99.99% of the cases. >>> >>> >>> >>> Sent from my iPhone >>> >>> On May 16, 2013, at 18:37, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> >>>> what is all this about a use-case? it's the same thing we've been asking for from you since very early in the project. we want to be able to say which metagenomes a given KBase genome can be found in (or a close relative). it's pretty straightforward. if I just state it like that, how would you implement a solution? >>>> >>>> thanks, >>>> - dylan >>>> >>>> >>>> >>>> On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: >>>> ah. I see so this for a use case like RAST? >>>> >>>> as Silva's job is to mine all databases for 16s and expose them to the rest of us, I assume this would be a novel thing. So yes if it is not in the database we will not be able to search against it. >>>> >>>> In any case my comment is even more applicable: I don't understand your use-case well enough to be helpful. >>>> >>>> best, >>>> Folker >>>> >>>> On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> >>>> > "user-defined" in this case means a 16S from a KBase microbial genome (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the inability to synchronize it with KBase regularly as we've discussed). >>>> > >>>> > thanks, >>>> > - dylan >>>> > >>>> > >>>> > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: >>>> > >>>> > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >>>> > > Sorry, which matching? (a) metagenome gene to 16S database or (b) metagenome read to user-defined 16S query sequence? >>>> > >>>> > Now you are loosing me. Why would the user want to define a 16s query sequence? It seems clear to me that I do not understand your use-case. >>>> > >>>> > The API is providing matches that were computed against the m5nr and m5rnaNR databases. >>>> > >>>> > If you want to search against another database that is not part of the API. However it is very likely that the other 16s sequence will be in our database. once we are fully in shock you will be able to ask for the top 300 hits for a read/cluster if that is what you want. this may or may not be interesting for your use-case. >>>> > >>>> > > >>>> > > - dylan >>>> > > >>>> > > >>>> > > >>>> > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> wrote: >>>> > > The matching has already been computed. >>>> > > >>>> > > Sent from my iPad >>>> > > >>>> > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > > >>>> > >> ... and there's no sequence-based search query, right? we'll need to do the alignments ourselves? >>>> > >> >>>> > >> Thanks >>>> > >> - Dylan >>>> > >> >>>> > >> >>>> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> Great! What's the command for getting the public metagenome IDs? >>>> > >> >>>> > >> Thanks, >>>> > >> - Dylan >>>> > >> >>>> > >> >>>> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >>>> > >> sure. this is quite simple in the API. >>>> > >> >>>> > >> >>>> > >> pseudo code: >>>> > >> ----------- >>>> > >> for each public_metagenome >>>> > >> >>>> > >> do >>>> > >> # get abundance profile >>>> > >> curl "http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" | json_xs >>>> > >> >>>> > >> # e.g. curl "http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" | json_xs >>>> > >> >>>> > >> done >>>> > >> ----------- >>>> > >> This gives you the taxon info derived from either proteins or 16s using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are also mapped to NCBI taxonomy. >>>> > >> >>>> > >> Using Silva (aka SSU) as namespace instead of GenBank above will just give you 16s. >>>> > >> >>>> > >> best, >>>> > >> Folker >>>> > >> >>>> > >> >>>> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >>>> > >> >>>> > >> > Hi Folker, >>>> > >> > >>>> > >> > Right now we are just building prototypes, so it would still be useful to have the functionality even if it is not optimized. >>>> > >> > >>>> > >> > Can we see an example of how to do this with the api? I think that is all we are asking for at this point. >>>> > >> > >>>> > >> > Thanks, >>>> > >> > Matt >>>> > >> > >>>> > >> > >>>> > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >>>> > >> > this is something that is supported by the API now. But not fast. >>>> > >> > >>>> > >> > The query is one of the uses cases we are re-designing the search for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. >>>> > >> > >>>> > >> > once that is done, we will implement simple searches in MG-RAST like which taxon is present in which geographic region etc. >>>> > >> > >>>> > >> > currently it is very slow. you are going 90 degrees to the way the system is optimized >>>> > >> > >>>> > >> > Best, >>>> > >> > Folker >>>> > >> > >>>> > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: >>>> > >> > >>>> > >> > > Yeah, those methods would work, I just need to know the function call and how to execute it. >>>> > >> > > >>>> > >> > > >>>> > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> wrote: >>>> > >> > > For isolate genomes we can use metadata and 16s precomputed mapping. >>>> > >> > > >>>> > >> > > For WGS and WGS contigs we need to workout precisely what you mean by assign to a taxon grouping. >>>> > >> > > >>>> > >> > > >>>> > >> > > >>>> > >> > > >>>> > >> > > Sent from my iPad >>>> > >> > > >>>> > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> wrote: >>>> > >> > > >>>> > >> > >> Dylan >>>> > >> > >> >>>> > >> > >> I'm trying Ti understand what you want to build. >>>> > >> > >> >>>> > >> > >> I think you want some way to have some one select a taxon and bring up in a taxon page all genomes and metagenomes and metagenome derived contigs associated with that taxon.. >>>> > >> > >> >>>> > >> > >> Is that correct ? >>>> > >> > >> >>>> > >> > >> Sent from my iPad >>>> > >> > >> >>>> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> > >> >>>> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys flew out there to work with your guys. Is that not happening? >>>> > >> > >>> >>>> > >> > >>> Thanks, >>>> > >> > >>> - Dylan >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: >>>> > >> > >>> We are super busy right now. Can we do this next week over the phone. I still need some input on the actual goals of this. >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> -- >>>> > >> > >>> Sent from mobile device >>>> > >> > >>> >>>> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >>>> > >> > >>> >>>> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you guys (Andreas?) are welcome to come over and be part of the excitement of the KBase build and "stuff". Mike and I will also be here tomorrow morning, but will have to leave around lunchtime to get back to the airport for the trip back home. >>>> > >> > >>>> >>>> > >> > >>>> Paramvir is less concerned at the moment about thresholding and just wants to see some functionality. >>>> > >> > >>>> >>>> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email pointing us in the proper direction would also be appreciated. >>>> > >> > >>>> >>>> > >> > >>>> Thanks, >>>> > >> > >>>> Matt >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> > >>>> Paramvir is a stakeholder here, so we should get his input, including on which identity thresholds he'd like to see. >>>> > >> > >>>> >>>> > >> > >>>> Thanks, >>>> > >> > >>>> - Dylan >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> > >>>> Hi guys, >>>> > >> > >>>> >>>> > >> > >>>> instead of the call today, probably the time would be best spent with Folker and Andreas working with Matt and Mike directly on the metagenomics species search functionality for the Taxon pages. I expect it is an expensive operation, so it is probably something that should be a one-time compute and stored somewhere associated with each taxon rather than on-the-fly when someone calls up the Taxon page. There should probably also be separate lists for 99% identity and 97% identity (not sure other thresholds make any sense). >>>> > >> > >>>> >>>> > >> > >>>> Thoughts? Are there other things to discuss and we should have the call anyway? >>>> > >> > >>>> >>>> > >> > >>>> Thanks, >>>> > >> > >>>> - Dylan >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> _______________________________________________ >>>> > >> > >>>> Kbase-microcomm mailing list >>>> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> _______________________________________________ >>>> > >> > >>>> Kbase-microcomm mailing list >>>> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>> >>>> > >> > >>> _______________________________________________ >>>> > >> > >>> Kbase-microcomm mailing list >>>> > >> > >>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> _______________________________________________ >>>> > >> > >>> Kbase-microcomm mailing list >>>> > >> > >>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > > >>>> > >> > > _______________________________________________ >>>> > >> > > Kbase-microcomm mailing list >>>> > >> > > Kbase-microcomm(a)lists.kbase.us >>>> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > > >>>> > >> > > >>>> > >> > > _______________________________________________ >>>> > >> > > Kbase-microcomm mailing list >>>> > >> > > Kbase-microcomm(a)lists.kbase.us >>>> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>>> > >> > _______________________________________________ >>>> > >> > Kbase-microcomm mailing list >>>> > >> > Kbase-microcomm(a)lists.kbase.us >>>> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>>> > >> > _______________________________________________ >>>> > >> > Kbase-microcomm mailing list >>>> > >> > Kbase-microcomm(a)lists.kbase.us >>>> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> >>>> > >> _______________________________________________ >>>> > >> Kbase-microcomm mailing list >>>> > >> Kbase-microcomm(a)lists.kbase.us >>>> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> >>>> > >> >>>> > >> _______________________________________________ >>>> > >> Kbase-microcomm mailing list >>>> > >> Kbase-microcomm(a)lists.kbase.us >>>> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > > >>>> > > _______________________________________________ >>>> > > Kbase-microcomm mailing list >>>> > > Kbase-microcomm(a)lists.kbase.us >>>> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >>>> > >>>> > _______________________________________________ >>>> > Kbase-microcomm mailing list >>>> > Kbase-microcomm(a)lists.kbase.us >>>> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> >>>> >>>> _______________________________________________ >>>> Kbase-microcomm mailing list >>>> Kbase-microcomm(a)lists.kbase.us >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> _______________________________________________ >>> Kbase-microcomm mailing list >>> Kbase-microcomm(a)lists.kbase.us >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> _______________________________________________ >> Kbase-microcomm mailing list >> Kbase-microcomm(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm

1 0

Re: [Kbase-MicroComm] repurpose call?
by Dylan Chivian 16 May '13

16 May '13

As Rick points out, the problem is one then needs to check whether the transitive similarity is still within threshold (i.e. KB_16S - m5_16S >= 97% and m5_16S - MG_16S >= 97% does not necessarily mean KB_16S - MG_16S >= 97%) and also you could miss stuff in the other direction where the KB_16S is closer to the MG_16S. I think you just draw a 2x radius circle around the KB_16S and then realign the KB_16S against the MG_16S and check identity. So if you want to be within 97%, then the KB_16S - m5_16S >= 94% (i.e. within 6%), collect the set of MG_16S that satisfy m5_16S - MG_16S >= 97% (that should be a Communities_API service), and then check KB_16S - MG_16S >= 97% (i.e. within 3%). Is my math right? Is there a query to return all sequences (or maybe just the abundance counts) within thresholding of identity, e-value, length to a given m5 reference sequence? Thanks, - Dylan On Thu, May 16, 2013 at 5:26 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: > Yes using the pre computed metagenome analysis as an index into the reads > is the right idea IMO > > Sent from mobile device > > On May 16, 2013, at 19:08, "Rick @ Argonne" <stevens(a)anl.gov> wrote: > > One approach (im sure there ate variations and better) to the clever case > could be resolved by finding a set of more loosely related genomes > gathering the superset then a sequence search in that smaller set. > > Ideally you would be searching contigs or representatives rather than raw > MG reads. So it would be tractable. But I bet we would almost never be in > this case. > > Sent from my iPhone > > On May 16, 2013, at 18:52, "Rick @ Argonne" <stevens(a)anl.gov> wrote: > > I would build tables that map metagenome IDs to the IDs for isolate > genomes in the CDM and vice versa. > > If one derefrenced via the M5nr, the M5rna and silva > > Then the vast majority would be captured by the normal annotation process > of each domain. > > That leaves the case where something is too new to have been captured in > the M5s > > If the new genome is a close relative then a simple transitive search > works. > > If its really different then one would need to do something more clever. > > I would focus on the non clever cases first which will cover 99.99% of the > cases. > > > > Sent from my iPhone > > On May 16, 2013, at 18:37, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > > what is all this about a use-case? it's the same thing we've been asking > for from you since very early in the project. we want to be able to say > which metagenomes a given KBase genome can be found in (or a close > relative). it's pretty straightforward. if I just state it like that, how > would you implement a solution? > > thanks, > - dylan > > > > On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: > >> ah. I see so this for a use case like RAST? >> >> as Silva's job is to mine all databases for 16s and expose them to the >> rest of us, I assume this would be a novel thing. So yes if it is not in >> the database we will not be able to search against it. >> >> In any case my comment is even more applicable: I don't understand your >> use-case well enough to be helpful. >> >> best, >> Folker >> >> On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> >> > "user-defined" in this case means a 16S from a KBase microbial genome >> (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the >> inability to synchronize it with KBase regularly as we've discussed). >> > >> > thanks, >> > - dylan >> > >> > >> > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: >> > >> > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > > Sorry, which matching? (a) metagenome gene to 16S database or (b) >> metagenome read to user-defined 16S query sequence? >> > >> > Now you are loosing me. Why would the user want to define a 16s query >> sequence? It seems clear to me that I do not understand your use-case. >> > >> > The API is providing matches that were computed against the m5nr and >> m5rnaNR databases. >> > >> > If you want to search against another database that is not part of the >> API. However it is very likely that the other 16s sequence will be in our >> database. once we are fully in shock you will be able to ask for the top >> 300 hits for a read/cluster if that is what you want. this may or may not >> be interesting for your use-case. >> > >> > > >> > > - dylan >> > > >> > > >> > > >> > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> >> wrote: >> > > The matching has already been computed. >> > > >> > > Sent from my iPad >> > > >> > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > > >> > >> ... and there's no sequence-based search query, right? we'll need >> to do the alignments ourselves? >> > >> >> > >> Thanks >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> >> wrote: >> > >> Great! What's the command for getting the public metagenome IDs? >> > >> >> > >> Thanks, >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer < >> folker.meyer(a)gmail.com> wrote: >> > >> sure. this is quite simple in the API. >> > >> >> > >> >> > >> pseudo code: >> > >> ----------- >> > >> for each public_metagenome >> > >> >> > >> do >> > >> # get abundance profile >> > >> curl " >> http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" >> | json_xs >> > >> >> > >> # e.g. curl " >> http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" >> | json_xs >> > >> >> > >> done >> > >> ----------- >> > >> This gives you the taxon info derived from either proteins or 16s >> using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are >> also mapped to NCBI taxonomy. >> > >> >> > >> Using Silva (aka SSU) as namespace instead of GenBank above will >> just give you 16s. >> > >> >> > >> best, >> > >> Folker >> > >> >> > >> >> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> >> wrote: >> > >> >> > >> > Hi Folker, >> > >> > >> > >> > Right now we are just building prototypes, so it would still be >> useful to have the functionality even if it is not optimized. >> > >> > >> > >> > Can we see an example of how to do this with the api? I think >> that is all we are asking for at this point. >> > >> > >> > >> > Thanks, >> > >> > Matt >> > >> > >> > >> > >> > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer < >> folker.meyer(a)gmail.com> wrote: >> > >> > this is something that is supported by the API now. But not fast. >> > >> > >> > >> > The query is one of the uses cases we are re-designing the search >> for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. >> > >> > >> > >> > once that is done, we will implement simple searches in MG-RAST >> like which taxon is present in which geographic region etc. >> > >> > >> > >> > currently it is very slow. you are going 90 degrees to the way the >> system is optimized >> > >> > >> > >> > Best, >> > >> > Folker >> > >> > >> > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> >> wrote: >> > >> > >> > >> > > Yeah, those methods would work, I just need to know the function >> call and how to execute it. >> > >> > > >> > >> > > >> > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> >> wrote: >> > >> > > For isolate genomes we can use metadata and 16s precomputed >> mapping. >> > >> > > >> > >> > > For WGS and WGS contigs we need to workout precisely what you >> mean by assign to a taxon grouping. >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > Sent from my iPad >> > >> > > >> > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> >> wrote: >> > >> > > >> > >> > >> Dylan >> > >> > >> >> > >> > >> I'm trying Ti understand what you want to build. >> > >> > >> >> > >> > >> I think you want some way to have some one select a taxon and >> bring up in a taxon page all genomes and metagenomes and metagenome derived >> contigs associated with that taxon.. >> > >> > >> >> > >> > >> Is that correct ? >> > >> > >> >> > >> > >> Sent from my iPad >> > >> > >> >> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> >> wrote: >> > >> > >> >> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys >> flew out there to work with your guys. Is that not happening? >> > >> > >>> >> > >> > >>> Thanks, >> > >> > >>> - Dylan >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer < >> folker(a)mcs.anl.gov> wrote: >> > >> > >>> We are super busy right now. Can we do this next week over the >> phone. I still need some input on the actual goals of this. >> > >> > >>> >> > >> > >>> >> > >> > >>> -- >> > >> > >>> Sent from mobile device >> > >> > >>> >> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson < >> mhenderson(a)lbl.gov> wrote: >> > >> > >>> >> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you >> guys (Andreas?) are welcome to come over and be part of the excitement of >> the KBase build and "stuff". Mike and I will also be here tomorrow >> morning, but will have to leave around lunchtime to get back to the airport >> for the trip back home. >> > >> > >>>> >> > >> > >>>> Paramvir is less concerned at the moment about thresholding >> and just wants to see some functionality. >> > >> > >>>> >> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email >> pointing us in the proper direction would also be appreciated. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> Matt >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian < >> dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Paramvir is a stakeholder here, so we should get his input, >> including on which identity thresholds he'd like to see. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian < >> dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Hi guys, >> > >> > >>>> >> > >> > >>>> instead of the call today, probably the time would be best >> spent with Folker and Andreas working with Matt and Mike directly on the >> metagenomics species search functionality for the Taxon pages. I expect it >> is an expensive operation, so it is probably something that should be a >> one-time compute and stored somewhere associated with each taxon rather >> than on-the-fly when someone calls up the Taxon page. There should >> probably also be separate lists for 99% identity and 97% identity (not sure >> other thresholds make any sense). >> > >> > >>>> >> > >> > >>>> Thoughts? Are there other things to discuss and we should >> have the call anyway? >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > > >> > > _______________________________________________ >> > > Kbase-microcomm mailing list >> > > Kbase-microcomm(a)lists.kbase.us >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > _______________________________________________ >> > Kbase-microcomm mailing list >> > Kbase-microcomm(a)lists.kbase.us >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> >> > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >

1 0

Re: [Kbase-MicroComm] repurpose call?
by Folker Meyer 16 May '13

16 May '13

Yes using the pre computed metagenome analysis as an index into the reads is the right idea IMO Sent from mobile device On May 16, 2013, at 19:08, "Rick @ Argonne" <stevens(a)anl.gov> wrote: > One approach (im sure there ate variations and better) to the clever case could be resolved by finding a set of more loosely related genomes gathering the superset then a sequence search in that smaller set. > > Ideally you would be searching contigs or representatives rather than raw MG reads. So it would be tractable. But I bet we would almost never be in this case. > > Sent from my iPhone > > On May 16, 2013, at 18:52, "Rick @ Argonne" <stevens(a)anl.gov> wrote: > >> I would build tables that map metagenome IDs to the IDs for isolate genomes in the CDM and vice versa. >> >> If one derefrenced via the M5nr, the M5rna and silva >> >> Then the vast majority would be captured by the normal annotation process of each domain. >> >> That leaves the case where something is too new to have been captured in the M5s >> >> If the new genome is a close relative then a simple transitive search works. >> >> If its really different then one would need to do something more clever. >> >> I would focus on the non clever cases first which will cover 99.99% of the cases. >> >> >> >> Sent from my iPhone >> >> On May 16, 2013, at 18:37, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> >>> what is all this about a use-case? it's the same thing we've been asking for from you since very early in the project. we want to be able to say which metagenomes a given KBase genome can be found in (or a close relative). it's pretty straightforward. if I just state it like that, how would you implement a solution? >>> >>> thanks, >>> - dylan >>> >>> >>> >>> On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: >>>> ah. I see so this for a use case like RAST? >>>> >>>> as Silva's job is to mine all databases for 16s and expose them to the rest of us, I assume this would be a novel thing. So yes if it is not in the database we will not be able to search against it. >>>> >>>> In any case my comment is even more applicable: I don't understand your use-case well enough to be helpful. >>>> >>>> best, >>>> Folker >>>> >>>> On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> >>>> > "user-defined" in this case means a 16S from a KBase microbial genome (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the inability to synchronize it with KBase regularly as we've discussed). >>>> > >>>> > thanks, >>>> > - dylan >>>> > >>>> > >>>> > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: >>>> > >>>> > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >>>> > > Sorry, which matching? (a) metagenome gene to 16S database or (b) metagenome read to user-defined 16S query sequence? >>>> > >>>> > Now you are loosing me. Why would the user want to define a 16s query sequence? It seems clear to me that I do not understand your use-case. >>>> > >>>> > The API is providing matches that were computed against the m5nr and m5rnaNR databases. >>>> > >>>> > If you want to search against another database that is not part of the API. However it is very likely that the other 16s sequence will be in our database. once we are fully in shock you will be able to ask for the top 300 hits for a read/cluster if that is what you want. this may or may not be interesting for your use-case. >>>> > >>>> > > >>>> > > - dylan >>>> > > >>>> > > >>>> > > >>>> > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> wrote: >>>> > > The matching has already been computed. >>>> > > >>>> > > Sent from my iPad >>>> > > >>>> > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > > >>>> > >> ... and there's no sequence-based search query, right? we'll need to do the alignments ourselves? >>>> > >> >>>> > >> Thanks >>>> > >> - Dylan >>>> > >> >>>> > >> >>>> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> Great! What's the command for getting the public metagenome IDs? >>>> > >> >>>> > >> Thanks, >>>> > >> - Dylan >>>> > >> >>>> > >> >>>> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >>>> > >> sure. this is quite simple in the API. >>>> > >> >>>> > >> >>>> > >> pseudo code: >>>> > >> ----------- >>>> > >> for each public_metagenome >>>> > >> >>>> > >> do >>>> > >> # get abundance profile >>>> > >> curl "http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" | json_xs >>>> > >> >>>> > >> # e.g. curl "http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" | json_xs >>>> > >> >>>> > >> done >>>> > >> ----------- >>>> > >> This gives you the taxon info derived from either proteins or 16s using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are also mapped to NCBI taxonomy. >>>> > >> >>>> > >> Using Silva (aka SSU) as namespace instead of GenBank above will just give you 16s. >>>> > >> >>>> > >> best, >>>> > >> Folker >>>> > >> >>>> > >> >>>> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >>>> > >> >>>> > >> > Hi Folker, >>>> > >> > >>>> > >> > Right now we are just building prototypes, so it would still be useful to have the functionality even if it is not optimized. >>>> > >> > >>>> > >> > Can we see an example of how to do this with the api? I think that is all we are asking for at this point. >>>> > >> > >>>> > >> > Thanks, >>>> > >> > Matt >>>> > >> > >>>> > >> > >>>> > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >>>> > >> > this is something that is supported by the API now. But not fast. >>>> > >> > >>>> > >> > The query is one of the uses cases we are re-designing the search for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. >>>> > >> > >>>> > >> > once that is done, we will implement simple searches in MG-RAST like which taxon is present in which geographic region etc. >>>> > >> > >>>> > >> > currently it is very slow. you are going 90 degrees to the way the system is optimized >>>> > >> > >>>> > >> > Best, >>>> > >> > Folker >>>> > >> > >>>> > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: >>>> > >> > >>>> > >> > > Yeah, those methods would work, I just need to know the function call and how to execute it. >>>> > >> > > >>>> > >> > > >>>> > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> wrote: >>>> > >> > > For isolate genomes we can use metadata and 16s precomputed mapping. >>>> > >> > > >>>> > >> > > For WGS and WGS contigs we need to workout precisely what you mean by assign to a taxon grouping. >>>> > >> > > >>>> > >> > > >>>> > >> > > >>>> > >> > > >>>> > >> > > Sent from my iPad >>>> > >> > > >>>> > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> wrote: >>>> > >> > > >>>> > >> > >> Dylan >>>> > >> > >> >>>> > >> > >> I'm trying Ti understand what you want to build. >>>> > >> > >> >>>> > >> > >> I think you want some way to have some one select a taxon and bring up in a taxon page all genomes and metagenomes and metagenome derived contigs associated with that taxon.. >>>> > >> > >> >>>> > >> > >> Is that correct ? >>>> > >> > >> >>>> > >> > >> Sent from my iPad >>>> > >> > >> >>>> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> > >> >>>> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys flew out there to work with your guys. Is that not happening? >>>> > >> > >>> >>>> > >> > >>> Thanks, >>>> > >> > >>> - Dylan >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: >>>> > >> > >>> We are super busy right now. Can we do this next week over the phone. I still need some input on the actual goals of this. >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> -- >>>> > >> > >>> Sent from mobile device >>>> > >> > >>> >>>> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >>>> > >> > >>> >>>> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you guys (Andreas?) are welcome to come over and be part of the excitement of the KBase build and "stuff". Mike and I will also be here tomorrow morning, but will have to leave around lunchtime to get back to the airport for the trip back home. >>>> > >> > >>>> >>>> > >> > >>>> Paramvir is less concerned at the moment about thresholding and just wants to see some functionality. >>>> > >> > >>>> >>>> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email pointing us in the proper direction would also be appreciated. >>>> > >> > >>>> >>>> > >> > >>>> Thanks, >>>> > >> > >>>> Matt >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> > >>>> Paramvir is a stakeholder here, so we should get his input, including on which identity thresholds he'd like to see. >>>> > >> > >>>> >>>> > >> > >>>> Thanks, >>>> > >> > >>>> - Dylan >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>>> > >> > >>>> Hi guys, >>>> > >> > >>>> >>>> > >> > >>>> instead of the call today, probably the time would be best spent with Folker and Andreas working with Matt and Mike directly on the metagenomics species search functionality for the Taxon pages. I expect it is an expensive operation, so it is probably something that should be a one-time compute and stored somewhere associated with each taxon rather than on-the-fly when someone calls up the Taxon page. There should probably also be separate lists for 99% identity and 97% identity (not sure other thresholds make any sense). >>>> > >> > >>>> >>>> > >> > >>>> Thoughts? Are there other things to discuss and we should have the call anyway? >>>> > >> > >>>> >>>> > >> > >>>> Thanks, >>>> > >> > >>>> - Dylan >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> _______________________________________________ >>>> > >> > >>>> Kbase-microcomm mailing list >>>> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> _______________________________________________ >>>> > >> > >>>> Kbase-microcomm mailing list >>>> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>> >>>> > >> > >>> _______________________________________________ >>>> > >> > >>> Kbase-microcomm mailing list >>>> > >> > >>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>> >>>> > >> > >>> >>>> > >> > >>> _______________________________________________ >>>> > >> > >>> Kbase-microcomm mailing list >>>> > >> > >>> Kbase-microcomm(a)lists.kbase.us >>>> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > > >>>> > >> > > _______________________________________________ >>>> > >> > > Kbase-microcomm mailing list >>>> > >> > > Kbase-microcomm(a)lists.kbase.us >>>> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > > >>>> > >> > > >>>> > >> > > _______________________________________________ >>>> > >> > > Kbase-microcomm mailing list >>>> > >> > > Kbase-microcomm(a)lists.kbase.us >>>> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>>> > >> > _______________________________________________ >>>> > >> > Kbase-microcomm mailing list >>>> > >> > Kbase-microcomm(a)lists.kbase.us >>>> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> > >>>> > >> > _______________________________________________ >>>> > >> > Kbase-microcomm mailing list >>>> > >> > Kbase-microcomm(a)lists.kbase.us >>>> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> >>>> > >> _______________________________________________ >>>> > >> Kbase-microcomm mailing list >>>> > >> Kbase-microcomm(a)lists.kbase.us >>>> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >> >>>> > >> >>>> > >> _______________________________________________ >>>> > >> Kbase-microcomm mailing list >>>> > >> Kbase-microcomm(a)lists.kbase.us >>>> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > > >>>> > > _______________________________________________ >>>> > > Kbase-microcomm mailing list >>>> > > Kbase-microcomm(a)lists.kbase.us >>>> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>>> > >>>> > >>>> > _______________________________________________ >>>> > Kbase-microcomm mailing list >>>> > Kbase-microcomm(a)lists.kbase.us >>>> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> >>> _______________________________________________ >>> Kbase-microcomm mailing list >>> Kbase-microcomm(a)lists.kbase.us >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> _______________________________________________ >> Kbase-microcomm mailing list >> Kbase-microcomm(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm

1 0

Re: [Kbase-MicroComm] repurpose call?
by Rick ＠ Argonne 16 May '13

16 May '13

One approach (im sure there ate variations and better) to the clever case could be resolved by finding a set of more loosely related genomes gathering the superset then a sequence search in that smaller set. Ideally you would be searching contigs or representatives rather than raw MG reads. So it would be tractable. But I bet we would almost never be in this case. Sent from my iPhone On May 16, 2013, at 18:52, "Rick @ Argonne" <stevens(a)anl.gov> wrote: > I would build tables that map metagenome IDs to the IDs for isolate genomes in the CDM and vice versa. > > If one derefrenced via the M5nr, the M5rna and silva > > Then the vast majority would be captured by the normal annotation process of each domain. > > That leaves the case where something is too new to have been captured in the M5s > > If the new genome is a close relative then a simple transitive search works. > > If its really different then one would need to do something more clever. > > I would focus on the non clever cases first which will cover 99.99% of the cases. > > > > Sent from my iPhone > > On May 16, 2013, at 18:37, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > >> what is all this about a use-case? it's the same thing we've been asking for from you since very early in the project. we want to be able to say which metagenomes a given KBase genome can be found in (or a close relative). it's pretty straightforward. if I just state it like that, how would you implement a solution? >> >> thanks, >> - dylan >> >> >> >> On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: >>> ah. I see so this for a use case like RAST? >>> >>> as Silva's job is to mine all databases for 16s and expose them to the rest of us, I assume this would be a novel thing. So yes if it is not in the database we will not be able to search against it. >>> >>> In any case my comment is even more applicable: I don't understand your use-case well enough to be helpful. >>> >>> best, >>> Folker >>> >>> On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> >>> > "user-defined" in this case means a 16S from a KBase microbial genome (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the inability to synchronize it with KBase regularly as we've discussed). >>> > >>> > thanks, >>> > - dylan >>> > >>> > >>> > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: >>> > >>> > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> > >>> > > Sorry, which matching? (a) metagenome gene to 16S database or (b) metagenome read to user-defined 16S query sequence? >>> > >>> > Now you are loosing me. Why would the user want to define a 16s query sequence? It seems clear to me that I do not understand your use-case. >>> > >>> > The API is providing matches that were computed against the m5nr and m5rnaNR databases. >>> > >>> > If you want to search against another database that is not part of the API. However it is very likely that the other 16s sequence will be in our database. once we are fully in shock you will be able to ask for the top 300 hits for a read/cluster if that is what you want. this may or may not be interesting for your use-case. >>> > >>> > > >>> > > - dylan >>> > > >>> > > >>> > > >>> > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> wrote: >>> > > The matching has already been computed. >>> > > >>> > > Sent from my iPad >>> > > >>> > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> > > >>> > >> ... and there's no sequence-based search query, right? we'll need to do the alignments ourselves? >>> > >> >>> > >> Thanks >>> > >> - Dylan >>> > >> >>> > >> >>> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> > >> Great! What's the command for getting the public metagenome IDs? >>> > >> >>> > >> Thanks, >>> > >> - Dylan >>> > >> >>> > >> >>> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >>> > >> sure. this is quite simple in the API. >>> > >> >>> > >> >>> > >> pseudo code: >>> > >> ----------- >>> > >> for each public_metagenome >>> > >> >>> > >> do >>> > >> # get abundance profile >>> > >> curl "http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" | json_xs >>> > >> >>> > >> # e.g. curl "http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" | json_xs >>> > >> >>> > >> done >>> > >> ----------- >>> > >> This gives you the taxon info derived from either proteins or 16s using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are also mapped to NCBI taxonomy. >>> > >> >>> > >> Using Silva (aka SSU) as namespace instead of GenBank above will just give you 16s. >>> > >> >>> > >> best, >>> > >> Folker >>> > >> >>> > >> >>> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >>> > >> >>> > >> > Hi Folker, >>> > >> > >>> > >> > Right now we are just building prototypes, so it would still be useful to have the functionality even if it is not optimized. >>> > >> > >>> > >> > Can we see an example of how to do this with the api? I think that is all we are asking for at this point. >>> > >> > >>> > >> > Thanks, >>> > >> > Matt >>> > >> > >>> > >> > >>> > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >>> > >> > this is something that is supported by the API now. But not fast. >>> > >> > >>> > >> > The query is one of the uses cases we are re-designing the search for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. >>> > >> > >>> > >> > once that is done, we will implement simple searches in MG-RAST like which taxon is present in which geographic region etc. >>> > >> > >>> > >> > currently it is very slow. you are going 90 degrees to the way the system is optimized >>> > >> > >>> > >> > Best, >>> > >> > Folker >>> > >> > >>> > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: >>> > >> > >>> > >> > > Yeah, those methods would work, I just need to know the function call and how to execute it. >>> > >> > > >>> > >> > > >>> > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> wrote: >>> > >> > > For isolate genomes we can use metadata and 16s precomputed mapping. >>> > >> > > >>> > >> > > For WGS and WGS contigs we need to workout precisely what you mean by assign to a taxon grouping. >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > Sent from my iPad >>> > >> > > >>> > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> wrote: >>> > >> > > >>> > >> > >> Dylan >>> > >> > >> >>> > >> > >> I'm trying Ti understand what you want to build. >>> > >> > >> >>> > >> > >> I think you want some way to have some one select a taxon and bring up in a taxon page all genomes and metagenomes and metagenome derived contigs associated with that taxon.. >>> > >> > >> >>> > >> > >> Is that correct ? >>> > >> > >> >>> > >> > >> Sent from my iPad >>> > >> > >> >>> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> > >> > >> >>> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys flew out there to work with your guys. Is that not happening? >>> > >> > >>> >>> > >> > >>> Thanks, >>> > >> > >>> - Dylan >>> > >> > >>> >>> > >> > >>> >>> > >> > >>> >>> > >> > >>> >>> > >> > >>> >>> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: >>> > >> > >>> We are super busy right now. Can we do this next week over the phone. I still need some input on the actual goals of this. >>> > >> > >>> >>> > >> > >>> >>> > >> > >>> -- >>> > >> > >>> Sent from mobile device >>> > >> > >>> >>> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >>> > >> > >>> >>> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you guys (Andreas?) are welcome to come over and be part of the excitement of the KBase build and "stuff". Mike and I will also be here tomorrow morning, but will have to leave around lunchtime to get back to the airport for the trip back home. >>> > >> > >>>> >>> > >> > >>>> Paramvir is less concerned at the moment about thresholding and just wants to see some functionality. >>> > >> > >>>> >>> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email pointing us in the proper direction would also be appreciated. >>> > >> > >>>> >>> > >> > >>>> Thanks, >>> > >> > >>>> Matt >>> > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> > >> > >>>> Paramvir is a stakeholder here, so we should get his input, including on which identity thresholds he'd like to see. >>> > >> > >>>> >>> > >> > >>>> Thanks, >>> > >> > >>>> - Dylan >>> > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >>> > >> > >>>> Hi guys, >>> > >> > >>>> >>> > >> > >>>> instead of the call today, probably the time would be best spent with Folker and Andreas working with Matt and Mike directly on the metagenomics species search functionality for the Taxon pages. I expect it is an expensive operation, so it is probably something that should be a one-time compute and stored somewhere associated with each taxon rather than on-the-fly when someone calls up the Taxon page. There should probably also be separate lists for 99% identity and 97% identity (not sure other thresholds make any sense). >>> > >> > >>>> >>> > >> > >>>> Thoughts? Are there other things to discuss and we should have the call anyway? >>> > >> > >>>> >>> > >> > >>>> Thanks, >>> > >> > >>>> - Dylan >>> > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> _______________________________________________ >>> > >> > >>>> Kbase-microcomm mailing list >>> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >>> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> _______________________________________________ >>> > >> > >>>> Kbase-microcomm mailing list >>> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >>> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> > >>> >>> > >> > >>> _______________________________________________ >>> > >> > >>> Kbase-microcomm mailing list >>> > >> > >>> Kbase-microcomm(a)lists.kbase.us >>> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> > >>> >>> > >> > >>> >>> > >> > >>> _______________________________________________ >>> > >> > >>> Kbase-microcomm mailing list >>> > >> > >>> Kbase-microcomm(a)lists.kbase.us >>> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> > > >>> > >> > > _______________________________________________ >>> > >> > > Kbase-microcomm mailing list >>> > >> > > Kbase-microcomm(a)lists.kbase.us >>> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> > > >>> > >> > > >>> > >> > > _______________________________________________ >>> > >> > > Kbase-microcomm mailing list >>> > >> > > Kbase-microcomm(a)lists.kbase.us >>> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> > >>> > >> > _______________________________________________ >>> > >> > Kbase-microcomm mailing list >>> > >> > Kbase-microcomm(a)lists.kbase.us >>> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> > >>> > >> > _______________________________________________ >>> > >> > Kbase-microcomm mailing list >>> > >> > Kbase-microcomm(a)lists.kbase.us >>> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> >>> > >> _______________________________________________ >>> > >> Kbase-microcomm mailing list >>> > >> Kbase-microcomm(a)lists.kbase.us >>> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >> >>> > >> >>> > >> _______________________________________________ >>> > >> Kbase-microcomm mailing list >>> > >> Kbase-microcomm(a)lists.kbase.us >>> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > > >>> > > _______________________________________________ >>> > > Kbase-microcomm mailing list >>> > > Kbase-microcomm(a)lists.kbase.us >>> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >>> > >>> > >>> > _______________________________________________ >>> > Kbase-microcomm mailing list >>> > Kbase-microcomm(a)lists.kbase.us >>> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> >> _______________________________________________ >> Kbase-microcomm mailing list >> Kbase-microcomm(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm

1 0

Re: [Kbase-MicroComm] repurpose call?
by Dylan Chivian 16 May '13

16 May '13

that sounds about right. thanks, - dylan On Thu, May 16, 2013 at 4:52 PM, Rick @ Argonne <stevens(a)anl.gov> wrote: > I would build tables that map metagenome IDs to the IDs for isolate > genomes in the CDM and vice versa. > > If one derefrenced via the M5nr, the M5rna and silva > > Then the vast majority would be captured by the normal annotation process > of each domain. > > That leaves the case where something is too new to have been captured in > the M5s > > If the new genome is a close relative then a simple transitive search > works. > > If its really different then one would need to do something more clever. > > I would focus on the non clever cases first which will cover 99.99% of the > cases. > > > > Sent from my iPhone > > On May 16, 2013, at 18:37, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > > what is all this about a use-case? it's the same thing we've been asking > for from you since very early in the project. we want to be able to say > which metagenomes a given KBase genome can be found in (or a close > relative). it's pretty straightforward. if I just state it like that, how > would you implement a solution? > > thanks, > - dylan > > > > On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: > >> ah. I see so this for a use case like RAST? >> >> as Silva's job is to mine all databases for 16s and expose them to the >> rest of us, I assume this would be a novel thing. So yes if it is not in >> the database we will not be able to search against it. >> >> In any case my comment is even more applicable: I don't understand your >> use-case well enough to be helpful. >> >> best, >> Folker >> >> On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> >> > "user-defined" in this case means a 16S from a KBase microbial genome >> (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the >> inability to synchronize it with KBase regularly as we've discussed). >> > >> > thanks, >> > - dylan >> > >> > >> > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: >> > >> > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > > Sorry, which matching? (a) metagenome gene to 16S database or (b) >> metagenome read to user-defined 16S query sequence? >> > >> > Now you are loosing me. Why would the user want to define a 16s query >> sequence? It seems clear to me that I do not understand your use-case. >> > >> > The API is providing matches that were computed against the m5nr and >> m5rnaNR databases. >> > >> > If you want to search against another database that is not part of the >> API. However it is very likely that the other 16s sequence will be in our >> database. once we are fully in shock you will be able to ask for the top >> 300 hits for a read/cluster if that is what you want. this may or may not >> be interesting for your use-case. >> > >> > > >> > > - dylan >> > > >> > > >> > > >> > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> >> wrote: >> > > The matching has already been computed. >> > > >> > > Sent from my iPad >> > > >> > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > > >> > >> ... and there's no sequence-based search query, right? we'll need >> to do the alignments ourselves? >> > >> >> > >> Thanks >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> >> wrote: >> > >> Great! What's the command for getting the public metagenome IDs? >> > >> >> > >> Thanks, >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer < >> folker.meyer(a)gmail.com> wrote: >> > >> sure. this is quite simple in the API. >> > >> >> > >> >> > >> pseudo code: >> > >> ----------- >> > >> for each public_metagenome >> > >> >> > >> do >> > >> # get abundance profile >> > >> curl " >> http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" >> | json_xs >> > >> >> > >> # e.g. curl " >> http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" >> | json_xs >> > >> >> > >> done >> > >> ----------- >> > >> This gives you the taxon info derived from either proteins or 16s >> using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are >> also mapped to NCBI taxonomy. >> > >> >> > >> Using Silva (aka SSU) as namespace instead of GenBank above will >> just give you 16s. >> > >> >> > >> best, >> > >> Folker >> > >> >> > >> >> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> >> wrote: >> > >> >> > >> > Hi Folker, >> > >> > >> > >> > Right now we are just building prototypes, so it would still be >> useful to have the functionality even if it is not optimized. >> > >> > >> > >> > Can we see an example of how to do this with the api? I think >> that is all we are asking for at this point. >> > >> > >> > >> > Thanks, >> > >> > Matt >> > >> > >> > >> > >> > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer < >> folker.meyer(a)gmail.com> wrote: >> > >> > this is something that is supported by the API now. But not fast. >> > >> > >> > >> > The query is one of the uses cases we are re-designing the search >> for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. >> > >> > >> > >> > once that is done, we will implement simple searches in MG-RAST >> like which taxon is present in which geographic region etc. >> > >> > >> > >> > currently it is very slow. you are going 90 degrees to the way the >> system is optimized >> > >> > >> > >> > Best, >> > >> > Folker >> > >> > >> > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> >> wrote: >> > >> > >> > >> > > Yeah, those methods would work, I just need to know the function >> call and how to execute it. >> > >> > > >> > >> > > >> > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> >> wrote: >> > >> > > For isolate genomes we can use metadata and 16s precomputed >> mapping. >> > >> > > >> > >> > > For WGS and WGS contigs we need to workout precisely what you >> mean by assign to a taxon grouping. >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > Sent from my iPad >> > >> > > >> > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> >> wrote: >> > >> > > >> > >> > >> Dylan >> > >> > >> >> > >> > >> I'm trying Ti understand what you want to build. >> > >> > >> >> > >> > >> I think you want some way to have some one select a taxon and >> bring up in a taxon page all genomes and metagenomes and metagenome derived >> contigs associated with that taxon.. >> > >> > >> >> > >> > >> Is that correct ? >> > >> > >> >> > >> > >> Sent from my iPad >> > >> > >> >> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> >> wrote: >> > >> > >> >> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys >> flew out there to work with your guys. Is that not happening? >> > >> > >>> >> > >> > >>> Thanks, >> > >> > >>> - Dylan >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer < >> folker(a)mcs.anl.gov> wrote: >> > >> > >>> We are super busy right now. Can we do this next week over the >> phone. I still need some input on the actual goals of this. >> > >> > >>> >> > >> > >>> >> > >> > >>> -- >> > >> > >>> Sent from mobile device >> > >> > >>> >> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson < >> mhenderson(a)lbl.gov> wrote: >> > >> > >>> >> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you >> guys (Andreas?) are welcome to come over and be part of the excitement of >> the KBase build and "stuff". Mike and I will also be here tomorrow >> morning, but will have to leave around lunchtime to get back to the airport >> for the trip back home. >> > >> > >>>> >> > >> > >>>> Paramvir is less concerned at the moment about thresholding >> and just wants to see some functionality. >> > >> > >>>> >> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email >> pointing us in the proper direction would also be appreciated. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> Matt >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian < >> dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Paramvir is a stakeholder here, so we should get his input, >> including on which identity thresholds he'd like to see. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian < >> dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Hi guys, >> > >> > >>>> >> > >> > >>>> instead of the call today, probably the time would be best >> spent with Folker and Andreas working with Matt and Mike directly on the >> metagenomics species search functionality for the Taxon pages. I expect it >> is an expensive operation, so it is probably something that should be a >> one-time compute and stored somewhere associated with each taxon rather >> than on-the-fly when someone calls up the Taxon page. There should >> probably also be separate lists for 99% identity and 97% identity (not sure >> other thresholds make any sense). >> > >> > >>>> >> > >> > >>>> Thoughts? Are there other things to discuss and we should >> have the call anyway? >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > > >> > > _______________________________________________ >> > > Kbase-microcomm mailing list >> > > Kbase-microcomm(a)lists.kbase.us >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > _______________________________________________ >> > Kbase-microcomm mailing list >> > Kbase-microcomm(a)lists.kbase.us >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> >> > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >

1 0

Re: [Kbase-MicroComm] repurpose call?
by Rick ＠ Argonne 16 May '13

16 May '13

I would build tables that map metagenome IDs to the IDs for isolate genomes in the CDM and vice versa. If one derefrenced via the M5nr, the M5rna and silva Then the vast majority would be captured by the normal annotation process of each domain. That leaves the case where something is too new to have been captured in the M5s If the new genome is a close relative then a simple transitive search works. If its really different then one would need to do something more clever. I would focus on the non clever cases first which will cover 99.99% of the cases. Sent from my iPhone On May 16, 2013, at 18:37, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > what is all this about a use-case? it's the same thing we've been asking for from you since very early in the project. we want to be able to say which metagenomes a given KBase genome can be found in (or a close relative). it's pretty straightforward. if I just state it like that, how would you implement a solution? > > thanks, > - dylan > > > > On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: >> ah. I see so this for a use case like RAST? >> >> as Silva's job is to mine all databases for 16s and expose them to the rest of us, I assume this would be a novel thing. So yes if it is not in the database we will not be able to search against it. >> >> In any case my comment is even more applicable: I don't understand your use-case well enough to be helpful. >> >> best, >> Folker >> >> On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> >> > "user-defined" in this case means a 16S from a KBase microbial genome (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the inability to synchronize it with KBase regularly as we've discussed). >> > >> > thanks, >> > - dylan >> > >> > >> > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: >> > >> > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > > Sorry, which matching? (a) metagenome gene to 16S database or (b) metagenome read to user-defined 16S query sequence? >> > >> > Now you are loosing me. Why would the user want to define a 16s query sequence? It seems clear to me that I do not understand your use-case. >> > >> > The API is providing matches that were computed against the m5nr and m5rnaNR databases. >> > >> > If you want to search against another database that is not part of the API. However it is very likely that the other 16s sequence will be in our database. once we are fully in shock you will be able to ask for the top 300 hits for a read/cluster if that is what you want. this may or may not be interesting for your use-case. >> > >> > > >> > > - dylan >> > > >> > > >> > > >> > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> wrote: >> > > The matching has already been computed. >> > > >> > > Sent from my iPad >> > > >> > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > > >> > >> ... and there's no sequence-based search query, right? we'll need to do the alignments ourselves? >> > >> >> > >> Thanks >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> Great! What's the command for getting the public metagenome IDs? >> > >> >> > >> Thanks, >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >> > >> sure. this is quite simple in the API. >> > >> >> > >> >> > >> pseudo code: >> > >> ----------- >> > >> for each public_metagenome >> > >> >> > >> do >> > >> # get abundance profile >> > >> curl "http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" | json_xs >> > >> >> > >> # e.g. curl "http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" | json_xs >> > >> >> > >> done >> > >> ----------- >> > >> This gives you the taxon info derived from either proteins or 16s using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are also mapped to NCBI taxonomy. >> > >> >> > >> Using Silva (aka SSU) as namespace instead of GenBank above will just give you 16s. >> > >> >> > >> best, >> > >> Folker >> > >> >> > >> >> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >> > >> >> > >> > Hi Folker, >> > >> > >> > >> > Right now we are just building prototypes, so it would still be useful to have the functionality even if it is not optimized. >> > >> > >> > >> > Can we see an example of how to do this with the api? I think that is all we are asking for at this point. >> > >> > >> > >> > Thanks, >> > >> > Matt >> > >> > >> > >> > >> > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >> > >> > this is something that is supported by the API now. But not fast. >> > >> > >> > >> > The query is one of the uses cases we are re-designing the search for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. >> > >> > >> > >> > once that is done, we will implement simple searches in MG-RAST like which taxon is present in which geographic region etc. >> > >> > >> > >> > currently it is very slow. you are going 90 degrees to the way the system is optimized >> > >> > >> > >> > Best, >> > >> > Folker >> > >> > >> > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: >> > >> > >> > >> > > Yeah, those methods would work, I just need to know the function call and how to execute it. >> > >> > > >> > >> > > >> > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> wrote: >> > >> > > For isolate genomes we can use metadata and 16s precomputed mapping. >> > >> > > >> > >> > > For WGS and WGS contigs we need to workout precisely what you mean by assign to a taxon grouping. >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > Sent from my iPad >> > >> > > >> > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> wrote: >> > >> > > >> > >> > >> Dylan >> > >> > >> >> > >> > >> I'm trying Ti understand what you want to build. >> > >> > >> >> > >> > >> I think you want some way to have some one select a taxon and bring up in a taxon page all genomes and metagenomes and metagenome derived contigs associated with that taxon.. >> > >> > >> >> > >> > >> Is that correct ? >> > >> > >> >> > >> > >> Sent from my iPad >> > >> > >> >> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > >> >> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys flew out there to work with your guys. Is that not happening? >> > >> > >>> >> > >> > >>> Thanks, >> > >> > >>> - Dylan >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: >> > >> > >>> We are super busy right now. Can we do this next week over the phone. I still need some input on the actual goals of this. >> > >> > >>> >> > >> > >>> >> > >> > >>> -- >> > >> > >>> Sent from mobile device >> > >> > >>> >> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >> > >> > >>> >> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you guys (Andreas?) are welcome to come over and be part of the excitement of the KBase build and "stuff". Mike and I will also be here tomorrow morning, but will have to leave around lunchtime to get back to the airport for the trip back home. >> > >> > >>>> >> > >> > >>>> Paramvir is less concerned at the moment about thresholding and just wants to see some functionality. >> > >> > >>>> >> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email pointing us in the proper direction would also be appreciated. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> Matt >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Paramvir is a stakeholder here, so we should get his input, including on which identity thresholds he'd like to see. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Hi guys, >> > >> > >>>> >> > >> > >>>> instead of the call today, probably the time would be best spent with Folker and Andreas working with Matt and Mike directly on the metagenomics species search functionality for the Taxon pages. I expect it is an expensive operation, so it is probably something that should be a one-time compute and stored somewhere associated with each taxon rather than on-the-fly when someone calls up the Taxon page. There should probably also be separate lists for 99% identity and 97% identity (not sure other thresholds make any sense). >> > >> > >>>> >> > >> > >>>> Thoughts? Are there other things to discuss and we should have the call anyway? >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > > >> > > _______________________________________________ >> > > Kbase-microcomm mailing list >> > > Kbase-microcomm(a)lists.kbase.us >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > _______________________________________________ >> > Kbase-microcomm mailing list >> > Kbase-microcomm(a)lists.kbase.us >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm

1 0

Re: [Kbase-MicroComm] repurpose call?
by Rick ＠ Argonne 16 May '13

16 May '13

Is a KBase genome something from the CDM or from a user workspace ? Sent from my iPhone On May 16, 2013, at 18:37, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > what is all this about a use-case? it's the same thing we've been asking for from you since very early in the project. we want to be able to say which metagenomes a given KBase genome can be found in (or a close relative). it's pretty straightforward. if I just state it like that, how would you implement a solution? > > thanks, > - dylan > > > > On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: >> ah. I see so this for a use case like RAST? >> >> as Silva's job is to mine all databases for 16s and expose them to the rest of us, I assume this would be a novel thing. So yes if it is not in the database we will not be able to search against it. >> >> In any case my comment is even more applicable: I don't understand your use-case well enough to be helpful. >> >> best, >> Folker >> >> On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> >> > "user-defined" in this case means a 16S from a KBase microbial genome (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the inability to synchronize it with KBase regularly as we've discussed). >> > >> > thanks, >> > - dylan >> > >> > >> > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: >> > >> > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > > Sorry, which matching? (a) metagenome gene to 16S database or (b) metagenome read to user-defined 16S query sequence? >> > >> > Now you are loosing me. Why would the user want to define a 16s query sequence? It seems clear to me that I do not understand your use-case. >> > >> > The API is providing matches that were computed against the m5nr and m5rnaNR databases. >> > >> > If you want to search against another database that is not part of the API. However it is very likely that the other 16s sequence will be in our database. once we are fully in shock you will be able to ask for the top 300 hits for a read/cluster if that is what you want. this may or may not be interesting for your use-case. >> > >> > > >> > > - dylan >> > > >> > > >> > > >> > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> wrote: >> > > The matching has already been computed. >> > > >> > > Sent from my iPad >> > > >> > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > > >> > >> ... and there's no sequence-based search query, right? we'll need to do the alignments ourselves? >> > >> >> > >> Thanks >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> Great! What's the command for getting the public metagenome IDs? >> > >> >> > >> Thanks, >> > >> - Dylan >> > >> >> > >> >> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >> > >> sure. this is quite simple in the API. >> > >> >> > >> >> > >> pseudo code: >> > >> ----------- >> > >> for each public_metagenome >> > >> >> > >> do >> > >> # get abundance profile >> > >> curl "http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" | json_xs >> > >> >> > >> # e.g. curl "http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" | json_xs >> > >> >> > >> done >> > >> ----------- >> > >> This gives you the taxon info derived from either proteins or 16s using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are also mapped to NCBI taxonomy. >> > >> >> > >> Using Silva (aka SSU) as namespace instead of GenBank above will just give you 16s. >> > >> >> > >> best, >> > >> Folker >> > >> >> > >> >> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >> > >> >> > >> > Hi Folker, >> > >> > >> > >> > Right now we are just building prototypes, so it would still be useful to have the functionality even if it is not optimized. >> > >> > >> > >> > Can we see an example of how to do this with the api? I think that is all we are asking for at this point. >> > >> > >> > >> > Thanks, >> > >> > Matt >> > >> > >> > >> > >> > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: >> > >> > this is something that is supported by the API now. But not fast. >> > >> > >> > >> > The query is one of the uses cases we are re-designing the search for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. >> > >> > >> > >> > once that is done, we will implement simple searches in MG-RAST like which taxon is present in which geographic region etc. >> > >> > >> > >> > currently it is very slow. you are going 90 degrees to the way the system is optimized >> > >> > >> > >> > Best, >> > >> > Folker >> > >> > >> > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: >> > >> > >> > >> > > Yeah, those methods would work, I just need to know the function call and how to execute it. >> > >> > > >> > >> > > >> > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> wrote: >> > >> > > For isolate genomes we can use metadata and 16s precomputed mapping. >> > >> > > >> > >> > > For WGS and WGS contigs we need to workout precisely what you mean by assign to a taxon grouping. >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > Sent from my iPad >> > >> > > >> > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> wrote: >> > >> > > >> > >> > >> Dylan >> > >> > >> >> > >> > >> I'm trying Ti understand what you want to build. >> > >> > >> >> > >> > >> I think you want some way to have some one select a taxon and bring up in a taxon page all genomes and metagenomes and metagenome derived contigs associated with that taxon.. >> > >> > >> >> > >> > >> Is that correct ? >> > >> > >> >> > >> > >> Sent from my iPad >> > >> > >> >> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > >> >> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys flew out there to work with your guys. Is that not happening? >> > >> > >>> >> > >> > >>> Thanks, >> > >> > >>> - Dylan >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: >> > >> > >>> We are super busy right now. Can we do this next week over the phone. I still need some input on the actual goals of this. >> > >> > >>> >> > >> > >>> >> > >> > >>> -- >> > >> > >>> Sent from mobile device >> > >> > >>> >> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson <mhenderson(a)lbl.gov> wrote: >> > >> > >>> >> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you guys (Andreas?) are welcome to come over and be part of the excitement of the KBase build and "stuff". Mike and I will also be here tomorrow morning, but will have to leave around lunchtime to get back to the airport for the trip back home. >> > >> > >>>> >> > >> > >>>> Paramvir is less concerned at the moment about thresholding and just wants to see some functionality. >> > >> > >>>> >> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email pointing us in the proper direction would also be appreciated. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> Matt >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Paramvir is a stakeholder here, so we should get his input, including on which identity thresholds he'd like to see. >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: >> > >> > >>>> Hi guys, >> > >> > >>>> >> > >> > >>>> instead of the call today, probably the time would be best spent with Folker and Andreas working with Matt and Mike directly on the metagenomics species search functionality for the Taxon pages. I expect it is an expensive operation, so it is probably something that should be a one-time compute and stored somewhere associated with each taxon rather than on-the-fly when someone calls up the Taxon page. There should probably also be separate lists for 99% identity and 97% identity (not sure other thresholds make any sense). >> > >> > >>>> >> > >> > >>>> Thoughts? Are there other things to discuss and we should have the call anyway? >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> - Dylan >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> _______________________________________________ >> > >> > >>>> Kbase-microcomm mailing list >> > >> > >>>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >>> >> > >> > >>> >> > >> > >>> _______________________________________________ >> > >> > >>> Kbase-microcomm mailing list >> > >> > >>> Kbase-microcomm(a)lists.kbase.us >> > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > > >> > >> > > >> > >> > > _______________________________________________ >> > >> > > Kbase-microcomm mailing list >> > >> > > Kbase-microcomm(a)lists.kbase.us >> > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > >> > _______________________________________________ >> > >> > Kbase-microcomm mailing list >> > >> > Kbase-microcomm(a)lists.kbase.us >> > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> >> > >> >> > >> _______________________________________________ >> > >> Kbase-microcomm mailing list >> > >> Kbase-microcomm(a)lists.kbase.us >> > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > > >> > > _______________________________________________ >> > > Kbase-microcomm mailing list >> > > Kbase-microcomm(a)lists.kbase.us >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm >> > >> > >> > _______________________________________________ >> > Kbase-microcomm mailing list >> > Kbase-microcomm(a)lists.kbase.us >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm

1 0

Re: [Kbase-MicroComm] repurpose call?
by Dylan Chivian 16 May '13

16 May '13

what is all this about a use-case? it's the same thing we've been asking for from you since very early in the project. we want to be able to say which metagenomes a given KBase genome can be found in (or a close relative). it's pretty straightforward. if I just state it like that, how would you implement a solution? thanks, - dylan On Thu, May 16, 2013 at 3:39 PM, Folker Meyer <folker(a)anl.gov> wrote: > ah. I see so this for a use case like RAST? > > as Silva's job is to mine all databases for 16s and expose them to the > rest of us, I assume this would be a novel thing. So yes if it is not in > the database we will not be able to search against it. > > In any case my comment is even more applicable: I don't understand your > use-case well enough to be helpful. > > best, > Folker > > On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > > > "user-defined" in this case means a 16S from a KBase microbial genome > (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the > inability to synchronize it with KBase regularly as we've discussed). > > > > thanks, > > - dylan > > > > > > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: > > > > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > > > > > Sorry, which matching? (a) metagenome gene to 16S database or (b) > metagenome read to user-defined 16S query sequence? > > > > Now you are loosing me. Why would the user want to define a 16s query > sequence? It seems clear to me that I do not understand your use-case. > > > > The API is providing matches that were computed against the m5nr and > m5rnaNR databases. > > > > If you want to search against another database that is not part of the > API. However it is very likely that the other 16s sequence will be in our > database. once we are fully in shock you will be able to ask for the top > 300 hits for a read/cluster if that is what you want. this may or may not > be interesting for your use-case. > > > > > > > > - dylan > > > > > > > > > > > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> wrote: > > > The matching has already been computed. > > > > > > Sent from my iPad > > > > > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > > > > > >> ... and there's no sequence-based search query, right? we'll need to > do the alignments ourselves? > > >> > > >> Thanks > > >> - Dylan > > >> > > >> > > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> > wrote: > > >> Great! What's the command for getting the public metagenome IDs? > > >> > > >> Thanks, > > >> - Dylan > > >> > > >> > > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer <folker.meyer(a)gmail.com> > wrote: > > >> sure. this is quite simple in the API. > > >> > > >> > > >> pseudo code: > > >> ----------- > > >> for each public_metagenome > > >> > > >> do > > >> # get abundance profile > > >> curl " > http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" > | json_xs > > >> > > >> # e.g. curl " > http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" > | json_xs > > >> > > >> done > > >> ----------- > > >> This gives you the taxon info derived from either proteins or 16s > using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are > also mapped to NCBI taxonomy. > > >> > > >> Using Silva (aka SSU) as namespace instead of GenBank above will just > give you 16s. > > >> > > >> best, > > >> Folker > > >> > > >> > > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> > wrote: > > >> > > >> > Hi Folker, > > >> > > > >> > Right now we are just building prototypes, so it would still be > useful to have the functionality even if it is not optimized. > > >> > > > >> > Can we see an example of how to do this with the api? I think that > is all we are asking for at this point. > > >> > > > >> > Thanks, > > >> > Matt > > >> > > > >> > > > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer < > folker.meyer(a)gmail.com> wrote: > > >> > this is something that is supported by the API now. But not fast. > > >> > > > >> > The query is one of the uses cases we are re-designing the search > for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. > > >> > > > >> > once that is done, we will implement simple searches in MG-RAST > like which taxon is present in which geographic region etc. > > >> > > > >> > currently it is very slow. you are going 90 degrees to the way the > system is optimized > > >> > > > >> > Best, > > >> > Folker > > >> > > > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> > wrote: > > >> > > > >> > > Yeah, those methods would work, I just need to know the function > call and how to execute it. > > >> > > > > >> > > > > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> > wrote: > > >> > > For isolate genomes we can use metadata and 16s precomputed > mapping. > > >> > > > > >> > > For WGS and WGS contigs we need to workout precisely what you > mean by assign to a taxon grouping. > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > Sent from my iPad > > >> > > > > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> > wrote: > > >> > > > > >> > >> Dylan > > >> > >> > > >> > >> I'm trying Ti understand what you want to build. > > >> > >> > > >> > >> I think you want some way to have some one select a taxon and > bring up in a taxon page all genomes and metagenomes and metagenome derived > contigs associated with that taxon.. > > >> > >> > > >> > >> Is that correct ? > > >> > >> > > >> > >> Sent from my iPad > > >> > >> > > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> > wrote: > > >> > >> > > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys > flew out there to work with your guys. Is that not happening? > > >> > >>> > > >> > >>> Thanks, > > >> > >>> - Dylan > > >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer < > folker(a)mcs.anl.gov> wrote: > > >> > >>> We are super busy right now. Can we do this next week over the > phone. I still need some input on the actual goals of this. > > >> > >>> > > >> > >>> > > >> > >>> -- > > >> > >>> Sent from mobile device > > >> > >>> > > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson < > mhenderson(a)lbl.gov> wrote: > > >> > >>> > > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you > guys (Andreas?) are welcome to come over and be part of the excitement of > the KBase build and "stuff". Mike and I will also be here tomorrow > morning, but will have to leave around lunchtime to get back to the airport > for the trip back home. > > >> > >>>> > > >> > >>>> Paramvir is less concerned at the moment about thresholding > and just wants to see some functionality. > > >> > >>>> > > >> > >>>> Alternatively, if you are not feeling Freundlich, an email > pointing us in the proper direction would also be appreciated. > > >> > >>>> > > >> > >>>> Thanks, > > >> > >>>> Matt > > >> > >>>> > > >> > >>>> > > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian < > dcchivian(a)lbl.gov> wrote: > > >> > >>>> Paramvir is a stakeholder here, so we should get his input, > including on which identity thresholds he'd like to see. > > >> > >>>> > > >> > >>>> Thanks, > > >> > >>>> - Dylan > > >> > >>>> > > >> > >>>> > > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian < > dcchivian(a)lbl.gov> wrote: > > >> > >>>> Hi guys, > > >> > >>>> > > >> > >>>> instead of the call today, probably the time would be best > spent with Folker and Andreas working with Matt and Mike directly on the > metagenomics species search functionality for the Taxon pages. I expect it > is an expensive operation, so it is probably something that should be a > one-time compute and stored somewhere associated with each taxon rather > than on-the-fly when someone calls up the Taxon page. There should > probably also be separate lists for 99% identity and 97% identity (not sure > other thresholds make any sense). > > >> > >>>> > > >> > >>>> Thoughts? Are there other things to discuss and we should > have the call anyway? > > >> > >>>> > > >> > >>>> Thanks, > > >> > >>>> - Dylan > > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> _______________________________________________ > > >> > >>>> Kbase-microcomm mailing list > > >> > >>>> Kbase-microcomm(a)lists.kbase.us > > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > >>>> > > >> > >>>> > > >> > >>>> _______________________________________________ > > >> > >>>> Kbase-microcomm mailing list > > >> > >>>> Kbase-microcomm(a)lists.kbase.us > > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > >>> > > >> > >>> _______________________________________________ > > >> > >>> Kbase-microcomm mailing list > > >> > >>> Kbase-microcomm(a)lists.kbase.us > > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > >>> > > >> > >>> > > >> > >>> _______________________________________________ > > >> > >>> Kbase-microcomm mailing list > > >> > >>> Kbase-microcomm(a)lists.kbase.us > > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > > > > >> > > _______________________________________________ > > >> > > Kbase-microcomm mailing list > > >> > > Kbase-microcomm(a)lists.kbase.us > > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > > > > >> > > > > >> > > _______________________________________________ > > >> > > Kbase-microcomm mailing list > > >> > > Kbase-microcomm(a)lists.kbase.us > > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > > > >> > _______________________________________________ > > >> > Kbase-microcomm mailing list > > >> > Kbase-microcomm(a)lists.kbase.us > > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > > > >> > _______________________________________________ > > >> > Kbase-microcomm mailing list > > >> > Kbase-microcomm(a)lists.kbase.us > > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > > >> _______________________________________________ > > >> Kbase-microcomm mailing list > > >> Kbase-microcomm(a)lists.kbase.us > > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > >> > > >> > > >> _______________________________________________ > > >> Kbase-microcomm mailing list > > >> Kbase-microcomm(a)lists.kbase.us > > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > > > > > _______________________________________________ > > > Kbase-microcomm mailing list > > > Kbase-microcomm(a)lists.kbase.us > > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > > > > > _______________________________________________ > > Kbase-microcomm mailing list > > Kbase-microcomm(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >

1 0

Re: [Kbase-MicroComm] repurpose call?
by Folker Meyer 16 May '13

16 May '13

ah. I see so this for a use case like RAST? as Silva's job is to mine all databases for 16s and expose them to the rest of us, I assume this would be a novel thing. So yes if it is not in the database we will not be able to search against it. In any case my comment is even more applicable: I don't understand your use-case well enough to be helpful. best, Folker On May 16, 2013, at 5:34 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > "user-defined" in this case means a 16S from a KBase microbial genome (that isn't necessarily in GenBank or SILVA or the m5rnaNR due to the inability to synchronize it with KBase regularly as we've discussed). > > thanks, > - dylan > > > On Thu, May 16, 2013 at 3:28 PM, Folker Meyer <folker(a)anl.gov> wrote: > > On May 16, 2013, at 5:24 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > > > Sorry, which matching? (a) metagenome gene to 16S database or (b) metagenome read to user-defined 16S query sequence? > > Now you are loosing me. Why would the user want to define a 16s query sequence? It seems clear to me that I do not understand your use-case. > > The API is providing matches that were computed against the m5nr and m5rnaNR databases. > > If you want to search against another database that is not part of the API. However it is very likely that the other 16s sequence will be in our database. once we are fully in shock you will be able to ask for the top 300 hits for a read/cluster if that is what you want. this may or may not be interesting for your use-case. > > > > > - dylan > > > > > > > > On Thu, May 16, 2013 at 3:21 PM, Rick stevens <stevens(a)anl.gov> wrote: > > The matching has already been computed. > > > > Sent from my iPad > > > > On May 16, 2013, at 5:15 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > > > >> ... and there's no sequence-based search query, right? we'll need to do the alignments ourselves? > >> > >> Thanks > >> - Dylan > >> > >> > >> On Thu, May 16, 2013 at 3:12 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > >> Great! What's the command for getting the public metagenome IDs? > >> > >> Thanks, > >> - Dylan > >> > >> > >> On Thu, May 16, 2013 at 2:45 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: > >> sure. this is quite simple in the API. > >> > >> > >> pseudo code: > >> ----------- > >> for each public_metagenome > >> > >> do > >> # get abundance profile > >> curl "http://api.metagenomics.anl.gov/abundanceprofile/ID?type=organism&source=Ge…" | json_xs > >> > >> # e.g. curl "http://api.metagenomics.anl.gov/abundanceprofile/mgm4440026.3?type=organism…" | json_xs > >> > >> done > >> ----------- > >> This gives you the taxon info derived from either proteins or 16s using Genbank annotations and NCBI taxon IDs+Names. SEED annotations are also mapped to NCBI taxonomy. > >> > >> Using Silva (aka SSU) as namespace instead of GenBank above will just give you 16s. > >> > >> best, > >> Folker > >> > >> > >> On May 16, 2013, at 4:19 PM, Matthew Henderson <mhenderson(a)lbl.gov> wrote: > >> > >> > Hi Folker, > >> > > >> > Right now we are just building prototypes, so it would still be useful to have the functionality even if it is not optimized. > >> > > >> > Can we see an example of how to do this with the api? I think that is all we are asking for at this point. > >> > > >> > Thanks, > >> > Matt > >> > > >> > > >> > On Thu, May 16, 2013 at 3:34 PM, Folker Meyer <folker.meyer(a)gmail.com> wrote: > >> > this is something that is supported by the API now. But not fast. > >> > > >> > The query is one of the uses cases we are re-designing the search for MG-RAST for. We expect it to be done in a couple (read 2) of weeks. > >> > > >> > once that is done, we will implement simple searches in MG-RAST like which taxon is present in which geographic region etc. > >> > > >> > currently it is very slow. you are going 90 degrees to the way the system is optimized > >> > > >> > Best, > >> > Folker > >> > > >> > On May 16, 2013, at 3:17 PM, Paramvir Dehal <psdehal(a)lbl.gov> wrote: > >> > > >> > > Yeah, those methods would work, I just need to know the function call and how to execute it. > >> > > > >> > > > >> > > On Thu, May 16, 2013 at 3:14 PM, Rick stevens <stevens(a)anl.gov> wrote: > >> > > For isolate genomes we can use metadata and 16s precomputed mapping. > >> > > > >> > > For WGS and WGS contigs we need to workout precisely what you mean by assign to a taxon grouping. > >> > > > >> > > > >> > > > >> > > > >> > > Sent from my iPad > >> > > > >> > > On May 16, 2013, at 2:58 PM, Rick stevens <stevens(a)anl.gov> wrote: > >> > > > >> > >> Dylan > >> > >> > >> > >> I'm trying Ti understand what you want to build. > >> > >> > >> > >> I think you want some way to have some one select a taxon and bring up in a taxon page all genomes and metagenomes and metagenome derived contigs associated with that taxon.. > >> > >> > >> > >> Is that correct ? > >> > >> > >> > >> Sent from my iPad > >> > >> > >> > >> On May 16, 2013, at 2:26 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > >> > >> > >> > >>> I'm confused. Isn't this week a joint hack-a-thon? My guys flew out there to work with your guys. Is that not happening? > >> > >>> > >> > >>> Thanks, > >> > >>> - Dylan > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> On Thu, May 16, 2013 at 12:21 PM, Folker Meyer <folker(a)mcs.anl.gov> wrote: > >> > >>> We are super busy right now. Can we do this next week over the phone. I still need some input on the actual goals of this. > >> > >>> > >> > >>> > >> > >>> -- > >> > >>> Sent from mobile device > >> > >>> > >> > >>> On May 16, 2013, at 14:14, Matthew Henderson <mhenderson(a)lbl.gov> wrote: > >> > >>> > >> > >>>> Paramvir, Mike and I are at Freund Lodge currently, and you guys (Andreas?) are welcome to come over and be part of the excitement of the KBase build and "stuff". Mike and I will also be here tomorrow morning, but will have to leave around lunchtime to get back to the airport for the trip back home. > >> > >>>> > >> > >>>> Paramvir is less concerned at the moment about thresholding and just wants to see some functionality. > >> > >>>> > >> > >>>> Alternatively, if you are not feeling Freundlich, an email pointing us in the proper direction would also be appreciated. > >> > >>>> > >> > >>>> Thanks, > >> > >>>> Matt > >> > >>>> > >> > >>>> > >> > >>>> On Thu, May 16, 2013 at 1:28 PM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > >> > >>>> Paramvir is a stakeholder here, so we should get his input, including on which identity thresholds he'd like to see. > >> > >>>> > >> > >>>> Thanks, > >> > >>>> - Dylan > >> > >>>> > >> > >>>> > >> > >>>> On Thu, May 16, 2013 at 11:10 AM, Dylan Chivian <dcchivian(a)lbl.gov> wrote: > >> > >>>> Hi guys, > >> > >>>> > >> > >>>> instead of the call today, probably the time would be best spent with Folker and Andreas working with Matt and Mike directly on the metagenomics species search functionality for the Taxon pages. I expect it is an expensive operation, so it is probably something that should be a one-time compute and stored somewhere associated with each taxon rather than on-the-fly when someone calls up the Taxon page. There should probably also be separate lists for 99% identity and 97% identity (not sure other thresholds make any sense). > >> > >>>> > >> > >>>> Thoughts? Are there other things to discuss and we should have the call anyway? > >> > >>>> > >> > >>>> Thanks, > >> > >>>> - Dylan > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> _______________________________________________ > >> > >>>> Kbase-microcomm mailing list > >> > >>>> Kbase-microcomm(a)lists.kbase.us > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > >>>> > >> > >>>> > >> > >>>> _______________________________________________ > >> > >>>> Kbase-microcomm mailing list > >> > >>>> Kbase-microcomm(a)lists.kbase.us > >> > >>>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > >>> > >> > >>> _______________________________________________ > >> > >>> Kbase-microcomm mailing list > >> > >>> Kbase-microcomm(a)lists.kbase.us > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > >>> > >> > >>> > >> > >>> _______________________________________________ > >> > >>> Kbase-microcomm mailing list > >> > >>> Kbase-microcomm(a)lists.kbase.us > >> > >>> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > > > >> > > _______________________________________________ > >> > > Kbase-microcomm mailing list > >> > > Kbase-microcomm(a)lists.kbase.us > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > > > >> > > > >> > > _______________________________________________ > >> > > Kbase-microcomm mailing list > >> > > Kbase-microcomm(a)lists.kbase.us > >> > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > > >> > _______________________________________________ > >> > Kbase-microcomm mailing list > >> > Kbase-microcomm(a)lists.kbase.us > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > > >> > _______________________________________________ > >> > Kbase-microcomm mailing list > >> > Kbase-microcomm(a)lists.kbase.us > >> > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > >> _______________________________________________ > >> Kbase-microcomm mailing list > >> Kbase-microcomm(a)lists.kbase.us > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > >> > >> > >> _______________________________________________ > >> Kbase-microcomm mailing list > >> Kbase-microcomm(a)lists.kbase.us > >> https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > > > _______________________________________________ > > Kbase-microcomm mailing list > > Kbase-microcomm(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-microcomm > > > _______________________________________________ > Kbase-microcomm mailing list > Kbase-microcomm(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-microcomm

1 0