- Kbase-core - mailman3.cels.anl.gov

Re: [Kbase-core] Globus online
by Dan Olson 24 Apr '13

24 Apr '13

I see, that makes a lot of sense. With our proxy architecture, and desire to run multiple instances of some services having the DTN sit in the middle makes sense. How about taking these steps: We define a globus user account for each KBase service. We install GCMU on a DTN-like machine (possibly with raid, right now the dtn's are raid0 - scratch). The data flows from a user to the GCMU endpoint, and they share it with the intended KBase service. The KBase service has a method to reach out using the GCMU client to pickup / dropoff data. Would you want to use this bi-directionally, or just upload? Since I'm currently working on getting this setup on a Magellan node, it would be fairly easy for me to also configure a dev invocation instance like this in parallel. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Wednesday, April 24, 2013 3:51:27 PM > Subject: Re: [Kbase-core] Globus online > > We'd run GCMU on the machine hosting the service. Then the file is > local to the service or set of services that are apt to be using > those files. > > > On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: > > > That's correct, the dtn's aren't visible to the service machines. > > > > I think the easiest way imho to enable that integration is through: > > a. a shock fuse mount on the dtn > > b. a dtn "mailbox" to shock (which we've used before on the ornl > > variation service) > > c. a gridftp plugin for shock > > > > Maybe I'm not seeing GCMU in the same light, how would you envision > > a file transfer through > > GCMU getting to a service? > > > > ----- Original Message ----- > >> From: "Robert Olson" <olson(a)mcs.anl.gov> > >> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >> Cc: kbase-core(a)lists.kbase.us > >> Sent: Wednesday, April 24, 2013 3:15:04 PM > >> Subject: Re: [Kbase-core] Globus online > >> > >> I mentioned gcmu because as far as I know the dtn isn't visible > >> from > >> any of the service machines, is it? Say I upload my 4GB sample > >> file > >> via GO; how does my service see it to process it? > >> > >> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > >> > >>> We do have the dtn's defined in GO. kbase-chicago, and > >>> kbase-knoxville are defined. > >>> > >>> In order to use shock directly we need to figure out the optimal > >>> way to display the > >>> shock data as a filesystem abstraction. I think this should be a > >>> priority for the > >>> shock developers. > >>> > >>> GCMU isn't required to use globus connect. You can fire up > >>> globus > >>> connect and transfer > >>> data to the kbase DTNs today. However, you have to use your > >>> developer credentials to > >>> auth. > >>> > >>> We are trying to get GCMU installed on some KBase endpoints to > >>> support richer user-user > >>> sharing of data. The globus connect to standard gridftp > >>> interface > >>> just puts the data > >>> down on the filesystem with no good way to link it to accounts > >>> outside of unix perms > >>> on the DTN. > >>> > >>> ----- Original Message ----- > >>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>> To: kbase-core(a)lists.kbase.us > >>>> Sent: Wednesday, April 24, 2013 2:55:13 PM > >>>> Subject: [Kbase-core] Globus online > >>>> > >>>> > >>>> Do we have any KBase Globus Online endpoints defined? It seems > >>>> it > >>>> would make sense to have ADM storage visible via both Globus > >>>> Online > >>>> and Shock. Since Shock doesn't supply any real upload support > >>>> (the > >>>> client application doesn't seem to do much from my read of it) > >>>> it > >>>> would make sense to utilize GO to provide high performance > >>>> upload. > >>>> > >>>> > >>>> With Globus Connect (a client-side application that appears to > >>>> integrate nicely with the Mac at least), we could enable high > >>>> performance up/downloads between user systems and KBase. > >>>> > >>>> > >>>> This seems related - globus connect multiuser > >>>> https://www.globusonline.org/gcmu/ and would be required on the > >>>> KB > >>>> transfer node (it requires firewall be opened). > >>>> > >>>> > >>>> The context of this is enabling real-life large file upload for > >>>> KB > >>>> users without forcing them to handle the huge details required > >>>> since > >>>> the Shock folks don't care to make things easy for the end user. > >>>> _______________________________________________ > >>>> Kbase-core mailing list > >>>> Kbase-core(a)lists.kbase.us > >>>> https://lists.kbase.us/mailman/listinfo/kbase-core > >>>> > >> > >> > >

1 0

Re: [Kbase-core] Globus online
by Robert Olson 24 Apr '13

24 Apr '13

We'd run GCMU on the machine hosting the service. Then the file is local to the service or set of services that are apt to be using those files. On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: > That's correct, the dtn's aren't visible to the service machines. > > I think the easiest way imho to enable that integration is through: > a. a shock fuse mount on the dtn > b. a dtn "mailbox" to shock (which we've used before on the ornl variation service) > c. a gridftp plugin for shock > > Maybe I'm not seeing GCMU in the same light, how would you envision a file transfer through > GCMU getting to a service? > > ----- Original Message ----- >> From: "Robert Olson" <olson(a)mcs.anl.gov> >> To: "Dan Olson" <dolson(a)mcs.anl.gov> >> Cc: kbase-core(a)lists.kbase.us >> Sent: Wednesday, April 24, 2013 3:15:04 PM >> Subject: Re: [Kbase-core] Globus online >> >> I mentioned gcmu because as far as I know the dtn isn't visible from >> any of the service machines, is it? Say I upload my 4GB sample file >> via GO; how does my service see it to process it? >> >> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: >> >>> We do have the dtn's defined in GO. kbase-chicago, and >>> kbase-knoxville are defined. >>> >>> In order to use shock directly we need to figure out the optimal >>> way to display the >>> shock data as a filesystem abstraction. I think this should be a >>> priority for the >>> shock developers. >>> >>> GCMU isn't required to use globus connect. You can fire up globus >>> connect and transfer >>> data to the kbase DTNs today. However, you have to use your >>> developer credentials to >>> auth. >>> >>> We are trying to get GCMU installed on some KBase endpoints to >>> support richer user-user >>> sharing of data. The globus connect to standard gridftp interface >>> just puts the data >>> down on the filesystem with no good way to link it to accounts >>> outside of unix perms >>> on the DTN. >>> >>> ----- Original Message ----- >>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>> To: kbase-core(a)lists.kbase.us >>>> Sent: Wednesday, April 24, 2013 2:55:13 PM >>>> Subject: [Kbase-core] Globus online >>>> >>>> >>>> Do we have any KBase Globus Online endpoints defined? It seems it >>>> would make sense to have ADM storage visible via both Globus >>>> Online >>>> and Shock. Since Shock doesn't supply any real upload support (the >>>> client application doesn't seem to do much from my read of it) it >>>> would make sense to utilize GO to provide high performance upload. >>>> >>>> >>>> With Globus Connect (a client-side application that appears to >>>> integrate nicely with the Mac at least), we could enable high >>>> performance up/downloads between user systems and KBase. >>>> >>>> >>>> This seems related - globus connect multiuser >>>> https://www.globusonline.org/gcmu/ and would be required on the KB >>>> transfer node (it requires firewall be opened). >>>> >>>> >>>> The context of this is enabling real-life large file upload for KB >>>> users without forcing them to handle the huge details required >>>> since >>>> the Shock folks don't care to make things easy for the end user. >>>> _______________________________________________ >>>> Kbase-core mailing list >>>> Kbase-core(a)lists.kbase.us >>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>>> >> >>

1 0

Re: [Kbase-core] Globus online
by Dan Olson 24 Apr '13

24 Apr '13

That's correct, the dtn's aren't visible to the service machines. I think the easiest way imho to enable that integration is through: a. a shock fuse mount on the dtn b. a dtn "mailbox" to shock (which we've used before on the ornl variation service) c. a gridftp plugin for shock Maybe I'm not seeing GCMU in the same light, how would you envision a file transfer through GCMU getting to a service? ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Wednesday, April 24, 2013 3:15:04 PM > Subject: Re: [Kbase-core] Globus online > > I mentioned gcmu because as far as I know the dtn isn't visible from > any of the service machines, is it? Say I upload my 4GB sample file > via GO; how does my service see it to process it? > > On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > > > We do have the dtn's defined in GO. kbase-chicago, and > > kbase-knoxville are defined. > > > > In order to use shock directly we need to figure out the optimal > > way to display the > > shock data as a filesystem abstraction. I think this should be a > > priority for the > > shock developers. > > > > GCMU isn't required to use globus connect. You can fire up globus > > connect and transfer > > data to the kbase DTNs today. However, you have to use your > > developer credentials to > > auth. > > > > We are trying to get GCMU installed on some KBase endpoints to > > support richer user-user > > sharing of data. The globus connect to standard gridftp interface > > just puts the data > > down on the filesystem with no good way to link it to accounts > > outside of unix perms > > on the DTN. > > > > ----- Original Message ----- > >> From: "Robert Olson" <olson(a)mcs.anl.gov> > >> To: kbase-core(a)lists.kbase.us > >> Sent: Wednesday, April 24, 2013 2:55:13 PM > >> Subject: [Kbase-core] Globus online > >> > >> > >> Do we have any KBase Globus Online endpoints defined? It seems it > >> would make sense to have ADM storage visible via both Globus > >> Online > >> and Shock. Since Shock doesn't supply any real upload support (the > >> client application doesn't seem to do much from my read of it) it > >> would make sense to utilize GO to provide high performance upload. > >> > >> > >> With Globus Connect (a client-side application that appears to > >> integrate nicely with the Mac at least), we could enable high > >> performance up/downloads between user systems and KBase. > >> > >> > >> This seems related - globus connect multiuser > >> https://www.globusonline.org/gcmu/ and would be required on the KB > >> transfer node (it requires firewall be opened). > >> > >> > >> The context of this is enabling real-life large file upload for KB > >> users without forcing them to handle the huge details required > >> since > >> the Shock folks don't care to make things easy for the end user. > >> _______________________________________________ > >> Kbase-core mailing list > >> Kbase-core(a)lists.kbase.us > >> https://lists.kbase.us/mailman/listinfo/kbase-core > >> > >

1 0

Re: [Kbase-core] Globus online
by Robert Olson 24 Apr '13

24 Apr '13

I mentioned gcmu because as far as I know the dtn isn't visible from any of the service machines, is it? Say I upload my 4GB sample file via GO; how does my service see it to process it? On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > We do have the dtn's defined in GO. kbase-chicago, and kbase-knoxville are defined. > > In order to use shock directly we need to figure out the optimal way to display the > shock data as a filesystem abstraction. I think this should be a priority for the > shock developers. > > GCMU isn't required to use globus connect. You can fire up globus connect and transfer > data to the kbase DTNs today. However, you have to use your developer credentials to > auth. > > We are trying to get GCMU installed on some KBase endpoints to support richer user-user > sharing of data. The globus connect to standard gridftp interface just puts the data > down on the filesystem with no good way to link it to accounts outside of unix perms > on the DTN. > > ----- Original Message ----- >> From: "Robert Olson" <olson(a)mcs.anl.gov> >> To: kbase-core(a)lists.kbase.us >> Sent: Wednesday, April 24, 2013 2:55:13 PM >> Subject: [Kbase-core] Globus online >> >> >> Do we have any KBase Globus Online endpoints defined? It seems it >> would make sense to have ADM storage visible via both Globus Online >> and Shock. Since Shock doesn't supply any real upload support (the >> client application doesn't seem to do much from my read of it) it >> would make sense to utilize GO to provide high performance upload. >> >> >> With Globus Connect (a client-side application that appears to >> integrate nicely with the Mac at least), we could enable high >> performance up/downloads between user systems and KBase. >> >> >> This seems related - globus connect multiuser >> https://www.globusonline.org/gcmu/ and would be required on the KB >> transfer node (it requires firewall be opened). >> >> >> The context of this is enabling real-life large file upload for KB >> users without forcing them to handle the huge details required since >> the Shock folks don't care to make things easy for the end user. >> _______________________________________________ >> Kbase-core mailing list >> Kbase-core(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-core >>

1 0

Re: [Kbase-core] Globus online
by Dan Olson 24 Apr '13

24 Apr '13

We do have the dtn's defined in GO. kbase-chicago, and kbase-knoxville are defined. In order to use shock directly we need to figure out the optimal way to display the shock data as a filesystem abstraction. I think this should be a priority for the shock developers. GCMU isn't required to use globus connect. You can fire up globus connect and transfer data to the kbase DTNs today. However, you have to use your developer credentials to auth. We are trying to get GCMU installed on some KBase endpoints to support richer user-user sharing of data. The globus connect to standard gridftp interface just puts the data down on the filesystem with no good way to link it to accounts outside of unix perms on the DTN. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: kbase-core(a)lists.kbase.us > Sent: Wednesday, April 24, 2013 2:55:13 PM > Subject: [Kbase-core] Globus online > > > Do we have any KBase Globus Online endpoints defined? It seems it > would make sense to have ADM storage visible via both Globus Online > and Shock. Since Shock doesn't supply any real upload support (the > client application doesn't seem to do much from my read of it) it > would make sense to utilize GO to provide high performance upload. > > > With Globus Connect (a client-side application that appears to > integrate nicely with the Mac at least), we could enable high > performance up/downloads between user systems and KBase. > > > This seems related - globus connect multiuser > https://www.globusonline.org/gcmu/ and would be required on the KB > transfer node (it requires firewall be opened). > > > The context of this is enabling real-life large file upload for KB > users without forcing them to handle the huge details required since > the Shock folks don't care to make things easy for the end user. > _______________________________________________ > Kbase-core mailing list > Kbase-core(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-core >

1 0

Globus online
by Robert Olson 24 Apr '13

24 Apr '13

Do we have any KBase Globus Online endpoints defined? It seems it would make sense to have ADM storage visible via both Globus Online and Shock. Since Shock doesn't supply any real upload support (the client application doesn't seem to do much from my read of it) it would make sense to utilize GO to provide high performance upload. With Globus Connect (a client-side application that appears to integrate nicely with the Mac at least), we could enable high performance up/downloads between user systems and KBase. This seems related - globus connect multiuser https://www.globusonline.org/gcmu/ and would be required on the KB transfer node (it requires firewall be opened). The context of this is enabling real-life large file upload for KB users without forcing them to handle the huge details required since the Shock folks don't care to make things easy for the end user.

1 0

Re: [Kbase-core] [Kbase-devel] Should we stop manual annotation of microarray sample? ---about plant gene expression service
by Robert Olson 22 Apr '13

22 Apr '13

As I understand it this is exactly what the PATRIC folks do with the microarray data. I heartily invite you to discuss the ontology on the mailing list. I'll make the popcorn. --bob On Apr 22, 2013, at 11:34 AM, Christopher Henry wrote: > BTW, we've explored doing similar manual curation of microarrays for microbes… for example, looking at the experimental descriptions from the french dataset. I can't really site definitive success, and I won't say it's worth doing, but having the conditions for the experiments is useful. > > What I will say is we are working on regulatory models that link gene expression to stimuli. Now, if this works and works well, one can imagine running the model in reverse: you tell me what genes were expressed, I'll tell you the conditions of the experiment. I would love to get to a place where we could do this…. then begin a systematic bi-dectional comparison of "actual" experimental conditions with those predicted by the model… this then becomes analogous to what we're doing with phenotype data in metabolic models. > > I'm curious about the ontology they're using. The same ontology should ideally be used for stimuli in our regulatory models. > > On Apr 22, 2013, at 11:28 AM, Thomas Brettin <brettin(a)mcs.anl.gov> wrote: > >> Manual annotation == a human reads text that describes the experiment, and creates structured metadata for the experiment and associated microarrays. >> >> >> >> On Apr 22, 2013, at 11:25 AM, Robert Olson <olson(a)mcs.anl.gov> wrote: >> >>> OK, that's good. I just don't understand what manual annotation of microarray samples means. Seems a strange concept. >>> >>> On Apr 22, 2013, at 11:25 AM, Thomas Brettin wrote: >>> >>>> I've just been watching. He seems to be moving towards a position that is similar to what you, Ross, Terry et. al took when you originally created the expression data model in the CDM. >>>> >>>> t >>>> >>>> >>>> >>>> On Apr 22, 2013, at 11:22 AM, Robert Olson <olson(a)mcs.anl.gov> wrote: >>>> >>>>> do you guys have any idea what he's talking about and if matters? >>>>> >>>>> On Apr 21, 2013, at 4:45 PM, He, Fei wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> In this email, I want to talk about why I suggest to stop manual annotation of plant microarray samples. >>>>>> >>>>>> Currently, our plant expression service provide a function, get_eo_samplelist. This function can retrieve all the microarray samples, which are detected under a specific environmental condition (i.e. environment ontology term). >>>>>> Obviously this is useful. For instance, I am interested in getting all the microarray experiments under drought condition, I can easily use this function to get them. >>>>>> >>>>>> However, this is based on manual annotation. We have to read the description for each microarray sample, then assign a ontology term with it. After reading hundreds of description file of microarray samples, I found it is impossible to automatic this step. Can we assign a person with the task of annotate each microarray sample added in KBase? >>>>>> >>>>>> On the other hand, a microarray experiment often contains a group of samples. For instance, sample A is under drought condition while sample B is under normal condition (control). The biological insights should be generated by comparing A and B (find differential expression). In fact, 'get_eo_samplelist' will only return all the samples (from different experiments) under drought condition. The corresponding control will not be returned. >>>>>> >>>>>> As I understand, this is because, based on all the samples associated with an ontology term, we can build co-expression networks. But, here is another question, the co-expression network based on a list of drought samples(from different experiments) makes no sense to me(I might be wrong). To make things simple, I'll talk about co-expression networks in another email. >>>>>> >>>>>> Certainly, we have spent a lot of time working on along this track. Myself is also involved in the manual annotation of microarray samples. >>>>>> After the Plants Science Retreat, I realized that, for the microarray samples, we should store the data and provide tools to handle, analysis and display the data. The manual curation is useful but easy to fail (Again, I might be wrong.). >>>>>> >>>>>> More specifically, for the plant microarray, we should store: 1) normalized expression abundance; 2) description for each sample. >>>>>> We should provides tools: 1) to search and retrieve the data; 2) to identify differentially expressed genes; 3) to build co-expression network; 4) to display. >>>>>> >>>>>> Everyone is welcome to convince me 1) we should not stop manual annotation for each microarray sample; 2) we should not stop store pre-computed co-expression networks for each ontology term. >>>>>> >>>>>> Best, >>>>>> >>>>>> >>>>>> Fei He >>>>>> >>>>>> Research Associate >>>>>> >>>>>> Dr. Sergei Maslov's Lab >>>>>> >>>>>> _______________________________________________ >>>>>> Kbase-devel mailing list >>>>>> Kbase-devel(a)lists.kbase.us >>>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>>> >>>>> _______________________________________________ >>>>> Kbase-core mailing list >>>>> Kbase-core(a)lists.kbase.us >>>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>>> >>> >> >> _______________________________________________ >> Kbase-core mailing list >> Kbase-core(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-core >

1 0

Re: [Kbase-core] [Kbase-devel] Should we stop manual annotation of microarray sample? ---about plant gene expression service
by Christopher Henry 22 Apr '13

22 Apr '13

BTW, we've explored doing similar manual curation of microarrays for microbes… for example, looking at the experimental descriptions from the french dataset. I can't really site definitive success, and I won't say it's worth doing, but having the conditions for the experiments is useful. What I will say is we are working on regulatory models that link gene expression to stimuli. Now, if this works and works well, one can imagine running the model in reverse: you tell me what genes were expressed, I'll tell you the conditions of the experiment. I would love to get to a place where we could do this…. then begin a systematic bi-dectional comparison of "actual" experimental conditions with those predicted by the model… this then becomes analogous to what we're doing with phenotype data in metabolic models. I'm curious about the ontology they're using. The same ontology should ideally be used for stimuli in our regulatory models. On Apr 22, 2013, at 11:28 AM, Thomas Brettin <brettin(a)mcs.anl.gov> wrote: > Manual annotation == a human reads text that describes the experiment, and creates structured metadata for the experiment and associated microarrays. > > > > On Apr 22, 2013, at 11:25 AM, Robert Olson <olson(a)mcs.anl.gov> wrote: > >> OK, that's good. I just don't understand what manual annotation of microarray samples means. Seems a strange concept. >> >> On Apr 22, 2013, at 11:25 AM, Thomas Brettin wrote: >> >>> I've just been watching. He seems to be moving towards a position that is similar to what you, Ross, Terry et. al took when you originally created the expression data model in the CDM. >>> >>> t >>> >>> >>> >>> On Apr 22, 2013, at 11:22 AM, Robert Olson <olson(a)mcs.anl.gov> wrote: >>> >>>> do you guys have any idea what he's talking about and if matters? >>>> >>>> On Apr 21, 2013, at 4:45 PM, He, Fei wrote: >>>> >>>>> Hi All, >>>>> >>>>> In this email, I want to talk about why I suggest to stop manual annotation of plant microarray samples. >>>>> >>>>> Currently, our plant expression service provide a function, get_eo_samplelist. This function can retrieve all the microarray samples, which are detected under a specific environmental condition (i.e. environment ontology term). >>>>> Obviously this is useful. For instance, I am interested in getting all the microarray experiments under drought condition, I can easily use this function to get them. >>>>> >>>>> However, this is based on manual annotation. We have to read the description for each microarray sample, then assign a ontology term with it. After reading hundreds of description file of microarray samples, I found it is impossible to automatic this step. Can we assign a person with the task of annotate each microarray sample added in KBase? >>>>> >>>>> On the other hand, a microarray experiment often contains a group of samples. For instance, sample A is under drought condition while sample B is under normal condition (control). The biological insights should be generated by comparing A and B (find differential expression). In fact, 'get_eo_samplelist' will only return all the samples (from different experiments) under drought condition. The corresponding control will not be returned. >>>>> >>>>> As I understand, this is because, based on all the samples associated with an ontology term, we can build co-expression networks. But, here is another question, the co-expression network based on a list of drought samples(from different experiments) makes no sense to me(I might be wrong). To make things simple, I'll talk about co-expression networks in another email. >>>>> >>>>> Certainly, we have spent a lot of time working on along this track. Myself is also involved in the manual annotation of microarray samples. >>>>> After the Plants Science Retreat, I realized that, for the microarray samples, we should store the data and provide tools to handle, analysis and display the data. The manual curation is useful but easy to fail (Again, I might be wrong.). >>>>> >>>>> More specifically, for the plant microarray, we should store: 1) normalized expression abundance; 2) description for each sample. >>>>> We should provides tools: 1) to search and retrieve the data; 2) to identify differentially expressed genes; 3) to build co-expression network; 4) to display. >>>>> >>>>> Everyone is welcome to convince me 1) we should not stop manual annotation for each microarray sample; 2) we should not stop store pre-computed co-expression networks for each ontology term. >>>>> >>>>> Best, >>>>> >>>>> >>>>> Fei He >>>>> >>>>> Research Associate >>>>> >>>>> Dr. Sergei Maslov's Lab >>>>> >>>>> _______________________________________________ >>>>> Kbase-devel mailing list >>>>> Kbase-devel(a)lists.kbase.us >>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>> >>>> _______________________________________________ >>>> Kbase-core mailing list >>>> Kbase-core(a)lists.kbase.us >>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>> >> > > _______________________________________________ > Kbase-core mailing list > Kbase-core(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-core

1 0

Re: [Kbase-core] [Kbase-devel] Should we stop manual annotation of microarray sample? ---about plant gene expression service
by Thomas Brettin 22 Apr '13

22 Apr '13

Manual annotation == a human reads text that describes the experiment, and creates structured metadata for the experiment and associated microarrays. On Apr 22, 2013, at 11:25 AM, Robert Olson <olson(a)mcs.anl.gov> wrote: > OK, that's good. I just don't understand what manual annotation of microarray samples means. Seems a strange concept. > > On Apr 22, 2013, at 11:25 AM, Thomas Brettin wrote: > >> I've just been watching. He seems to be moving towards a position that is similar to what you, Ross, Terry et. al took when you originally created the expression data model in the CDM. >> >> t >> >> >> >> On Apr 22, 2013, at 11:22 AM, Robert Olson <olson(a)mcs.anl.gov> wrote: >> >>> do you guys have any idea what he's talking about and if matters? >>> >>> On Apr 21, 2013, at 4:45 PM, He, Fei wrote: >>> >>>> Hi All, >>>> >>>> In this email, I want to talk about why I suggest to stop manual annotation of plant microarray samples. >>>> >>>> Currently, our plant expression service provide a function, get_eo_samplelist. This function can retrieve all the microarray samples, which are detected under a specific environmental condition (i.e. environment ontology term). >>>> Obviously this is useful. For instance, I am interested in getting all the microarray experiments under drought condition, I can easily use this function to get them. >>>> >>>> However, this is based on manual annotation. We have to read the description for each microarray sample, then assign a ontology term with it. After reading hundreds of description file of microarray samples, I found it is impossible to automatic this step. Can we assign a person with the task of annotate each microarray sample added in KBase? >>>> >>>> On the other hand, a microarray experiment often contains a group of samples. For instance, sample A is under drought condition while sample B is under normal condition (control). The biological insights should be generated by comparing A and B (find differential expression). In fact, 'get_eo_samplelist' will only return all the samples (from different experiments) under drought condition. The corresponding control will not be returned. >>>> >>>> As I understand, this is because, based on all the samples associated with an ontology term, we can build co-expression networks. But, here is another question, the co-expression network based on a list of drought samples(from different experiments) makes no sense to me(I might be wrong). To make things simple, I'll talk about co-expression networks in another email. >>>> >>>> Certainly, we have spent a lot of time working on along this track. Myself is also involved in the manual annotation of microarray samples. >>>> After the Plants Science Retreat, I realized that, for the microarray samples, we should store the data and provide tools to handle, analysis and display the data. The manual curation is useful but easy to fail (Again, I might be wrong.). >>>> >>>> More specifically, for the plant microarray, we should store: 1) normalized expression abundance; 2) description for each sample. >>>> We should provides tools: 1) to search and retrieve the data; 2) to identify differentially expressed genes; 3) to build co-expression network; 4) to display. >>>> >>>> Everyone is welcome to convince me 1) we should not stop manual annotation for each microarray sample; 2) we should not stop store pre-computed co-expression networks for each ontology term. >>>> >>>> Best, >>>> >>>> >>>> Fei He >>>> >>>> Research Associate >>>> >>>> Dr. Sergei Maslov's Lab >>>> >>>> _______________________________________________ >>>> Kbase-devel mailing list >>>> Kbase-devel(a)lists.kbase.us >>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>> >>> _______________________________________________ >>> Kbase-core mailing list >>> Kbase-core(a)lists.kbase.us >>> https://lists.kbase.us/mailman/listinfo/kbase-core >> >

1 0

Re: [Kbase-core] [Kbase-devel] Should we stop manual annotation of microarray sample? ---about plant gene expression service
by Robert Olson 22 Apr '13

22 Apr '13

OK, that's good. I just don't understand what manual annotation of microarray samples means. Seems a strange concept. On Apr 22, 2013, at 11:25 AM, Thomas Brettin wrote: > I've just been watching. He seems to be moving towards a position that is similar to what you, Ross, Terry et. al took when you originally created the expression data model in the CDM. > > t > > > > On Apr 22, 2013, at 11:22 AM, Robert Olson <olson(a)mcs.anl.gov> wrote: > >> do you guys have any idea what he's talking about and if matters? >> >> On Apr 21, 2013, at 4:45 PM, He, Fei wrote: >> >>> Hi All, >>> >>> In this email, I want to talk about why I suggest to stop manual annotation of plant microarray samples. >>> >>> Currently, our plant expression service provide a function, get_eo_samplelist. This function can retrieve all the microarray samples, which are detected under a specific environmental condition (i.e. environment ontology term). >>> Obviously this is useful. For instance, I am interested in getting all the microarray experiments under drought condition, I can easily use this function to get them. >>> >>> However, this is based on manual annotation. We have to read the description for each microarray sample, then assign a ontology term with it. After reading hundreds of description file of microarray samples, I found it is impossible to automatic this step. Can we assign a person with the task of annotate each microarray sample added in KBase? >>> >>> On the other hand, a microarray experiment often contains a group of samples. For instance, sample A is under drought condition while sample B is under normal condition (control). The biological insights should be generated by comparing A and B (find differential expression). In fact, 'get_eo_samplelist' will only return all the samples (from different experiments) under drought condition. The corresponding control will not be returned. >>> >>> As I understand, this is because, based on all the samples associated with an ontology term, we can build co-expression networks. But, here is another question, the co-expression network based on a list of drought samples(from different experiments) makes no sense to me(I might be wrong). To make things simple, I'll talk about co-expression networks in another email. >>> >>> Certainly, we have spent a lot of time working on along this track. Myself is also involved in the manual annotation of microarray samples. >>> After the Plants Science Retreat, I realized that, for the microarray samples, we should store the data and provide tools to handle, analysis and display the data. The manual curation is useful but easy to fail (Again, I might be wrong.). >>> >>> More specifically, for the plant microarray, we should store: 1) normalized expression abundance; 2) description for each sample. >>> We should provides tools: 1) to search and retrieve the data; 2) to identify differentially expressed genes; 3) to build co-expression network; 4) to display. >>> >>> Everyone is welcome to convince me 1) we should not stop manual annotation for each microarray sample; 2) we should not stop store pre-computed co-expression networks for each ontology term. >>> >>> Best, >>> >>> >>> Fei He >>> >>> Research Associate >>> >>> Dr. Sergei Maslov's Lab >>> >>> _______________________________________________ >>> Kbase-devel mailing list >>> Kbase-devel(a)lists.kbase.us >>> https://lists.kbase.us/mailman/listinfo/kbase-devel >> >> _______________________________________________ >> Kbase-core mailing list >> Kbase-core(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-core >

1 0