April 2013 - Kbase-core - mailman3.cels.anl.gov

Re: [Kbase-core] Globus online
by Dan Olson 25 Apr '13

25 Apr '13

They are the dolson# ones. I'm going to request that they add them to the kbase namespace. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Thursday, April 25, 2013 12:50:58 PM > Subject: Re: [Kbase-core] Globus online > > if I bring up globus-connect, I don't see any kbase-* endpoints in > the endpoint list - am I reading this wrong? > > > Or are they dolson#kbase-chicago dolson#kbase-knoxville? > > --bob > > On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > > > We do have the dtn's defined in GO. kbase-chicago, and > > kbase-knoxville are defined. > > > > In order to use shock directly we need to figure out the optimal > > way to display the > > shock data as a filesystem abstraction. I think this should be a > > priority for the > > shock developers. > > > > GCMU isn't required to use globus connect. You can fire up globus > > connect and transfer > > data to the kbase DTNs today. However, you have to use your > > developer credentials to > > auth. > > > > We are trying to get GCMU installed on some KBase endpoints to > > support richer user-user > > sharing of data. The globus connect to standard gridftp interface > > just puts the data > > down on the filesystem with no good way to link it to accounts > > outside of unix perms > > on the DTN. > > > > ----- Original Message ----- > >> From: "Robert Olson" <olson(a)mcs.anl.gov> > >> To: kbase-core(a)lists.kbase.us > >> Sent: Wednesday, April 24, 2013 2:55:13 PM > >> Subject: [Kbase-core] Globus online > >> > >> > >> Do we have any KBase Globus Online endpoints defined? It seems it > >> would make sense to have ADM storage visible via both Globus > >> Online > >> and Shock. Since Shock doesn't supply any real upload support (the > >> client application doesn't seem to do much from my read of it) it > >> would make sense to utilize GO to provide high performance upload. > >> > >> > >> With Globus Connect (a client-side application that appears to > >> integrate nicely with the Mac at least), we could enable high > >> performance up/downloads between user systems and KBase. > >> > >> > >> This seems related - globus connect multiuser > >> https://www.globusonline.org/gcmu/ and would be required on the KB > >> transfer node (it requires firewall be opened). > >> > >> > >> The context of this is enabling real-life large file upload for KB > >> users without forcing them to handle the huge details required > >> since > >> the Shock folks don't care to make things easy for the end user. > >> _______________________________________________ > >> Kbase-core mailing list > >> Kbase-core(a)lists.kbase.us > >> https://lists.kbase.us/mailman/listinfo/kbase-core > >> > >

1 0

Re: [Kbase-core] Globus online
by Robert Olson 25 Apr '13

25 Apr '13

I tried to start a transfer to one but: Didn't recognize my GO or my KB password. On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > We do have the dtn's defined in GO. kbase-chicago, and kbase-knoxville are defined. > > In order to use shock directly we need to figure out the optimal way to display the > shock data as a filesystem abstraction. I think this should be a priority for the > shock developers. > > GCMU isn't required to use globus connect. You can fire up globus connect and transfer > data to the kbase DTNs today. However, you have to use your developer credentials to > auth. > > We are trying to get GCMU installed on some KBase endpoints to support richer user-user > sharing of data. The globus connect to standard gridftp interface just puts the data > down on the filesystem with no good way to link it to accounts outside of unix perms > on the DTN. > > ----- Original Message ----- >> From: "Robert Olson" <olson(a)mcs.anl.gov> >> To: kbase-core(a)lists.kbase.us >> Sent: Wednesday, April 24, 2013 2:55:13 PM >> Subject: [Kbase-core] Globus online >> >> >> Do we have any KBase Globus Online endpoints defined? It seems it >> would make sense to have ADM storage visible via both Globus Online >> and Shock. Since Shock doesn't supply any real upload support (the >> client application doesn't seem to do much from my read of it) it >> would make sense to utilize GO to provide high performance upload. >> >> >> With Globus Connect (a client-side application that appears to >> integrate nicely with the Mac at least), we could enable high >> performance up/downloads between user systems and KBase. >> >> >> This seems related - globus connect multiuser >> https://www.globusonline.org/gcmu/ and would be required on the KB >> transfer node (it requires firewall be opened). >> >> >> The context of this is enabling real-life large file upload for KB >> users without forcing them to handle the huge details required since >> the Shock folks don't care to make things easy for the end user. >> _______________________________________________ >> Kbase-core mailing list >> Kbase-core(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-core >>

1 0

Re: [Kbase-core] Globus online
by Robert Olson 25 Apr '13

25 Apr '13

if I bring up globus-connect, I don't see any kbase-* endpoints in the endpoint list - am I reading this wrong? Or are they dolson#kbase-chicago dolson#kbase-knoxville? --bob On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > We do have the dtn's defined in GO. kbase-chicago, and kbase-knoxville are defined. > > In order to use shock directly we need to figure out the optimal way to display the > shock data as a filesystem abstraction. I think this should be a priority for the > shock developers. > > GCMU isn't required to use globus connect. You can fire up globus connect and transfer > data to the kbase DTNs today. However, you have to use your developer credentials to > auth. > > We are trying to get GCMU installed on some KBase endpoints to support richer user-user > sharing of data. The globus connect to standard gridftp interface just puts the data > down on the filesystem with no good way to link it to accounts outside of unix perms > on the DTN. > > ----- Original Message ----- >> From: "Robert Olson" <olson(a)mcs.anl.gov> >> To: kbase-core(a)lists.kbase.us >> Sent: Wednesday, April 24, 2013 2:55:13 PM >> Subject: [Kbase-core] Globus online >> >> >> Do we have any KBase Globus Online endpoints defined? It seems it >> would make sense to have ADM storage visible via both Globus Online >> and Shock. Since Shock doesn't supply any real upload support (the >> client application doesn't seem to do much from my read of it) it >> would make sense to utilize GO to provide high performance upload. >> >> >> With Globus Connect (a client-side application that appears to >> integrate nicely with the Mac at least), we could enable high >> performance up/downloads between user systems and KBase. >> >> >> This seems related - globus connect multiuser >> https://www.globusonline.org/gcmu/ and would be required on the KB >> transfer node (it requires firewall be opened). >> >> >> The context of this is enabling real-life large file upload for KB >> users without forcing them to handle the huge details required since >> the Shock folks don't care to make things easy for the end user. >> _______________________________________________ >> Kbase-core mailing list >> Kbase-core(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-core >>

1 0

Re: [Kbase-core] Globus online
by Robert Olson 24 Apr '13

24 Apr '13

Good question, and one for the designers of the ADM. ----- Original Message ----- > I like this from the interface perspective. My question is how global > throughout the system would the both the type and value of $handle > be? > > > > > On Apr 24, 2013, at 4:31 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > > > I'd want to use it bidirectionally. > > > > The use case is simple. Consider a service that does a > > transformation on sequence data, like prinseq to trim sequences. > > > > the naive version of the call in the type language would be > > > > funcdef trim_sequence(FILE in, params...) returns(FILE out); > > > > It's not clear what the FILE type would be. I think it makes sense > > to proxy it via some handle. So if we had an ADM service it might > > have a client library that works like this (NB client library, not > > a standard funcdef - you need the client to be in cahoots with the > > server to get efficient and reliable large file transfers): > > > > my $adm = ADM->new(); > > my $handle = $adm->upload($input_file) > > > > then you could pass around file handles: > > > > my $svc = PrinSeq->new(); > > my $out_handle = $svc->trim_sequence($handle, params); > > > > If you wished to download, you could: > > > > $adm->download($out_handle, $local_filename) > > > > Or pass the handle to another service; > > > > my $go = GenomeAnnotation->new(); > > my $obj = $go->annotate_reads($out_handle); > > > > etc. > > > > > > > > On Apr 24, 2013, at 4:02 PM, Dan Olson wrote: > > > >> I see, that makes a lot of sense. With our proxy architecture, > >> and desire to run multiple instances > >> of some services having the DTN sit in the middle makes sense. > >> > >> How about taking these steps: > >> We define a globus user account for each KBase service. > >> We install GCMU on a DTN-like machine (possibly with raid, right > >> now the dtn's are raid0 - scratch). > >> The data flows from a user to the GCMU endpoint, and they share it > >> with the intended KBase service. > >> The KBase service has a method to reach out using the GCMU client > >> to pickup / dropoff data. > >> > >> Would you want to use this bi-directionally, or just upload? > >> > >> Since I'm currently working on getting this setup on a Magellan > >> node, it would be fairly easy for me > >> to also configure a dev invocation instance like this in parallel. > >> > >> ----- Original Message ----- > >>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >>> Cc: kbase-core(a)lists.kbase.us > >>> Sent: Wednesday, April 24, 2013 3:51:27 PM > >>> Subject: Re: [Kbase-core] Globus online > >>> > >>> We'd run GCMU on the machine hosting the service. Then the file > >>> is > >>> local to the service or set of services that are apt to be using > >>> those files. > >>> > >>> > >>> On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: > >>> > >>>> That's correct, the dtn's aren't visible to the service > >>>> machines. > >>>> > >>>> I think the easiest way imho to enable that integration is > >>>> through: > >>>> a. a shock fuse mount on the dtn > >>>> b. a dtn "mailbox" to shock (which we've used before on the ornl > >>>> variation service) > >>>> c. a gridftp plugin for shock > >>>> > >>>> Maybe I'm not seeing GCMU in the same light, how would you > >>>> envision > >>>> a file transfer through > >>>> GCMU getting to a service? > >>>> > >>>> ----- Original Message ----- > >>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>>> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >>>>> Cc: kbase-core(a)lists.kbase.us > >>>>> Sent: Wednesday, April 24, 2013 3:15:04 PM > >>>>> Subject: Re: [Kbase-core] Globus online > >>>>> > >>>>> I mentioned gcmu because as far as I know the dtn isn't visible > >>>>> from > >>>>> any of the service machines, is it? Say I upload my 4GB sample > >>>>> file > >>>>> via GO; how does my service see it to process it? > >>>>> > >>>>> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > >>>>> > >>>>>> We do have the dtn's defined in GO. kbase-chicago, and > >>>>>> kbase-knoxville are defined. > >>>>>> > >>>>>> In order to use shock directly we need to figure out the > >>>>>> optimal > >>>>>> way to display the > >>>>>> shock data as a filesystem abstraction. I think this should > >>>>>> be a > >>>>>> priority for the > >>>>>> shock developers. > >>>>>> > >>>>>> GCMU isn't required to use globus connect. You can fire up > >>>>>> globus > >>>>>> connect and transfer > >>>>>> data to the kbase DTNs today. However, you have to use your > >>>>>> developer credentials to > >>>>>> auth. > >>>>>> > >>>>>> We are trying to get GCMU installed on some KBase endpoints to > >>>>>> support richer user-user > >>>>>> sharing of data. The globus connect to standard gridftp > >>>>>> interface > >>>>>> just puts the data > >>>>>> down on the filesystem with no good way to link it to accounts > >>>>>> outside of unix perms > >>>>>> on the DTN. > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>>>>> To: kbase-core(a)lists.kbase.us > >>>>>>> Sent: Wednesday, April 24, 2013 2:55:13 PM > >>>>>>> Subject: [Kbase-core] Globus online > >>>>>>> > >>>>>>> > >>>>>>> Do we have any KBase Globus Online endpoints defined? It > >>>>>>> seems > >>>>>>> it > >>>>>>> would make sense to have ADM storage visible via both Globus > >>>>>>> Online > >>>>>>> and Shock. Since Shock doesn't supply any real upload support > >>>>>>> (the > >>>>>>> client application doesn't seem to do much from my read of > >>>>>>> it) > >>>>>>> it > >>>>>>> would make sense to utilize GO to provide high performance > >>>>>>> upload. > >>>>>>> > >>>>>>> > >>>>>>> With Globus Connect (a client-side application that appears > >>>>>>> to > >>>>>>> integrate nicely with the Mac at least), we could enable high > >>>>>>> performance up/downloads between user systems and KBase. > >>>>>>> > >>>>>>> > >>>>>>> This seems related - globus connect multiuser > >>>>>>> https://www.globusonline.org/gcmu/ and would be required on > >>>>>>> the > >>>>>>> KB > >>>>>>> transfer node (it requires firewall be opened). > >>>>>>> > >>>>>>> > >>>>>>> The context of this is enabling real-life large file upload > >>>>>>> for > >>>>>>> KB > >>>>>>> users without forcing them to handle the huge details > >>>>>>> required > >>>>>>> since > >>>>>>> the Shock folks don't care to make things easy for the end > >>>>>>> user. > >>>>>>> _______________________________________________ > >>>>>>> Kbase-core mailing list > >>>>>>> Kbase-core(a)lists.kbase.us > >>>>>>> https://lists.kbase.us/mailman/listinfo/kbase-core > >>>>>>> > >>>>> > >>>>> > >>> > >>> > > > > _______________________________________________ > > Kbase-core mailing list > > Kbase-core(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-core > >

1 0

Re: [Kbase-core] Globus online
by Thomas Brettin 24 Apr '13

24 Apr '13

I like this from the interface perspective. My question is how global throughout the system would the both the type and value of $handle be? On Apr 24, 2013, at 4:31 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > I'd want to use it bidirectionally. > > The use case is simple. Consider a service that does a transformation on sequence data, like prinseq to trim sequences. > > the naive version of the call in the type language would be > > funcdef trim_sequence(FILE in, params...) returns(FILE out); > > It's not clear what the FILE type would be. I think it makes sense to proxy it via some handle. So if we had an ADM service it might have a client library that works like this (NB client library, not a standard funcdef - you need the client to be in cahoots with the server to get efficient and reliable large file transfers): > > my $adm = ADM->new(); > my $handle = $adm->upload($input_file) > > then you could pass around file handles: > > my $svc = PrinSeq->new(); > my $out_handle = $svc->trim_sequence($handle, params); > > If you wished to download, you could: > > $adm->download($out_handle, $local_filename) > > Or pass the handle to another service; > > my $go = GenomeAnnotation->new(); > my $obj = $go->annotate_reads($out_handle); > > etc. > > > > On Apr 24, 2013, at 4:02 PM, Dan Olson wrote: > >> I see, that makes a lot of sense. With our proxy architecture, and desire to run multiple instances >> of some services having the DTN sit in the middle makes sense. >> >> How about taking these steps: >> We define a globus user account for each KBase service. >> We install GCMU on a DTN-like machine (possibly with raid, right now the dtn's are raid0 - scratch). >> The data flows from a user to the GCMU endpoint, and they share it with the intended KBase service. >> The KBase service has a method to reach out using the GCMU client to pickup / dropoff data. >> >> Would you want to use this bi-directionally, or just upload? >> >> Since I'm currently working on getting this setup on a Magellan node, it would be fairly easy for me >> to also configure a dev invocation instance like this in parallel. >> >> ----- Original Message ----- >>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>> To: "Dan Olson" <dolson(a)mcs.anl.gov> >>> Cc: kbase-core(a)lists.kbase.us >>> Sent: Wednesday, April 24, 2013 3:51:27 PM >>> Subject: Re: [Kbase-core] Globus online >>> >>> We'd run GCMU on the machine hosting the service. Then the file is >>> local to the service or set of services that are apt to be using >>> those files. >>> >>> >>> On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: >>> >>>> That's correct, the dtn's aren't visible to the service machines. >>>> >>>> I think the easiest way imho to enable that integration is through: >>>> a. a shock fuse mount on the dtn >>>> b. a dtn "mailbox" to shock (which we've used before on the ornl >>>> variation service) >>>> c. a gridftp plugin for shock >>>> >>>> Maybe I'm not seeing GCMU in the same light, how would you envision >>>> a file transfer through >>>> GCMU getting to a service? >>>> >>>> ----- Original Message ----- >>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>>> To: "Dan Olson" <dolson(a)mcs.anl.gov> >>>>> Cc: kbase-core(a)lists.kbase.us >>>>> Sent: Wednesday, April 24, 2013 3:15:04 PM >>>>> Subject: Re: [Kbase-core] Globus online >>>>> >>>>> I mentioned gcmu because as far as I know the dtn isn't visible >>>>> from >>>>> any of the service machines, is it? Say I upload my 4GB sample >>>>> file >>>>> via GO; how does my service see it to process it? >>>>> >>>>> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: >>>>> >>>>>> We do have the dtn's defined in GO. kbase-chicago, and >>>>>> kbase-knoxville are defined. >>>>>> >>>>>> In order to use shock directly we need to figure out the optimal >>>>>> way to display the >>>>>> shock data as a filesystem abstraction. I think this should be a >>>>>> priority for the >>>>>> shock developers. >>>>>> >>>>>> GCMU isn't required to use globus connect. You can fire up >>>>>> globus >>>>>> connect and transfer >>>>>> data to the kbase DTNs today. However, you have to use your >>>>>> developer credentials to >>>>>> auth. >>>>>> >>>>>> We are trying to get GCMU installed on some KBase endpoints to >>>>>> support richer user-user >>>>>> sharing of data. The globus connect to standard gridftp >>>>>> interface >>>>>> just puts the data >>>>>> down on the filesystem with no good way to link it to accounts >>>>>> outside of unix perms >>>>>> on the DTN. >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>>>>> To: kbase-core(a)lists.kbase.us >>>>>>> Sent: Wednesday, April 24, 2013 2:55:13 PM >>>>>>> Subject: [Kbase-core] Globus online >>>>>>> >>>>>>> >>>>>>> Do we have any KBase Globus Online endpoints defined? It seems >>>>>>> it >>>>>>> would make sense to have ADM storage visible via both Globus >>>>>>> Online >>>>>>> and Shock. Since Shock doesn't supply any real upload support >>>>>>> (the >>>>>>> client application doesn't seem to do much from my read of it) >>>>>>> it >>>>>>> would make sense to utilize GO to provide high performance >>>>>>> upload. >>>>>>> >>>>>>> >>>>>>> With Globus Connect (a client-side application that appears to >>>>>>> integrate nicely with the Mac at least), we could enable high >>>>>>> performance up/downloads between user systems and KBase. >>>>>>> >>>>>>> >>>>>>> This seems related - globus connect multiuser >>>>>>> https://www.globusonline.org/gcmu/ and would be required on the >>>>>>> KB >>>>>>> transfer node (it requires firewall be opened). >>>>>>> >>>>>>> >>>>>>> The context of this is enabling real-life large file upload for >>>>>>> KB >>>>>>> users without forcing them to handle the huge details required >>>>>>> since >>>>>>> the Shock folks don't care to make things easy for the end user. >>>>>>> _______________________________________________ >>>>>>> Kbase-core mailing list >>>>>>> Kbase-core(a)lists.kbase.us >>>>>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>>>>>> >>>>> >>>>> >>> >>> > > _______________________________________________ > Kbase-core mailing list > Kbase-core(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-core

1 0

Re: [Kbase-core] Globus online
by Dan Olson 24 Apr '13

24 Apr '13

I was able to troubleshoot my GCMU install, the primary problem was a bunch of NAT settings because it is on Magellan. Out of the box it appears that it doesn't support nexus accounts, only local machine accounts. Excerpt from the GCMU docs: That’s it! Once the installation is complete, any Globus Online user will be able to see this Globus Connect Multiuser endpoint, but only users with an account on the server will be able to access it and move data to/from it. I'll ask Rachana, who I've been working with on this, to see if that is in the works. Without that, users with basic KBase user accounts won't be able to use GCMU. If you want to experiment with it, I published it in GO as KBASE-CONNECT. Use the username kbase and the password kbase4me123 ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Wednesday, April 24, 2013 4:31:52 PM > Subject: Re: [Kbase-core] Globus online > > I'd want to use it bidirectionally. > > The use case is simple. Consider a service that does a transformation > on sequence data, like prinseq to trim sequences. > > the naive version of the call in the type language would be > > funcdef trim_sequence(FILE in, params...) returns(FILE out); > > It's not clear what the FILE type would be. I think it makes sense to > proxy it via some handle. So if we had an ADM service it might have > a client library that works like this (NB client library, not a > standard funcdef - you need the client to be in cahoots with the > server to get efficient and reliable large file transfers): > > my $adm = ADM->new(); > my $handle = $adm->upload($input_file) > > then you could pass around file handles: > > my $svc = PrinSeq->new(); > my $out_handle = $svc->trim_sequence($handle, params); > > If you wished to download, you could: > > $adm->download($out_handle, $local_filename) > > Or pass the handle to another service; > > my $go = GenomeAnnotation->new(); > my $obj = $go->annotate_reads($out_handle); > > etc. > > > > On Apr 24, 2013, at 4:02 PM, Dan Olson wrote: > > > I see, that makes a lot of sense. With our proxy architecture, and > > desire to run multiple instances > > of some services having the DTN sit in the middle makes sense. > > > > How about taking these steps: > > We define a globus user account for each KBase service. > > We install GCMU on a DTN-like machine (possibly with raid, right > > now the dtn's are raid0 - scratch). > > The data flows from a user to the GCMU endpoint, and they share it > > with the intended KBase service. > > The KBase service has a method to reach out using the GCMU client > > to pickup / dropoff data. > > > > Would you want to use this bi-directionally, or just upload? > > > > Since I'm currently working on getting this setup on a Magellan > > node, it would be fairly easy for me > > to also configure a dev invocation instance like this in parallel. > > > > ----- Original Message ----- > >> From: "Robert Olson" <olson(a)mcs.anl.gov> > >> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >> Cc: kbase-core(a)lists.kbase.us > >> Sent: Wednesday, April 24, 2013 3:51:27 PM > >> Subject: Re: [Kbase-core] Globus online > >> > >> We'd run GCMU on the machine hosting the service. Then the file is > >> local to the service or set of services that are apt to be using > >> those files. > >> > >> > >> On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: > >> > >>> That's correct, the dtn's aren't visible to the service machines. > >>> > >>> I think the easiest way imho to enable that integration is > >>> through: > >>> a. a shock fuse mount on the dtn > >>> b. a dtn "mailbox" to shock (which we've used before on the ornl > >>> variation service) > >>> c. a gridftp plugin for shock > >>> > >>> Maybe I'm not seeing GCMU in the same light, how would you > >>> envision > >>> a file transfer through > >>> GCMU getting to a service? > >>> > >>> ----- Original Message ----- > >>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >>>> Cc: kbase-core(a)lists.kbase.us > >>>> Sent: Wednesday, April 24, 2013 3:15:04 PM > >>>> Subject: Re: [Kbase-core] Globus online > >>>> > >>>> I mentioned gcmu because as far as I know the dtn isn't visible > >>>> from > >>>> any of the service machines, is it? Say I upload my 4GB sample > >>>> file > >>>> via GO; how does my service see it to process it? > >>>> > >>>> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > >>>> > >>>>> We do have the dtn's defined in GO. kbase-chicago, and > >>>>> kbase-knoxville are defined. > >>>>> > >>>>> In order to use shock directly we need to figure out the > >>>>> optimal > >>>>> way to display the > >>>>> shock data as a filesystem abstraction. I think this should be > >>>>> a > >>>>> priority for the > >>>>> shock developers. > >>>>> > >>>>> GCMU isn't required to use globus connect. You can fire up > >>>>> globus > >>>>> connect and transfer > >>>>> data to the kbase DTNs today. However, you have to use your > >>>>> developer credentials to > >>>>> auth. > >>>>> > >>>>> We are trying to get GCMU installed on some KBase endpoints to > >>>>> support richer user-user > >>>>> sharing of data. The globus connect to standard gridftp > >>>>> interface > >>>>> just puts the data > >>>>> down on the filesystem with no good way to link it to accounts > >>>>> outside of unix perms > >>>>> on the DTN. > >>>>> > >>>>> ----- Original Message ----- > >>>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>>>> To: kbase-core(a)lists.kbase.us > >>>>>> Sent: Wednesday, April 24, 2013 2:55:13 PM > >>>>>> Subject: [Kbase-core] Globus online > >>>>>> > >>>>>> > >>>>>> Do we have any KBase Globus Online endpoints defined? It seems > >>>>>> it > >>>>>> would make sense to have ADM storage visible via both Globus > >>>>>> Online > >>>>>> and Shock. Since Shock doesn't supply any real upload support > >>>>>> (the > >>>>>> client application doesn't seem to do much from my read of it) > >>>>>> it > >>>>>> would make sense to utilize GO to provide high performance > >>>>>> upload. > >>>>>> > >>>>>> > >>>>>> With Globus Connect (a client-side application that appears to > >>>>>> integrate nicely with the Mac at least), we could enable high > >>>>>> performance up/downloads between user systems and KBase. > >>>>>> > >>>>>> > >>>>>> This seems related - globus connect multiuser > >>>>>> https://www.globusonline.org/gcmu/ and would be required on > >>>>>> the > >>>>>> KB > >>>>>> transfer node (it requires firewall be opened). > >>>>>> > >>>>>> > >>>>>> The context of this is enabling real-life large file upload > >>>>>> for > >>>>>> KB > >>>>>> users without forcing them to handle the huge details required > >>>>>> since > >>>>>> the Shock folks don't care to make things easy for the end > >>>>>> user. > >>>>>> _______________________________________________ > >>>>>> Kbase-core mailing list > >>>>>> Kbase-core(a)lists.kbase.us > >>>>>> https://lists.kbase.us/mailman/listinfo/kbase-core > >>>>>> > >>>> > >>>> > >> > >> > >

1 0

Re: [Kbase-core] Globus online
by Robert Olson 24 Apr '13

24 Apr '13

I'd want to use it bidirectionally. The use case is simple. Consider a service that does a transformation on sequence data, like prinseq to trim sequences. the naive version of the call in the type language would be funcdef trim_sequence(FILE in, params...) returns(FILE out); It's not clear what the FILE type would be. I think it makes sense to proxy it via some handle. So if we had an ADM service it might have a client library that works like this (NB client library, not a standard funcdef - you need the client to be in cahoots with the server to get efficient and reliable large file transfers): my $adm = ADM->new(); my $handle = $adm->upload($input_file) then you could pass around file handles: my $svc = PrinSeq->new(); my $out_handle = $svc->trim_sequence($handle, params); If you wished to download, you could: $adm->download($out_handle, $local_filename) Or pass the handle to another service; my $go = GenomeAnnotation->new(); my $obj = $go->annotate_reads($out_handle); etc. On Apr 24, 2013, at 4:02 PM, Dan Olson wrote: > I see, that makes a lot of sense. With our proxy architecture, and desire to run multiple instances > of some services having the DTN sit in the middle makes sense. > > How about taking these steps: > We define a globus user account for each KBase service. > We install GCMU on a DTN-like machine (possibly with raid, right now the dtn's are raid0 - scratch). > The data flows from a user to the GCMU endpoint, and they share it with the intended KBase service. > The KBase service has a method to reach out using the GCMU client to pickup / dropoff data. > > Would you want to use this bi-directionally, or just upload? > > Since I'm currently working on getting this setup on a Magellan node, it would be fairly easy for me > to also configure a dev invocation instance like this in parallel. > > ----- Original Message ----- >> From: "Robert Olson" <olson(a)mcs.anl.gov> >> To: "Dan Olson" <dolson(a)mcs.anl.gov> >> Cc: kbase-core(a)lists.kbase.us >> Sent: Wednesday, April 24, 2013 3:51:27 PM >> Subject: Re: [Kbase-core] Globus online >> >> We'd run GCMU on the machine hosting the service. Then the file is >> local to the service or set of services that are apt to be using >> those files. >> >> >> On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: >> >>> That's correct, the dtn's aren't visible to the service machines. >>> >>> I think the easiest way imho to enable that integration is through: >>> a. a shock fuse mount on the dtn >>> b. a dtn "mailbox" to shock (which we've used before on the ornl >>> variation service) >>> c. a gridftp plugin for shock >>> >>> Maybe I'm not seeing GCMU in the same light, how would you envision >>> a file transfer through >>> GCMU getting to a service? >>> >>> ----- Original Message ----- >>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>> To: "Dan Olson" <dolson(a)mcs.anl.gov> >>>> Cc: kbase-core(a)lists.kbase.us >>>> Sent: Wednesday, April 24, 2013 3:15:04 PM >>>> Subject: Re: [Kbase-core] Globus online >>>> >>>> I mentioned gcmu because as far as I know the dtn isn't visible >>>> from >>>> any of the service machines, is it? Say I upload my 4GB sample >>>> file >>>> via GO; how does my service see it to process it? >>>> >>>> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: >>>> >>>>> We do have the dtn's defined in GO. kbase-chicago, and >>>>> kbase-knoxville are defined. >>>>> >>>>> In order to use shock directly we need to figure out the optimal >>>>> way to display the >>>>> shock data as a filesystem abstraction. I think this should be a >>>>> priority for the >>>>> shock developers. >>>>> >>>>> GCMU isn't required to use globus connect. You can fire up >>>>> globus >>>>> connect and transfer >>>>> data to the kbase DTNs today. However, you have to use your >>>>> developer credentials to >>>>> auth. >>>>> >>>>> We are trying to get GCMU installed on some KBase endpoints to >>>>> support richer user-user >>>>> sharing of data. The globus connect to standard gridftp >>>>> interface >>>>> just puts the data >>>>> down on the filesystem with no good way to link it to accounts >>>>> outside of unix perms >>>>> on the DTN. >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>>>> To: kbase-core(a)lists.kbase.us >>>>>> Sent: Wednesday, April 24, 2013 2:55:13 PM >>>>>> Subject: [Kbase-core] Globus online >>>>>> >>>>>> >>>>>> Do we have any KBase Globus Online endpoints defined? It seems >>>>>> it >>>>>> would make sense to have ADM storage visible via both Globus >>>>>> Online >>>>>> and Shock. Since Shock doesn't supply any real upload support >>>>>> (the >>>>>> client application doesn't seem to do much from my read of it) >>>>>> it >>>>>> would make sense to utilize GO to provide high performance >>>>>> upload. >>>>>> >>>>>> >>>>>> With Globus Connect (a client-side application that appears to >>>>>> integrate nicely with the Mac at least), we could enable high >>>>>> performance up/downloads between user systems and KBase. >>>>>> >>>>>> >>>>>> This seems related - globus connect multiuser >>>>>> https://www.globusonline.org/gcmu/ and would be required on the >>>>>> KB >>>>>> transfer node (it requires firewall be opened). >>>>>> >>>>>> >>>>>> The context of this is enabling real-life large file upload for >>>>>> KB >>>>>> users without forcing them to handle the huge details required >>>>>> since >>>>>> the Shock folks don't care to make things easy for the end user. >>>>>> _______________________________________________ >>>>>> Kbase-core mailing list >>>>>> Kbase-core(a)lists.kbase.us >>>>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>>>>> >>>> >>>> >> >>

1 0

Re: [Kbase-core] Globus online
by Dan Olson 24 Apr '13

24 Apr '13

I see, that makes a lot of sense. With our proxy architecture, and desire to run multiple instances of some services having the DTN sit in the middle makes sense. How about taking these steps: We define a globus user account for each KBase service. We install GCMU on a DTN-like machine (possibly with raid, right now the dtn's are raid0 - scratch). The data flows from a user to the GCMU endpoint, and they share it with the intended KBase service. The KBase service has a method to reach out using the GCMU client to pickup / dropoff data. Would you want to use this bi-directionally, or just upload? Since I'm currently working on getting this setup on a Magellan node, it would be fairly easy for me to also configure a dev invocation instance like this in parallel. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Wednesday, April 24, 2013 3:51:27 PM > Subject: Re: [Kbase-core] Globus online > > We'd run GCMU on the machine hosting the service. Then the file is > local to the service or set of services that are apt to be using > those files. > > > On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: > > > That's correct, the dtn's aren't visible to the service machines. > > > > I think the easiest way imho to enable that integration is through: > > a. a shock fuse mount on the dtn > > b. a dtn "mailbox" to shock (which we've used before on the ornl > > variation service) > > c. a gridftp plugin for shock > > > > Maybe I'm not seeing GCMU in the same light, how would you envision > > a file transfer through > > GCMU getting to a service? > > > > ----- Original Message ----- > >> From: "Robert Olson" <olson(a)mcs.anl.gov> > >> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >> Cc: kbase-core(a)lists.kbase.us > >> Sent: Wednesday, April 24, 2013 3:15:04 PM > >> Subject: Re: [Kbase-core] Globus online > >> > >> I mentioned gcmu because as far as I know the dtn isn't visible > >> from > >> any of the service machines, is it? Say I upload my 4GB sample > >> file > >> via GO; how does my service see it to process it? > >> > >> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > >> > >>> We do have the dtn's defined in GO. kbase-chicago, and > >>> kbase-knoxville are defined. > >>> > >>> In order to use shock directly we need to figure out the optimal > >>> way to display the > >>> shock data as a filesystem abstraction. I think this should be a > >>> priority for the > >>> shock developers. > >>> > >>> GCMU isn't required to use globus connect. You can fire up > >>> globus > >>> connect and transfer > >>> data to the kbase DTNs today. However, you have to use your > >>> developer credentials to > >>> auth. > >>> > >>> We are trying to get GCMU installed on some KBase endpoints to > >>> support richer user-user > >>> sharing of data. The globus connect to standard gridftp > >>> interface > >>> just puts the data > >>> down on the filesystem with no good way to link it to accounts > >>> outside of unix perms > >>> on the DTN. > >>> > >>> ----- Original Message ----- > >>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>> To: kbase-core(a)lists.kbase.us > >>>> Sent: Wednesday, April 24, 2013 2:55:13 PM > >>>> Subject: [Kbase-core] Globus online > >>>> > >>>> > >>>> Do we have any KBase Globus Online endpoints defined? It seems > >>>> it > >>>> would make sense to have ADM storage visible via both Globus > >>>> Online > >>>> and Shock. Since Shock doesn't supply any real upload support > >>>> (the > >>>> client application doesn't seem to do much from my read of it) > >>>> it > >>>> would make sense to utilize GO to provide high performance > >>>> upload. > >>>> > >>>> > >>>> With Globus Connect (a client-side application that appears to > >>>> integrate nicely with the Mac at least), we could enable high > >>>> performance up/downloads between user systems and KBase. > >>>> > >>>> > >>>> This seems related - globus connect multiuser > >>>> https://www.globusonline.org/gcmu/ and would be required on the > >>>> KB > >>>> transfer node (it requires firewall be opened). > >>>> > >>>> > >>>> The context of this is enabling real-life large file upload for > >>>> KB > >>>> users without forcing them to handle the huge details required > >>>> since > >>>> the Shock folks don't care to make things easy for the end user. > >>>> _______________________________________________ > >>>> Kbase-core mailing list > >>>> Kbase-core(a)lists.kbase.us > >>>> https://lists.kbase.us/mailman/listinfo/kbase-core > >>>> > >> > >> > >

1 0

Re: [Kbase-core] Globus online
by Robert Olson 24 Apr '13

24 Apr '13

We'd run GCMU on the machine hosting the service. Then the file is local to the service or set of services that are apt to be using those files. On Apr 24, 2013, at 3:45 PM, Dan Olson wrote: > That's correct, the dtn's aren't visible to the service machines. > > I think the easiest way imho to enable that integration is through: > a. a shock fuse mount on the dtn > b. a dtn "mailbox" to shock (which we've used before on the ornl variation service) > c. a gridftp plugin for shock > > Maybe I'm not seeing GCMU in the same light, how would you envision a file transfer through > GCMU getting to a service? > > ----- Original Message ----- >> From: "Robert Olson" <olson(a)mcs.anl.gov> >> To: "Dan Olson" <dolson(a)mcs.anl.gov> >> Cc: kbase-core(a)lists.kbase.us >> Sent: Wednesday, April 24, 2013 3:15:04 PM >> Subject: Re: [Kbase-core] Globus online >> >> I mentioned gcmu because as far as I know the dtn isn't visible from >> any of the service machines, is it? Say I upload my 4GB sample file >> via GO; how does my service see it to process it? >> >> On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: >> >>> We do have the dtn's defined in GO. kbase-chicago, and >>> kbase-knoxville are defined. >>> >>> In order to use shock directly we need to figure out the optimal >>> way to display the >>> shock data as a filesystem abstraction. I think this should be a >>> priority for the >>> shock developers. >>> >>> GCMU isn't required to use globus connect. You can fire up globus >>> connect and transfer >>> data to the kbase DTNs today. However, you have to use your >>> developer credentials to >>> auth. >>> >>> We are trying to get GCMU installed on some KBase endpoints to >>> support richer user-user >>> sharing of data. The globus connect to standard gridftp interface >>> just puts the data >>> down on the filesystem with no good way to link it to accounts >>> outside of unix perms >>> on the DTN. >>> >>> ----- Original Message ----- >>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>> To: kbase-core(a)lists.kbase.us >>>> Sent: Wednesday, April 24, 2013 2:55:13 PM >>>> Subject: [Kbase-core] Globus online >>>> >>>> >>>> Do we have any KBase Globus Online endpoints defined? It seems it >>>> would make sense to have ADM storage visible via both Globus >>>> Online >>>> and Shock. Since Shock doesn't supply any real upload support (the >>>> client application doesn't seem to do much from my read of it) it >>>> would make sense to utilize GO to provide high performance upload. >>>> >>>> >>>> With Globus Connect (a client-side application that appears to >>>> integrate nicely with the Mac at least), we could enable high >>>> performance up/downloads between user systems and KBase. >>>> >>>> >>>> This seems related - globus connect multiuser >>>> https://www.globusonline.org/gcmu/ and would be required on the KB >>>> transfer node (it requires firewall be opened). >>>> >>>> >>>> The context of this is enabling real-life large file upload for KB >>>> users without forcing them to handle the huge details required >>>> since >>>> the Shock folks don't care to make things easy for the end user. >>>> _______________________________________________ >>>> Kbase-core mailing list >>>> Kbase-core(a)lists.kbase.us >>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>>> >> >>

1 0

Re: [Kbase-core] Globus online
by Dan Olson 24 Apr '13

24 Apr '13

That's correct, the dtn's aren't visible to the service machines. I think the easiest way imho to enable that integration is through: a. a shock fuse mount on the dtn b. a dtn "mailbox" to shock (which we've used before on the ornl variation service) c. a gridftp plugin for shock Maybe I'm not seeing GCMU in the same light, how would you envision a file transfer through GCMU getting to a service? ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Wednesday, April 24, 2013 3:15:04 PM > Subject: Re: [Kbase-core] Globus online > > I mentioned gcmu because as far as I know the dtn isn't visible from > any of the service machines, is it? Say I upload my 4GB sample file > via GO; how does my service see it to process it? > > On Apr 24, 2013, at 3:10 PM, Dan Olson wrote: > > > We do have the dtn's defined in GO. kbase-chicago, and > > kbase-knoxville are defined. > > > > In order to use shock directly we need to figure out the optimal > > way to display the > > shock data as a filesystem abstraction. I think this should be a > > priority for the > > shock developers. > > > > GCMU isn't required to use globus connect. You can fire up globus > > connect and transfer > > data to the kbase DTNs today. However, you have to use your > > developer credentials to > > auth. > > > > We are trying to get GCMU installed on some KBase endpoints to > > support richer user-user > > sharing of data. The globus connect to standard gridftp interface > > just puts the data > > down on the filesystem with no good way to link it to accounts > > outside of unix perms > > on the DTN. > > > > ----- Original Message ----- > >> From: "Robert Olson" <olson(a)mcs.anl.gov> > >> To: kbase-core(a)lists.kbase.us > >> Sent: Wednesday, April 24, 2013 2:55:13 PM > >> Subject: [Kbase-core] Globus online > >> > >> > >> Do we have any KBase Globus Online endpoints defined? It seems it > >> would make sense to have ADM storage visible via both Globus > >> Online > >> and Shock. Since Shock doesn't supply any real upload support (the > >> client application doesn't seem to do much from my read of it) it > >> would make sense to utilize GO to provide high performance upload. > >> > >> > >> With Globus Connect (a client-side application that appears to > >> integrate nicely with the Mac at least), we could enable high > >> performance up/downloads between user systems and KBase. > >> > >> > >> This seems related - globus connect multiuser > >> https://www.globusonline.org/gcmu/ and would be required on the KB > >> transfer node (it requires firewall be opened). > >> > >> > >> The context of this is enabling real-life large file upload for KB > >> users without forcing them to handle the huge details required > >> since > >> the Shock folks don't care to make things easy for the end user. > >> _______________________________________________ > >> Kbase-core mailing list > >> Kbase-core(a)lists.kbase.us > >> https://lists.kbase.us/mailman/listinfo/kbase-core > >> > >

1 0