I like this from the interface perspective. My question is how global throughout the system would the both the type and value of $handle be? On Apr 24, 2013, at 4:31 PM, Robert Olson <[email protected]> wrote:
I'd want to use it bidirectionally.
The use case is simple. Consider a service that does a transformation on sequence data, like prinseq to trim sequences.
the naive version of the call in the type language would be
funcdef trim_sequence(FILE in, params...) returns(FILE out);
It's not clear what the FILE type would be. I think it makes sense to proxy it via some handle. So if we had an ADM service it might have a client library that works like this (NB client library, not a standard funcdef - you need the client to be in cahoots with the server to get efficient and reliable large file transfers):
my $adm = ADM->new(); my $handle = $adm->upload($input_file)
then you could pass around file handles:
my $svc = PrinSeq->new(); my $out_handle = $svc->trim_sequence($handle, params);
If you wished to download, you could:
$adm->download($out_handle, $local_filename)
Or pass the handle to another service;
my $go = GenomeAnnotation->new(); my $obj = $go->annotate_reads($out_handle);
etc.
On Apr 24, 2013, at 4:02 PM, Dan Olson wrote:
I see, that makes a lot of sense. With our proxy architecture, and desire to run multiple instances of some services having the DTN sit in the middle makes sense.
How about taking these steps: We define a globus user account for each KBase service. We install GCMU on a DTN-like machine (possibly with raid, right now the dtn's are raid0 - scratch). The data flows from a user to the GCMU endpoint, and they share it with the intended KBase service. The KBase service has a method to reach out using the GCMU client to pickup / dropoff data.
Would you want to use this bi-directionally, or just upload?
Since I'm currently working on getting this setup on a Magellan node, it would be fairly easy for me to also configure a dev invocation instance like this in parallel.
----- Original Message -----
From: "Robert Olson" <[email protected]> To: "Dan Olson" <[email protected]> Cc: [email protected] Sent: Wednesday, April 24, 2013 3:51:27 PM Subject: Re: [Kbase-core] Globus online
We'd run GCMU on the machine hosting the service. Then the file is local to the service or set of services that are apt to be using those files.
On Apr 24, 2013, at 3:45 PM, Dan Olson wrote:
That's correct, the dtn's aren't visible to the service machines.
I think the easiest way imho to enable that integration is through: a. a shock fuse mount on the dtn b. a dtn "mailbox" to shock (which we've used before on the ornl variation service) c. a gridftp plugin for shock
Maybe I'm not seeing GCMU in the same light, how would you envision a file transfer through GCMU getting to a service?
----- Original Message -----
From: "Robert Olson" <[email protected]> To: "Dan Olson" <[email protected]> Cc: [email protected] Sent: Wednesday, April 24, 2013 3:15:04 PM Subject: Re: [Kbase-core] Globus online
I mentioned gcmu because as far as I know the dtn isn't visible from any of the service machines, is it? Say I upload my 4GB sample file via GO; how does my service see it to process it?
On Apr 24, 2013, at 3:10 PM, Dan Olson wrote:
We do have the dtn's defined in GO. kbase-chicago, and kbase-knoxville are defined.
In order to use shock directly we need to figure out the optimal way to display the shock data as a filesystem abstraction. I think this should be a priority for the shock developers.
GCMU isn't required to use globus connect. You can fire up globus connect and transfer data to the kbase DTNs today. However, you have to use your developer credentials to auth.
We are trying to get GCMU installed on some KBase endpoints to support richer user-user sharing of data. The globus connect to standard gridftp interface just puts the data down on the filesystem with no good way to link it to accounts outside of unix perms on the DTN.
----- Original Message -----
From: "Robert Olson" <[email protected]> To: [email protected] Sent: Wednesday, April 24, 2013 2:55:13 PM Subject: [Kbase-core] Globus online
Do we have any KBase Globus Online endpoints defined? It seems it would make sense to have ADM storage visible via both Globus Online and Shock. Since Shock doesn't supply any real upload support (the client application doesn't seem to do much from my read of it) it would make sense to utilize GO to provide high performance upload.
With Globus Connect (a client-side application that appears to integrate nicely with the Mac at least), we could enable high performance up/downloads between user systems and KBase.
This seems related - globus connect multiuser https://www.globusonline.org/gcmu/ and would be required on the KB transfer node (it requires firewall be opened).
The context of this is enabling real-life large file upload for KB users without forcing them to handle the huge details required since the Shock folks don't care to make things easy for the end user. _______________________________________________ Kbase-core mailing list [email protected] https://lists.kbase.us/mailman/listinfo/kbase-core
_______________________________________________ Kbase-core mailing list [email protected] https://lists.kbase.us/mailman/listinfo/kbase-core