March 2014 - Release-team - mailman3.cels.anl.gov

Re: [Release-team] [Kbase-devel] Authenticated Shock access from service
by Wei Tang 07 Mar '14

07 Mar '14

Hi Shane The new token passing feature requires the version of AWE server and clients match. Thus it would be good to update the AWE server to the latest version so that people who use the latest client code can work with that server. Thanks Wei On Fri, Mar 7, 2014 at 10:07 AM, Shane Canon <scanon(a)lbl.gov> wrote: > > Should I go ahead and redeploy AWE in production? > > --Shane > > On Mar 7, 2014, at 7:59 AM, Jared Bischof <jbischof(a)gmail.com> wrote: > > Wei's going to send an example. Since AWE has many more features now than > the last time it was presented, it's probably a good time to go over the > status of AWE on a conference call. > > -Jared > > > On Fri, Mar 7, 2014 at 9:54 AM, Wei Tang <wtang(a)anl.gov> wrote: > >> by default it is: KB_AUTH_TOKEN >> >> On Fri, Mar 7, 2014 at 9:51 AM, Shane Canon <scanon(a)lbl.gov> wrote: >> >>> >>> Awesome. What is the variable name? >>> >>> Shane >>> >>> Sent from my iPhone >>> >>> On Mar 7, 2014, at 7:43 AM, Wei Tang <wtang(a)anl.gov> wrote: >>> >>> The feature that awe client sets kbase token as env variable before >>> dispatching the actual script has been implemented and committed, which >>> means AWE tasks can now access other kbase services using tokens previously >>> set at job submission. >>> >>> Thanks >>> Wei >>> >>> >>> On Tue, Mar 4, 2014 at 5:17 PM, Shane Canon <scanon(a)lbl.gov> wrote: >>> >>>> >>>> These sounds like good suggestions. I think it is important that AWE >>>> tasks should be able to easily interact with KBase services, especially >>>> workspace. But we should spend some >>>> time thinking about the best model for doing this. >>>> >>>> --Shane >>>> >>>> On Mar 4, 2014, at 2:49 PM, Wei Tang <wtang(a)anl.gov> wrote: >>>> >>>> Putting aside the security issue, AWE can be enhanced to support >>>> following: >>>> >>>> In the task description section of the workflow document, we add a >>>> field as ENV_VARS (along the cmd name/arg list), and let awe-client to set >>>> environmental variables based on that field before dispatch the actual >>>> script. In this way, we can pass needed token for certain tasks to access >>>> other kbase services (other than shock). This works for some scripts that >>>> retrieves token from environmental variables instead of command line args. >>>> The job submitter will need to define the needed token for each task in >>>> workflow template. >>>> >>>> Pls let me know if this feature is useful. We can add it to the feature >>>> list. >>>> >>>> Thanks >>>> Wei >>>> >>>> >>>> On Tue, Mar 4, 2014 at 4:33 PM, Jared Bischof <jbischof(a)gmail.com>wrote: >>>> >>>>> You could pass the KBase token as a command line argument for a given >>>>> task or we could even create another option for setting an environment >>>>> variable for a task. This isn't a secure way of handling things but I >>>>> haven't seen a better solution for this yet in the KBase infrastructure. >>>>> Suggestions are welcome. >>>>> >>>>> -Jared >>>>> >>>>> >>>>> On Tue, Mar 4, 2014 at 4:26 PM, Desai, Narayan <desai(a)mcs.anl.gov>wrote: >>>>> >>>>>> The task isn't exposed to the runtime environment for tasks, it is >>>>>> only used for data movement. We do all of this with restricted use tokens >>>>>> that are only good for a single shock object, so it wouldn't be useful for >>>>>> the workspace, etc. >>>>>> -nld >>>>>> >>>>>> On Mar 4, 2014, at 3:56 PM, Shane Canon wrote: >>>>>> >>>>>> >>>>>> Wei, >>>>>> >>>>>> If a worker task needed to use the token to access another KBase >>>>>> service (i.e. the workspace), how could the task access the token? Or is >>>>>> this allowed? >>>>>> >>>>>> --Shane >>>>>> >>>>>> On Feb 28, 2014, at 1:56 PM, Wei Tang <wtang(a)mcs.anl.gov> wrote: >>>>>> >>>>>> Yes, AWE can pass tokens to use authenticated shock, following is >>>>>> the how it currently working. >>>>>> >>>>>> When a AWE job is submitted and it need to use private shock data, >>>>>> one need to specify the shock token for that data in the header of the POST >>>>>> call for job submission. And the token (associated with the job) will be >>>>>> stored in mongodb at awe server. >>>>>> >>>>>> When a awe-client checks out a workunit from AWE, the workunit >>>>>> description tells whether this workunit needs token to get input data. if >>>>>> yes, the awe-client will issue another GET request to get the token from >>>>>> the AWE server, and continue to use that token to talk to shock, either >>>>>> for file downloading and uploading. >>>>>> >>>>>> Thanks >>>>>> Wei >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Feb 28, 2014 at 3:40 PM, Narayan Desai <desai(a)mcs.anl.gov>wrote: >>>>>> >>>>>>> Wei has done this for AWE. Wei? >>>>>>> -nld >>>>>>> >>>>>>> On Feb 28, 2014, at 2:47 PM, Olson, Robert D. wrote: >>>>>>> >>>>>>> Does anyone have a service backend working that accesses >>>>>>> authenticated Shock based on the credentials coming in from an >>>>>>> authenticated service call? I seem to recall that when Tom and I tried >>>>>>> this during our last bootcamp we had a lot of trouble with it. >>>>>>> >>>>>>> thanks, >>>>>>> --bob >>>>>>> _______________________________________________ >>>>>>> Kbase-devel mailing list >>>>>>> Kbase-devel(a)lists.kbase.us >>>>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Kbase-devel mailing list >>>>>>> Kbase-devel(a)lists.kbase.us >>>>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Kbase-devel mailing list >>>>>> Kbase-devel(a)lists.kbase.us >>>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Kbase-devel mailing list >>>>> Kbase-devel(a)lists.kbase.us >>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>>> >>>>> >>>> _______________________________________________ >>>> Kbase-devel mailing list >>>> Kbase-devel(a)lists.kbase.us >>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>> >>>> >>>> >>>> _______________________________________________ >>>> Kbase-devel mailing list >>>> Kbase-devel(a)lists.kbase.us >>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>> >>>> >>> >>> _______________________________________________ >>> Kbase-devel mailing list >>> Kbase-devel(a)lists.kbase.us >>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>> >>> >> > >

1 0

Re: [Release-team] [Kbase-devel] Authenticated Shock access from service
by Shane Canon 07 Mar '14

07 Mar '14

Should I go ahead and redeploy AWE in production? --Shane On Mar 7, 2014, at 7:59 AM, Jared Bischof <jbischof(a)gmail.com> wrote: > Wei's going to send an example. Since AWE has many more features now than the last time it was presented, it's probably a good time to go over the status of AWE on a conference call. > > -Jared > > > On Fri, Mar 7, 2014 at 9:54 AM, Wei Tang <wtang(a)anl.gov> wrote: > by default it is: KB_AUTH_TOKEN > > On Fri, Mar 7, 2014 at 9:51 AM, Shane Canon <scanon(a)lbl.gov> wrote: > > Awesome. What is the variable name? > > Shane > > Sent from my iPhone > > On Mar 7, 2014, at 7:43 AM, Wei Tang <wtang(a)anl.gov> wrote: > >> The feature that awe client sets kbase token as env variable before dispatching the actual script has been implemented and committed, which means AWE tasks can now access other kbase services using tokens previously set at job submission. >> >> Thanks >> Wei >> >> >> On Tue, Mar 4, 2014 at 5:17 PM, Shane Canon <scanon(a)lbl.gov> wrote: >> >> These sounds like good suggestions. I think it is important that AWE tasks should be able to easily interact with KBase services, especially workspace. But we should spend some >> time thinking about the best model for doing this. >> >> --Shane >> >> On Mar 4, 2014, at 2:49 PM, Wei Tang <wtang(a)anl.gov> wrote: >> >>> Putting aside the security issue, AWE can be enhanced to support following: >>> >>> In the task description section of the workflow document, we add a field as ENV_VARS (along the cmd name/arg list), and let awe-client to set environmental variables based on that field before dispatch the actual script. In this way, we can pass needed token for certain tasks to access other kbase services (other than shock). This works for some scripts that retrieves token from environmental variables instead of command line args. The job submitter will need to define the needed token for each task in workflow template. >>> >>> Pls let me know if this feature is useful. We can add it to the feature list. >>> >>> Thanks >>> Wei >>> >>> >>> On Tue, Mar 4, 2014 at 4:33 PM, Jared Bischof <jbischof(a)gmail.com> wrote: >>> You could pass the KBase token as a command line argument for a given task or we could even create another option for setting an environment variable for a task. This isn't a secure way of handling things but I haven't seen a better solution for this yet in the KBase infrastructure. Suggestions are welcome. >>> >>> -Jared >>> >>> >>> On Tue, Mar 4, 2014 at 4:26 PM, Desai, Narayan <desai(a)mcs.anl.gov> wrote: >>> The task isn't exposed to the runtime environment for tasks, it is only used for data movement. We do all of this with restricted use tokens that are only good for a single shock object, so it wouldn't be useful for the workspace, etc. >>> -nld >>> >>> On Mar 4, 2014, at 3:56 PM, Shane Canon wrote: >>> >>>> >>>> Wei, >>>> >>>> If a worker task needed to use the token to access another KBase service (i.e. the workspace), how could the task access the token? Or is this allowed? >>>> >>>> --Shane >>>> >>>> On Feb 28, 2014, at 1:56 PM, Wei Tang <wtang(a)mcs.anl.gov> wrote: >>>> >>>>> Yes, AWE can pass tokens to use authenticated shock, following is the how it currently working. >>>>> >>>>> When a AWE job is submitted and it need to use private shock data, one need to specify the shock token for that data in the header of the POST call for job submission. And the token (associated with the job) will be stored in mongodb at awe server. >>>>> >>>>> When a awe-client checks out a workunit from AWE, the workunit description tells whether this workunit needs token to get input data. if yes, the awe-client will issue another GET request to get the token from the AWE server, and continue to use that token to talk to shock, either for file downloading and uploading. >>>>> >>>>> Thanks >>>>> Wei >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Feb 28, 2014 at 3:40 PM, Narayan Desai <desai(a)mcs.anl.gov> wrote: >>>>> Wei has done this for AWE. Wei? >>>>> -nld >>>>> >>>>> On Feb 28, 2014, at 2:47 PM, Olson, Robert D. wrote: >>>>> >>>>>> Does anyone have a service backend working that accesses authenticated Shock based on the credentials coming in from an authenticated service call? I seem to recall that when Tom and I tried this during our last bootcamp we had a lot of trouble with it. >>>>>> >>>>>> thanks, >>>>>> --bob >>>>>> _______________________________________________ >>>>>> Kbase-devel mailing list >>>>>> Kbase-devel(a)lists.kbase.us >>>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>>> >>>>> >>>>> _______________________________________________ >>>>> Kbase-devel mailing list >>>>> Kbase-devel(a)lists.kbase.us >>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>>> >>>>> >>>>> _______________________________________________ >>>>> Kbase-devel mailing list >>>>> Kbase-devel(a)lists.kbase.us >>>>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>>> >>> >>> >>> >>> _______________________________________________ >>> Kbase-devel mailing list >>> Kbase-devel(a)lists.kbase.us >>> https://lists.kbase.us/mailman/listinfo/kbase-devel >>> >>> >>> _______________________________________________ >>> Kbase-devel mailing list >>> Kbase-devel(a)lists.kbase.us >>> https://lists.kbase.us/mailman/listinfo/kbase-devel >> >> >> _______________________________________________ >> Kbase-devel mailing list >> Kbase-devel(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-devel >> >> > > _______________________________________________ > Kbase-devel mailing list > Kbase-devel(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-devel > > >

1 0

Re: [Release-team] Release and dmg
by Shane Canon 05 Mar '14

05 Mar '14

I still need to update FBA. I will try to do that now. I haven't done anything to IRIS yet since I'm unclear what needs to be fixed and the deployment is a little more involved. --Shane On Mar 5, 2014, at 2:53 PM, Nomi Harris <nlharris(a)lbl.gov> wrote: > Bob Olson wrote (in response to https://atlassian.kbase.us/browse/KBASE-198) >> I am awaiting information from the release folks about building a new [dmg]. > > I think we’re almost ready for that. The FBA and workspace services are now working in IRIS—right? There are some issues with CBD and Shock (and a few other issues with some services), but I don’t think we have to wait for those to be fixed before putting together a new dmg. > > Thanks, > Nomi > _______________________________________________ > Release-team mailing list > Release-team(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/release-team

1 0

Release and dmg
by Nomi Harris 05 Mar '14

05 Mar '14

Bob Olson wrote (in response to https://atlassian.kbase.us/browse/KBASE-198) > I am awaiting information from the release folks about building a new [dmg]. I think we’re almost ready for that. The FBA and workspace services are now working in IRIS—right? There are some issues with CBD and Shock (and a few other issues with some services), but I don’t think we have to wait for those to be fixed before putting together a new dmg. Thanks, Nomi

1 0

Re: [Release-team] Empty or obsolete git repos--can we get rid of some of them?
by Nomi Harris 03 Mar '14

03 Mar '14

Fair enough. Does anyone want to defend any of the other listed repos before I send the list to dev? Nomi On Feb 25, 2014, at 2:38 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > strep_repeats is untouched because it is stable and working. > > > On Feb 25, 2014, at 4:20 PM, Nomi Harris <nlharris(a)lbl.gov> wrote: > >> By the way, while we’re talking about defunct services, there are some repos that we could probably get rid of. >> >> Here are repos that are empty: >> Empty: prinseq.git support%40kbase.us >> Empty: coexpression.git gang.fang%40yale.edu >> Empty: fat_cat.git dave.dineen%40gmail.com >> Empty: kmer_annotation_function.git olson%40mcs.anl.gov >> Empty: kbtypedef_common_proposals.git msneddon%40lbl.gov >> Empty: fbaModelData.git support%40kbase.us >> Empty: kbtyped_obj_service.git msneddon%40lbl.gov >> >> And here are some repos that haven’t been touched in 7 months or more (see https://git.kbase.us/?o=age) and may no longer be needed: >> >> structured_assertions.git repo for structured assertions... psdehal(a)lbl.gov 7 months ago >> genome_data_browser.git a kbase lab project to browse... wjriehl(a)lbl.gov 8 months ago >> registry.git "service registry" brettin(a)anl.gov 8 months ago >> seq_search.git sequence search nconrad(a)mcs.anl.gov 8 months ago >> # erdb_service.git service to query cs via erdb... gaprice(a)lbl.gov 9 months ago # erdb_service is apparently still in use >> user_page.git repo for a user page wjriehl(a)lbl.gov 9 months ago >> askkb.git natural language interface... disz 9 months ago >> data_scatter_plot.git js for displaying data in... psdehal(a)lbl.gov 9 months ago >> strep_repeats.git Unnamed repository; edit this... olson(a)mcs.anl.gov 9 months ago >> plant.git repository for all things... shiran(a)cshl.edu 10 months ago >> adapters.git adapters repo brettin(a)mcs.anl.gov 10 months ago >> network_viewer.git browser for networks jmchandonia(a)lbl.gov 11 months ago >> otu_correlation.git generate correlations between... wjriehl(a)lbl.gov 11 months ago >> exchange_format_specs.git data exchange formats for... disz(a)mcs.anl.gov 11 months ago >> regprecise.git things related to regulons... psnovichkov(a)lbl.gov 12 months ago >> prinseq.git one of Rob Edwards services support(a)kbase.us 13 months ago >> core_model.git old code for gui nconrad(a)mcs.anl.gov 13 months ago >> fbaModelData.git "new module for storing fba... support(a)kbase.us 13 months ago >> _______________________________________________ >> Release-team mailing list >> Release-team(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/release-team >

1 0

Metagenomic Sequence QC service--is it working again?
by Nomi Harris 03 Mar '14

03 Mar '14

Can someone update me on the status of the Metagenomic Sequence QC service? It is visible in IRIS, and I can run the commands with —help but I don’t have the right input files to test it more thoroughly. Miriam, could you do that? We had taken that service off the services page and taken down its tutorial (http://kbase.us/for-users/tutorials/analyzing-data/metagenome-analysis/sequ…) because the service wasn’t working. I need to know if those should be restored. Thanks, Nomi

1 0