- Release-team - mailman3.cels.anl.gov

Re: [Release-team] auto generation of command -h output
by Land, Miriam L. 22 May '13

22 May '13

How much of a nag should I be when –h and —help are not valid options? For example: ----------TEST idserver external_ids_to_kbase_ids with --help option Unknown option: help usage: external_ids_to_kbase_ids [-c column] DB-name < input > output It contains usage information but it is non-standard. Miriam From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> Date: Wednesday, May 22, 2013 4:37 PM To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> Subject: Re: [Release-team] auto generation of command -h output Please tell the module owner to fix the problem, Also, we're seeing errors from the gen-comand-docs.pl program while deploying invocation service in production. Tom On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: It appears that there are problems with processing the COMMANDS file. For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. -michael On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- Michael, The following have a COMMANDS file: assembly/COMMANDS :::::::::::::: #group-name arast Assembly-RAST scripts #command-set scripts/(.*).pl arast communities_api/COMMANDS :::::::::::::: #group-name mgapi Metagenomics Scripts #group-set scripts/(.*).pl mgapi communities_qc/COMMANDS :::::::::::::: #group-name qc Sequence QC #group-set scripts/(.*) qc jnomics/COMMANDS :::::::::::::: #group-name variation Variation Detection Scripts #command-set bin/(jk_*) variation ****************** Nothing actually here Miriam From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Date: Tuesday, May 21, 2013 5:55 PM To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: Re: [Release-team] auto generation of command -h output Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI t On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? -michael On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Date: Tuesday, May 21, 2013 2:32 PM To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Subject: Re: [Release-team] auto generation of command -h output Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. -michael On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- I noticed that there are differences between the modules. There are some that: * Are empty (phispy) * Are one line and difficult to read (genopheno) * Try to download a file instead of displaying content (sr) * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Date: Tuesday, May 21, 2013 2:06 PM To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Subject: [Release-team] auto generation of command -h output During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. Let me know if you have questions. The format of this output is fairly easy to change. For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? -michael <gen-valid-commands-output.txt>

1 0

Re: [Release-team] auto generation of command -h output
by Thomas Brettin 22 May '13

22 May '13

Please tell the module owner to fix the problem, Also, we're seeing errors from the gen-comand-docs.pl program while deploying invocation service in production. Tom On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon(a)lbl.gov> wrote: > It appears that there are problems with processing the COMMANDS file. > > For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. > > For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. > > Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... > > I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. > > -michael > > > > > On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > Michael, > > The following have a COMMANDS file: > > assembly/COMMANDS > :::::::::::::: > #group-name arast Assembly-RAST scripts > #command-set scripts/(.*).pl arast > > communities_api/COMMANDS > :::::::::::::: > #group-name mgapi Metagenomics Scripts > #group-set scripts/(.*).pl mgapi > > communities_qc/COMMANDS > :::::::::::::: > #group-name qc Sequence QC > #group-set scripts/(.*) qc > > jnomics/COMMANDS > :::::::::::::: > #group-name variation Variation Detection Scripts > #command-set bin/(jk_*) variation ****************** Nothing actually here > > Miriam > > From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> > Date: Tuesday, May 21, 2013 5:55 PM > To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> > Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> > Subject: Re: [Release-team] auto generation of command -h output > > Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI > > t > > > On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: > > Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? > > -michael > > > On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Date: Tuesday, May 21, 2013 2:32 PM > To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Subject: Re: [Release-team] auto generation of command -h output > > Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. > > -michael > > > On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > I noticed that there are differences between the modules. There are some that: > > * Are empty (phispy) > * Are one line and difficult to read (genopheno) > * Try to download a file instead of displaying content (sr) > * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) > > It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> > Date: Tuesday, May 21, 2013 2:06 PM > To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> > Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> > Subject: [Release-team] auto generation of command -h output > > During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ > > The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. > > If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. > > Let me know if you have questions. The format of this output is fairly easy to change. > > For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? > > -michael > > > > > > > > <gen-valid-commands-output.txt>

1 0

Re: [Release-team] auto generation of command -h output
by Elizabeth M. Glass 22 May '13

22 May '13

Hi all, Sorry not to do this sooner - but was on travel. I am looping in Lynn Schriml who is reviewing the -h output as well. Elizabeth ----- Original Message ----- From: Miriam L. Land <landml(a)ornl.gov> To: Thomas Brettin <Brettin(a)mcs.anl.gov>, Michael Sneddon <mwsneddon(a)lbl.gov> Cc: Elizabeth M. Glass <marland(a)mcs.anl.gov>, Keith Keller <kkeller(a)lbl.gov>, release-team(a)kbase.us Sent: Wed, 22 May 2013 13:11:54 -0500 (CDT) Subject: Re: [Release-team] auto generation of command -h output Tom, I've been looking using the script capture the types of errors. Miriam From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> Date: Wednesday, May 22, 2013 2:06 PM To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> Subject: Re: [Release-team] auto generation of command -h output It probably does, once you are ready to commit the changes. Does the script return non-zero when it fails to execute the -h param? We can chat more about this later. Not a big deal rt. now t On May 22, 2013, at 12:52 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: Keith and I discussed it briefly with you and Bob during the build, and we thought we got the ok. The only additions are a new Makefile target and a script- no existing invocation code was touched. But perhaps this still warrants an official change request? -michael On Wed, May 22, 2013 at 10:45 AM, Thomas Brettin <Brettin(a)mcs.anl.gov<mailto:[email protected]>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- did you tell bob you were changing the invocation service make file and adding a new program tom On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: It appears that there are problems with processing the COMMANDS file. For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. -michael On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- Michael, The following have a COMMANDS file: assembly/COMMANDS :::::::::::::: #group-name arast Assembly-RAST scripts #command-set scripts/(.*).pl arast communities_api/COMMANDS :::::::::::::: #group-name mgapi Metagenomics Scripts #group-set scripts/(.*).pl mgapi communities_qc/COMMANDS :::::::::::::: #group-name qc Sequence QC #group-set scripts/(.*) qc jnomics/COMMANDS :::::::::::::: #group-name variation Variation Detection Scripts #command-set bin/(jk_*) variation ****************** Nothing actually here Miriam From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Date: Tuesday, May 21, 2013 5:55 PM To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: Re: [Release-team] auto generation of command -h output Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI t On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? -michael On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Date: Tuesday, May 21, 2013 2:32 PM To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Subject: Re: [Release-team] auto generation of command -h output Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. -michael On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- I noticed that there are differences between the modules. There are some that: * Are empty (phispy) * Are one line and difficult to read (genopheno) * Try to download a file instead of displaying content (sr) * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Date: Tuesday, May 21, 2013 2:06 PM To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Subject: [Release-team] auto generation of command -h output During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. Let me know if you have questions. The format of this output is fairly easy to change. For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? -michael <gen-valid-commands-output.txt>

1 0

Re: [Release-team] auto generation of command -h output
by Land, Miriam L. 22 May '13

22 May '13

Tom, I've been looking using the script capture the types of errors. Miriam From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> Date: Wednesday, May 22, 2013 2:06 PM To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> Subject: Re: [Release-team] auto generation of command -h output It probably does, once you are ready to commit the changes. Does the script return non-zero when it fails to execute the -h param? We can chat more about this later. Not a big deal rt. now t On May 22, 2013, at 12:52 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: Keith and I discussed it briefly with you and Bob during the build, and we thought we got the ok. The only additions are a new Makefile target and a script- no existing invocation code was touched. But perhaps this still warrants an official change request? -michael On Wed, May 22, 2013 at 10:45 AM, Thomas Brettin <Brettin(a)mcs.anl.gov<mailto:[email protected]>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- did you tell bob you were changing the invocation service make file and adding a new program tom On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: It appears that there are problems with processing the COMMANDS file. For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. -michael On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- Michael, The following have a COMMANDS file: assembly/COMMANDS :::::::::::::: #group-name arast Assembly-RAST scripts #command-set scripts/(.*).pl arast communities_api/COMMANDS :::::::::::::: #group-name mgapi Metagenomics Scripts #group-set scripts/(.*).pl mgapi communities_qc/COMMANDS :::::::::::::: #group-name qc Sequence QC #group-set scripts/(.*) qc jnomics/COMMANDS :::::::::::::: #group-name variation Variation Detection Scripts #command-set bin/(jk_*) variation ****************** Nothing actually here Miriam From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Date: Tuesday, May 21, 2013 5:55 PM To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: Re: [Release-team] auto generation of command -h output Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI t On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? -michael On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Date: Tuesday, May 21, 2013 2:32 PM To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Subject: Re: [Release-team] auto generation of command -h output Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. -michael On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- I noticed that there are differences between the modules. There are some that: * Are empty (phispy) * Are one line and difficult to read (genopheno) * Try to download a file instead of displaying content (sr) * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Date: Tuesday, May 21, 2013 2:06 PM To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>> Subject: [Release-team] auto generation of command -h output During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. Let me know if you have questions. The format of this output is fairly easy to change. For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? -michael <gen-valid-commands-output.txt>

1 0

Re: [Release-team] auto generation of command -h output
by Thomas Brettin 22 May '13

22 May '13

It probably does, once you are ready to commit the changes. Does the script return non-zero when it fails to execute the -h param? We can chat more about this later. Not a big deal rt. now t On May 22, 2013, at 12:52 PM, Michael Sneddon <mwsneddon(a)lbl.gov> wrote: > Keith and I discussed it briefly with you and Bob during the build, and we thought we got the ok. The only additions are a new Makefile target and a script- no existing invocation code was touched. But perhaps this still warrants an official change request? > > -michael > > > On Wed, May 22, 2013 at 10:45 AM, Thomas Brettin <Brettin(a)mcs.anl.gov> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > did you tell bob you were changing the invocation service make file and adding a new program > > tom > > > > On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon(a)lbl.gov> wrote: > >> It appears that there are problems with processing the COMMANDS file. >> >> For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. >> >> For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. >> >> Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... >> >> I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. >> >> -michael >> >> >> >> >> On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov> wrote: >> -------- >> WARNING: At least one of the links in the message below goes to an IP address (e.g. >> 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: >> http://www.lbl.gov/cyber/services/suspicious-links.html >> -------- >> >> Michael, >> >> The following have a COMMANDS file: >> >> assembly/COMMANDS >> :::::::::::::: >> #group-name arast Assembly-RAST scripts >> #command-set scripts/(.*).pl arast >> >> communities_api/COMMANDS >> :::::::::::::: >> #group-name mgapi Metagenomics Scripts >> #group-set scripts/(.*).pl mgapi >> >> communities_qc/COMMANDS >> :::::::::::::: >> #group-name qc Sequence QC >> #group-set scripts/(.*) qc >> >> jnomics/COMMANDS >> :::::::::::::: >> #group-name variation Variation Detection Scripts >> #command-set bin/(jk_*) variation ****************** Nothing actually here >> >> Miriam >> >> From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> >> Date: Tuesday, May 21, 2013 5:55 PM >> To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> >> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> >> Subject: Re: [Release-team] auto generation of command -h output >> >> Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI >> >> t >> >> >> On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: >> >> Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? >> >> -michael >> >> >> On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: >> -------- >> WARNING: At least one of the links in the message below goes to an IP address (e.g. >> 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: >> http://www.lbl.gov/cyber/services/suspicious-links.html >> -------- >> >> There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? >> >> Miriam >> >> From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> >> Date: Tuesday, May 21, 2013 2:32 PM >> To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> >> Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> >> Subject: Re: [Release-team] auto generation of command -h output >> >> Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. >> >> -michael >> >> >> On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: >> -------- >> WARNING: At least one of the links in the message below goes to an IP address (e.g. >> 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: >> http://www.lbl.gov/cyber/services/suspicious-links.html >> -------- >> >> I noticed that there are differences between the modules. There are some that: >> >> * Are empty (phispy) >> * Are one line and difficult to read (genopheno) >> * Try to download a file instead of displaying content (sr) >> * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) >> >> It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. >> >> Miriam >> >> From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> >> Date: Tuesday, May 21, 2013 2:06 PM >> To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> >> Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> >> Subject: [Release-team] auto generation of command -h output >> >> During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ >> >> The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. >> >> If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. >> >> Let me know if you have questions. The format of this output is fairly easy to change. >> >> For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? >> >> -michael >> >> >> >> >> >> >> >> <gen-valid-commands-output.txt> > >

1 0

Re: [Release-team] auto generation of command -h output
by Michael Sneddon 22 May '13

22 May '13

Keith and I discussed it briefly with you and Bob during the build, and we thought we got the ok. The only additions are a new Makefile target and a script- no existing invocation code was touched. But perhaps this still warrants an official change request? -michael On Wed, May 22, 2013 at 10:45 AM, Thomas Brettin <Brettin(a)mcs.anl.gov>wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP > address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, > please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > did you tell bob you were changing the invocation service make file and > adding a new program > > tom > > > > On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon(a)lbl.gov> wrote: > > It appears that there are problems with processing the COMMANDS file. > > For "communities_api" and "communities_qc", the COMMANDS file is not > properly formed- the module authors have used "group-set" as a keyword > instead of "command-set", so no scripts are recognized by invocation. > > For "jnomics" and "assembly", the COMMANDS file looks like it was parsed > correctly, but the files were not located by the "gen-valid-commands" > script. It could be a problem with the regex provided in COMMANDS, or with > handling the regex in the gen-valid-commands script, but I'm not sure. > > Therefore none of these scripts will appear in the command-docs listing or > in the IRIS deploy as well... > > I've attached output from 'gen-valid-commands' which I modified slightly > to print out the files that are detected. This may help in solving the > problem. > > -michael > > > > > On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov> wrote: > >> -------- >> WARNING: At least one of the links in the message below goes to an IP >> address (e.g. >> 10.1.1.1), which could be malicious. To learn how to protect yourself, >> please go here: >> http://www.lbl.gov/cyber/services/suspicious-links.html >> -------- >> >> Michael, >> >> The following have a COMMANDS file: >> >> assembly/COMMANDS >> :::::::::::::: >> #group-name arast Assembly-RAST scripts >> #command-set scripts/(.*).pl arast >> >> communities_api/COMMANDS >> :::::::::::::: >> #group-name mgapi Metagenomics Scripts >> #group-set scripts/(.*).pl mgapi >> >> communities_qc/COMMANDS >> :::::::::::::: >> #group-name qc Sequence QC >> #group-set scripts/(.*) qc >> >> jnomics/COMMANDS >> :::::::::::::: >> #group-name variation Variation Detection Scripts >> #command-set bin/(jk_*) variation ****************** Nothing actually here >> >> Miriam >> >> From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> >> Date: Tuesday, May 21, 2013 5:55 PM >> To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> >> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth >> Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller < >> kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto: >> release-team(a)kbase.us>" <release-team(a)kbase.us<mailto: >> release-team(a)kbase.us>> >> Subject: Re: [Release-team] auto generation of command -h output >> >> Not exactly sure what you guys are seeing, but for now I think we should >> take the position that we are only documenting those commands listed in >> COMMANDS files. IRIS users see these commands, and presumably these >> commands are our user facing CLI >> >> t >> >> >> On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto: >> mwsneddon(a)lbl.gov>> wrote: >> >> Hmm, it should have picked up all of them. I checked out the repos that >> contain non-empty COMMANDS files, and our code relies on an invocation >> service script to assemble the list of commands. One of these steps could >> have an error. What modules/scripts seem to be missing? >> >> -michael >> >> >> On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov >> <mailto:[email protected]>> wrote: >> -------- >> WARNING: At least one of the links in the message below goes to an IP >> address (e.g. >> 10.1.1.1), which could be malicious. To learn how to protect yourself, >> please go here: >> http://www.lbl.gov/cyber/services/suspicious-links.html >> -------- >> >> There are other modules with a COMMANDS file. Were only certain modules >> picked for this demonstration? >> >> Miriam >> >> From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected] >> ><mailto:[email protected]<mailto:[email protected]>>> >> Date: Tuesday, May 21, 2013 2:32 PM >> To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto: >> landml(a)ornl.gov<mailto:[email protected]>>> >> Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto: >> brettin(a)mcs.anl.gov<mailto:[email protected]>>>, Elizabeth Glass < >> marland(a)mcs.anl.gov<mailto:[email protected]><mailto: >> marland(a)mcs.anl.gov<mailto:[email protected]>>>, Keith Keller < >> kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto: >> kkeller(a)lbl.gov>>>, "release-team(a)kbase.us<mailto:[email protected] >> ><mailto:[email protected]<mailto:[email protected]>>" < >> release-team(a)kbase.us<mailto:[email protected]><mailto: >> release-team(a)kbase.us<mailto:[email protected]>>> >> Subject: Re: [Release-team] auto generation of command -h output >> >> Yeah, getting everyone to follow the same conventions is going to be a >> challenge, but at least we have something to work with. One of the goals >> of this exercise is to provide Outreach with at least an index of what >> commands are available in KBase so that we can start checking for >> accuracy/completeness and post some of this reference information to the >> website. >> >> -michael >> >> >> On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov >> <mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> >> wrote: >> -------- >> WARNING: At least one of the links in the message below goes to an IP >> address (e.g. >> 10.1.1.1), which could be malicious. To learn how to protect yourself, >> please go here: >> http://www.lbl.gov/cyber/services/suspicious-links.html >> -------- >> >> I noticed that there are differences between the modules. There are some >> that: >> >> * Are empty (phispy) >> * Are one line and difficult to read (genopheno) >> * Try to download a file instead of displaying content (sr) >> * Are much more elaborate and complete than others ( >> http://140.221.85.77/command-docs/pe/coex_cluster.txt vs >> http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) >> >> It is going to be difficult for a script to compensate for all the >> variation. Otherwise, good place to start. >> >> Miriam >> >> From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected] >> ><mailto:[email protected]<mailto:[email protected]>><mailto: >> mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected] >> <mailto:[email protected]>>>> >> Date: Tuesday, May 21, 2013 2:06 PM >> To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto: >> brettin(a)mcs.anl.gov<mailto:[email protected]>><mailto: >> brettin(a)mcs.anl.gov<mailto:[email protected]><mailto: >> brettin(a)mcs.anl.gov<mailto:[email protected]>>>>, Elizabeth Glass < >> marland(a)mcs.anl.gov<mailto:[email protected]><mailto: >> marland(a)mcs.anl.gov<mailto:[email protected]>><mailto: >> marland(a)mcs.anl.gov<mailto:[email protected]><mailto: >> marland(a)mcs.anl.gov<mailto:[email protected]>>>> >> Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto: >> kkeller(a)lbl.gov<mailto:[email protected]>><mailto:[email protected]<mailto: >> kkeller(a)lbl.gov><mailto:[email protected]<mailto:[email protected]>>>>, " >> release-team(a)kbase.us<mailto:[email protected]><mailto: >> release-team(a)kbase.us<mailto:[email protected]>><mailto: >> release-team(a)kbase.us<mailto:[email protected]><mailto: >> release-team(a)kbase.us<mailto:[email protected]>>>" < >> release-team(a)kbase.us<mailto:[email protected]><mailto: >> release-team(a)kbase.us<mailto:[email protected]>><mailto: >> release-team(a)kbase.us<mailto:[email protected]><mailto: >> release-team(a)kbase.us<mailto:[email protected]>>>> >> Subject: [Release-team] auto generation of command -h output >> >> During this last build meeting, Keith and I put together, using some of >> Bob's invocation code, a method for capturing the help output of the >> command-line scripts which service owners have declared using the COMMANDS >> file. I put up an example of the output here: >> http://140.221.85.77/command-docs/ >> >> The commands are grouped into folders which are named with the >> "group-name" declared in the COMMANDS files. In each of these folders, >> there is an 'index.txt' file which includes the verbose name of the set of >> commands on the first line, followed by a list of all the commands that >> were declared. For each of these commands, there is a text file in the >> folder with the actual commands '-h' help output. >> >> If the text file for a command is empty, then for some reason the script >> with the '-h' option did not run successfully or no help was printed. >> >> Let me know if you have questions. The format of this output is fairly >> easy to change. >> >> For those on the release team, this help output generation can be invoked >> from the invocation service Makefile with the "deploy-command-docs" target. >> This output is placed in /kb/deployment/services/invocation/webroot/. >> Perhaps we should start including this step by default whenever the >> invocation service is deployed? >> >> -michael >> >> >> >> >> >> >> > <gen-valid-commands-output.txt> > > >

1 0

Re: [Release-team] auto generation of command -h output
by Thomas Brettin 22 May '13

22 May '13

did you tell bob you were changing the invocation service make file and adding a new program tom On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon(a)lbl.gov> wrote: > It appears that there are problems with processing the COMMANDS file. > > For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. > > For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. > > Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... > > I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. > > -michael > > > > > On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > Michael, > > The following have a COMMANDS file: > > assembly/COMMANDS > :::::::::::::: > #group-name arast Assembly-RAST scripts > #command-set scripts/(.*).pl arast > > communities_api/COMMANDS > :::::::::::::: > #group-name mgapi Metagenomics Scripts > #group-set scripts/(.*).pl mgapi > > communities_qc/COMMANDS > :::::::::::::: > #group-name qc Sequence QC > #group-set scripts/(.*) qc > > jnomics/COMMANDS > :::::::::::::: > #group-name variation Variation Detection Scripts > #command-set bin/(jk_*) variation ****************** Nothing actually here > > Miriam > > From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> > Date: Tuesday, May 21, 2013 5:55 PM > To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> > Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> > Subject: Re: [Release-team] auto generation of command -h output > > Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI > > t > > > On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: > > Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? > > -michael > > > On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Date: Tuesday, May 21, 2013 2:32 PM > To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Subject: Re: [Release-team] auto generation of command -h output > > Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. > > -michael > > > On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > I noticed that there are differences between the modules. There are some that: > > * Are empty (phispy) > * Are one line and difficult to read (genopheno) > * Try to download a file instead of displaying content (sr) > * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) > > It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> > Date: Tuesday, May 21, 2013 2:06 PM > To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> > Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> > Subject: [Release-team] auto generation of command -h output > > During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ > > The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. > > If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. > > Let me know if you have questions. The format of this output is fairly easy to change. > > For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? > > -michael > > > > > > > > <gen-valid-commands-output.txt>

1 0

Re: [Release-team] auto generation of command -h output
by Michael Sneddon 22 May '13

22 May '13

It appears that there are problems with processing the COMMANDS file. For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. -michael On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml(a)ornl.gov> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP > address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, > please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > Michael, > > The following have a COMMANDS file: > > assembly/COMMANDS > :::::::::::::: > #group-name arast Assembly-RAST scripts > #command-set scripts/(.*).pl arast > > communities_api/COMMANDS > :::::::::::::: > #group-name mgapi Metagenomics Scripts > #group-set scripts/(.*).pl mgapi > > communities_qc/COMMANDS > :::::::::::::: > #group-name qc Sequence QC > #group-set scripts/(.*) qc > > jnomics/COMMANDS > :::::::::::::: > #group-name variation Variation Detection Scripts > #command-set bin/(jk_*) variation ****************** Nothing actually here > > Miriam > > From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> > Date: Tuesday, May 21, 2013 5:55 PM > To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> > Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth > Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller < > kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto: > release-team(a)kbase.us>" <release-team(a)kbase.us<mailto: > release-team(a)kbase.us>> > Subject: Re: [Release-team] auto generation of command -h output > > Not exactly sure what you guys are seeing, but for now I think we should > take the position that we are only documenting those commands listed in > COMMANDS files. IRIS users see these commands, and presumably these > commands are our user facing CLI > > t > > > On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto: > mwsneddon(a)lbl.gov>> wrote: > > Hmm, it should have picked up all of them. I checked out the repos that > contain non-empty COMMANDS files, and our code relies on an invocation > service script to assemble the list of commands. One of these steps could > have an error. What modules/scripts seem to be missing? > > -michael > > > On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto: > landml(a)ornl.gov>> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP > address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, > please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > There are other modules with a COMMANDS file. Were only certain modules > picked for this demonstration? > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto: > mwsneddon(a)lbl.gov<mailto:[email protected]>>> > Date: Tuesday, May 21, 2013 2:32 PM > To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto: > landml(a)ornl.gov<mailto:[email protected]>>> > Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto: > brettin(a)mcs.anl.gov<mailto:[email protected]>>>, Elizabeth Glass < > marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected] > <mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto: > kkeller(a)lbl.gov><mailto:[email protected]<mailto:[email protected]>>>, " > release-team(a)kbase.us<mailto:[email protected]><mailto: > release-team(a)kbase.us<mailto:[email protected]>>" < > release-team(a)kbase.us<mailto:[email protected]><mailto: > release-team(a)kbase.us<mailto:[email protected]>>> > Subject: Re: [Release-team] auto generation of command -h output > > Yeah, getting everyone to follow the same conventions is going to be a > challenge, but at least we have something to work with. One of the goals > of this exercise is to provide Outreach with at least an index of what > commands are available in KBase so that we can start checking for > accuracy/completeness and post some of this reference information to the > website. > > -michael > > > On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto: > landml(a)ornl.gov><mailto:[email protected]<mailto:[email protected]>>> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP > address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, > please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > I noticed that there are differences between the modules. There are some > that: > > * Are empty (phispy) > * Are one line and difficult to read (genopheno) > * Try to download a file instead of displaying content (sr) > * Are much more elaborate and complete than others ( > http://140.221.85.77/command-docs/pe/coex_cluster.txt vs > http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) > > It is going to be difficult for a script to compensate for all the > variation. Otherwise, good place to start. > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto: > mwsneddon(a)lbl.gov<mailto:[email protected]>><mailto:[email protected] > <mailto:[email protected]><mailto:[email protected]<mailto: > mwsneddon(a)lbl.gov>>>> > Date: Tuesday, May 21, 2013 2:06 PM > To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto: > brettin(a)mcs.anl.gov<mailto:[email protected]>><mailto: > brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected] > <mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov > <mailto:[email protected]><mailto:[email protected]<mailto: > marland(a)mcs.anl.gov>><mailto:[email protected]<mailto: > marland(a)mcs.anl.gov><mailto:[email protected]<mailto:[email protected] > >>>> > Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto: > kkeller(a)lbl.gov<mailto:[email protected]>><mailto:[email protected]<mailto: > kkeller(a)lbl.gov><mailto:[email protected]<mailto:[email protected]>>>>, " > release-team(a)kbase.us<mailto:[email protected]><mailto: > release-team(a)kbase.us<mailto:[email protected]>><mailto: > release-team(a)kbase.us<mailto:[email protected]><mailto: > release-team(a)kbase.us<mailto:[email protected]>>>" < > release-team(a)kbase.us<mailto:[email protected]><mailto: > release-team(a)kbase.us<mailto:[email protected]>><mailto: > release-team(a)kbase.us<mailto:[email protected]><mailto: > release-team(a)kbase.us<mailto:[email protected]>>>> > Subject: [Release-team] auto generation of command -h output > > During this last build meeting, Keith and I put together, using some of > Bob's invocation code, a method for capturing the help output of the > command-line scripts which service owners have declared using the COMMANDS > file. I put up an example of the output here: > http://140.221.85.77/command-docs/ > > The commands are grouped into folders which are named with the > "group-name" declared in the COMMANDS files. In each of these folders, > there is an 'index.txt' file which includes the verbose name of the set of > commands on the first line, followed by a list of all the commands that > were declared. For each of these commands, there is a text file in the > folder with the actual commands '-h' help output. > > If the text file for a command is empty, then for some reason the script > with the '-h' option did not run successfully or no help was printed. > > Let me know if you have questions. The format of this output is fairly > easy to change. > > For those on the release team, this help output generation can be invoked > from the invocation service Makefile with the "deploy-command-docs" target. > This output is placed in /kb/deployment/services/invocation/webroot/. > Perhaps we should start including this step by default whenever the > invocation service is deployed? > > -michael > > > > > > >

1 0

Re: [Release-team] auto generation of command -h output
by Land, Miriam L. 22 May '13

22 May '13

Michael, The following have a COMMANDS file: assembly/COMMANDS :::::::::::::: #group-name arast Assembly-RAST scripts #command-set scripts/(.*).pl arast communities_api/COMMANDS :::::::::::::: #group-name mgapi Metagenomics Scripts #group-set scripts/(.*).pl mgapi communities_qc/COMMANDS :::::::::::::: #group-name qc Sequence QC #group-set scripts/(.*) qc jnomics/COMMANDS :::::::::::::: #group-name variation Variation Detection Scripts #command-set bin/(jk_*) variation ****************** Nothing actually here Miriam From: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>> Date: Tuesday, May 21, 2013 5:55 PM To: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> Cc: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> Subject: Re: [Release-team] auto generation of command -h output Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI t On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> wrote: Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? -michael On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Date: Tuesday, May 21, 2013 2:32 PM To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: Re: [Release-team] auto generation of command -h output Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. -michael On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html -------- I noticed that there are differences between the modules. There are some that: * Are empty (phispy) * Are one line and difficult to read (genopheno) * Try to download a file instead of displaying content (sr) * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. Miriam From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Date: Tuesday, May 21, 2013 2:06 PM To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Subject: [Release-team] auto generation of command -h output During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. Let me know if you have questions. The format of this output is fairly easy to change. For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? -michael

1 0

Re: [Release-team] auto generation of command -h output
by Thomas Brettin 21 May '13

21 May '13

Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI t On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon(a)lbl.gov> wrote: > Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing? > > -michael > > > On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml(a)ornl.gov> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration? > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]>> > Date: Tuesday, May 21, 2013 2:32 PM > To: Miriam Land <landml(a)ornl.gov<mailto:[email protected]>> > Cc: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]>>, Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]>>, "release-team(a)kbase.us<mailto:[email protected]>" <release-team(a)kbase.us<mailto:[email protected]>> > Subject: Re: [Release-team] auto generation of command -h output > > Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website. > > -michael > > > On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml(a)ornl.gov<mailto:[email protected]>> wrote: > -------- > WARNING: At least one of the links in the message below goes to an IP address (e.g. > 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: > http://www.lbl.gov/cyber/services/suspicious-links.html > -------- > > I noticed that there are differences between the modules. There are some that: > > * Are empty (phispy) > * Are one line and difficult to read (genopheno) > * Try to download a file instead of displaying content (sr) > * Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt) > > It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start. > > Miriam > > From: Michael Sneddon <mwsneddon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Date: Tuesday, May 21, 2013 2:06 PM > To: Tom Brettin <brettin(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, Elizabeth Glass <marland(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Cc: Keith Keller <kkeller(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>, "release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <release-team(a)kbase.us<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Subject: [Release-team] auto generation of command -h output > > During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/ > > The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output. > > If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed. > > Let me know if you have questions. The format of this output is fairly easy to change. > > For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed? > > -michael > > > >

1 0