Keith and I discussed it briefly with you and Bob during the build, and we thought we got the ok.  The only additions are a new Makefile target and a script- no existing invocation code was touched.  But perhaps this still warrants an official change request?

-michael

On Wed, May 22, 2013 at 10:45 AM, Thomas Brettin <Brettin@mcs.anl.gov> wrote:

--------
WARNING: At least one of the links in the message below goes to an IP address (e.g.
10.1.1.1), which could be malicious. To learn how to protect yourself, please go here:
http://www.lbl.gov/cyber/services/suspicious-links.html
--------

did you tell bob you were changing the invocation service make file and adding a new program

tom

On May 22, 2013, at 12:33 PM, Michael Sneddon <mwsneddon@lbl.gov> wrote:

It appears that there are problems with processing the COMMANDS file.

For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation.

For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script.  It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure.

Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well...

I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected.  This may help in solving the problem.

-michael

On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <landml@ornl.gov> wrote:

--------
WARNING: At least one of the links in the message below goes to an IP address (e.g.
10.1.1.1), which could be malicious. To learn how to protect yourself, please go here:
http://www.lbl.gov/cyber/services/suspicious-links.html
--------

Michael,

The following have a COMMANDS file:

assembly/COMMANDS
::::::::::::::
#group-name     arast       Assembly-RAST scripts
#command-set    scripts/(.*).pl arast

 communities_api/COMMANDS
::::::::::::::
#group-name mgapi Metagenomics Scripts
#group-set scripts/(.*).pl mgapi

communities_qc/COMMANDS
::::::::::::::
#group-name qc Sequence QC
#group-set scripts/(.*) qc

jnomics/COMMANDS
::::::::::::::
#group-name variation Variation Detection Scripts
#command-set bin/(jk_*) variation ****************** Nothing actually here

Miriam

From: Tom Brettin <brettin@mcs.anl.gov<mailto:brettin@mcs.anl.gov>>
Date: Tuesday, May 21, 2013 5:55 PM
To: Michael Sneddon <mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov>>
Cc: Miriam Land <landml@ornl.gov<mailto:landml@ornl.gov>>, Elizabeth Glass <marland@mcs.anl.gov<mailto:marland@mcs.anl.gov>>, Keith Keller <kkeller@lbl.gov<mailto:kkeller@lbl.gov>>, "release-team@kbase.us<mailto:release-team@kbase.us>" <release-team@kbase.us<mailto:release-team@kbase.us>>

Subject: Re: [Release-team] auto generation of command -h output

Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI

t

On May 21, 2013, at 2:18 PM, Michael Sneddon <mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov>> wrote:

Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands.  One of these steps could have an error.  What modules/scripts seem to be missing?

-michael

On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <landml@ornl.gov<mailto:landml@ornl.gov>> wrote:
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g.
10.1.1.1), which could be malicious. To learn how to protect yourself, please go here:
http://www.lbl.gov/cyber/services/suspicious-links.html
--------

There are other modules with a COMMANDS file.  Were only certain modules picked for this demonstration?

Miriam

From: Michael Sneddon <mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov><mailto:mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov>>>

Date: Tuesday, May 21, 2013 2:32 PM

To: Miriam Land <landml@ornl.gov<mailto:landml@ornl.gov><mailto:landml@ornl.gov<mailto:landml@ornl.gov>>>
Cc: Tom Brettin <brettin@mcs.anl.gov<mailto:brettin@mcs.anl.gov><mailto:brettin@mcs.anl.gov<mailto:brettin@mcs.anl.gov>>>, Elizabeth Glass <marland@mcs.anl.gov<mailto:marland@mcs.anl.gov><mailto:marland@mcs.anl.gov<mailto:marland@mcs.anl.gov>>>, Keith Keller <kkeller@lbl.gov<mailto:kkeller@lbl.gov><mailto:kkeller@lbl.gov<mailto:kkeller@lbl.gov>>>, "release-team@kbase.us<mailto:release-team@kbase.us><mailto:release-team@kbase.us<mailto:release-team@kbase.us>>" <release-team@kbase.us<mailto:release-team@kbase.us><mailto:release-team@kbase.us<mailto:release-team@kbase.us>>>

Subject: Re: [Release-team] auto generation of command -h output

Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with.  One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website.

-michael

On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <landml@ornl.gov<mailto:landml@ornl.gov><mailto:landml@ornl.gov<mailto:landml@ornl.gov>>> wrote:
--------
WARNING: At least one of the links in the message below goes to an IP address (e.g.
10.1.1.1), which could be malicious. To learn how to protect yourself, please go here:
http://www.lbl.gov/cyber/services/suspicious-links.html
--------

I noticed that there are differences between the modules.  There are some that:

 *   Are empty (phispy)
 *   Are one line and difficult to read (genopheno)
 *   Try to download a file instead of displaying content (sr)
 *   Are much more elaborate and complete than others (http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt)

It is going to be difficult for a script to compensate for all the variation.  Otherwise, good place to start.

Miriam

From: Michael Sneddon <mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov><mailto:mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov>><mailto:mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov><mailto:mwsneddon@lbl.gov<mailto:mwsneddon@lbl.gov>>>>

Date: Tuesday, May 21, 2013 2:06 PM

To: Tom Brettin <brettin@mcs.anl.gov<mailto:brettin@mcs.anl.gov><mailto:brettin@mcs.anl.gov<mailto:brettin@mcs.anl.gov>><mailto:brettin@mcs.anl.gov<mailto:brettin@mcs.anl.gov><mailto:brettin@mcs.anl.gov<mailto:brettin@mcs.anl.gov>>>>, Elizabeth Glass <marland@mcs.anl.gov<mailto:marland@mcs.anl.gov><mailto:marland@mcs.anl.gov<mailto:marland@mcs.anl.gov>><mailto:marland@mcs.anl.gov<mailto:marland@mcs.anl.gov><mailto:marland@mcs.anl.gov<mailto:marland@mcs.anl.gov>>>>
Cc: Keith Keller <kkeller@lbl.gov<mailto:kkeller@lbl.gov><mailto:kkeller@lbl.gov<mailto:kkeller@lbl.gov>><mailto:kkeller@lbl.gov<mailto:kkeller@lbl.gov><mailto:kkeller@lbl.gov<mailto:kkeller@lbl.gov>>>>, "release-team@kbase.us<mailto:release-team@kbase.us><mailto:release-team@kbase.us<mailto:release-team@kbase.us>><mailto:release-team@kbase.us<mailto:release-team@kbase.us><mailto:release-team@kbase.us<mailto:release-team@kbase.us>>>" <release-team@kbase.us<mailto:release-team@kbase.us><mailto:release-team@kbase.us<mailto:release-team@kbase.us>><mailto:release-team@kbase.us<mailto:release-team@kbase.us><mailto:release-team@kbase.us<mailto:release-team@kbase.us>>>>

Subject: [Release-team] auto generation of command -h output

During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file.  I put up an example of the output here: http://140.221.85.77/command-docs/

The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files.  In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared.  For each of these commands, there is a text file in the folder with the actual commands '-h' help output.

If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed.

Let me know if you have questions.  The format of this output is fairly easy to change.

For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target.  This output is placed in /kb/deployment/services/invocation/webroot/.  Perhaps we should start including this step by default whenever the invocation service is deployed?

-michael

<gen-valid-commands-output.txt>