It appears that there are problems with processing the COMMANDS file. For "communities_api" and "communities_qc", the COMMANDS file is not properly formed- the module authors have used "group-set" as a keyword instead of "command-set", so no scripts are recognized by invocation. For "jnomics" and "assembly", the COMMANDS file looks like it was parsed correctly, but the files were not located by the "gen-valid-commands" script. It could be a problem with the regex provided in COMMANDS, or with handling the regex in the gen-valid-commands script, but I'm not sure. Therefore none of these scripts will appear in the command-docs listing or in the IRIS deploy as well... I've attached output from 'gen-valid-commands' which I modified slightly to print out the files that are detected. This may help in solving the problem. -michael On Wed, May 22, 2013 at 6:26 AM, Land, Miriam L. <[email protected]> wrote:

-------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html --------

Michael,

The following have a COMMANDS file:

assembly/COMMANDS :::::::::::::: #group-name arast Assembly-RAST scripts #command-set scripts/(.*).pl arast

communities_api/COMMANDS :::::::::::::: #group-name mgapi Metagenomics Scripts #group-set scripts/(.*).pl mgapi

communities_qc/COMMANDS :::::::::::::: #group-name qc Sequence QC #group-set scripts/(.*) qc

jnomics/COMMANDS :::::::::::::: #group-name variation Variation Detection Scripts #command-set bin/(jk_*) variation ****************** Nothing actually here

Miriam

From: Tom Brettin <[email protected]<mailto:[email protected]>> Date: Tuesday, May 21, 2013 5:55 PM To: Michael Sneddon <[email protected]<mailto:[email protected]>> Cc: Miriam Land <[email protected]<mailto:[email protected]>>, Elizabeth Glass <[email protected]<mailto:[email protected]>>, Keith Keller < [email protected]<mailto:[email protected]>>, "[email protected]<mailto: [email protected]>" <[email protected]<mailto: [email protected]>> Subject: Re: [Release-team] auto generation of command -h output

Not exactly sure what you guys are seeing, but for now I think we should take the position that we are only documenting those commands listed in COMMANDS files. IRIS users see these commands, and presumably these commands are our user facing CLI

t

On May 21, 2013, at 2:18 PM, Michael Sneddon <[email protected]<mailto: [email protected]>> wrote:

Hmm, it should have picked up all of them. I checked out the repos that contain non-empty COMMANDS files, and our code relies on an invocation service script to assemble the list of commands. One of these steps could have an error. What modules/scripts seem to be missing?

-michael

On Tue, May 21, 2013 at 11:46 AM, Land, Miriam L. <[email protected]<mailto: [email protected]>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html --------

There are other modules with a COMMANDS file. Were only certain modules picked for this demonstration?

Miriam

From: Michael Sneddon <[email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>>> Date: Tuesday, May 21, 2013 2:32 PM To: Miriam Land <[email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>>> Cc: Tom Brettin <[email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>>>, Elizabeth Glass < [email protected]<mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>>>, Keith Keller <[email protected]<mailto: [email protected]><mailto:[email protected]<mailto:[email protected]>>>, " [email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>>" < [email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>>> Subject: Re: [Release-team] auto generation of command -h output

Yeah, getting everyone to follow the same conventions is going to be a challenge, but at least we have something to work with. One of the goals of this exercise is to provide Outreach with at least an index of what commands are available in KBase so that we can start checking for accuracy/completeness and post some of this reference information to the website.

-michael

On Tue, May 21, 2013 at 11:20 AM, Land, Miriam L. <[email protected]<mailto: [email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: -------- WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: http://www.lbl.gov/cyber/services/suspicious-links.html --------

I noticed that there are differences between the modules. There are some that:

* Are empty (phispy) * Are one line and difficult to read (genopheno) * Try to download a file instead of displaying content (sr) * Are much more elaborate and complete than others ( http://140.221.85.77/command-docs/pe/coex_cluster.txt vs http://140.221.85.77/command-docs/genome_annotation/a_and_b.txt)

It is going to be difficult for a script to compensate for all the variation. Otherwise, good place to start.

Miriam

From: Michael Sneddon <[email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>><mailto:[email protected] <mailto:[email protected]><mailto:[email protected]<mailto: [email protected]>>>> Date: Tuesday, May 21, 2013 2:06 PM To: Tom Brettin <[email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>><mailto: [email protected]<mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>>>>, Elizabeth Glass <[email protected] <mailto:[email protected]><mailto:[email protected]<mailto: [email protected]>><mailto:[email protected]<mailto: [email protected]><mailto:[email protected]<mailto:[email protected]

...

...

...

...

Cc: Keith Keller <[email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>><mailto:[email protected]<mailto: [email protected]><mailto:[email protected]<mailto:[email protected]>>>>, " [email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>><mailto: [email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>>>" < [email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>><mailto: [email protected]<mailto:[email protected]><mailto: [email protected]<mailto:[email protected]>>>> Subject: [Release-team] auto generation of command -h output

During this last build meeting, Keith and I put together, using some of Bob's invocation code, a method for capturing the help output of the command-line scripts which service owners have declared using the COMMANDS file. I put up an example of the output here: http://140.221.85.77/command-docs/

The commands are grouped into folders which are named with the "group-name" declared in the COMMANDS files. In each of these folders, there is an 'index.txt' file which includes the verbose name of the set of commands on the first line, followed by a list of all the commands that were declared. For each of these commands, there is a text file in the folder with the actual commands '-h' help output.

If the text file for a command is empty, then for some reason the script with the '-h' option did not run successfully or no help was printed.

Let me know if you have questions. The format of this output is fairly easy to change.

For those on the release team, this help output generation can be invoked from the invocation service Makefile with the "deploy-command-docs" target. This output is placed in /kb/deployment/services/invocation/webroot/. Perhaps we should start including this step by default whenever the invocation service is deployed?

-michael