Hi Stephen and Marcin and Doreen, good discussion so far. thanks for your comments. here are a couple of additional comments/explanations or more food for thought: - MetaZen is indeed a frontend to create spreadsheets (and prefill some fields) - The "schema" we devised is indeed implemented in a database allowing the queries. I use "schema" because it is a very lightweight solution more an overlay really. - Before we ingest the metadata files, we also validate them. Requiring the user to provide at least the minimal data in correct form and with the correct CVs - The main problem we are trying to solve is allowing hundreds of users per month to submit data accompanied by metadata into MG-RAST - We don't have the manpower to manually scrutinized the data sets submitted by users, and I don't think that is a game we want to get into in KBase. So we need to provide automated solutions with automated validation where possible. - I believe the use case that we are aiming for with KBase is to be the primary location that users upload their data to for analysis, while we will continue to automatically ingest data from third parties. I believe in the long run these (e.g. GEO) will be a special case (IFF we are successful), there is simply too much new data to worry about. Legacy data is not what we should spend too much time worrying about, in 2-3 years there will be thousands of data sets of each given type. - Many users find metadata annoying and don't like to provide it. Our approach has been to tell them they get better service if they provide the metadata and also provide tools that make minimal metadata easy --> Metazen. - Marcin is right that richer metadata is important (this is solved via the "environmental packages") and the notion of different levels of metadata. Richness and ease of use are somewhat opposite goals, Doreen points out that we will have to coerce our users to do the right thing. One way might be to have tools in KBase that require certain sets of metadata to work e.g. the use of certain CVs for experimental conditions. - We probably should go for multiple levels of metadata completeness, encouraging the users to add as much as they can, while requiring a minimal subset. At least that is what we did in MG-RAST. - regarding Stephen's question about a single integrated query capability: the "schema" we use links IDs to metadata. I believe we need the ability to link all KBase objects (from narrative to metagenome reads etc.) to metadata. With many different lightweight "schemas" used along the way. If you look at the way the "schemata" are implemented, you will see they are suited for this. I'd say a single KBase ID needs to link to the only one object irrespective of location in KBase. - in my mind the search integration is where the metadata would be queried most of the time, but I can also see lots of tools that would benefit from CVs etc. Data discovery is just one area were we are a little bit stuck w/o metadata currently I think one key question we have to ask ourselves is: Do we want to work with the international standards community (e.g. GSC) and expand on their "templates" or go our own way? With MG-RAST we have chosen to work with the community (and are now part of this community), evolving standards and CVs where needed (sometimes too slow). For us that has produced the situation where users are happy to provide minimal data and many are asking for more metadata. We basically get the same number of complaints about too little and too much metadata --> must mean we did the right thing. ;-) BTW: One important aspect of our solution is that we can capture arbitrary metadata, it is just not displayed in the controlled subset. It can all be queried in the database. Let me what you think, this is just my $0.02 Best, Folker PS: There is a new emerging standard for Biolog type experiments that is just now being established. We (as KBase) might want to get involved in this? On Mar 14, 2013, at 11:58 AM, Stephen Chan <[email protected]> wrote:

Hi Folker, Thanks for sending this out - a quick read of the document indicates that Metazen is a tool to generate a metadata spreadsheet that is shipped around with the main datafiles. It seems to be a good start at a standalone app for that purpose. Is there a backend service that consumes the metadata file and makes it queryable?

It seems to me that we would want to have metadata query-able across the different data stores (in much the same way that search operates). This would indicate that we need to either have a single central metadata service, and/or else federate the metadata across the datastores and enforce some level of conformity across them.

If we have an overarching metadata service or even a federated metadata service, the other question that immediately comes up is having single namespace that allows us to reference kbase objects across the different datastores. For example, how do we distinguish between something in the workspace that is called "kb|g.1716.peg.3960" versus something in the cdmi called "kb|g.1716.peg.3960" versus a file in Shock that may be named the same? Do we use some kind of URI based approach, such as shock://kbase.us/kb|g.1716.peg.3960 and workspace://kbase.us/kb|g.1716.peg.3960 ? We can't really provide an JSON-RPC url that is a accessor for the object the way we might with a REST api.

Anyway - just some more grist for the mill.

Steve

On Thu, Mar 14, 2013 at 8:53 AM, Folker Meyer <[email protected]> wrote:

...

Hi all,

we have had several emails/discussions with several groups now on metadata in KBase. Now a couple of people are talking about using the API to store metadata.

There is a lot of prior discussion of metadata and metadata standards outside KBase and I think it is important to find a solution that integrates well with the external standards community. And in addition, rather than having a solution creep into place, I think it would be good to get in front of this and define what it is that we (as KBase) want.

As a result I mentioned in a recent discussion the fact that we have an implementation of this in MG-RAST that might be worth looking at. As a punishment for this I was asked to put together a document outlining our solution for this in MG-RAST (see attached).

There are some questions that I see us faced with:

- do we need a single metadata strategy for KBase - what aspects of metadata are required for each domain - what type of solution are people comfortable with - what are our customers comfortable with

I am sure I am missing tons of stuff that we need to think about and agree on a solution for, but hopefully this will be food for thought.

Best, Folker

_______________________________________________ Kbase-devel mailing list [email protected] https://lists.kbase.us/mailman/listinfo/kbase-devel