- Kbase-core - mailman3.cels.anl.gov

Re: [Kbase-core] Server URLs should not be set at deploy
by Christopher Henry 20 Jun '13

20 Jun '13

"Each service should be consuming the parameters set in the deployment.cfg at runtime. You shouldn't need to redeploy to modify those values." What I meant by "deploy-time" was that the server is started with a certain configuration, and once started the configuration is static… as you say, I'd like to set the configuration when I call a method in the service. "There are already some services whose command line scripts take a url argument" Absolutely. For example, all methods in the FBA service take a "wsurl" argument that lets the user specify which workspace they want to use when running their command. And it works very nicely… and it proves the principle. What I'm proposing is to have that formalized, scaled up, and applied KBase wide. On Jun 20, 2013, at 12:00 AM, Dan Olson <dolson(a)mcs.anl.gov> wrote: > Each service should be consuming the parameters set in the deployment.cfg at runtime. You shouldn't need to redeploy to modify those values. > > That being said, we should have a standard for specifying an alternate endpoint at the time you invoke a method. There are already some services whose command line scripts take a url argument, but that may be too simple for something that invokes multiple methods accessing multiple endpoints. > > > ----- Original Message ----- >> From: "Christopher Henry" <chrisshenry(a)gmail.com> >> To: kbase-core(a)lists.kbase.us >> Sent: Wednesday, June 19, 2013 11:44:17 PM >> Subject: [Kbase-core] Server URLs should not be set at deploy >> >> I'm growing increasingly concerned about the fact that we're setting >> the server inter-connects at deploy time instead of run-time. >> >> It's making deploy complex, and it's really reducing the flexibility >> we could have with our system infrastructure. >> >> I see two alternative options: >> >> Every API command takes a "configuration" has as optional input: >> { >> workspace_url => "http//kbase.us/services/workspace" >> fba_url => "http//kbase.us/services/fba" >> etc…. >> } >> >> There would be a default "production" config, but this would allow >> users to build their own custom config for use with development >> servers for example. >> >> Alternative option: >> >> We have a single "configuration" store where configurations like the >> one above can be generated and stored behind user-specified names. >> Then when I call a server command, I can tell it which >> "configuration" I'm using. It will talk to the config store and pull >> the URLs out of there. >> >> Just throwing it out there… not sure if this is what we have in mind >> with the service registry… but it's worth thinking about. Where this >> becomes really important is when you're working with an outside >> group to build a new service that interfaces with many other >> services… because then you may have to coordinate multiple >> development servers… and it gets tough. >> >> Chris >> >> >> _______________________________________________ >> Kbase-core mailing list >> Kbase-core(a)lists.kbase.us >> https://lists.kbase.us/mailman/listinfo/kbase-core >>

1 0

Re: [Kbase-core] Server URLs should not be set at deploy
by Dan Olson 20 Jun '13

20 Jun '13

Each service should be consuming the parameters set in the deployment.cfg at runtime. You shouldn't need to redeploy to modify those values. That being said, we should have a standard for specifying an alternate endpoint at the time you invoke a method. There are already some services whose command line scripts take a url argument, but that may be too simple for something that invokes multiple methods accessing multiple endpoints. ----- Original Message ----- > From: "Christopher Henry" <chrisshenry(a)gmail.com> > To: kbase-core(a)lists.kbase.us > Sent: Wednesday, June 19, 2013 11:44:17 PM > Subject: [Kbase-core] Server URLs should not be set at deploy > > I'm growing increasingly concerned about the fact that we're setting > the server inter-connects at deploy time instead of run-time. > > It's making deploy complex, and it's really reducing the flexibility > we could have with our system infrastructure. > > I see two alternative options: > > Every API command takes a "configuration" has as optional input: > { > workspace_url => "http//kbase.us/services/workspace" > fba_url => "http//kbase.us/services/fba" > etc…. > } > > There would be a default "production" config, but this would allow > users to build their own custom config for use with development > servers for example. > > Alternative option: > > We have a single "configuration" store where configurations like the > one above can be generated and stored behind user-specified names. > Then when I call a server command, I can tell it which > "configuration" I'm using. It will talk to the config store and pull > the URLs out of there. > > Just throwing it out there… not sure if this is what we have in mind > with the service registry… but it's worth thinking about. Where this > becomes really important is when you're working with an outside > group to build a new service that interfaces with many other > services… because then you may have to coordinate multiple > development servers… and it gets tough. > > Chris > > > _______________________________________________ > Kbase-core mailing list > Kbase-core(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-core >

1 0

Server URLs should not be set at deploy
by Christopher Henry 19 Jun '13

19 Jun '13

I'm growing increasingly concerned about the fact that we're setting the server inter-connects at deploy time instead of run-time. It's making deploy complex, and it's really reducing the flexibility we could have with our system infrastructure. I see two alternative options: Every API command takes a "configuration" has as optional input: { workspace_url => "http//kbase.us/services/workspace" fba_url => "http//kbase.us/services/fba" etc…. } There would be a default "production" config, but this would allow users to build their own custom config for use with development servers for example. Alternative option: We have a single "configuration" store where configurations like the one above can be generated and stored behind user-specified names. Then when I call a server command, I can tell it which "configuration" I'm using. It will talk to the config store and pull the URLs out of there. Just throwing it out there… not sure if this is what we have in mind with the service registry… but it's worth thinking about. Where this becomes really important is when you're working with an outside group to build a new service that interfaces with many other services… because then you may have to coordinate multiple development servers… and it gets tough. Chris

1 0

Re: [Kbase-core] shock
by Robert Olson 27 Apr '13

27 Apr '13

This is why I want to do the test. Right now we do this with the filesystem and seek indexes and it works great; folks keep proposing new tech thats better than anything currently done and I'd like to put it to the test. ----- Original Message ----- > Well, I would love to see the results of a field test. Can you > efficiently save down 1 million objects and then quickly query and > load up 4000 of them? > Of course… if the objects are going into the file system… consider > what happens when you try and read in 1000 files. If that's really > how it's implemented… forget about it. > Every file read will initiate a new disk access. But maybe I just > don't get the implementation. It's worth an experiment. > > I would personally love to see how mongodb and gridfs compares for > shock. So if we set up a test server somewhere, let me know. > > Chris > > On Apr 26, 2013, at 10:46 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > > > didn't you say you had 9TB? That data is just 160GB so should fit > > fine, right? > > > > I'm thinking of testing shock as an example of one of your object > > stores, since as I understand it they're putting it up as being an > > efficient data store for that sort of data. > > > > What I want to do with the data is retrieve arbitrary pieces of it > > efficiently. Can't ask for much more than that. > > > > ----- Original Message ----- > >> > >>> So say I wanted to store 40 million pieces data, 4K each. Is > >>> shock > >>> appropriate for that? > >> > >> Just storing the data, sure. I haven't tried this scenario, but I > >> haven't seen anything in shock that would prevent you from doing > >> this. But you could also store that data on thumbdrives, tapes, > >> or > >> cd-roms :) It really depends on what you want to do with the > >> data, > >> and what kind of availability and performance you need. > >> > >> Shock stores data on a filesystem at the end of the day, and the > >> last > >> time I checked it would only store it on one filesystem - on one > >> server. The last time I spoke with Jared Wilkening about this > >> last > >> year we were talking about sync methods for replicas and referrals > >> to other servers - so I know this was in the works. > >> > >> If you want to test shock at that scale, it's not difficult to > >> start > >> an instance following the readme. > >> I don't think we have anything with that much storage free for > >> trying > >> this out. > >> > >> ----- Original Message ----- > >>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>> To: "Dan Olson" <dolson(a)mcs.anl.gov> > >>> Cc: kbase-core(a)lists.kbase.us > >>> Sent: Friday, April 26, 2013 5:17:10 PM > >>> Subject: Re: [Kbase-core] shock > >>> > >>> Thanks. > >>> > >>> So say I wanted to store 40 million pieces data, 4K each. Is > >>> shock > >>> appropriate for that? > >>> > >>> ----- Original Message ----- > >>>> Yes, shock is running at kbase.us/services/shock and > >>>> kbase.us/services/shock-api > >>>> Globus creds should get you write access. There is 9.5T > >>>> available. > >>>> > >>>> Let me know if you need anything else to get things going. > >>>> > >>>> ----- Original Message ----- > >>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> > >>>>> To: kbase-core(a)lists.kbase.us > >>>>> Sent: Friday, April 26, 2013 3:07:37 PM > >>>>> Subject: [Kbase-core] shock > >>>>> > >>>>> Is there a production shock server running in KB land? Is there > >>>>> doc? > >>>>> I searched for "shock" on kbase.us and got no hits. > >>>>> > >>>>> I'd like to look into pushing all of the sequence data (contigs > >>>>> + > >>>>> features) into shock and using its range lookups for pulling > >>>>> sequence data chunks back out, the way we currently do with > >>>>> flat > >>>>> files in SEED and the file+offset+length indexes we build. > >>>>> > >>>>> --bob > >>>>> _______________________________________________ > >>>>> Kbase-core mailing list > >>>>> Kbase-core(a)lists.kbase.us > >>>>> https://lists.kbase.us/mailman/listinfo/kbase-core > >>>>> > >>>> > >>> > >> > > _______________________________________________ > > Kbase-core mailing list > > Kbase-core(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-core > >

1 0

Re: [Kbase-core] shock
by Dan Olson 27 Apr '13

27 Apr '13

About the size, yes. I was off by an order of magnitude. We have space for that. You should test it in a test environment first before Pushing that into the production instance. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Robert Olson <olson(a)mcs.anl.gov> wrote: > didn't you say you had 9TB? That data is just 160GB so should fit fine, right? > > I'm thinking of testing shock as an example of one of your object stores, since as I understand it they're putting it up as being an efficient data store for that sort of data. > > What I want to do with the data is retrieve arbitrary pieces of it efficiently. Can't ask for much more than that. > > ----- Original Message ----- > > > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > > appropriate for that? > > > > Just storing the data, sure. I haven't tried this scenario, but I > > haven't seen anything in shock that would prevent you from doing > > this. But you could also store that data on thumbdrives, tapes, or > > cd-roms :) It really depends on what you want to do with the data, > > and what kind of availability and performance you need. > > > > Shock stores data on a filesystem at the end of the day, and the last > > time I checked it would only store it on one filesystem - on one > > server. The last time I spoke with Jared Wilkening about this last > > year we were talking about sync methods for replicas and referrals > > to other servers - so I know this was in the works. > > > > If you want to test shock at that scale, it's not difficult to start > > an instance following the readme. > > I don't think we have anything with that much storage free for trying > > this out. > > > > ----- Original Message ----- > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > To: "Dan Olson" <dolson(a)mcs.anl.gov> > > > Cc: kbase-core(a)lists.kbase.us > > > Sent: Friday, April 26, 2013 5:17:10 PM > > > Subject: Re: [Kbase-core] shock > > > > > > Thanks. > > > > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > > appropriate for that? > > > > > > ----- Original Message ----- > > > > Yes, shock is running at kbase.us/services/shock and > > > > kbase.us/services/shock-api > > > > Globus creds should get you write access. There is 9.5T > > > > available. > > > > > > > > Let me know if you need anything else to get things going. > > > > > > > > ----- Original Message ----- > > > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > > > To: kbase-core(a)lists.kbase.us > > > > > Sent: Friday, April 26, 2013 3:07:37 PM > > > > > Subject: [Kbase-core] shock > > > > > > > > > > Is there a production shock server running in KB land? Is there > > > > > doc? > > > > > I searched for "shock" on kbase.us and got no hits. > > > > > > > > > > I'd like to look into pushing all of the sequence data (contigs > > > > > + > > > > > features) into shock and using its range lookups for pulling > > > > > sequence data chunks back out, the way we currently do with > > > > > flat > > > > > files in SEED and the file+offset+length indexes we build. > > > > > > > > > > --bob > > > > > _______________________________________________ > > > > > Kbase-core mailing list > > > > > Kbase-core(a)lists.kbase.us > > > > > https://lists.kbase.us/mailman/listinfo/kbase-core > > > > > > > > > > > > > >

1 0

Re: [Kbase-core] shock
by Christopher Henry 26 Apr '13

26 Apr '13

Well, I would love to see the results of a field test. Can you efficiently save down 1 million objects and then quickly query and load up 4000 of them? Of course… if the objects are going into the file system… consider what happens when you try and read in 1000 files. If that's really how it's implemented… forget about it. Every file read will initiate a new disk access. But maybe I just don't get the implementation. It's worth an experiment. I would personally love to see how mongodb and gridfs compares for shock. So if we set up a test server somewhere, let me know. Chris On Apr 26, 2013, at 10:46 PM, Robert Olson <olson(a)mcs.anl.gov> wrote: > didn't you say you had 9TB? That data is just 160GB so should fit fine, right? > > I'm thinking of testing shock as an example of one of your object stores, since as I understand it they're putting it up as being an efficient data store for that sort of data. > > What I want to do with the data is retrieve arbitrary pieces of it efficiently. Can't ask for much more than that. > > ----- Original Message ----- >> >>> So say I wanted to store 40 million pieces data, 4K each. Is shock >>> appropriate for that? >> >> Just storing the data, sure. I haven't tried this scenario, but I >> haven't seen anything in shock that would prevent you from doing >> this. But you could also store that data on thumbdrives, tapes, or >> cd-roms :) It really depends on what you want to do with the data, >> and what kind of availability and performance you need. >> >> Shock stores data on a filesystem at the end of the day, and the last >> time I checked it would only store it on one filesystem - on one >> server. The last time I spoke with Jared Wilkening about this last >> year we were talking about sync methods for replicas and referrals >> to other servers - so I know this was in the works. >> >> If you want to test shock at that scale, it's not difficult to start >> an instance following the readme. >> I don't think we have anything with that much storage free for trying >> this out. >> >> ----- Original Message ----- >>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>> To: "Dan Olson" <dolson(a)mcs.anl.gov> >>> Cc: kbase-core(a)lists.kbase.us >>> Sent: Friday, April 26, 2013 5:17:10 PM >>> Subject: Re: [Kbase-core] shock >>> >>> Thanks. >>> >>> So say I wanted to store 40 million pieces data, 4K each. Is shock >>> appropriate for that? >>> >>> ----- Original Message ----- >>>> Yes, shock is running at kbase.us/services/shock and >>>> kbase.us/services/shock-api >>>> Globus creds should get you write access. There is 9.5T >>>> available. >>>> >>>> Let me know if you need anything else to get things going. >>>> >>>> ----- Original Message ----- >>>>> From: "Robert Olson" <olson(a)mcs.anl.gov> >>>>> To: kbase-core(a)lists.kbase.us >>>>> Sent: Friday, April 26, 2013 3:07:37 PM >>>>> Subject: [Kbase-core] shock >>>>> >>>>> Is there a production shock server running in KB land? Is there >>>>> doc? >>>>> I searched for "shock" on kbase.us and got no hits. >>>>> >>>>> I'd like to look into pushing all of the sequence data (contigs >>>>> + >>>>> features) into shock and using its range lookups for pulling >>>>> sequence data chunks back out, the way we currently do with >>>>> flat >>>>> files in SEED and the file+offset+length indexes we build. >>>>> >>>>> --bob >>>>> _______________________________________________ >>>>> Kbase-core mailing list >>>>> Kbase-core(a)lists.kbase.us >>>>> https://lists.kbase.us/mailman/listinfo/kbase-core >>>>> >>>> >>> >> > _______________________________________________ > Kbase-core mailing list > Kbase-core(a)lists.kbase.us > https://lists.kbase.us/mailman/listinfo/kbase-core

1 0

Re: [Kbase-core] shock
by Robert Olson 26 Apr '13

26 Apr '13

didn't you say you had 9TB? That data is just 160GB so should fit fine, right? I'm thinking of testing shock as an example of one of your object stores, since as I understand it they're putting it up as being an efficient data store for that sort of data. What I want to do with the data is retrieve arbitrary pieces of it efficiently. Can't ask for much more than that. ----- Original Message ----- > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > appropriate for that? > > Just storing the data, sure. I haven't tried this scenario, but I > haven't seen anything in shock that would prevent you from doing > this. But you could also store that data on thumbdrives, tapes, or > cd-roms :) It really depends on what you want to do with the data, > and what kind of availability and performance you need. > > Shock stores data on a filesystem at the end of the day, and the last > time I checked it would only store it on one filesystem - on one > server. The last time I spoke with Jared Wilkening about this last > year we were talking about sync methods for replicas and referrals > to other servers - so I know this was in the works. > > If you want to test shock at that scale, it's not difficult to start > an instance following the readme. > I don't think we have anything with that much storage free for trying > this out. > > ----- Original Message ----- > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > To: "Dan Olson" <dolson(a)mcs.anl.gov> > > Cc: kbase-core(a)lists.kbase.us > > Sent: Friday, April 26, 2013 5:17:10 PM > > Subject: Re: [Kbase-core] shock > > > > Thanks. > > > > So say I wanted to store 40 million pieces data, 4K each. Is shock > > appropriate for that? > > > > ----- Original Message ----- > > > Yes, shock is running at kbase.us/services/shock and > > > kbase.us/services/shock-api > > > Globus creds should get you write access. There is 9.5T > > > available. > > > > > > Let me know if you need anything else to get things going. > > > > > > ----- Original Message ----- > > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > > To: kbase-core(a)lists.kbase.us > > > > Sent: Friday, April 26, 2013 3:07:37 PM > > > > Subject: [Kbase-core] shock > > > > > > > > Is there a production shock server running in KB land? Is there > > > > doc? > > > > I searched for "shock" on kbase.us and got no hits. > > > > > > > > I'd like to look into pushing all of the sequence data (contigs > > > > + > > > > features) into shock and using its range lookups for pulling > > > > sequence data chunks back out, the way we currently do with > > > > flat > > > > files in SEED and the file+offset+length indexes we build. > > > > > > > > --bob > > > > _______________________________________________ > > > > Kbase-core mailing list > > > > Kbase-core(a)lists.kbase.us > > > > https://lists.kbase.us/mailman/listinfo/kbase-core > > > > > > > > > >

1 0

Re: [Kbase-core] shock
by Dan Olson 26 Apr '13

26 Apr '13

> So say I wanted to store 40 million pieces data, 4K each. Is shock > appropriate for that? Just storing the data, sure. I haven't tried this scenario, but I haven't seen anything in shock that would prevent you from doing this. But you could also store that data on thumbdrives, tapes, or cd-roms :) It really depends on what you want to do with the data, and what kind of availability and performance you need. Shock stores data on a filesystem at the end of the day, and the last time I checked it would only store it on one filesystem - on one server. The last time I spoke with Jared Wilkening about this last year we were talking about sync methods for replicas and referrals to other servers - so I know this was in the works. If you want to test shock at that scale, it's not difficult to start an instance following the readme. I don't think we have anything with that much storage free for trying this out. ----- Original Message ----- > From: "Robert Olson" <olson(a)mcs.anl.gov> > To: "Dan Olson" <dolson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Friday, April 26, 2013 5:17:10 PM > Subject: Re: [Kbase-core] shock > > Thanks. > > So say I wanted to store 40 million pieces data, 4K each. Is shock > appropriate for that? > > ----- Original Message ----- > > Yes, shock is running at kbase.us/services/shock and > > kbase.us/services/shock-api > > Globus creds should get you write access. There is 9.5T available. > > > > Let me know if you need anything else to get things going. > > > > ----- Original Message ----- > > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > > To: kbase-core(a)lists.kbase.us > > > Sent: Friday, April 26, 2013 3:07:37 PM > > > Subject: [Kbase-core] shock > > > > > > Is there a production shock server running in KB land? Is there > > > doc? > > > I searched for "shock" on kbase.us and got no hits. > > > > > > I'd like to look into pushing all of the sequence data (contigs + > > > features) into shock and using its range lookups for pulling > > > sequence data chunks back out, the way we currently do with flat > > > files in SEED and the file+offset+length indexes we build. > > > > > > --bob > > > _______________________________________________ > > > Kbase-core mailing list > > > Kbase-core(a)lists.kbase.us > > > https://lists.kbase.us/mailman/listinfo/kbase-core > > > > > >

1 0

Re: [Kbase-core] shock
by Robert Olson 26 Apr '13

26 Apr '13

Thanks. So say I wanted to store 40 million pieces data, 4K each. Is shock appropriate for that? ----- Original Message ----- > Yes, shock is running at kbase.us/services/shock and > kbase.us/services/shock-api > Globus creds should get you write access. There is 9.5T available. > > Let me know if you need anything else to get things going. > > ----- Original Message ----- > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > To: kbase-core(a)lists.kbase.us > > Sent: Friday, April 26, 2013 3:07:37 PM > > Subject: [Kbase-core] shock > > > > Is there a production shock server running in KB land? Is there > > doc? > > I searched for "shock" on kbase.us and got no hits. > > > > I'd like to look into pushing all of the sequence data (contigs + > > features) into shock and using its range lookups for pulling > > sequence data chunks back out, the way we currently do with flat > > files in SEED and the file+offset+length indexes we build. > > > > --bob > > _______________________________________________ > > Kbase-core mailing list > > Kbase-core(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-core > > >

1 0

Re: [Kbase-core] shock
by Dan Olson 26 Apr '13

26 Apr '13

Oh.. The only doc I'm aware of is the readme, accesible https://github.com/MG-RAST/Shock ----- Original Message ----- > From: "Dan Olson" <dolson(a)mcs.anl.gov> > To: "Robert Olson" <olson(a)mcs.anl.gov> > Cc: kbase-core(a)lists.kbase.us > Sent: Friday, April 26, 2013 3:59:09 PM > Subject: Re: [Kbase-core] shock > > Yes, shock is running at kbase.us/services/shock and > kbase.us/services/shock-api > Globus creds should get you write access. There is 9.5T available. > > Let me know if you need anything else to get things going. > > ----- Original Message ----- > > From: "Robert Olson" <olson(a)mcs.anl.gov> > > To: kbase-core(a)lists.kbase.us > > Sent: Friday, April 26, 2013 3:07:37 PM > > Subject: [Kbase-core] shock > > > > Is there a production shock server running in KB land? Is there > > doc? > > I searched for "shock" on kbase.us and got no hits. > > > > I'd like to look into pushing all of the sequence data (contigs + > > features) into shock and using its range lookups for pulling > > sequence data chunks back out, the way we currently do with flat > > files in SEED and the file+offset+length indexes we build. > > > > --bob > > _______________________________________________ > > Kbase-core mailing list > > Kbase-core(a)lists.kbase.us > > https://lists.kbase.us/mailman/listinfo/kbase-core > > >

1 0