Re: [Release-team] [Kbase-devel] help with pushing updated fba code into services/workspace

24 Feb 2014

      Nomi, 

We pushed these updates into production about an hour ago.  

We're still working through the problem reports to verify they are resolved.

----
Daniel Murphy-Olson
Sr. Systems Administrator
Computing, Environment and Life Sciences Division
Argonne National Laboratory
630-252-0055

________________________________________
From: Nomi Harris [[email protected]]
Sent: Monday, February 24, 2014 12:23 PM
To: Murphy-Olson, Daniel E.
Cc: Henry, Christopher S.; Stevens, Rick L.; [email protected]
Subject: Re: [Kbase-devel] help with pushing updated fba code into services/workspace

Hi Dan,

Do you have an estimate about when the ability to load genome into the new ws (so that one can run FBA modeling) will be restored? This has been broken for several weeks now, and it is a very popular KBase service (and is also described in our IRIS Quick Start tutorial). We continue to get bug reports from external users about it (see, for example, https://atlassian.kbase.us/browse/KBASE-630 which came in this morning). If it’s going to take a while to fix, then I think we should post a notice about it on our home page and the Quick Start tutorial and possibly other places as well. (I would have done this sooner except that I thought it would be fixed Real Soon Now—I guess you probably did, too!)

Not trying to point any fingers—I know this is complicated to fix. I just think it would be better to let users know if this is going to be out of order for a while, since we keep getting questions about it.

Thanks,
        Nomi

On Feb 24, 2014, at 7:04 AM, Murphy-Olson, Daniel E. <[email protected]> wrote:
...
I just remounted the filesystem again.
I'm going to try to get a permanent fix in today for the filesystem issues.
----
Daniel Murphy-Olson
Sr. Systems Administrator
Computing, Environment and Life Sciences Division
Argonne National Laboratory
630-252-0055
________________________________________
From: Christopher Henry [[email protected]]
Sent: Monday, February 24, 2014 9:01 AM
To: Stevens, Rick L.
Cc: Shane Canon; Murphy-Olson, Daniel E.; kbase devel
Subject: Re: help with pushing updated fba code into services/workspace
Well, last time this happened yesterday, Dan was able to kick something and got the machine working again right away. Let's look for a note from dan. But you can also keep trying.
Chris
On Feb 24, 2014, at 8:59 AM, "Stevens, Rick L." <[email protected]> wrote:
...
Sound I keep trying to run it?
On Feb 24, 2014, at 8:45 AM, Christopher Henry <[email protected]> wrote:
...
Unfortunately, this is an issue with an intermittent instability in the file system of the shock server the workspace is connected to.
Dan fixed it over the weekend, but it's acting up again. It could also be related to a higher than usual load on the mongo. We need to check the logs to see what is inducing this.
I'll stop by Dan's office today and talk about deploying the bug-fixes… but that won't resolve this issue…
On Feb 24, 2014, at 8:42 AM, "Stevens, Rick L." <[email protected]> wrote:
...
Hi
I’m getting an error when trying to load genome into the new workspace (and also loading from a file).
I believe Chris has fixed the code, but that it needs a push to the production service.
Over the weekend chris fixed this and pointed me at his dev workspace.
I’d like to get the fix into the production infrastructure as soon as possible.
I have students that have projects going that need to use the new workspace and production tools.
Sorry for pressing folks early in the day, week etc.
thanks
—rick
Paleo:FBA stevens$ kbfba-loadgenome 'kb|g.0' -w RickBatch1 -e
JSONRPC error:
JSONRPC error:
IO Error during streaming of JsonNode: The doRead() threw exception. Use getCause() for details.
JSONRPC error code: -32500
JSONRPC error data:java.lang.RuntimeException: IO Error during streaming of JsonNode: The doRead() threw exception. Use getCause() for details.
   at us.kbase.workspace.database.mongo.ShockBackend.saveBlob(ShockBackend.java:162)
   at us.kbase.workspace.database.mongo.MongoWorkspaceDB.saveData(MongoWorkspaceDB.java:1965)
   at us.kbase.workspace.database.mongo.MongoWorkspaceDB.saveObjects(MongoWorkspaceDB.java:1651)
   at us.kbase.workspace.lib.Workspace.saveObjects(Workspace.java:571)
   at us.kbase.workspace.kbase.WorkspaceServerMethods.saveObjects(WorkspaceServerMethods.java:173)
   at us.kbase.workspace.WorkspaceServer.saveObjects(WorkspaceServer.java:532)
   at sun.reflect.GeneratedMethodAccessor778.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at us.kbase.common.service.JsonServerServlet.doPost(JsonServerServlet.java:310)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
   at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
   at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
   at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
   at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
   at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
   at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
   at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860)
   at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
   at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
   at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229)
   at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
   at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
   at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
   at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
   at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
   at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
   at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
   at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
   at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
   at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: The doRead() threw exception. Use getCause() for details.
   at com.gc.iotools.stream.os.OutputStreamToInputStream.internalClose(OutputStreamToInputStream.java:542)
   at com.gc.iotools.stream.os.OutputStreamToInputStream.close(OutputStreamToInputStream.java:409)
   at com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1083)
   at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:2720)
   at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2177)
   at us.kbase.workspace.database.mongo.ShockBackend.saveBlob(ShockBackend.java:160)
   ... 35 more
Caused by: us.kbase.workspace.database.mongo.exceptions.BlobStoreCommunicationException: Could not connect to the shock backend: Premature end of Content-Length delimited message body (expected: 139; received: 0
   at us.kbase.workspace.database.mongo.ShockBackend$1.doRead(ShockBackend.java:146)
   at us.kbase.workspace.database.mongo.ShockBackend$1.doRead(ShockBackend.java:129)
   at com.gc.iotools.stream.os.OutputStreamToInputStream$DataConsumer.call(OutputStreamToInputStream.java:116)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   ... 1 more
Caused by: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 139; received: 0
   at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:180)
   at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
   at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
   at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
   at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
   at java.io.InputStreamReader.read(InputStreamReader.java:184)
   at java.io.Reader.read(Reader.java:140)
   at org.apache.http.util.EntityUtils.toString(EntityUtils.java:244)
   at org.apache.http.util.EntityUtils.toString(EntityUtils.java:288)
   at us.kbase.shock.client.BasicShockClient.getShockData(BasicShockClient.java:207)
   at us.kbase.shock.client.BasicShockClient.processRequest(BasicShockClient.java:198)
   at us.kbase.shock.client.BasicShockClient._addNode(BasicShockClient.java:413)
   at us.kbase.shock.client.BasicShockClient._addNodeStreaming(BasicShockClient.java:426)
   at us.kbase.shock.client.BasicShockClient.addNode(BasicShockClient.java:357)
   at us.kbase.workspace.database.mongo.ShockBackend$1.doRead(ShockBackend.java:135)
   ... 6 more
Trace begun at /fbadev/KBaseDeploy/../KBaseClient/lib/Bio/KBase/workspace/Client.pm line 1456
Bio::KBase::workspace::Client::save_objects('Bio::KBase::workspace::Client=HASH(0x1603c0b0)', 'HASH(0x17e798a0)') called at /fbadev/KBaseDeploy/../KBaseFBAModeling/lib/Bio/KBase/ObjectAPI/KBaseStore.pm line 244
Bio::KBase::ObjectAPI::KBaseStore::save_objects('Bio::KBase::ObjectAPI::KBaseStore=HASH(0x8b105b0)', 'HASH(0x8b2cb68)') called at /fbadev/KBaseDeploy/../KBaseFBAModeling/lib/Bio/KBase/ObjectAPI/KBaseStore.pm line 197
Bio::KBase::ObjectAPI::KBaseStore::save_object('Bio::KBase::ObjectAPI::KBaseStore=HASH(0x8b105b0)', 'Bio::KBase::ObjectAPI::KBaseGenomes::Genome=HASH(0x17a44058)', 'RickBatch1/kb|g.0', undef) called at /fbadev/KBaseDeploy/../KBaseFBAModeling/lib/Bio/KBase/fbaModelServices/Impl.pm line 438
Bio::KBase::fbaModelServices::Impl::_save_msobject('Bio::KBase::fbaModelServices::Impl=HASH(0x85171d8)', 'Bio::KBase::ObjectAPI::KBaseGenomes::Genome=HASH(0x17a44058)', 'Genome', 'RickBatch1', 'kb|g.0') called at /fbadev/KBaseDeploy/../KBaseFBAModeling/lib/Bio/KBase/fbaModelServices/Impl.pm line 5104
Bio::KBase::fbaModelServices::Impl::genome_to_workspace('Bio::KBase::fbaModelServices::Impl=HASH(0x85171d8)', 'HASH(0x8b2ee58)') called at /fbadev/KBaseDeploy/../KBaseFBAModeling/lib/Bio/KBase/fbaModelServices/Server.pm line 396
eval {...} at /fbadev/KBaseDeploy/../KBaseFBAModeling/lib/Bio/KBase/fbaModelServices/Server.pm line 394

Murphy-Olson, Daniel E.

tags

participants (1)