asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abdullah alamoudi <bamou...@gmail.com>
Subject Re: Asterix Cluster Global recovery is not yet complete and The system is in ACTIVE state
Date Thu, 24 Dec 2015 15:25:50 GMT
Siripen,
This situation can happen in the following scenario:

1. The cluster is active and everything works fine.
2. One or more node controllers get disconnected. This sends a signal to
the GlobalRecoveryManager.
3. GlobalRecoveryManager sets a flag that when the cluster becomes active,
it should run global recovery to clean up hanging global transactions.
4. Nodes which left the cluster join again.
5. The cluster becomes active. This should send a signal to
GlobalRecoveryManager to start global recovery and allow queries to be
executed. Someone disabled this and so GlobalRecoveryManager stops all
queries.

I am going to investigate this and fix it very soon. For the time being,
the only way to handle such situations is to restart the instance.

Cheers,
Abdullah.

Amoudi, Abdullah.

On Thu, Dec 24, 2015 at 5:47 AM, Siripen Pongpaichet <spongpai@gmail.com>
wrote:

> Hi all,
>
> I got this following error when I tried to call the REST api. Could anybody
> please help? I included part of the log files at the end. Thank you very
> much.
>
>
> {
>     •    "summary": " Asterix Cluster Global recovery is not yet complete
> and The system is in ACTIVE state\n caused by:
>
> edu.uci.ics.asterix.translator.AbstractAqlTranslator.validateOperation(AbstractAqlTranslator.java:97)",
>     •    "error-code":
>     •    [
>     ◦    99,
>     ◦    " Asterix Cluster Global recovery is not yet complete and The
> system is in ACTIVE state [AsterixException]"
>     •    ],
>     •    "stacktrace":
> "edu.uci.ics.asterix.common.exceptions.AsterixException: Asterix Cluster
> Global recovery is not yet complete and The system is in ACTIVE state\n\tat
>
> edu.uci.ics.asterix.translator.AbstractAqlTranslator.validateOperation(AbstractAqlTranslator.java:97)\n\tat
>
> edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:257)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.handleRequest(RESTAPIServlet.java:156)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.handleRequest(QueryAPIServlet.java:25)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.doGet(RESTAPIServlet.java:129)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.doGet(QueryAPIServlet.java:25)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:734)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:847)\n\tat
>
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:347)\n\tat
>
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)\n\tat
>
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:907)\n\tat
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:562)\n\tat
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)\n\tat
>
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)\n\tat
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)\n\tat
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)\n\tat
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)\n\tat
> java.lang.Thread.run(Thread.java:745)\n"
> }
>
>
> I looked at the nodecontroller log. I found a lot of line with.
>
> OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError
> occurred dispatching signal SIGTERM to handler- the VM may need to be
> forcibly terminated
>
>
> Here is some log from cc.log
>
> Dec 23, 2015 6:12:20 PM
> edu.uci.ics.hyracks.control.common.dataset.ResultStateSweeper sweep
> INFO: Result state cleanup instance successfully completed.
> Dec 23, 2015 6:12:23 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":0.86,"nodeId":"my_asterix_node1","heap_usage":0.9068803669364529,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:12:37 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":1.05,"nodeId":"my_asterix_node1","heap_usage":0.9068707699501982,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:13:13 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":0.82,"nodeId":"my_asterix_node1","heap_usage":0.906874105845857,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:13:20 PM
> edu.uci.ics.hyracks.control.common.dataset.ResultStateSweeper sweep
> INFO: Result state cleanup instance successfully completed.
> Dec 23, 2015 6:13:47 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":1.41,"nodeId":"my_asterix_node1","heap_usage":0.9068897459998756,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:14:07 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":1.07,"nodeId":"my_asterix_node1","heap_usage":0.9068736071378377,"n_runtimes":0,"message-type":"NODE_REPORT"}
>
>
> Best,
> Siripen
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message