aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Sweeney (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AURORA-584) Storage is not READY uncaught exception during getJobSummary
Date Thu, 17 Jul 2014 01:19:04 GMT

    [ https://issues.apache.org/jira/browse/AURORA-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064456#comment-14064456
] 

Kevin Sweeney commented on AURORA-584:
--------------------------------------

This typically happens during scheduler startup (which is usually after a planned or unplanned
failover). Follower schedulers do redirect to the leader when they're aware of one (https://github.com/apache/incubator-aurora/blob/master/src/main/java/org/apache/aurora/scheduler/http/LeaderRedirectFilter.java).
The leader always announces itself in ZooKeeper (at -serverset_path) for service discovery
once it's up so a load balancer can look there (or some intermediary that knows how to speak
the service discovery protocol and the LB configuration protocol).

That said there's still a period of time during startup where the leader is still recovering
its storage and unable to service requests, which will always result in some version of these
errors. The best we can do in that case is indicate that the error is "retryable" with an
appropriate error code, as the scheduler will likely be able to service the request in the
future.

> Storage is not READY uncaught exception during getJobSummary
> ------------------------------------------------------------
>
>                 Key: AURORA-584
>                 URL: https://issues.apache.org/jira/browse/AURORA-584
>             Project: Aurora
>          Issue Type: Bug
>            Reporter: Jay Buffington
>
> When loading http://aurora-scheduler:8081/scheduler/jaybuff/devel/memcached I get the
following error in the response:
> {noformat}
> An error occurred when querying the server. Please reload this page.
> Error fetching tasks: No error message returned by the scheduler
> {noformat}
> I've seen this message enough to be annoying over the last month or two.  I'm currently
seeing it while running commit 84ee1571 (with a couple of simple and (hopefully) unrelated
branches applied to it).
> Generally I wait a minute and reload the /scheduler page a couple of times and the issue
goes away.
> tailing /var/log/aurora/aurora-scheduler.log on the scheduler I see the following stacktrace
at the time of the error:
> {noformat}
> 0716 16:30:15.008 THREAD140 org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
Uncaught exception while handling getJobSummary(jaybuff)
> org.apache.aurora.scheduler.storage.Storage$StorageException: Storage is not READY
> 	at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.checkInState(CallOrderEnforcingStorage.java:78)
> 	at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.weaklyConsistentRead(CallOrderEnforcingStorage.java:122)
> 	at org.apache.aurora.scheduler.storage.Storage$Util.weaklyConsistentFetchTasks(Storage.java:295)
> 	at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getTasks(SchedulerThriftInterface.java:584)
> 	at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:511)
> 	at org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:32)
> 	at org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
> 	at org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
> 	at org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:87)
> 	at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:723)
> 	at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:708)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> 	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> 	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> 	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
> 	at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
> 	at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
> 	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
> 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> 	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
> 	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
> 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> 	at org.mortbay.jetty.Server.handle(Server.java:326)
> 	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> 	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
> 	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
> 	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
> 	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> 	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> 	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message