Return-Path: X-Original-To: apmail-aurora-issues-archive@minotaur.apache.org Delivered-To: apmail-aurora-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 946A7119C1 for ; Fri, 18 Jul 2014 00:19:32 +0000 (UTC) Received: (qmail 9864 invoked by uid 500); 18 Jul 2014 00:19:32 -0000 Delivered-To: apmail-aurora-issues-archive@aurora.apache.org Received: (qmail 9832 invoked by uid 500); 18 Jul 2014 00:19:32 -0000 Mailing-List: contact issues-help@aurora.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@aurora.incubator.apache.org Delivered-To: mailing list issues@aurora.incubator.apache.org Received: (qmail 9823 invoked by uid 99); 18 Jul 2014 00:19:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jul 2014 00:19:32 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 18 Jul 2014 00:19:31 +0000 Received: (qmail 8612 invoked by uid 99); 18 Jul 2014 00:19:04 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jul 2014 00:19:04 +0000 Date: Fri, 18 Jul 2014 00:19:04 +0000 (UTC) From: "Bill Farner (JIRA)" To: issues@aurora.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (AURORA-584) Storage is not READY uncaught exception during getJobSummary MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/AURORA-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065814#comment-14065814 ] Bill Farner commented on AURORA-584: ------------------------------------ [~kevints] discovered that we only issue an HTTP redirect for '/scheduler', but we should be doing so for all other pages under '/scheduler' as well. This is certainly a big part of the problem. > Storage is not READY uncaught exception during getJobSummary > ------------------------------------------------------------ > > Key: AURORA-584 > URL: https://issues.apache.org/jira/browse/AURORA-584 > Project: Aurora > Issue Type: Bug > Reporter: Jay Buffington > > When loading http://aurora-scheduler:8081/scheduler/jaybuff/devel/memcached I get the following error in the response: > {noformat} > An error occurred when querying the server. Please reload this page. > Error fetching tasks: No error message returned by the scheduler > {noformat} > I've seen this message enough to be annoying over the last month or two. I'm currently seeing it while running commit 84ee1571 (with a couple of simple and (hopefully) unrelated branches applied to it). > Generally I wait a minute and reload the /scheduler page a couple of times and the issue goes away. > tailing /var/log/aurora/aurora-scheduler.log on the scheduler I see the following stacktrace at the time of the error: > {noformat} > 0716 16:30:15.008 THREAD140 org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke: Uncaught exception while handling getJobSummary(jaybuff) > org.apache.aurora.scheduler.storage.Storage$StorageException: Storage is not READY > at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.checkInState(CallOrderEnforcingStorage.java:78) > at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.weaklyConsistentRead(CallOrderEnforcingStorage.java:122) > at org.apache.aurora.scheduler.storage.Storage$Util.weaklyConsistentFetchTasks(Storage.java:295) > at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getTasks(SchedulerThriftInterface.java:584) > at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:511) > at org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:32) > at org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47) > at org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38) > at org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:87) > at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:723) > at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:708) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78) > at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)