Return-Path: X-Original-To: apmail-aurora-dev-archive@minotaur.apache.org Delivered-To: apmail-aurora-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 25FAF17DB5 for ; Tue, 31 Mar 2015 20:20:20 +0000 (UTC) Received: (qmail 64690 invoked by uid 500); 31 Mar 2015 20:20:20 -0000 Delivered-To: apmail-aurora-dev-archive@aurora.apache.org Received: (qmail 64634 invoked by uid 500); 31 Mar 2015 20:20:20 -0000 Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@aurora.apache.org Delivered-To: mailing list dev@aurora.apache.org Received: (qmail 64622 invoked by uid 99); 31 Mar 2015 20:20:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Mar 2015 20:20:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of josh@gmail.com designates 74.125.82.42 as permitted sender) Received: from [74.125.82.42] (HELO mail-wg0-f42.google.com) (74.125.82.42) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Mar 2015 20:20:14 +0000 Received: by wgra20 with SMTP id a20so31758612wgr.3 for ; Tue, 31 Mar 2015 13:18:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=xjZkyBmCfRYNdMILFgB8g4df9+crTbU/rQSpVM14B3M=; b=oeBBwJXnobrALI5CnyXpDbDCF/l+NCu9lxRHLbs0ooLWwrPl7t0VPW1ulPIXT7xRUg JZfTNkbjGgYzIgwaocJhnhEof/7Dt01fh0FROV9tDKTsFO8t/IlZG6ySL72PkVjnfgyE er00zET9r2XhKZsrC1JJ4iHPbq4PmuEy/LHnIeIg5FZaIxavC/GGVvnazPYj3/oIM0ii Ca1Rwc5LciwCK4+oeKRMK0IbcmiA7uEws08Oj2r7OCXSYeGXPl+2/v9sPRpxuU0Hca/V 3kDYWY+q39AJBVMq77BGZiGykBKlkUJlN892rzRVyMV9MaI6+FsCtya4qDTTtPEmljAi 7isw== X-Received: by 10.194.5.37 with SMTP id p5mr78093196wjp.20.1427833103032; Tue, 31 Mar 2015 13:18:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.192.198 with HTTP; Tue, 31 Mar 2015 13:18:02 -0700 (PDT) In-Reply-To: References: From: Josh Adams Date: Tue, 31 Mar 2015 13:18:02 -0700 Message-ID: Subject: Re: Error fetching job summaries: No error message returned by the scheduler To: dev@aurora.apache.org Content-Type: multipart/alternative; boundary=047d7b5d5736e7a86e05129b4c89 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b5d5736e7a86e05129b4c89 Content-Type: text/plain; charset=UTF-8 Slightly more information: Looking at what requests the browser is making to the scheduler api, I recreated this one from my Chrome Inspector and got the same response that Chrome did: $ curl -sd '[1,"getJobSummary",1,0,{"1":{"str":"produser"}}]' -X POST http://fsae17:8081/api; echo [1,"getJobSummary",2,0,{"0":{"rec":{"1":{"i32":2},"5":{"rec":{"1":{"str":"staging"},"2":{"i32":3},"3":{"str":""}}},"6":{"lst":["rec",1,{"1":{"str":"Cannot compare value: null"}}]}}}}] $ Josh On Tue, Mar 31, 2015 at 1:03 PM, Josh Adams wrote: > Hey Maxim, > > I forgot to mention that yesterday I killed all the running jobs in an > attempt to get it working again. When I run aurora.pex job list > // it just prints this out before handing me back a > terminal: > > INFO] Retrieving jobs for role produser > > Cheers, > Josh > > On Tue, Mar 31, 2015 at 12:54 PM, Maxim Khutornenko > wrote: > >> Hi Josh, >> >> How many jobs do you have under that role? Any chance you could run >> "aurora job status" for every one of them to see if you have any tasks >> with abnormal event sequences? Every task must have at least one event >> with a timestamp. Here is a reference example: >> >> events: >> 2015-03-23 01:01:28 PENDING: Rescheduled >> 2015-03-23 01:01:28 ASSIGNED: None >> 2015-03-23 01:01:30 STARTING: Initializing sandbox. >> 2015-03-23 01:01:31 RUNNING: None >> >> On Tue, Mar 31, 2015 at 12:17 PM, Josh Adams wrote: >> > Hi there, >> > >> > I'm getting this error when clicking on a Role on the /scheduler page. >> We >> > only have a single role and this is our staging cluster. It seems like >> > other operations like `aurora.pex job list ...` and similar commands >> work >> > fine, but I can't seem to figure out how to get the scheduler out of >> this >> > state so I can view the Job summary page. >> > >> > I've upgraded from Mesos/0.20.1 & Aurora/0.6.1 to Mesos/0.21.1 and >> > Aurora/0.7.1@3ab8a9e and the problem persists. Below is the stack trace >> > from the scheduler process. >> > >> > Thank you! >> > Josh >> > >> > D0331 19:14:31.229 THREAD2827 org.eclipse.jetty.server.Server.handle: >> > REQUEST /api on AsyncHttpConnection@6fa15c80 >> > ,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=362,c=43},r=4 >> > W0331 19:14:31.229 THREAD2828 >> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke: >> Uncaught >> > exception while handling getJobSummary(produser) >> > com.google.common.collect.Ordering$IncomparableValueException: Cannot >> > compare value: null >> > at >> > >> com.google.common.collect.ExplicitOrdering.rank(ExplicitOrdering.java:46) >> > at >> > >> com.google.common.collect.ExplicitOrdering.compare(ExplicitOrdering.java:40) >> > at >> > >> com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46) >> > at >> > >> com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43) >> > at com.google.common.collect.Ordering.max(Ordering.java:572) >> > at com.google.common.collect.Ordering.max(Ordering.java:539) >> > at com.google.common.collect.Ordering.max(Ordering.java:555) >> > at >> > >> org.apache.aurora.scheduler.base.Tasks.getLatestActiveTask(Tasks.java:215) >> > at >> > >> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:382) >> > at >> > >> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:374) >> > at com.google.common.collect.Maps$10.getValue(Maps.java:1855) >> > at java.util.HashMap.putAll(HashMap.java:647) >> > at >> > >> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobs(ReadOnlySchedulerImpl.java:373) >> > at >> > >> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobSummary(ReadOnlySchedulerImpl.java:257) >> > at >> > >> org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:461) >> > at >> > >> org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47) >> > at >> > >> org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38) >> > at >> > >> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102) >> > at >> > >> org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30) >> > at >> > >> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:906) >> > at >> > >> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:891) >> > at >> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) >> > at >> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >> > at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> > >> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >> > at >> > >> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >> > at >> > >> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69) >> > at >> > >> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) >> > at >> > org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >> > at >> > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >> > at >> > >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288) >> > at >> > >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) >> > at >> > >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) >> > at >> > >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) >> > at >> > >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) >> > at >> > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >> > at >> > >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) >> > at >> > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >> > at >> > >> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317) >> > at >> > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >> > at org.eclipse.jetty.server.Server.handle(Server.java:365) >> > at >> > >> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) >> > at >> > >> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944) >> > at >> > >> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005) >> > at >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) >> > at >> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) >> > at >> > >> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >> > at >> > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) >> > at >> > >> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >> > at >> > >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >> > at >> > >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >> > at java.lang.Thread.run(Thread.java:744) >> > > --047d7b5d5736e7a86e05129b4c89--