aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Adams <j...@gmail.com>
Subject Re: Error fetching job summaries: No error message returned by the scheduler
Date Tue, 31 Mar 2015 20:18:02 GMT
Slightly more information:

Looking at what requests the browser is making to the scheduler api, I
recreated this one from my Chrome Inspector and got the same response that
Chrome did:

$ curl -sd '[1,"getJobSummary",1,0,{"1":{"str":"produser"}}]' -X POST
http://fsae17:8081/api; echo
[1,"getJobSummary",2,0,{"0":{"rec":{"1":{"i32":2},"5":{"rec":{"1":{"str":"staging"},"2":{"i32":3},"3":{"str":""}}},"6":{"lst":["rec",1,{"1":{"str":"Cannot
compare value: null"}}]}}}}]
$

Josh

On Tue, Mar 31, 2015 at 1:03 PM, Josh Adams <josh@gmail.com> wrote:

> Hey Maxim,
>
> I forgot to mention that yesterday I killed all the running jobs in an
> attempt to get it working again. When I run aurora.pex job list
> <cluster>/<role>/<env> it just prints this out before handing me back
a
> terminal:
>
> INFO] Retrieving jobs for role produser
>
> Cheers,
> Josh
>
> On Tue, Mar 31, 2015 at 12:54 PM, Maxim Khutornenko <maxim@apache.org>
> wrote:
>
>> Hi Josh,
>>
>> How many jobs do you have under that role? Any chance you could run
>> "aurora job status" for every one of them to see if you have any tasks
>> with abnormal event sequences? Every task must have at least one event
>> with a timestamp. Here is a reference example:
>>
>> events:
>>   2015-03-23 01:01:28 PENDING: Rescheduled
>>   2015-03-23 01:01:28 ASSIGNED: None
>>   2015-03-23 01:01:30 STARTING: Initializing sandbox.
>>   2015-03-23 01:01:31 RUNNING: None
>>
>> On Tue, Mar 31, 2015 at 12:17 PM, Josh Adams <josh@gmail.com> wrote:
>> > Hi there,
>> >
>> > I'm getting this error when clicking on a Role on the /scheduler page.
>> We
>> > only have a single role and this is our staging cluster. It seems like
>> > other operations like `aurora.pex job list ...` and similar commands
>> work
>> > fine, but I can't seem to figure out how to get the scheduler out of
>> this
>> > state so I can view the Job summary page.
>> >
>> > I've upgraded from Mesos/0.20.1 & Aurora/0.6.1 to Mesos/0.21.1 and
>> > Aurora/0.7.1@3ab8a9e and the problem persists. Below is the stack trace
>> > from the scheduler process.
>> >
>> > Thank you!
>> > Josh
>> >
>> > D0331 19:14:31.229 THREAD2827 org.eclipse.jetty.server.Server.handle:
>> > REQUEST /api on AsyncHttpConnection@6fa15c80
>> > ,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=362,c=43},r=4
>> > W0331 19:14:31.229 THREAD2828
>> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
>> Uncaught
>> > exception while handling getJobSummary(produser)
>> > com.google.common.collect.Ordering$IncomparableValueException: Cannot
>> > compare value: null
>> >         at
>> >
>> com.google.common.collect.ExplicitOrdering.rank(ExplicitOrdering.java:46)
>> >         at
>> >
>> com.google.common.collect.ExplicitOrdering.compare(ExplicitOrdering.java:40)
>> >         at
>> >
>> com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46)
>> >         at
>> >
>> com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43)
>> >         at com.google.common.collect.Ordering.max(Ordering.java:572)
>> >         at com.google.common.collect.Ordering.max(Ordering.java:539)
>> >         at com.google.common.collect.Ordering.max(Ordering.java:555)
>> >         at
>> >
>> org.apache.aurora.scheduler.base.Tasks.getLatestActiveTask(Tasks.java:215)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:382)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:374)
>> >         at com.google.common.collect.Maps$10.getValue(Maps.java:1855)
>> >         at java.util.HashMap.putAll(HashMap.java:647)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobs(ReadOnlySchedulerImpl.java:373)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobSummary(ReadOnlySchedulerImpl.java:257)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:461)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
>> >         at
>> >
>> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:906)
>> >         at
>> >
>> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:891)
>> >         at
>> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>> >         at
>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>> >         at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >         at
>> >
>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>> >         at
>> >
>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>> >         at
>> >
>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69)
>> >         at
>> >
>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
>> >         at
>> > org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>> >         at
>> > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>> >         at
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288)
>> >         at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
>> >         at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> >         at
>> >
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> >         at org.eclipse.jetty.server.Server.handle(Server.java:365)
>> >         at
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
>> >         at
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
>> >         at
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
>> >         at
>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
>> >         at
>> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>> >         at
>> >
>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> >         at
>> >
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
>> >         at
>> >
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> >         at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> >         at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> >         at java.lang.Thread.run(Thread.java:744)
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message