aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Khutornenko <ma...@apache.org>
Subject Re: Error fetching job summaries: No error message returned by the scheduler
Date Tue, 31 Mar 2015 20:27:48 GMT
If you remember your job names you should still be able to run "aurora
job status" though for as long as there is task history. Would be
great to know what caused that problem in the first place.

In case you want to unblock, you can temporarily change
-history_prune_threshold and let scheduler purge your task history. Be
careful as that will purge history for all jobs in your cluster, so
perhaps it's better wait for the scheduler to clean by itself (2 days
after your kill).

On Tue, Mar 31, 2015 at 1:03 PM, Josh Adams <josh@gmail.com> wrote:
> Hey Maxim,
>
> I forgot to mention that yesterday I killed all the running jobs in an
> attempt to get it working again. When I run aurora.pex job list
> <cluster>/<role>/<env> it just prints this out before handing me back
a
> terminal:
>
> INFO] Retrieving jobs for role produser
>
> Cheers,
> Josh
>
> On Tue, Mar 31, 2015 at 12:54 PM, Maxim Khutornenko <maxim@apache.org>
> wrote:
>
>> Hi Josh,
>>
>> How many jobs do you have under that role? Any chance you could run
>> "aurora job status" for every one of them to see if you have any tasks
>> with abnormal event sequences? Every task must have at least one event
>> with a timestamp. Here is a reference example:
>>
>> events:
>>   2015-03-23 01:01:28 PENDING: Rescheduled
>>   2015-03-23 01:01:28 ASSIGNED: None
>>   2015-03-23 01:01:30 STARTING: Initializing sandbox.
>>   2015-03-23 01:01:31 RUNNING: None
>>
>> On Tue, Mar 31, 2015 at 12:17 PM, Josh Adams <josh@gmail.com> wrote:
>> > Hi there,
>> >
>> > I'm getting this error when clicking on a Role on the /scheduler page. We
>> > only have a single role and this is our staging cluster. It seems like
>> > other operations like `aurora.pex job list ...` and similar commands work
>> > fine, but I can't seem to figure out how to get the scheduler out of this
>> > state so I can view the Job summary page.
>> >
>> > I've upgraded from Mesos/0.20.1 & Aurora/0.6.1 to Mesos/0.21.1 and
>> > Aurora/0.7.1@3ab8a9e and the problem persists. Below is the stack trace
>> > from the scheduler process.
>> >
>> > Thank you!
>> > Josh
>> >
>> > D0331 19:14:31.229 THREAD2827 org.eclipse.jetty.server.Server.handle:
>> > REQUEST /api on AsyncHttpConnection@6fa15c80
>> > ,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=362,c=43},r=4
>> > W0331 19:14:31.229 THREAD2828
>> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
>> Uncaught
>> > exception while handling getJobSummary(produser)
>> > com.google.common.collect.Ordering$IncomparableValueException: Cannot
>> > compare value: null
>> >         at
>> > com.google.common.collect.ExplicitOrdering.rank(ExplicitOrdering.java:46)
>> >         at
>> >
>> com.google.common.collect.ExplicitOrdering.compare(ExplicitOrdering.java:40)
>> >         at
>> >
>> com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46)
>> >         at
>> >
>> com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43)
>> >         at com.google.common.collect.Ordering.max(Ordering.java:572)
>> >         at com.google.common.collect.Ordering.max(Ordering.java:539)
>> >         at com.google.common.collect.Ordering.max(Ordering.java:555)
>> >         at
>> >
>> org.apache.aurora.scheduler.base.Tasks.getLatestActiveTask(Tasks.java:215)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:382)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:374)
>> >         at com.google.common.collect.Maps$10.getValue(Maps.java:1855)
>> >         at java.util.HashMap.putAll(HashMap.java:647)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobs(ReadOnlySchedulerImpl.java:373)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobSummary(ReadOnlySchedulerImpl.java:257)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:461)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102)
>> >         at
>> >
>> org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
>> >         at
>> >
>> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:906)
>> >         at
>> >
>> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:891)
>> >         at
>> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>> >         at
>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>> >         at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >         at
>> >
>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>> >         at
>> >
>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>> >         at
>> >
>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69)
>> >         at
>> >
>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
>> >         at
>> > org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
>> >         at
>> >
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> >         at
>> >
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >         at
>> >
>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>> >         at
>> > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>> >         at
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288)
>> >         at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
>> >         at
>> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> >         at
>> >
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
>> >         at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> >         at org.eclipse.jetty.server.Server.handle(Server.java:365)
>> >         at
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
>> >         at
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
>> >         at
>> >
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
>> >         at
>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
>> >         at
>> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>> >         at
>> >
>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> >         at
>> >
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
>> >         at
>> >
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> >         at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> >         at
>> >
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> >         at java.lang.Thread.run(Thread.java:744)
>>

Mime
View raw message