aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Adams <j...@gmail.com>
Subject Re: Error fetching job summaries: No error message returned by the scheduler
Date Tue, 31 Mar 2015 20:34:53 GMT
Ah ok. So for those that exist in the history I see events like this for
each:

2015-03-20 22:43:33 PENDING: None
2015-03-20 22:43:33 ASSIGNED: None
2015-03-20 22:43:36 STARTING: Initializing sandbox.
2015-03-20 22:43:39 RUNNING: None
2015-03-31 00:54:46 KILLING: Killed by UNSECURE
2015-03-31 00:55:06 KILLED: Instructed to kill task.

There are also some that I expected to exist but threw the exception:

Traceback (most recent call last):
  File
"/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py",
line 272, in execute
    self.execute_entry(entry_point, args)
  File
"/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py",
line 320, in execute_entry
    runner(entry_point)
  File
"/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py",
line 343, in execute_pkg_resources
    runner()
  File "apache/aurora/client/cli/client.py", line 95, in proxy_main
  File "apache/aurora/client/cli/__init__.py", line 329, in execute
  File "apache/aurora/client/cli/__init__.py", line 306, in _execute
  File "apache/aurora/client/cli/__init__.py", line 382, in execute
  File "apache/aurora/client/cli/jobs.py", line 659, in execute
  File "apache/aurora/client/cli/jobs.py", line 626, in get_status_for_jobs
  File "apache/aurora/client/cli/jobs.py", line 598, in render_tasks_pretty
  File "apache/aurora/client/cli/jobs.py", line 570, in render_task_pretty
KeyError: None

Josh

On Tue, Mar 31, 2015 at 1:27 PM, Maxim Khutornenko <maxim@apache.org> wrote:

> If you remember your job names you should still be able to run "aurora
> job status" though for as long as there is task history. Would be
> great to know what caused that problem in the first place.
>
> In case you want to unblock, you can temporarily change
> -history_prune_threshold and let scheduler purge your task history. Be
> careful as that will purge history for all jobs in your cluster, so
> perhaps it's better wait for the scheduler to clean by itself (2 days
> after your kill).
>
> On Tue, Mar 31, 2015 at 1:03 PM, Josh Adams <josh@gmail.com> wrote:
> > Hey Maxim,
> >
> > I forgot to mention that yesterday I killed all the running jobs in an
> > attempt to get it working again. When I run aurora.pex job list
> > <cluster>/<role>/<env> it just prints this out before handing
me back a
> > terminal:
> >
> > INFO] Retrieving jobs for role produser
> >
> > Cheers,
> > Josh
> >
> > On Tue, Mar 31, 2015 at 12:54 PM, Maxim Khutornenko <maxim@apache.org>
> > wrote:
> >
> >> Hi Josh,
> >>
> >> How many jobs do you have under that role? Any chance you could run
> >> "aurora job status" for every one of them to see if you have any tasks
> >> with abnormal event sequences? Every task must have at least one event
> >> with a timestamp. Here is a reference example:
> >>
> >> events:
> >>   2015-03-23 01:01:28 PENDING: Rescheduled
> >>   2015-03-23 01:01:28 ASSIGNED: None
> >>   2015-03-23 01:01:30 STARTING: Initializing sandbox.
> >>   2015-03-23 01:01:31 RUNNING: None
> >>
> >> On Tue, Mar 31, 2015 at 12:17 PM, Josh Adams <josh@gmail.com> wrote:
> >> > Hi there,
> >> >
> >> > I'm getting this error when clicking on a Role on the /scheduler
> page. We
> >> > only have a single role and this is our staging cluster. It seems like
> >> > other operations like `aurora.pex job list ...` and similar commands
> work
> >> > fine, but I can't seem to figure out how to get the scheduler out of
> this
> >> > state so I can view the Job summary page.
> >> >
> >> > I've upgraded from Mesos/0.20.1 & Aurora/0.6.1 to Mesos/0.21.1 and
> >> > Aurora/0.7.1@3ab8a9e and the problem persists. Below is the stack
> trace
> >> > from the scheduler process.
> >> >
> >> > Thank you!
> >> > Josh
> >> >
> >> > D0331 19:14:31.229 THREAD2827 org.eclipse.jetty.server.Server.handle:
> >> > REQUEST /api on AsyncHttpConnection@6fa15c80
> >> > ,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=362,c=43},r=4
> >> > W0331 19:14:31.229 THREAD2828
> >> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
> >> Uncaught
> >> > exception while handling getJobSummary(produser)
> >> > com.google.common.collect.Ordering$IncomparableValueException: Cannot
> >> > compare value: null
> >> >         at
> >> >
> com.google.common.collect.ExplicitOrdering.rank(ExplicitOrdering.java:46)
> >> >         at
> >> >
> >>
> com.google.common.collect.ExplicitOrdering.compare(ExplicitOrdering.java:40)
> >> >         at
> >> >
> >>
> com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46)
> >> >         at
> >> >
> >>
> com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43)
> >> >         at com.google.common.collect.Ordering.max(Ordering.java:572)
> >> >         at com.google.common.collect.Ordering.max(Ordering.java:539)
> >> >         at com.google.common.collect.Ordering.max(Ordering.java:555)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.base.Tasks.getLatestActiveTask(Tasks.java:215)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:382)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:374)
> >> >         at com.google.common.collect.Maps$10.getValue(Maps.java:1855)
> >> >         at java.util.HashMap.putAll(HashMap.java:647)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobs(ReadOnlySchedulerImpl.java:373)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobSummary(ReadOnlySchedulerImpl.java:257)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:461)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
> >> >         at
> >> >
> >>
> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:906)
> >> >         at
> >> >
> >>
> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:891)
> >> >         at
> >> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> >> >         at
> >> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> >> >         at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> >> >         at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> >> >         at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69)
> >> >         at
> >> >
> >>
> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
> >> >         at
> >> > org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> >> >         at
> >> >
> >>
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> >> >         at
> >> > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> >> >         at
> >> >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> >> >         at org.eclipse.jetty.server.Server.handle(Server.java:365)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
> >> >         at
> >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
> >> >         at
> >> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> >> >         at
> >> >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> >> >         at java.lang.Thread.run(Thread.java:744)
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message