aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Khutornenko <ma...@apache.org>
Subject Re: Error fetching job summaries: No error message returned by the scheduler
Date Tue, 31 Mar 2015 20:50:00 GMT
Interesting. If your client is built off of master it points towards
this line: https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/cli/jobs.py#L570

Somehow you have tasks without status? No sure what could cause that.
Do you have any client or scheduler customizations that may interfere
with thrift layer?

Anyway, the only sane way forward from here is to wait or force the
task history cleanup I described earlier.

On Tue, Mar 31, 2015 at 1:34 PM, Josh Adams <josh@gmail.com> wrote:
> Ah ok. So for those that exist in the history I see events like this for
> each:
>
> 2015-03-20 22:43:33 PENDING: None
> 2015-03-20 22:43:33 ASSIGNED: None
> 2015-03-20 22:43:36 STARTING: Initializing sandbox.
> 2015-03-20 22:43:39 RUNNING: None
> 2015-03-31 00:54:46 KILLING: Killed by UNSECURE
> 2015-03-31 00:55:06 KILLED: Instructed to kill task.
>
> There are also some that I expected to exist but threw the exception:
>
> Traceback (most recent call last):
>   File
> "/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py",
> line 272, in execute
>     self.execute_entry(entry_point, args)
>   File
> "/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py",
> line 320, in execute_entry
>     runner(entry_point)
>   File
> "/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py",
> line 343, in execute_pkg_resources
>     runner()
>   File "apache/aurora/client/cli/client.py", line 95, in proxy_main
>   File "apache/aurora/client/cli/__init__.py", line 329, in execute
>   File "apache/aurora/client/cli/__init__.py", line 306, in _execute
>   File "apache/aurora/client/cli/__init__.py", line 382, in execute
>   File "apache/aurora/client/cli/jobs.py", line 659, in execute
>   File "apache/aurora/client/cli/jobs.py", line 626, in get_status_for_jobs
>   File "apache/aurora/client/cli/jobs.py", line 598, in render_tasks_pretty
>   File "apache/aurora/client/cli/jobs.py", line 570, in render_task_pretty
> KeyError: None
>
> Josh
>
> On Tue, Mar 31, 2015 at 1:27 PM, Maxim Khutornenko <maxim@apache.org> wrote:
>
>> If you remember your job names you should still be able to run "aurora
>> job status" though for as long as there is task history. Would be
>> great to know what caused that problem in the first place.
>>
>> In case you want to unblock, you can temporarily change
>> -history_prune_threshold and let scheduler purge your task history. Be
>> careful as that will purge history for all jobs in your cluster, so
>> perhaps it's better wait for the scheduler to clean by itself (2 days
>> after your kill).
>>
>> On Tue, Mar 31, 2015 at 1:03 PM, Josh Adams <josh@gmail.com> wrote:
>> > Hey Maxim,
>> >
>> > I forgot to mention that yesterday I killed all the running jobs in an
>> > attempt to get it working again. When I run aurora.pex job list
>> > <cluster>/<role>/<env> it just prints this out before handing
me back a
>> > terminal:
>> >
>> > INFO] Retrieving jobs for role produser
>> >
>> > Cheers,
>> > Josh
>> >
>> > On Tue, Mar 31, 2015 at 12:54 PM, Maxim Khutornenko <maxim@apache.org>
>> > wrote:
>> >
>> >> Hi Josh,
>> >>
>> >> How many jobs do you have under that role? Any chance you could run
>> >> "aurora job status" for every one of them to see if you have any tasks
>> >> with abnormal event sequences? Every task must have at least one event
>> >> with a timestamp. Here is a reference example:
>> >>
>> >> events:
>> >>   2015-03-23 01:01:28 PENDING: Rescheduled
>> >>   2015-03-23 01:01:28 ASSIGNED: None
>> >>   2015-03-23 01:01:30 STARTING: Initializing sandbox.
>> >>   2015-03-23 01:01:31 RUNNING: None
>> >>
>> >> On Tue, Mar 31, 2015 at 12:17 PM, Josh Adams <josh@gmail.com> wrote:
>> >> > Hi there,
>> >> >
>> >> > I'm getting this error when clicking on a Role on the /scheduler
>> page. We
>> >> > only have a single role and this is our staging cluster. It seems like
>> >> > other operations like `aurora.pex job list ...` and similar commands
>> work
>> >> > fine, but I can't seem to figure out how to get the scheduler out of
>> this
>> >> > state so I can view the Job summary page.
>> >> >
>> >> > I've upgraded from Mesos/0.20.1 & Aurora/0.6.1 to Mesos/0.21.1
and
>> >> > Aurora/0.7.1@3ab8a9e and the problem persists. Below is the stack
>> trace
>> >> > from the scheduler process.
>> >> >
>> >> > Thank you!
>> >> > Josh
>> >> >
>> >> > D0331 19:14:31.229 THREAD2827 org.eclipse.jetty.server.Server.handle:
>> >> > REQUEST /api on AsyncHttpConnection@6fa15c80
>> >> > ,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=362,c=43},r=4
>> >> > W0331 19:14:31.229 THREAD2828
>> >> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
>> >> Uncaught
>> >> > exception while handling getJobSummary(produser)
>> >> > com.google.common.collect.Ordering$IncomparableValueException: Cannot
>> >> > compare value: null
>> >> >         at
>> >> >
>> com.google.common.collect.ExplicitOrdering.rank(ExplicitOrdering.java:46)
>> >> >         at
>> >> >
>> >>
>> com.google.common.collect.ExplicitOrdering.compare(ExplicitOrdering.java:40)
>> >> >         at
>> >> >
>> >>
>> com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46)
>> >> >         at
>> >> >
>> >>
>> com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43)
>> >> >         at com.google.common.collect.Ordering.max(Ordering.java:572)
>> >> >         at com.google.common.collect.Ordering.max(Ordering.java:539)
>> >> >         at com.google.common.collect.Ordering.max(Ordering.java:555)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.base.Tasks.getLatestActiveTask(Tasks.java:215)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:382)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:374)
>> >> >         at com.google.common.collect.Maps$10.getValue(Maps.java:1855)
>> >> >         at java.util.HashMap.putAll(HashMap.java:647)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobs(ReadOnlySchedulerImpl.java:373)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobSummary(ReadOnlySchedulerImpl.java:257)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:461)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:906)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:891)
>> >> >         at
>> >> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>> >> >         at
>> >> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>> >> >         at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>> >> >         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>> >> >         at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69)
>> >> >         at
>> >> >
>> >>
>> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
>> >> >         at
>> >> > org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>> >> >         at
>> >> >
>> >>
>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>> >> >         at
>> >> > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
>> >> >         at
>> >> >
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> >> >         at org.eclipse.jetty.server.Server.handle(Server.java:365)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
>> >> >         at
>> >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
>> >> >         at
>> >> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> >> >         at
>> >> >
>> >>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> >> >         at java.lang.Thread.run(Thread.java:744)
>> >>
>>

Mime
View raw message