ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hurley <jhur...@hortonworks.com>
Subject Re: Ambari - error 500 - getAllRequests after upgrade
Date Tue, 08 Mar 2016 15:00:49 GMT
That's very odd, especially since the upgrade doesn't touch the topology tables. Are you using
MySQL by any chance? If so, can you check to make sure that your database engine is Innodb
and not MyISAM. You have an integrity violation here which doesn't seem possible unless you're
using a database which doesn't support foreign key constraints.

There's probably some SQL which you can run to insert an entry into the topology_logical_request
table, but it's probably best to understand why this happened first.

On Mar 8, 2016, at 5:55 AM, cs user <acldstkusr@gmail.com<mailto:acldstkusr@gmail.com>>
wrote:

Hi All,

I've upgraded Ambari from version 2.1.2-377 to version 2.2.1.0-161.

After performing the upgrade on the server, agents, upgrading the database and starting everything
up, I keep seeing the following error in the logs on the server:

08 Mar 2016 10:07:05,087  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,088  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 3
08 Mar 2016 10:07:05,120  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,120  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 5
08 Mar 2016 10:07:05,134  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,134  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 8
08 Mar 2016 10:07:05,147  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,148  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 7
08 Mar 2016 10:07:05,158  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,158  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 6
08 Mar 2016 10:07:05,170  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,170  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 2
08 Mar 2016 10:07:05,184  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,185  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 1
08 Mar 2016 10:07:05,194  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: Host Assignment Pending
08 Mar 2016 10:07:05,194  INFO [qtp-ambari-agent-55] LogicalRequest:420 - LogicalRequest.createHostRequests:
created new outstanding host request ID = 4
08 Mar 2016 10:07:05,290  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: ambdevtestdc2host-group-21.node.example
08 Mar 2016 10:07:05,328  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: ambdevtestdc2host-group-51.node.example
08 Mar 2016 10:07:05,384  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: ambdevtestdc2host-group-11.node.example
08 Mar 2016 10:07:05,428  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: ambdevtestdc2host-group-41.node.example
08 Mar 2016 10:07:05,507  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: ambdevtestdc2host-group-31.node.example
08 Mar 2016 10:07:05,575  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: ambdevtestdc2host-group-53.node.example
08 Mar 2016 10:07:05,627  INFO [qtp-ambari-agent-55] HostRequest:125 - HostRequest: Successfully
recovered host request for host: ambdevtestdc2host-group-52.node.example
08 Mar 2016 10:07:05,644  WARN [qtp-ambari-agent-55] ServletHandler:563 - /agent/v1/register/ambdevtestdc2host-group-51.node.example
java.lang.NullPointerException
        at org.apache.ambari.server.topology.PersistedStateImpl.getAllRequests(PersistedStateImpl.java:157)
        at org.apache.ambari.server.topology.TopologyManager.ensureInitialized(TopologyManager.java:131)
        at org.apache.ambari.server.topology.TopologyManager.onHostRegistered(TopologyManager.java:315)
        at org.apache.ambari.server.state.host.HostImpl$HostRegistrationReceived.transition(HostImpl.java:301)
        at org.apache.ambari.server.state.host.HostImpl$HostRegistrationReceived.transition(HostImpl.java:266)
        at org.apache.ambari.server.state.fsm.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:354)
        at org.apache.ambari.server.state.fsm.StateMachineFactory.doTransition(StateMachineFactory.java:294)
        at org.apache.ambari.server.state.fsm.StateMachineFactory.access$300(StateMachineFactory.java:39)
        at org.apache.ambari.server.state.fsm.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:440)
        at org.apache.ambari.server.state.host.HostImpl.handleEvent(HostImpl.java:570)
        at org.apache.ambari.server.agent.HeartBeatHandler.handleRegistration(HeartBeatHandler.java:966)
        at org.apache.ambari.server.agent.rest.AgentResource.register(AgentResource.java:95)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
        at org.apache.ambari.server.security.SecurityFilter.doFilter(SecurityFilter.java:67)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)


This is not specific to host group ambdevtestdc2host-group-51.node.example, it is happening
for all host groups.

On the agents I see the following:

<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /agent/v1/register/ambdevtestdc2host-group-51.node.example Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>

Is there a work around for this? It's just a test cluster, but it would be good to know how
to work around this, as I've seen it a number of times now. Is there anything that can be
modified in the database to resolve it?

Thanks!





Mime
View raw message