incubator-ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumit Mohanty <smoha...@hortonworks.com>
Subject Re: Ganglia gmetad conf
Date Fri, 11 Oct 2013 20:57:45 GMT
If its not a bother, you can ignore the errors. Otherwise, let me know and I
can work out a solution to get rid of this error.

From:  Artem Ervits <are9004@nyp.org>
Reply-To:  <ambari-user@incubator.apache.org>
Date:  Friday, October 11, 2013 1:40 PM
To:  "ambari-user@incubator.apache.org" <ambari-user@incubator.apache.org>
Subject:  RE: Ganglia gmetad conf

I must¹ve hit this bug also
 
https://issues.apache.org/jira/browse/AMBARI-2953
 
because I¹m also getting this error:
 
Oct 11 16:38:16 servername /usr/sbin/gmetad[25261]: data_thread() for
[HDPHistoryServer] failed to contact node serverIP
Oct 11 16:38:16 servername /usr/sbin/gmetad[25261]: data_thread() got no
answer from any [HDPHistoryServer] datasource
 

From: Artem Ervits [mailto:are9004@nyp.org]
Sent: Friday, October 11, 2013 4:32 PM
To: ambari-user@incubator.apache.org
Subject: RE: Ganglia gmetad conf
 

1.2.5 and I see 6 versions. I think I seem to fix the issue, I was able to
execute the ./configs.sh file correctly and updated the configuration. I
also removed the symlink and changed the permissions on the rrds folder to
nobody:nobody, now I can see the dashboard as well as am getting the
metrics. 
 
Thank you very much for your patience!
 

From: Sumit Mohanty [mailto:smohanty@hortonworks.com]
Sent: Friday, October 11, 2013 4:18 PM
To: ambari-user@incubator.apache.org
Subject: Re: Ganglia gmetad conf
 

What version of Ambari are you using?

 

You can use curl -u admin:admin
http://localhost:8080/api/v1/clusters/c1/configurations?type=global to get
all versions of the config type "global"

 

How many version do you see?

 

-Sumit

From: Artem Ervits <are9004@nyp.org>
Reply-To: <ambari-user@incubator.apache.org>
Date: Friday, October 11, 2013 12:44 PM
To: "ambari-user@incubator.apache.org" <ambari-user@incubator.apache.org>
Subject: RE: Ganglia gmetad conf

 

How do I revert the config, I was able to create a symlink to another folder
with plenty of space.
 
From: Sumit Mohanty [mailto:smohanty@hortonworks.com]
Sent: Friday, October 11, 2013 3:30 PM
To: ambari-user@incubator.apache.org
Subject: Re: Ganglia gmetad conf
 

I have seen this error when the ganglia metrics read by Ambari Server is
malformed/incomplete. This issue is fixed in the latest release. However,
the root cause is still the malformed/incomplete data.

 

Can you check if the new folder you created for the rrd files have correct
permissions and metric rrd files are being created?

 

Any relevant logs in /var/log/messages?

 

Can you also try restarting ambari-server and ganglia server?

 
We can revert back the config temporarily as well while we figure out the
reason for error.

 

-Sumit

 

On Fri, Oct 11, 2013 at 11:36 AM, Artem Ervits <are9004@nyp.org> wrote:

Sumit,
 
After running the script I am getting the following errors on the Ambari
server and dashboard is not loading:
 
14:34:59,363 ERROR ReadHandler:83 - Caught a runtime exception executing a
query
java.lang.NullPointerException
        at 
org.apache.ambari.server.controller.ganglia.GangliaPropertyProvider$RRDReque
st.populateResources(GangliaPropertyProvider.java:426)
        at 
org.apache.ambari.server.controller.ganglia.GangliaPropertyProvider.populate
Resources(GangliaPropertyProvider.java:116)
        at 
org.apache.ambari.server.controller.internal.ClusterControllerImpl.populateR
esources(ClusterControllerImpl.java:350)
        at 
org.apache.ambari.server.controller.internal.ClusterControllerImpl.getResour
ces(ClusterControllerImpl.java:120)
        at 
org.apache.ambari.server.controller.internal.ClusterControllerImpl.getResour
ces(ClusterControllerImpl.java:94)
        at 
org.apache.ambari.server.api.query.QueryImpl.execute(QueryImpl.java:149)
        at 
org.apache.ambari.server.api.query.QueryImpl.execute(QueryImpl.java:169)
        at 
org.apache.ambari.server.api.query.QueryImpl.execute(QueryImpl.java:169)
        at 
org.apache.ambari.server.api.handlers.ReadHandler.handleRequest(ReadHandler.
java:61)
        at 
org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:1
07)
        at 
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.
java:80)
        at 
org.apache.ambari.server.api.services.ServiceService.getServices(ServiceServ
ice.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInv
okerFactory.java:60)
        at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispa
tchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvi
der.java:205)
        at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatche
r.dispatch(ResourceJavaMethodDispatcher.java:75)
        at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.ja
va:288)
        at 
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.ja
va:134)
        at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathR
ule.java:147)
        at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassR
ule.java:108)
        at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathR
ule.java:147)
        at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootReso
urceClassesRule.java:84)
        at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(Web
ApplicationImpl.java:1469)
        at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(Web
ApplicationImpl.java:1400)
        at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebA
pplicationImpl.java:1349)
        at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebA
pplicationImpl.java:1339)
        at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:
416)
        at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContain
er.java:537)
        at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContain
er.java:708)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1329)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:330)
        at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.
invoke(FilterSecurityInterceptor.java:118)
        at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.
doFilter(FilterSecurityInterceptor.java:84)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(
ExceptionTranslationFilter.java:113)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.session.SessionManagementFilter.doFilter(Se
ssionManagementFilter.java:103)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.authentication.AnonymousAuthenticationFilte
r.doFilter(AnonymousAuthenticationFilter.java:113)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.servletapi.SecurityContextHolderAwareReques
tFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilt
er(RequestCacheAwareFilter.java:45)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.authentication.www.BasicAuthenticationFilte
r.doFilter(BasicAuthenticationFilter.java:150)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.context.SecurityContextPersistenceFilter.do
Filter(SecurityContextPersistenceFilter.java:87)
        at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilte
r(FilterChainProxy.java:342)
        at 
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterCha
inProxy.java:192)
        at 
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.
java:160)
        at 
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Delegati
ngFilterProxy.java:237)
        at 
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilt
erProxy.java:167)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1300)
        at 
org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilte
r.java:48)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
.java:1300)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137
)
        at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java
:227)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java
:1038)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:
189)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:
972)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135
)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:1
16)
        at org.eclipse.jetty.server.Server.handle(Server.java:363)
        at 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpCo
nnection.java:483)
        at 
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpC
onnection.java:920)
        at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplet
e(AbstractHttpConnection.java:982)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
        at 
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at 
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java
:82)
        at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.
java:627)
        at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.j
ava:51)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:
608)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:5
43)
        at java.lang.Thread.run(Thread.java:662)
 

From: Sumit Mohanty [mailto:smohanty@hortonworks.com]
Sent: Friday, October 11, 2013 11:45 AM
To: ambari-user@incubator.apache.org
Subject: Re: Ganglia gmetad conf
 

The following procedure can be used:
Use the following script to read and update the config parameter that
specifies the target folder for ganglia rrd files.
https://git-wip-us.apache.org/repos/asf/incubator-ambari/repo?p=incubator-am
bari.git;a=blob;f=ambari-server/src/main/resources/scripts/configs.sh;h=9ac2
d9245b9401ee86014175fc24739eb0bd8218;hb=trunk
<https://git-wip-us.apache.org/repos/asf/incubator-ambari/repo?p=incubator-a
mbari.git;a=blob;f=ambari-server/src/main/resources/scripts/configs.sh;h=2aa
5250bde1d916e79fc08ecc2a6b00f292bc558;hb=HEAD>
Script name: configs.sh
Procedure:
·        Decide on the new location for the rrd files (e.g.
/newlocation/ganglia/rrds)
·        Stop Ganglia service and wait for it to stop completely
·        Get the existing folder path for Ganglia rrd files using configs.sh
o   E.g. ./configs.sh get localhost c1 global | grep "rrdcached_base_dir"
(example assume localhost for ambari-server host, c1 for the name of the
cluster)

·        Backup this folder and then remove it
·        Use configs.sh to update the config property with the new location
for the rrd files
o   E.g. ./configs.sh -u admin ­p admin set localhost c1 global
"rrdcached_base_dir" "/newlocation/ganglia/rrds" (example assume localhost
for ambari­server host, c1 for the name of the cluster and admin/admin as
username/password)

·        Start Ganglia service
·        You may need to wait for 5-10 minutes to see the metrics getting
populated

 

Artem, in addition we can also modify the gmetad config template to reduce
number of data points stored for each metrics. Let me know if you need that
information.

 

Thanks

Sumit

From: Artem Ervits <are9004@nyp.org>
Reply-To: <ambari-user@incubator.apache.org>
Date: Friday, October 11, 2013 8:25 AM
To: "ambari-user@incubator.apache.org" <ambari-user@incubator.apache.org>
Subject: Ganglia gmetad conf

 

Hello all,
How can I change settings in Ganglia gmetad using Ambari? Do I need to use
the API? I need to change the location of the /var/lib/ganglia/rrds database
due to low space
 
Artem Ervits
New York Presbyterian Hospital
 

This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged.
If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not
the named recipient, please notify us immediately by contacting the sender
at the electronic mail address noted above, and delete and destroy all
copies of this message. Thank you.

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.


This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged.
If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not
the named recipient, please notify us immediately by contacting the sender
at the electronic mail address noted above, and delete and destroy all
copies of this message. Thank you.
 

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.

This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged.
If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not
the named recipient, please notify us immediately by contacting the sender
at the electronic mail address noted above, and delete and destroy all
copies of this message. Thank you.

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.

This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged.
If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not
the named recipient, please notify us immediately by contacting the sender
at the electronic mail address noted above, and delete and destroy all
copies of this message. Thank you.



Confidential Information subject to NYP's (and its affiliates') information
management and security policies (http://infonet.nyp.org/QA/HospitalManual).

This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged.
If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited.  If you have received this message in error or are not
the named recipient, please notify us immediately by contacting the sender
at the electronic mail address noted above, and delete and destroy all
copies of this message.  Thank you.



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message