ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srimanth Gunturi <sgunt...@hortonworks.com>
Subject Re: Ambari agent memory growing
Date Thu, 03 Dec 2015 17:30:34 GMT
Hi Andrew,
Thank you for bringing up this issue.

I was hoping if you could open a JIRA and provide more details about these problem hosts...
- what components were installed on them
- what operations were performed that could have included the host (like stop/start YARN etc.)
- how long did it take for the agent's memory to spike up, etc.

Also, if its possible to attach /var/log/ambari-agent/ambari-agent.log*?
Regards,
Srimanth




________________________________________
From: Andrew Robertson <andyrobertson101@gmail.com>
Sent: Wednesday, December 02, 2015 9:31 PM
To: user@ambari.apache.org
Subject: Ambari agent memory growing

I'm running Ambari 2.1.2.1, which includes the fix for AMBARI-13307 /
Agent instance memory footprint likely gradually growing.

However, I noticed today (after an agent crashed) that the agent had
been consuming 37GB of memory, and I've got an agent on another
machine consuming 15GB of memory after it was restarted 11/13. (And a
couple others running in the ~8-10GB range).

The rest of the agents are only consuming ~1.6GB of memory - which
seems to be machines that only have datanodes/nodemanagers.  So
whatever the issue is, it seems to only be affecting a small number of
agents.

The symptoms here are very similar to the leak that appeared in/around
2.1.x that I (thought) would've been fixed by AMBARI-13307.  Checking
logs, it appears the rate of memory increase is approximately the same
as before the 2.1.2.1 upgrade, so 13307 doesn't seem to have fixed
this particular issue.

If you need more information about this just let me know.
Thank you!

Mime
View raw message