hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shmuel Blitz <shmuel.bl...@similarweb.com>
Subject Re: How to monitor YARN application memory per container?
Date Wed, 14 Jun 2017 06:28:55 GMT
Hi,

Thanks for your response.

The /metrics API returns a blank page on our RM.

The /jmx API has some metrics, but these are the same metrics we are
already loading into data-dog.
It's not good enough, because it doesn't break down the memory use by
container.

I need the by-container breakdown because resource allocation is per
container and I would like to se if my job is really using up all the
allocated memory.

Shmuel

On Tue, Jun 13, 2017 at 6:05 PM, Sidharth Kumar <sidharthkumar2707@gmail.com
> wrote:

> Hi,
>
> I guess you can get it from http://<resourcemanager-host>:<rm-port>/jmx
> or /metrics
>
> Regards
> Sidharth
> LinkedIn: www.linkedin.com/in/sidharthkumar2792
>
> On 13-Jun-2017 6:26 PM, "Shmuel Blitz" <shmuel.blitz@similarweb.com>
> wrote:
>
>> (This question has also been published on StackOveflow
>> <https://stackoverflow.com/q/44484940/416300>)
>>
>> I am looking for a way to monitor memory usage of YARN containers over
>> time.
>>
>> Specifically - given a YARN application-id, how can you get a graph,
>> showing the memory usage of each of its containers over time?
>>
>> The main goal is to better fit memory allocation requirements for our
>> YARN applications (Spark / Map-Reduce), to avoid over allocation and
>> cluster resource waste. A side goal would be the ability to debug memory
>> issues when developing our jobs and attempting to pick reasonable resource
>> allocations.
>>
>> We've tried using the Data-Dog integration, But it doesn't break down the
>> metrics by container.
>>
>> Another approach was to parse the hadoop-yarn logs. These logs have
>> messages like:
>>
>> Memory usage of ProcessTree 57251 for container-id
>> container_e116_1495951495692_35134_01_000001: 1.9 GB of 11 GB physical
>> memory used; 14.4 GB of 23.1 GB virtual memory used
>> Parsing the logs correctly can yield data that can be used to plot a
>> graph of memory usage over time.
>>
>> That's exactly what we want, but there are two downsides:
>>
>> It involves reading human-readable log lines and parsing them into
>> numeric data. We'd love to avoid that.
>> If this data can be consumed otherwise, we're hoping it'll have more
>> information that we might be interest in in the future. We wouldn't want to
>> put the time into parsing the logs just to realize we need something else.
>> Is there any other way to extract these metrics, either by plugging in to
>> an existing producer or by writing a simple listener?
>>
>> Perhaps a whole other approach?
>>
>> --
>> [image: Logo]
>> <https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>> Shmuel Blitz
>> *Big Data Developer*
>> www.similarweb.com
>> <http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Like
>> Us
>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Follow
>> Us
>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Watch
>> Us
>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Read
>> Us
>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>


-- 
[image: Logo]
<https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Shmuel Blitz
*Big Data Developer*
www.similarweb.com
<http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Like
Us
<https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Follow
Us
<https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Watch
Us
<https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Read
Us
<https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>

Mime
View raw message