hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil G <sun...@apache.org>
Subject Re: How to monitor YARN application memory per container?
Date Wed, 14 Jun 2017 13:04:13 GMT
Hi Shmuel

In Hadoop 2.8 release line, you could check "yarn node -status {nodeId}"
CLI command or "http://<rm http address:port>/ws/v1/cluster/nodes/{nodeid}"
REST end point to get container's actual resource usage per node. You could
also check the same in any of Hadoop 3.0 alpha releases as well.

Thanks
Sunil

On Tue, Jun 13, 2017 at 11:29 PM Shmuel Blitz <shmuel.blitz@similarweb.com>
wrote:

> Hi,
>
> Thanks for your response.
>
> The /metrics API returns a blank page on our RM.
>
> The /jmx API has some metrics, but these are the same metrics we are
> already loading into data-dog.
> It's not good enough, because it doesn't break down the memory use by
> container.
>
> I need the by-container breakdown because resource allocation is per
> container and I would like to se if my job is really using up all the
> allocated memory.
>
> Shmuel
>
> On Tue, Jun 13, 2017 at 6:05 PM, Sidharth Kumar <
> sidharthkumar2707@gmail.com> wrote:
>
>> Hi,
>>
>> I guess you can get it from http://<resourcemanager-host>:<rm-port>/jmx
>> or /metrics
>>
>> Regards
>> Sidharth
>> LinkedIn: www.linkedin.com/in/sidharthkumar2792
>>
>> On 13-Jun-2017 6:26 PM, "Shmuel Blitz" <shmuel.blitz@similarweb.com>
>> wrote:
>>
>>> (This question has also been published on StackOveflow
>>> <https://stackoverflow.com/q/44484940/416300>)
>>>
>>> I am looking for a way to monitor memory usage of YARN containers over
>>> time.
>>>
>>> Specifically - given a YARN application-id, how can you get a graph,
>>> showing the memory usage of each of its containers over time?
>>>
>>> The main goal is to better fit memory allocation requirements for our
>>> YARN applications (Spark / Map-Reduce), to avoid over allocation and
>>> cluster resource waste. A side goal would be the ability to debug memory
>>> issues when developing our jobs and attempting to pick reasonable resource
>>> allocations.
>>>
>>> We've tried using the Data-Dog integration, But it doesn't break down
>>> the metrics by container.
>>>
>>> Another approach was to parse the hadoop-yarn logs. These logs have
>>> messages like:
>>>
>>> Memory usage of ProcessTree 57251 for container-id
>>> container_e116_1495951495692_35134_01_000001: 1.9 GB of 11 GB physical
>>> memory used; 14.4 GB of 23.1 GB virtual memory used
>>> Parsing the logs correctly can yield data that can be used to plot a
>>> graph of memory usage over time.
>>>
>>> That's exactly what we want, but there are two downsides:
>>>
>>> It involves reading human-readable log lines and parsing them into
>>> numeric data. We'd love to avoid that.
>>> If this data can be consumed otherwise, we're hoping it'll have more
>>> information that we might be interest in in the future. We wouldn't want to
>>> put the time into parsing the logs just to realize we need something else.
>>> Is there any other way to extract these metrics, either by plugging in
>>> to an existing producer or by writing a simple listener?
>>>
>>> Perhaps a whole other approach?
>>>
>>> --
>>> [image: Logo]
>>> <https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>> Shmuel Blitz
>>> *Big Data Developer*
>>> www.similarweb.com
>>> <http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Like
>>> Us
>>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Follow
>>> Us
>>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Watch
>>> Us
>>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Read
>>> Us
>>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>
>>
>
>
> --
> [image: Logo]
> <https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
> Shmuel Blitz
> *Big Data Developer*
> www.similarweb.com
> <http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Like
> Us
> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Follow
> Us
> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Watch
> Us
> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>
> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Read
> Us
> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>

Mime
View raw message