hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shmuel Blitz <shmuel.bl...@similarweb.com>
Subject Re: How to monitor YARN application memory per container?
Date Wed, 14 Jun 2017 13:31:06 GMT
Hi Sunil,

Thanks for your response.

Here is the response I get when running  "yarn node -status {nodeId}" :

Node Report :
        Node-Id : myNode:4545
        Rack : /default
        Node-State : RUNNING
        Node-Http-Address : muNode:8042
        Last-Health-Update : Wed 14/Jun/17 08:25:43:261EST
        Health-Report :
        Containers : 7
        Memory-Used : 44032MB
        Memory-Capacity : 49152MB
        CPU-Used : 16 vcores
        CPU-Capacity : 48 vcores
        Node-Labels :

However, this is information regarding the entire node, containing all
containers.

I have no way of using this to see the value I give to '
spark.executor.memory' makes sense or not.

I'm looking for memory usage/allocated information *per-container*.

Shmuel

On Wed, Jun 14, 2017 at 4:04 PM, Sunil G <sunilg@apache.org> wrote:

> Hi Shmuel
>
> In Hadoop 2.8 release line, you could check "yarn node -status {nodeId}"
> CLI command or "http://<rm http address:port>/ws/v1/cluster/nodes/{nodeid}"
> REST end point to get container's actual resource usage per node. You could
> also check the same in any of Hadoop 3.0 alpha releases as well.
>
> Thanks
> Sunil
>
> On Tue, Jun 13, 2017 at 11:29 PM Shmuel Blitz <shmuel.blitz@similarweb.com>
> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> The /metrics API returns a blank page on our RM.
>>
>> The /jmx API has some metrics, but these are the same metrics we are
>> already loading into data-dog.
>> It's not good enough, because it doesn't break down the memory use by
>> container.
>>
>> I need the by-container breakdown because resource allocation is per
>> container and I would like to se if my job is really using up all the
>> allocated memory.
>>
>> Shmuel
>>
>> On Tue, Jun 13, 2017 at 6:05 PM, Sidharth Kumar <
>> sidharthkumar2707@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I guess you can get it from http://<resourcemanager-host>:<rm-port>/jmx
>>> or /metrics
>>>
>>> Regards
>>> Sidharth
>>> LinkedIn: www.linkedin.com/in/sidharthkumar2792
>>>
>>> On 13-Jun-2017 6:26 PM, "Shmuel Blitz" <shmuel.blitz@similarweb.com>
>>> wrote:
>>>
>>>> (This question has also been published on StackOveflow
>>>> <https://stackoverflow.com/q/44484940/416300>)
>>>>
>>>> I am looking for a way to monitor memory usage of YARN containers over
>>>> time.
>>>>
>>>> Specifically - given a YARN application-id, how can you get a graph,
>>>> showing the memory usage of each of its containers over time?
>>>>
>>>> The main goal is to better fit memory allocation requirements for our
>>>> YARN applications (Spark / Map-Reduce), to avoid over allocation and
>>>> cluster resource waste. A side goal would be the ability to debug memory
>>>> issues when developing our jobs and attempting to pick reasonable resource
>>>> allocations.
>>>>
>>>> We've tried using the Data-Dog integration, But it doesn't break down
>>>> the metrics by container.
>>>>
>>>> Another approach was to parse the hadoop-yarn logs. These logs have
>>>> messages like:
>>>>
>>>> Memory usage of ProcessTree 57251 for container-id
>>>> container_e116_1495951495692_35134_01_000001: 1.9 GB of 11 GB physical
>>>> memory used; 14.4 GB of 23.1 GB virtual memory used
>>>> Parsing the logs correctly can yield data that can be used to plot a
>>>> graph of memory usage over time.
>>>>
>>>> That's exactly what we want, but there are two downsides:
>>>>
>>>> It involves reading human-readable log lines and parsing them into
>>>> numeric data. We'd love to avoid that.
>>>> If this data can be consumed otherwise, we're hoping it'll have more
>>>> information that we might be interest in in the future. We wouldn't want
to
>>>> put the time into parsing the logs just to realize we need something else.
>>>> Is there any other way to extract these metrics, either by plugging in
>>>> to an existing producer or by writing a simple listener?
>>>>
>>>> Perhaps a whole other approach?
>>>>
>>>> --
>>>> [image: Logo]
>>>> <https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>> Shmuel Blitz
>>>> *Big Data Developer*
>>>> www.similarweb.com
>>>> <http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>>
>>>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Like
>>>> Us
>>>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>>
>>>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Follow
>>>> Us
>>>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>>
>>>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Watch
>>>> Us
>>>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>>
>>>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Read
>>>> Us
>>>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>>>
>>>
>>
>>
>> --
>> [image: Logo]
>> <https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>> Shmuel Blitz
>> *Big Data Developer*
>> www.similarweb.com
>> <http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Like
>> Us
>> <https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Follow
>> Us
>> <https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Watch
>> Us
>> <https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Read
>> Us
>> <https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
>>
>


-- 
[image: Logo]
<https://www.similarweb.com/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Shmuel Blitz
*Big Data Developer*
www.similarweb.com
<http://www.similarweb.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Like
Us
<https://www.facebook.com/SimilarWeb/?fref=ts&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Follow
Us
<https://twitter.com/SimilarWeb?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Watch
Us
<https://www.youtube.com/watch?v=Sb09jaZYY7s&utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
<https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>
Read
Us
<https://www.similarweb.com/blog/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature>

Mime
View raw message