hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Naegele" <jnaeg...@grierforensics.com>
Subject YarnChild and Container running beyond physical memory limits
Date Wed, 13 Apr 2016 23:40:30 GMT
Hi!

 

Can anyone tell me what exactly YarnChild is and how I can control the
quantity of child JVMs running in each container? In this case I'm concerned
with the map phase of my MR job. I'm having issues with my containers
running beyond *physical* memory limits and I'm trying to determine the
cause.

 

Is each child JVM just an individual map task? If so, why do I see a
variable number of them? I don't know if each of these JVMs is a clone of
the original YarnChild process, what they are doing, why they are each using
so much memory (1G).

 

Here is a sample excerpt of my MR job when YARN kills a container:
https://gist.githubusercontent.com/naegelejd/ad3a58192a2df79775d80e3eac0ae49
c/raw/808f998b1987c77ba1fe7fb41abab62ae07c5e02/job.log

Here's the same process tree reorganized and ordered by ancestry:
https://gist.githubusercontent.com/naegelejd/37afb27a6cf16ce918daeaeaf7450cd
c/raw/b8809ce023840799f2cbbee28e49930671198ead/job.clean.log

 

If I increase the amount of memory per container, in turn lowering the total
number of containers, I see these errors less often as expected, BUT when I
do see them, there are NO child JVM processes and it's always due to some
other unrelated external process chewing up RAM. Here is an example of that:
https://gist.githubusercontent.com/naegelejd/32d63b0f9b9c148d1c1c7c0de3c2c31
7/raw/934a93a7afe09c7cd62a50edc08ce902b9e71aac/job.log. You can see that the
[redacted] process is the culprit in that case.

 

I can share my mapred/yarn configuration if it's helpful.

 

If anyone has any ideas I'd greatly appreciate them!

 

Thanks,

Joe


Mime
View raw message