hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nico Pappagianis <nico.pappagia...@salesforce.com>
Subject YARN Resource Allocation When Memory is Very Small
Date Tue, 30 Aug 2016 19:09:36 GMT
How does YARN decide if remaining memory is enough AM allocation

Hi all,

Somewhat of a newb here so hopefully this question isn't too trivial.

I want to know how YARN decides if the remaining memory after AM resource
allocation is "enough". Here's a breakdown of what's going on in my
particular case.

We have a YARN queue that has ~1.7GB memory allocated to it. We have a
parent MR job that spawns multiple child jobs. The AM container memory for
the parent job is configured to 1GB. Once the AM container gets allocated
its 1GB there is a remaining ~0.7GB memory available. In my case that 0.7GB
is enough for the child jobs to succeed.

My question is, what if that remaining amount wasn't 0.7GB but something
much less, like 0.1GB, or 0.0001GB? How does YARN handle those cases - will
it allocate *any* remaining resources to the child jobs? Will there be a
ton of thrashing if there is only a small amount of memory available?

Thanks for any help, or a pointer in the right direction!

Mime
View raw message