hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark question <markq2...@gmail.com>
Subject Re: memory of mappers and reducers
Date Thu, 16 Feb 2012 17:05:33 GMT
Great! thanks a lot Srinivas !
Mark

On Thu, Feb 16, 2012 at 7:02 AM, Srinivas Surasani <vasajb@gmail.com> wrote:

> 1) Yes option 2 is enough.
> 2) Configuration variable "mapred.child.ulimit" can be used to control
> the maximum virtual memory of the child (map/reduce) processes.
>
> ** value of mapred.child.ulimit > value of mapred.child.java.opts
>
> On Thu, Feb 16, 2012 at 12:38 AM, Mark question <markq2011@gmail.com>
> wrote:
> > Thanks for the reply Srinivas, so option 2 will be enough, however, when
> I
> > tried setting it to 512MB, I see through the system monitor that the map
> > task is given 275MB of real memory!!
> > Is that normal in hadoop to go over the upper bound of memory given by
> the
> > property mapred.child.java.opts.
> >
> > Mark
> >
> > On Wed, Feb 15, 2012 at 4:00 PM, Srinivas Surasani <vasajb@gmail.com>
> wrote:
> >
> >> Hey Mark,
> >>
> >> Yes, you can limit the memory for each task with
> >> "mapred.child.java.opts" property. Set this to final if no developer
> >> has to change it .
> >>
> >> Little intro to "mapred.task.default.maxvmem"
> >>
> >> This property has to be set on both the JobTracker  for making
> >> scheduling decisions and on the TaskTracker nodes for the sake of
> >> memory management. If a job doesn't specify its virtual memory
> >> requirement by setting mapred.task.maxvmem to -1, tasks are assured a
> >> memory limit set to this property. This property is set to -1 by
> >> default. This value should in general be less than the cluster-wide
> >> configuration mapred.task.limit.maxvmem. If not or if it is not set,
> >> TaskTracker's memory management will be disabled and a scheduler's
> >> memory based scheduling decisions may be affected.
> >>
> >> On Wed, Feb 15, 2012 at 5:57 PM, Mark question <markq2011@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> >  My question is what's the difference between the following two
> settings:
> >> >
> >> > 1. mapred.task.default.maxvmem
> >> > 2. mapred.child.java.opts
> >> >
> >> > The first one is used by the TT to monitor the memory usage of tasks,
> >> while
> >> > the second one is the maximum heap space assigned for each task. I
> want
> >> to
> >> > limit each task to use upto say 100MB of memory. Can I use only #2 ??
> >> >
> >> > Thank you,
> >> > Mark
> >>
> >>
> >>
> >> --
> >> -- Srinivas
> >> Srinivas@Cloudwick.com
> >>
>
>
>
> --
> -- Srinivas
> Srinivas@Cloudwick.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message