Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of markq2011@gmail.com designates
 209.85.212.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPO4T7v0uhcTy1_c9Ts6vorWm322mmPMyVpVFkR=LNZW+pHmSQ@mail.gmail.com>
References: 
 <CALR_EGKpeX9gokBuhckW1yy=-Tm8eGjh75-7yeHh+waYyB--wQ@mail.gmail.com>
	<CAPO4T7t=8iioGyCCC3RbbZyL=TJa8-CSuUeCEXgZjWhu_5AirA@mail.gmail.com>
	<CALR_EGJc3HqBPaGAfJz2JSoMbCHanktsqzomSzwe06CiAOdGkg@mail.gmail.com>
	<CAPO4T7v0uhcTy1_c9Ts6vorWm322mmPMyVpVFkR=LNZW+pHmSQ@mail.gmail.com>
Date: Thu, 16 Feb 2012 09:05:33 -0800
Message-ID: 
 <CALR_EG+w2pHkKK-QPq+9XFU149HPb4YmZTFqtCfiABDsVBOaKA@mail.gmail.com>
Subject: Re: memory of mappers and reducers
From: Mark question <markq2011@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307cfd46122c3e04b917d6b6

--20cf307cfd46122c3e04b917d6b6
Content-Type: text/plain; charset=ISO-8859-1

Great! thanks a lot Srinivas !
Mark

On Thu, Feb 16, 2012 at 7:02 AM, Srinivas Surasani <vasajb@gmail.com> wrote:

> 1) Yes option 2 is enough.
> 2) Configuration variable "mapred.child.ulimit" can be used to control
> the maximum virtual memory of the child (map/reduce) processes.
>
> ** value of mapred.child.ulimit > value of mapred.child.java.opts
>
> On Thu, Feb 16, 2012 at 12:38 AM, Mark question <markq2011@gmail.com>
> wrote:
> > Thanks for the reply Srinivas, so option 2 will be enough, however, when
> I
> > tried setting it to 512MB, I see through the system monitor that the map
> > task is given 275MB of real memory!!
> > Is that normal in hadoop to go over the upper bound of memory given by
> the
> > property mapred.child.java.opts.
> >
> > Mark
> >
> > On Wed, Feb 15, 2012 at 4:00 PM, Srinivas Surasani <vasajb@gmail.com>
> wrote:
> >
> >> Hey Mark,
> >>
> >> Yes, you can limit the memory for each task with
> >> "mapred.child.java.opts" property. Set this to final if no developer
> >> has to change it .
> >>
> >> Little intro to "mapred.task.default.maxvmem"
> >>
> >> This property has to be set on both the JobTracker  for making
> >> scheduling decisions and on the TaskTracker nodes for the sake of
> >> memory management. If a job doesn't specify its virtual memory
> >> requirement by setting mapred.task.maxvmem to -1, tasks are assured a
> >> memory limit set to this property. This property is set to -1 by
> >> default. This value should in general be less than the cluster-wide
> >> configuration mapred.task.limit.maxvmem. If not or if it is not set,
> >> TaskTracker's memory management will be disabled and a scheduler's
> >> memory based scheduling decisions may be affected.
> >>
> >> On Wed, Feb 15, 2012 at 5:57 PM, Mark question <markq2011@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> >  My question is what's the difference between the following two
> settings:
> >> >
> >> > 1. mapred.task.default.maxvmem
> >> > 2. mapred.child.java.opts
> >> >
> >> > The first one is used by the TT to monitor the memory usage of tasks,
> >> while
> >> > the second one is the maximum heap space assigned for each task. I
> want
> >> to
> >> > limit each task to use upto say 100MB of memory. Can I use only #2 ??
> >> >
> >> > Thank you,
> >> > Mark
> >>
> >>
> >>
> >> --
> >> -- Srinivas
> >> Srinivas@Cloudwick.com
> >>
>
>
>
> --
> -- Srinivas
> Srinivas@Cloudwick.com
>

--20cf307cfd46122c3e04b917d6b6--