Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of glingappa@pivotal.io designates
 209.85.220.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <54415EEC.2070302@gmail.com>
References: 
 <CACeqxwQ7sMmo92RG=SFuuOhCHn2x1Rf=WvE1_9Hs38q-EBX+Kw@mail.gmail.com>
	<CALr1C9oYpmFZUnc1LK-aEtJxPGMBvqQZA7kxFbMk9+0egK=9qw@mail.gmail.com>
	<CACeqxwTcFqN6PJtz8kB9NHJts9L3a=HH-JDwjS+TL1cAWFRO3Q@mail.gmail.com>
	<COL126-DS146CAD630D36FF9B04F93A98A80@phx.gbl>
	<CAEo-6+TQA3=Yd1rw5gZJ1xi7YbtJY4Xkt_bM3Rs7zjKKZPOnsw@mail.gmail.com>
	<54415EEC.2070302@gmail.com>
Date: Fri, 17 Oct 2014 13:29:46 -0700
Message-ID: 
 <CAEMetGsA+Zxg+8Jfzb1PwJksKG=5552y=6w6t9d74xb2XWEXVg@mail.gmail.com>
Subject: Re: Dynamically set map / reducer memory
From: Girish Lingappa <glingappa@pivotal.io>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a1134c1e2d071390505a4392d

--001a1134c1e2d071390505a4392d
Content-Type: text/plain; charset=UTF-8

Peter

If you are using oozie to launch the MR jobs you can specify the memory
requirements in the workflow action specific to each job, in the workflow
xml you are using to launch the job. If you are writing your own driver
program to launch the jobs you can still set these parameters in the job
configuration you are using to launch the job.
 In the case where you modified mapred-site.xml to set your memory
requirements did you change that on the client machine where you are
launching the job?
 Please share more details on the setup and the way you are launching the
jobs so we can better understand the problem you are facing

Girish

On Fri, Oct 17, 2014 at 11:24 AM, peter 2 <regestrer@gmail.com> wrote:

>  HI Guys,
> I am trying to run a few MR jobs in a succession, some of the jobs don't
> need that much memory and others do. I want to be able to tell hadoop how
> much memory should be allocated  for the mappers of each job.
> I know how to increase the memory for a mapper JVM, through the mapred
> xml.
> I tried manually setting the  mapreduce.reduce.java.opts = -Xmx<someNumber>m
> , but wasn't picked up by the mapper jvm, the global setting was always
> been picked up .
>
> In summation
> Job 1 - Mappers need only 250 Mg of Ram
> Job2 - Mapper
>            Reducer need around - 2Gb
>
> I don't want to be able to set those restrictions prior to submitting the
> job to my hadoop cluster.
>

--001a1134c1e2d071390505a4392d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Peter<div><br></div><div>If you are using oozie to launch =
the MR jobs you can specify the memory requirements in the workflow action =
specific to each job, in the workflow xml you are using to launch the job. =
If you are writing your own driver program to launch the jobs you can still=
 set these parameters in the job configuration you are using to launch the =
job.</div><div>=C2=A0In the case where you modified mapred-site.xml to set =
your memory requirements did you change that on the client machine where yo=
u are launching the job?</div><div>=C2=A0Please share more details on the s=
etup and the way you are launching the jobs so we can better understand the=
 problem you are facing</div><div><br></div><div>Girish</div></div><div cla=
ss=3D"gmail_extra"><br><div class=3D"gmail_quote">On Fri, Oct 17, 2014 at 1=
1:24 AM, peter 2 <span dir=3D"ltr">&lt;<a href=3D"mailto:regestrer@gmail.co=
m" target=3D"_blank">regestrer@gmail.com</a>&gt;</span> wrote:<br><blockquo=
te class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc so=
lid;padding-left:1ex">
 =20
   =20
 =20
  <div bgcolor=3D"#FFFFFF" text=3D"#000000">
    HI Guys,<br>
    I am trying to run a few MR jobs in a succession, some of the jobs
    don&#39;t need that much memory and others do. I want to be able to tel=
l
    hadoop how much memory should be allocated=C2=A0 for the mappers of eac=
h
    job. <br>
    I know how to increase the memory for a mapper JVM, through the
    mapred xml. <br>
    I tried manually setting the=C2=A0 <span style=3D"color:rgb(0,0,128)"><=
/span>mapreduce.reduce.java.opts<span style=3D"color:rgb(0,0,128)"></span>
    =3D <span style=3D"color:rgb(0,0,128)"></span>-Xmx&lt;someNumber&gt;m ,=
 but wasn&#39;t picked up by
    the mapper jvm, the global setting was always been picked up . <br>
    <br>
    In summation <br>
    Job 1 - Mappers need only 250 Mg of Ram <br>
    Job2 - Mapper <br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Reducer ne=
ed around - 2Gb <br>
    <br>
    I don&#39;t want to be able to set those restrictions prior to
    submitting the job to my hadoop cluster. <br>
  </div>

</blockquote></div><br></div>

--001a1134c1e2d071390505a4392d--