Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of bejoy.hadoop@gmail.com
 designates 209.85.216.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAO7hTbNrysAs0NuKHMHdfdiYmEN5NxRRYO48AopyvOV5MbDaPg@mail.gmail.com>
References: 
 <CAO7hTbNrysAs0NuKHMHdfdiYmEN5NxRRYO48AopyvOV5MbDaPg@mail.gmail.com>
Date: Tue, 16 Apr 2013 14:40:47 +0530
Message-ID: 
 <CACD21EOXO2=FRH_Wud_v6+ZozFD5QifTrsaai_BeTn0Gztai9g@mail.gmail.com>
Subject: Re: VM reuse!
From: Bejoy Ks <bejoy.hadoop@gmail.com>
To: "mapreduce-user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=e89a8f502904ba09eb04da76be50

--e89a8f502904ba09eb04da76be50
Content-Type: text/plain; charset=ISO-8859-1

Hi Rahul

If you look at larger cluster and jobs that involve larger input data sets.
The data would be spread across the whole cluster, and a single node might
have  various blocks of that entire data set. Imagine you have a cluster
with 100 map slots and your job has 500 map tasks, now in that case there
should be multiple map tasks in a single task tracker based on slot
availability.

Here if you enable jvm reuse, all tasks related to a job on a single
TaskTracker would use the same jvm. The benefit here is just the time you
are saving in spawning and cleaning up jvm for individual tasks.


On Tue, Apr 16, 2013 at 2:04 PM, Rahul Bhattacharjee <
rahul.rec.dgp@gmail.com> wrote:

> Hi,
>
> I have a question related to VM reuse in Hadoop.I now understand the
> purpose of VM reuse , but I am wondering how is it useful.
>
> Example. for VM reuse to be effective or kicked in , we need more than one
> mapper task to be submitted to a single node (for the same job).Hadoop
> would consider spawning mappers into nodes which actually contains the data
> , it might rarely happen that multiple mappers are allocated to a single
> task tracker. And even if a single task nodes gets to run multiple mappers
> then it might as well run in parallel in multiple VM rather than
> sequentially in a single VM.
>
> I am sure I am missing some link here , please help me find that.
>
> Thanks,
> Rahul
>

--e89a8f502904ba09eb04da76be50
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi Rahul<br><br></div>If you look at larger clus=
ter and jobs that involve larger input data sets. The data would be spread =
across the whole cluster, and a single node might have=A0 various blocks of=
 that entire data set. Imagine you have a cluster with 100 map slots and yo=
ur job has 500 map tasks, now in that case there should be multiple map tas=
ks in a single task tracker based on slot availability.<br>
<br></div>Here if you enable jvm reuse, all tasks related to a job on a sin=
gle TaskTracker would use the same jvm. The benefit here is just the time y=
ou are saving in spawning and cleaning up jvm for individual tasks. <br>
<br><br></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote"=
>On Tue, Apr 16, 2013 at 2:04 PM, Rahul Bhattacharjee <span dir=3D"ltr">&lt=
;<a href=3D"mailto:rahul.rec.dgp@gmail.com" target=3D"_blank">rahul.rec.dgp=
@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_default=
" style=3D"font-family:courier new,monospace">Hi,<br><br></div><div class=
=3D"gmail_default" style=3D"font-family:courier new,monospace">
I have a question related to VM reuse in Hadoop.I now understand the purpos=
e of VM reuse , but I am wondering how is it useful.<br>

<br></div><div class=3D"gmail_default" style=3D"font-family:courier new,mon=
ospace">Example. for VM reuse to be effective or kicked in , we need more t=
han one mapper task to be submitted to a single node (for the same job).Had=
oop would consider spawning mappers into nodes which actually contains the =
data , it might rarely happen that multiple mappers are allocated to a sing=
le task tracker. And even if a single task nodes gets to run multiple mappe=
rs then it might as well run in parallel in multiple VM rather than sequent=
ially in a single VM.<br>


<br></div><div class=3D"gmail_default" style=3D"font-family:courier new,mon=
ospace">I am sure I am missing some link here , please help me find that.<b=
r><br></div><div class=3D"gmail_default" style=3D"font-family:courier new,m=
onospace">


Thanks,<br>Rahul<br></div></div>
</blockquote></div><br></div>

--e89a8f502904ba09eb04da76be50--