Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type;
  b=J0Y+/uLUevAmVtO+tjaXWLpPQZ3gwV9YzwZ1IhJDqymtKNSoJiBSorm26NsTog+pj93GIPxov2Algiu7fLeFdxl9pDdnYb8xgoelb/pwAlzNMc+4UEjuppVlxB57Kt2WJhrDQAqW/JJjz0JNYMGWWE7AsD+hnEax+NvzxVevqxE=;
References: 
 <CAPJKw67FGGVUZpoXyHzjHM_KOvU6zdK97TrmYiTYCd7F_Met9w@mail.gmail.com>
 <9C9A947B-55F4-4225-85A3-631299DC4C29@hortonworks.com>
Message-ID: <1316078059.14176.YahooMailNeo@web121613.mail.ne1.yahoo.com>
Date: Thu, 15 Sep 2011 02:14:19 -0700 (PDT)
From: Junping Du <junpingdu@yahoo.com>
Reply-To: Junping Du <junpingdu@yahoo.com>
Subject: Re: Adding Elasticity to Hadoop MapReduce
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Cc: "acm@hortonworks.com" <acm@hortonworks.com>
In-Reply-To: <9C9A947B-55F4-4225-85A3-631299DC4C29@hortonworks.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="-1184114812-1800494582-1316078059=:14176"

---1184114812-1800494582-1316078059=:14176
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hello Arun and all,=0A=A0=A0 =A0 =A0 =A0=A0I think current hadoop have a go=
od capability of scale out but not so good at scale in. As its design for d=
edicated cluster and machines, there is not too much attention for "scale i=
n" capability in a long time. However, I noticed that there are more and mo=
re users to deploy hadoop clusters in Cloud (ec2, eucalyptus, etc.) or shar=
ed infrastructures(vmware, xen) that "scale in" capability can contribute t=
o save resource utilization for other clusters or applications. The current=
 "scale in" solution (as you proposed in previous mail) have some significa=
nt drawbacks:=0A=A0=A0 =A0 =A0 =A0 1. It doesn't use a formal way to handle=
 scale-in case but rather a temporary workaround base on a disaster recover=
y mechanism.=0A=A0=A0 =A0 =A0 =A0 2. It is not convenient, Hadoop admin hav=
e to manually kill datanode one by one(in fact, maximum to be N(replica num=
ber) -1 each time to avoid possible data loss) and wait replica back To shr=
ink a cluster from 1000 nodes to 500 nodes, how much time and effort it cou=
ld be?=0A=A0=A0 =A0 =A0 =A0 3. It is not efficient as it is not well planne=
d. Let's say both node A, B and C should be eliminated from cluster. At fir=
st, A and B will be eliminated from cluster ( suppose N =3D3), and it is po=
ssible that C can get some replicas for block in A or B. This problem is se=
rious if big shrink happens.=0A=A0=A0 =A0 =A0 =A0 Thus, I think it is neces=
sary to have a good discussion to let hadoop have this cool "elastic" featu=
res. Here I am=A0volunteer for proposing one possible solution and welcome =
better solutions:=0A=A0=A0 =A0 =A0 =A0 1. We can think of breaking out the =
assumption of coexist of Datanode and TaskTracker on one machine and let so=
me machines only have task node. I think network traffic inside a rack is n=
ot so expensive, but you may say that it waste some local I/O resource for =
machines only with task node. Hey, don't look at these machines as dedicate=
d resource for this hadoop cluster. They can be used by other clusters and =
application(so they should be eliminated at some time). To this cluster, th=
ese machines are better than nothing, right?=0A=A0=A0 =A0 =A0 =A0 =A02. The=
 percentage of machines with only task node in whole cluster is a "elastic"=
 factor for this cluster. Take a example, if this cluster want to be scalab=
le between "500"-"1000", the elastic factor could be 1/2, and it should hav=
e 500 normal machines with both data and task nodes and another 500 machine=
s with task node only.=0A=A0=A0 =A0 =A0 =A0 =A03. Elastic factor can be con=
figured by hadoop admin and non-dedicated machines in this cluster can be m=
arked through some script like what have been done in rack-awareness.=0A=A0=
=A0 =A0 =A0 =A0 =A04. One command is provided to hadoop admin to shrink the=
 cluster to the target size directly. Some policy can be applied here for w=
aiting or not waiting task completed. If target size is smaller than elasti=
c factor * current size, some data node will be killed too but in a well pl=
anned way.=0A=A0=A0 =A0 =A0 =A0 =A0My 2 cents.=0A=0AThanks,=0A=0AJunping=0A=
=0A=0A________________________________=0AFrom: Arun C Murthy <acm@hortonwor=
ks.com>=0ATo: common-dev@hadoop.apache.org=0ASent: Thursday, September 15, =
2011 5:24 AM=0ASubject: Re: Adding Elasticity to Hadoop MapReduce=0A=0A=0AO=
n Sep 14, 2011, at 1:27 PM, Bharath Ravi wrote:=0A=0A> Hi all,=0A> =0A> I'm=
 a newcomer to Hadoop development, and I'm planning to work on an idea=0A> =
that I wanted to run by the dev community.=0A> =0A> My apologies if this is=
 not the right place to post this.=0A> =0A> Amazon has an "Elastic MapReduc=
e" Service (=0A> http://aws.amazon.com/elasticmapreduce/) that runs on Hado=
op.=0A> The service allows dynamic/runtime changes in resource allocation: =
more=0A> specifically, varying the number of=0A> compute nodes that a job i=
s running on.=0A> =0A> I was wondering if such a facility could be added to=
 the publicly available=0A> Hadoop MapReduce.=0A=0AFrom a long while=A0 you=
 can bring up either DataNodes or TaskTrackers and point them (via config) =
to the NameNode/JobTracker and they will be part of the cluster.=0A=0ASimil=
arly you can just kill the DataNode or TaskTracker and the respective maste=
rs will deal with their loss.=0A=0AArun
---1184114812-1800494582-1316078059=:14176--