hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junping Du <junpin...@yahoo.com>
Subject Re: Adding Elasticity to Hadoop MapReduce
Date Thu, 15 Sep 2011 09:14:19 GMT
Hello Arun and all,
         I think current hadoop have a good capability of scale out but not so good
at scale in. As its design for dedicated cluster and machines, there is not too much attention
for "scale in" capability in a long time. However, I noticed that there are more and more
users to deploy hadoop clusters in Cloud (ec2, eucalyptus, etc.) or shared infrastructures(vmware,
xen) that "scale in" capability can contribute to save resource utilization for other clusters
or applications. The current "scale in" solution (as you proposed in previous mail) have some
significant drawbacks:
         1. It doesn't use a formal way to handle scale-in case but rather a temporary
workaround base on a disaster recovery mechanism.
         2. It is not convenient, Hadoop admin have to manually kill datanode one by
one(in fact, maximum to be N(replica number) -1 each time to avoid possible data loss) and
wait replica back To shrink a cluster from 1000 nodes to 500 nodes, how much time and effort
it could be?
         3. It is not efficient as it is not well planned. Let's say both node A, B and
C should be eliminated from cluster. At first, A and B will be eliminated from cluster ( suppose
N =3), and it is possible that C can get some replicas for block in A or B. This problem is
serious if big shrink happens.
         Thus, I think it is necessary to have a good discussion to let hadoop have this
cool "elastic" features. Here I am volunteer for proposing one possible solution and welcome
better solutions:
         1. We can think of breaking out the assumption of coexist of Datanode and TaskTracker
on one machine and let some machines only have task node. I think network traffic inside a
rack is not so expensive, but you may say that it waste some local I/O resource for machines
only with task node. Hey, don't look at these machines as dedicated resource for this hadoop
cluster. They can be used by other clusters and application(so they should be eliminated at
some time). To this cluster, these machines are better than nothing, right?
          2. The percentage of machines with only task node in whole cluster is a "elastic"
factor for this cluster. Take a example, if this cluster want to be scalable between "500"-"1000",
the elastic factor could be 1/2, and it should have 500 normal machines with both data and
task nodes and another 500 machines with task node only.
          3. Elastic factor can be configured by hadoop admin and non-dedicated machines
in this cluster can be marked through some script like what have been done in rack-awareness.
          4. One command is provided to hadoop admin to shrink the cluster to the target
size directly. Some policy can be applied here for waiting or not waiting task completed.
If target size is smaller than elastic factor * current size, some data node will be killed
too but in a well planned way.
          My 2 cents.



From: Arun C Murthy <acm@hortonworks.com>
To: common-dev@hadoop.apache.org
Sent: Thursday, September 15, 2011 5:24 AM
Subject: Re: Adding Elasticity to Hadoop MapReduce

On Sep 14, 2011, at 1:27 PM, Bharath Ravi wrote:

> Hi all,
> I'm a newcomer to Hadoop development, and I'm planning to work on an idea
> that I wanted to run by the dev community.
> My apologies if this is not the right place to post this.
> Amazon has an "Elastic MapReduce" Service (
> http://aws.amazon.com/elasticmapreduce/) that runs on Hadoop.
> The service allows dynamic/runtime changes in resource allocation: more
> specifically, varying the number of
> compute nodes that a job is running on.
> I was wondering if such a facility could be added to the publicly available
> Hadoop MapReduce.

From a long while  you can bring up either DataNodes or TaskTrackers and point them (via
config) to the NameNode/JobTracker and they will be part of the cluster.

Similarly you can just kill the DataNode or TaskTracker and the respective masters will deal
with their loss.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message