hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeffrey Buell <jbu...@vmware.com>
Subject Re: Hadoop efficient resource isolation
Date Mon, 25 Feb 2013 21:37:17 GMT
This is one reason to consider virtualizing Hadoop clusters. The idea is to create multiple
virtual clusters on a single physical cluster and apply various kinds of resource controls
(CPU, memory, I/O) on the virtual machines that make up each virtual cluster. Then if any
application or VM within a virtual cluster crashes, hangs, or tries to hog resources, the
other virtual clusters will be unaffected. Multi-tenancy is also enabled since the isolation
between virtual clusters is secure. 


----- Original Message -----

From: "Hemanth Yamijala" <yhemanth@thoughtworks.com> 
To: user@hadoop.apache.org 
Sent: Thursday, February 21, 2013 8:51:04 AM 
Subject: Re: Hadoop efficient resource isolation 

Supporting a multiuser scenario like this is always hard under Hadoop. There are a few configuration
knobs that offer some administrative control and protection. 

Specifically for the problem you describe, you could probably set Mapreduce.{map|reduce}.child.ulimit
on the tasktrackers, so that any job that is exceeding these limits will be killed. Of course,
a side effect of this would be that jobs would be bound by some limits even if they legitimately
require more memory. 

But you could try starting with this. 


On Thursday, February 21, 2013, Dhanasekaran Anbalagan wrote: 

Hi Guys, 

It's possible isolation job submission for hadoop cluster, we currently running 48 machine
cluster. we monitor Hadoop is not provides efficient resource isolation. In my case we ran
for tech and research pool, When tech job some memory leak will haven, It's occupy the hole
cluster. Finally we figure out issue with tech job. It's screwed up hole hadoop cluster. finally
10 data node are dead. 

Any prevention of job submission efficient way resource allocation. When something wrong in
particular job, effect particular pool, Not effect others job. Any way to archive this 

Please guide me guys. 

My idea is, When tech user submit job means only apply job in for my case submit 24 machine.
other machine only for research user. 

It's will prevent the memory leak problem. 


Did I learn something today? If not, I wasted it. 

View raw message