hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Yamijala <yhema...@thoughtworks.com>
Subject Re: Hadoop efficient resource isolation
Date Thu, 21 Feb 2013 16:51:04 GMT
Supporting a multiuser scenario like this is always hard under Hadoop.
There are a few configuration knobs that offer some administrative control
and protection.

Specifically for the problem you describe, you could probably set
Mapreduce.{map|reduce}.child.ulimit on the tasktrackers, so that any job
that is exceeding these limits will be killed. Of course, a side effect of
this would be that jobs would be bound by some limits even if they
legitimately require more memory.

But you could try starting with this.


On Thursday, February 21, 2013, Dhanasekaran Anbalagan wrote:

> Hi Guys,
> It's possible isolation job submission for hadoop cluster, we currently
> running 48 machine cluster. we  monitor Hadoop is not provides efficient
> resource isolation. In my case we ran for tech and research pool, When tech
> job some memory leak will haven, It's occupy the hole cluster.  Finally
> we figure out  issue with tech job. It's  screwed up hole hadoop cluster.
> finally 10 data node  are dead.
> Any prevention of job submission efficient way resource allocation. When
> something wrong in   particular job, effect particular pool, Not effect
> others job. Any way to archive this
> Please guide me guys.
> My idea is, When tech user submit job means only apply job in for my
> case submit 24 machine. other machine only for research user.
> It's will prevent the memory leak problem.
> -Dhanasekaran.
> Did I learn something today? If not, I wasted it.

View raw message