Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of amits@infolinks.com
 designates 207.126.144.115 as permitted sender)
MIME-Version: 1.0
Date: Sat, 6 Jul 2013 17:12:01 +0300
Message-ID: 
 <CAAMYKhqX6GCmZRB5O3PnnRR-sFT_0bzd1nq0uxKMb0TRE4MDrw@mail.gmail.com>
Subject: Using CapacityScheduler to divide resources between jobs (not users)
From: Amit Sela <amits@infolinks.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e01494eee23716904e0d865d4

--089e01494eee23716904e0d865d4
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I'm running Hadoop 1.0.4 on a modest cluster (~20 machines).
The jobs running on the cluster can be divided (resource wise) as follows:

1. Very short jobs: less then 1 minute.
2. Normal jobs: 2-3 minutes up to an hour or two.
3. Very long jobs: days of processing. (still not active and the reason for
my inquiries here).

I was thinking of using the CapacityScheduler and divide the cluster
resources so that the long jobs can run without disturbing the other jobs.
I read that such job queues should be upper bound as well since it may use
the entire cluster resources once it's free but since it takes a long time
to finish, it won't release them to other queues as it should. Is it so ?
Any advise about using the CapacityScheduler in that use case ?

Thanks, and sorry for re-sending this message.

Amit.

--089e01494eee23716904e0d865d4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi all,=A0</div><div><br></div><div>I&#39;m running H=
adoop 1.0.4 on a modest cluster (~20 machines).</div><div>The jobs running =
on the cluster can be divided (resource wise) as follows:</div><div><br></d=
iv>
<div>1. Very short jobs: less then 1 minute.</div><div>2. Normal jobs: 2-3 =
minutes up to an hour or two.</div><div>3. Very long jobs: days of processi=
ng. (still not active and the reason for my inquiries here).</div><div>
<br></div><div>I was thinking of using the CapacityScheduler and divide the=
 cluster resources so that the long jobs can run without disturbing the oth=
er jobs.</div><div style>I read that such job queues should be upper bound =
as well since it may use the entire cluster resources once it&#39;s free bu=
t since it takes a long time to finish, it won&#39;t release them to other =
queues as it should. Is it so ?</div>
<div style>Any advise about using the=A0CapacityScheduler in that use case =
?</div><div style><br></div><div style>Thanks, and sorry for re-sending thi=
s message.</div><div style><br></div><div style>Amit.</div></div>

--089e01494eee23716904e0d865d4--