Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 671B5C1CF for ; Sat, 6 Jul 2013 14:07:54 +0000 (UTC) Received: (qmail 90127 invoked by uid 500); 6 Jul 2013 14:07:48 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 89817 invoked by uid 500); 6 Jul 2013 14:07:48 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 89810 invoked by uid 99); 6 Jul 2013 14:07:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Jul 2013 14:07:47 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of amits@infolinks.com designates 207.126.144.133 as permitted sender) Received: from [207.126.144.133] (HELO eu1sys200aog112.obsmtp.com) (207.126.144.133) by apache.org (qpsmtpd/0.29) with SMTP; Sat, 06 Jul 2013 14:07:39 +0000 Received: from mail-ie0-f176.google.com ([209.85.223.176]) (using TLSv1) by eu1sys200aob112.postini.com ([207.126.147.11]) with SMTP ID DSNKUdgkl3dM4/UFu+E6NE7GHoBN0NOAjdCn@postini.com; Sat, 06 Jul 2013 14:07:19 UTC Received: by mail-ie0-f176.google.com with SMTP id ar20so7086458iec.21 for ; Sat, 06 Jul 2013 07:07:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type :x-gm-message-state; bh=4QeOoSXLPzpiKFvjDUSpNL0Qrjjn90CzhhSrScM4fzU=; b=L8auG0XIdzSPAglB896Ecp7+E/NHs99jRQM2RQ5JlRTwwZCcBJGd3cgUd9Z7TEsj8c mbuQM5GllriEoZ2pFXVAJM/7antrnuUUjPE8r4ysGqpN2RNWX+JGwAJ1Txun25/dhjcc bshiWwaU/x5m8XD4zB6WBzP1SkUKf8gzhTxOf5RT2GPxDY3DLscHeSj+mTS70s0uLCKp RicIq3kspY9cvIjGiv0NGMMCdz1cz3x2nsNGPrysR6STJFBLJOw8WhyBGCFJYhdCcPsI VCLhFgD/sAx5nwORlg8A3JyuDNMk+NCDNm7CJtIHEM4d7QUs0RMnqygVPTRKOi0Q+jRN jdpg== X-Received: by 10.42.68.18 with SMTP id v18mr5124299ici.87.1373119638206; Sat, 06 Jul 2013 07:07:18 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.42.68.18 with SMTP id v18mr5124297ici.87.1373119638151; Sat, 06 Jul 2013 07:07:18 -0700 (PDT) Received: by 10.64.230.74 with HTTP; Sat, 6 Jul 2013 07:07:18 -0700 (PDT) Date: Sat, 6 Jul 2013 17:07:18 +0300 Message-ID: Subject: Using CapacityScheduler to divide resources between jobs (not users) From: Amit Sela To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf303636d944277e04e0d8544a X-Gm-Message-State: ALoCoQnTVRVJNm7aFlNJf3iz/syYcfnTk41HY3t67V2lHHULUAYhSCdPeID/RyrLC//aTQjhdzJN0OImaIjF6PXKyuNHDgYq7EPCIkQRSnffDsNdkF+UVKl5HxebPfAKDW8mbOFDMzkHFQOzHplPByUqkURzw4SgdV3IB5HDh2sH2VqSecGFsss= X-Virus-Checked: Checked by ClamAV on apache.org --20cf303636d944277e04e0d8544a Content-Type: text/plain; charset=ISO-8859-1 Hi all, I'm running Hadoop 1.0.4 on a modest cluster (~20 machines). The jobs running on the cluster can be divided (resource wise) as follows: 1. Very short jobs: less then 1 minute. 2. Normal jobs: 2-3 minutes up to an hour or two. 3. Very long jobs: days of processing. (still not active and the reason for my inquiries here). I was thinking of using the CapacityScheduler and divide the cluster resources something like: 1. min: --20cf303636d944277e04e0d8544a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi all,=A0

I&#= 39;m running Hadoop 1.0.4 on a modest cluster (~20 machines).
The jobs running= on the cluster can be divided (resource wise) as follows:

1. Very short jobs: less then 1 minute.
2. Normal jobs: 2-3 minutes up to an hour or t= wo.
3. Very= long jobs: days of processing. (still not active and the reason for my inq= uiries here).

I was thinking of usin= g the CapacityScheduler and divide the cluster resources something like:
1. min:=A0
=
--20cf303636d944277e04e0d8544a--