Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EDC529991 for ; Thu, 23 May 2013 17:25:44 +0000 (UTC) Received: (qmail 28550 invoked by uid 500); 23 May 2013 17:25:40 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 28275 invoked by uid 500); 23 May 2013 17:25:39 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 28086 invoked by uid 99); 23 May 2013 17:25:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 May 2013 17:25:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of amalgjos@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 May 2013 17:25:26 +0000 Received: by mail-ie0-f176.google.com with SMTP id at1so9482005iec.35 for ; Thu, 23 May 2013 10:25:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=v8RGw460QC9EWIkjoDHqZDipMbp7VP6jE58I+UIhEEc=; b=a87RqbJrbRlPfoOeN87WuKDI7EwulGDLAzjsvGoH6KeSJmRMdV4S5sJyaY5/NHLUFU DWjVe8lOwcuWWRazPVFrgDVyQAnCXTAh2zvKZ3a1RDdBWTF/KSwftJdC+aT5ri2FT4FE uhBO3nXsR/xpJVtuhHTrPCVD3buPzkZpvtG90nPkDRzLhweSi1gktP1QGisCxdCydjI2 RBZ//avEp7cQhLKxl2edpdeN0u3B/I7Pcf7sVUl+VWb/NOSII526MDRTUrGdnPyBJvk6 OiMKr5Y2ZUpkT6ljjco0HChFmgwALDJalHmFSNA+xm65HWzLQb+y9zZWlAxPqsezghoz sLug== MIME-Version: 1.0 X-Received: by 10.43.162.7 with SMTP id mi7mr10111822icc.34.1369329906071; Thu, 23 May 2013 10:25:06 -0700 (PDT) Received: by 10.64.137.197 with HTTP; Thu, 23 May 2013 10:25:06 -0700 (PDT) In-Reply-To: References: <3D527B17-E3EC-4988-BEEC-D3CAF8448658@oit.gatech.edu> Date: Thu, 23 May 2013 22:55:06 +0530 Message-ID: Subject: Re: Is there a way to limit # of hadoop tasks per user at runtime? From: Amal G Jose To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c21242a1be4604dd65f68c X-Virus-Checked: Checked by ClamAV on apache.org --001a11c21242a1be4604dd65f68c Content-Type: text/plain; charset=ISO-8859-1 You can use capacity scheduler also. In that you can create some queues, each of specific capacity. Then you can submit jobs to that specific queue at runtime or you can configure it as direct submission. On Wed, May 22, 2013 at 3:27 AM, Sandy Ryza wrote: > Hi Mehmet, > > Are you using MR1 or MR2? > > The fair scheduler, present in both versions, but configured slightly > differently, allows you to limit the number of map and reduce tasks in a > queue. The configuration can be updated at runtime by modifying the > scheduler's allocations file. It also has a feature that automatically > maps jobs to queues based on the user submitted them. > > Here are links to documentation in MR1 and MR2: > http://hadoop.apache.org/docs/stable/fair_scheduler.html > > http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html > > -Sandy > > > > On Tue, May 21, 2013 at 2:43 PM, Mehmet Belgin < > mehmet.belgin@oit.gatech.edu> wrote: > >> Hi Everyone, >> >> I was wondering if there is a way for limiting the number of tasks >> (map+reduce) *per user* at runtime? Using an environment variable perhaps? >> I am asking this from a resource provisioning perspective. I am trying to >> come up with a N-token licensing system for multiple users to use our >> limited hadoop resources simultaneously. That is, when user A checks out 6 >> tokens, he/she can only run 6 hadoop tasks. >> >> If there is no such thing in hadoop, has anyone tried to integrate hadoop >> with torque/moab (or any other RM or scheduler)? Any advice in that >> direction will be appreciated :) >> >> Thanks in advance, >> -Mehmet >> >> >> >> >> >> >> > --001a11c21242a1be4604dd65f68c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
You can use capacity scheduler also. In that you can creat= e some queues, each of specific capacity. Then you can submit jobs to that = specific queue at runtime or you can configure it as direct submission.


On Wed, May 2= 2, 2013 at 3:27 AM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
Hi Mehmet,

Are you using MR1 or MR2?

The fair scheduler, p= resent in both versions, but configured slightly differently, allows you to= limit the number of map and reduce tasks in a queue. =A0The configuration = can be updated at runtime by modifying the scheduler's allocations file= . =A0It also has a feature that automatically maps jobs to queues based on = the user submitted them.

Here are links to documentation in MR1 and MR2:
http://hadoop.apache.org/docs/stable/fair_scheduler.html=

-Sandy



On Tue, May 21, 2013 at 2:43 PM, Mehmet Belgin <m= ehmet.belgin@oit.gatech.edu> wrote:
Hi Every= one,

I was wondering if there is a way for limiting the = number of tasks (map+reduce) *per user* at runtime? Using an environment va= riable perhaps? I am asking this from a resource provisioning perspective. = I am trying to come up with a N-token licensing system for multiple users t= o use our limited hadoop resources simultaneously. That is, when user A che= cks out 6 tokens, =A0he/she can only run 6 hadoop tasks.=A0

If there is no such thing in hadoop, has anyone tried t= o integrate hadoop with torque/moab (or any other RM or scheduler)? Any adv= ice in that direction will be appreciated :)

Thanks in advance,
-Mehmet








--001a11c21242a1be4604dd65f68c--