Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0074C17847 for ; Tue, 21 Oct 2014 06:43:07 +0000 (UTC) Received: (qmail 76346 invoked by uid 500); 21 Oct 2014 06:43:02 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 76228 invoked by uid 500); 21 Oct 2014 06:43:02 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 76217 invoked by uid 99); 21 Oct 2014 06:43:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Oct 2014 06:43:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_IMAGE_ONLY_28,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of stransky.ja@gmail.com designates 209.85.220.180 as permitted sender) Received: from [209.85.220.180] (HELO mail-vc0-f180.google.com) (209.85.220.180) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Oct 2014 06:42:36 +0000 Received: by mail-vc0-f180.google.com with SMTP id le20so21080vcb.11 for ; Mon, 20 Oct 2014 23:42:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=XJGuHGvETRdoJTpncTxB7VzGOmokFiLebbduF2qHCn4=; b=TKe0NwcD+2H8D5Ad9gqzlpSAmYa75ODcyF27I7gRcQvgAPFWSi1//rek6n4c7tVN0m 8S/ev+XNWNhHQZRJbvEHDllWTLiib6o+WsG7mOiPjEL4t6MqfsFKWPmkJkg55e8jTYdY nhvv9fseMyCVPEY7FRK2NVCeDQzhZvv6ZTSrXSnTsjuL3NBpaehVECNFbvXt5HugHdA1 JWN7MP897pVYxnYxUy/9BCfzqRxfxSsbtS5FO75tPGW1BYLcVQnPtJQDPtd49skEeDZU /FCoOFuF5pcKo+XAVzYtlVznARG3JOhCOMLgQOqIM+ab5HeGdQ2FxmzGepvLbbKj5VQB WLhw== MIME-Version: 1.0 X-Received: by 10.220.124.4 with SMTP id s4mr42076vcr.38.1413873755383; Mon, 20 Oct 2014 23:42:35 -0700 (PDT) Received: by 10.52.116.71 with HTTP; Mon, 20 Oct 2014 23:42:35 -0700 (PDT) In-Reply-To: References: Date: Tue, 21 Oct 2014 08:42:35 +0200 Message-ID: Subject: Re: How to limit the number of containers requested by a pig script? From: Jakub Stransky To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0115fe3af26e790505e922fb X-Virus-Checked: Checked by ClamAV on apache.org --089e0115fe3af26e790505e922fb Content-Type: text/plain; charset=UTF-8 Hello, as far as I understand. Number of mappers you cannot drive. The number of reducers you can control via PARALEL keyword. Number of containers on a node is given by following combination of settings: yarn.nodemanager.resource.memory-mb - set on a cluster. And following properties can be "modified" from your script setting to a different number, mapreduce.map.memory.mb and mapreduce.reduce.memory.mb. Hope this helps On 21 October 2014 07:31, Sunil S Nandihalli wrote: > Hi Everybody, > I would like to know how I can limit the number of concurrent containers > requested(and used ofcourse) by my pig-script (not as a yarn queue > configuration or some such stuff.. I want to limit it from outside on a > per job basis. I would ideally like to set the number in my pig-script.) > Can I do this? > Thanks, > Sunil. > -- Jakub Stransky cz.linkedin.com/in/jakubstransky --089e0115fe3af26e790505e922fb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello,

as far as I understand. Number o= f mappers you cannot drive. The number of reducers you can control via PARA= LEL keyword. Number of containers on a node is given by following combinati= on of settings:
yarn.nodemanager.resource.memory-mb - set on a cl= uster. And following properties can be "modified" from your scrip= t setting to a different number,=C2=A0mapreduce.map.memory.mb and=C2=A0mapr= educe.reduce.memory.mb.

Hope this helps
<= /div>

On 21 Octobe= r 2014 07:31, Sunil S Nandihalli <sunil.nandihalli@gmail.com&= gt; wrote:
Hi Eve= rybody,
=C2=A0I would like to know how I can limit the number of concur= rent containers requested(and used ofcourse) =C2=A0by my pig-script (not as= a yarn queue configuration or some such stuff..=C2=A0 I want to limit it f= rom outside on a =C2=A0per job basis. I would ideally like to set the numbe= r in my pig-script.) Can I do this?
Thanks,
Sunil.



--
--089e0115fe3af26e790505e922fb--