Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9A03F1892D for ; Wed, 7 Oct 2015 18:46:40 +0000 (UTC) Received: (qmail 83916 invoked by uid 500); 7 Oct 2015 18:46:36 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 83819 invoked by uid 500); 7 Oct 2015 18:46:36 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 83809 invoked by uid 99); 7 Oct 2015 18:46:36 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Oct 2015 18:46:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 00F42C42E5 for ; Wed, 7 Oct 2015 18:46:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.998 X-Spam-Level: **** X-Spam-Status: No, score=4.998 tagged_above=-999 required=6.31 tests=[FSL_HELO_BARE_IP_2=1.999, HTML_MESSAGE=3, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id xEf-k_FTmPvg for ; Wed, 7 Oct 2015 18:46:35 +0000 (UTC) Received: from relayvx11b.securemail.intermedia.net (relayvx11b.securemail.intermedia.net [64.78.52.184]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id F3747439C4 for ; Wed, 7 Oct 2015 18:46:34 +0000 (UTC) Received: from securemail.intermedia.net (localhost [127.0.0.1]) by emg-ca-1-1.localdomain (Postfix) with ESMTP id 55E6D53E88; Wed, 7 Oct 2015 11:46:34 -0700 (PDT) Subject: Re: spark multi tenancy MIME-Version: 1.0 x-echoworx-msg-id: f26166fb-dbb0-489a-a9ce-cf8033fae86b x-echoworx-emg-received: Wed, 7 Oct 2015 11:46:34.311 -0700 x-echoworx-action: delivered Received: from 10.254.155.14 ([10.254.155.14]) by emg-ca-1-1 (JAMES SMTP Server 2.3.2) with SMTP ID 402; Wed, 7 Oct 2015 11:46:34 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (unknown [10.224.117.102]) by emg-ca-1-1.localdomain (Postfix) with ESMTP id 193FA53E88; Wed, 7 Oct 2015 11:46:34 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) by MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Wed, 7 Oct 2015 11:46:33 -0700 Received: from MBX080-W4-CO-2.exch080.serverpod.net ([10.224.117.102]) by mbx080-w4-co-2.exch080.serverpod.net ([10.224.117.102]) with mapi id 15.00.1044.021; Wed, 7 Oct 2015 11:46:33 -0700 From: Steve Loughran To: ayan guha CC: user , Dominik Fries Thread-Topic: spark multi tenancy Thread-Index: AdEA15NULgkGB8VRSTqF6EslUumz2gARlogAAAEibIAAEiqBAA== Date: Wed, 7 Oct 2015 18:46:32 +0000 Message-ID: <20545910-AF5D-4644-8FD0-2BA5994BD6FF@hortonworks.com> References: <39659BA7-FD40-4449-B54C-EEA7F36ADEA5@hortonworks.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [31.53.132.251] x-source-routing-agent: Processed Content-Type: multipart/alternative; boundary="_000_20545910AF5D46448FD02BA5994BD6FFhortonworkscom_" --_000_20545910AF5D46448FD02BA5994BD6FFhortonworkscom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On 7 Oct 2015, at 11:06, ayan guha > wrote: Can queues also be used to separate workloads? yes; that's standard practise. Different YARN queues can have different max= imum memory & CPU, and you can even tag queues as "pre-emptible", so more i= mportant work can kill containers if there isn't space on the cluster https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Capacit= yScheduler.html#Queue_Properties --_000_20545910AF5D46448FD02BA5994BD6FFhortonworkscom_ Content-Type: text/html; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable
On 7 Oct 2015, at 11:06, ayan guha <guha.ayan@gmail.com> wrote:

Can queues also be used to separate workloads?


yes; that's standard practise. Different YARN queues can have different max= imum memory & CPU, and you can even tag queues as "pre-emptible&qu= ot;, so more important work can kill containers if there isn't space on the= cluster


--_000_20545910AF5D46448FD02BA5994BD6FFhortonworkscom_--