Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 364FA186BA for ; Thu, 13 Aug 2015 16:51:01 +0000 (UTC) Received: (qmail 72448 invoked by uid 500); 13 Aug 2015 16:50:59 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 72341 invoked by uid 500); 13 Aug 2015 16:50:59 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 72326 invoked by uid 99); 13 Aug 2015 16:50:59 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2015 16:50:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 13895DD7F4 for ; Thu, 13 Aug 2015 16:50:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.294 X-Spam-Level: ***** X-Spam-Status: No, score=5.294 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Dd6wNIIruvJ3 for ; Thu, 13 Aug 2015 16:50:49 +0000 (UTC) Received: from mail-ig0-f182.google.com (mail-ig0-f182.google.com [209.85.213.182]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id DCC9537F2F for ; Thu, 13 Aug 2015 13:49:26 +0000 (UTC) Received: by igbpg9 with SMTP id pg9so36165831igb.0 for ; Thu, 13 Aug 2015 06:49:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=WLwYlyCMJ0JiJgcg9bUv1H0tkRmgT+IXevuP0HG1z2U=; b=mpC+kMFwIuGAHtjlcplI8JR8mQvrR4B1nI8WepuYW1bKU3/1mSsWrudSjrP2y8Ebzq rUrz1Q2j/Eyic3iVOdNIecyMDf2aRMSyHzNJExNU8ZxIEwphJLp/oQGuupSP/UjTPbTj z8ItsLtRQxHf3gH+6BJktyIx6coGS33IZEKW8BW7TZ+PRqvQaRaAy+fF3umxAl7c3RZd 2FU9woVF9OXxnG20tlmQ6xylp5x8EtUc5OkTQImSrcpodE8dfAzRAAZURDFLrpMZ2Ibg asKRigSvA+1IzxPnZL1spJbSKywV87bp0nEnYRoWkF+W/h1+zNmlNfVNegrTSWgGUel0 mgWA== X-Gm-Message-State: ALoCoQmjNO33PzX69uxLGmHqVqDkZ7Z0xX7KWjCD5/Bm9KEVaRn4BGeCPmf59I32lhh4Zc7v/6LZ MIME-Version: 1.0 X-Received: by 10.50.78.133 with SMTP id b5mr3237570igx.32.1439473765839; Thu, 13 Aug 2015 06:49:25 -0700 (PDT) Received: by 10.107.181.12 with HTTP; Thu, 13 Aug 2015 06:49:25 -0700 (PDT) In-Reply-To: <1439457988128-13661.post@n3.nabble.com> References: <1439457988128-13661.post@n3.nabble.com> Date: Thu, 13 Aug 2015 19:19:25 +0530 Message-ID: Subject: Re: Switch from Sort based to Hash based shuffle From: Akhil Das To: cheez <11besemjaved@seecs.edu.pk> Cc: dev Content-Type: multipart/alternative; boundary=089e0122f08e7a3e46051d319a4f --089e0122f08e7a3e46051d319a4f Content-Type: text/plain; charset=UTF-8 Have a look at spark.shuffle.manager, You can switch between sort and hash with this configuration. spark.shuffle.managersortImplementation to use for shuffling data. There are two implementations available:sort and hash. Sort-based shuffle is more memory-efficient and is the default option starting in 1.2. Thanks Best Regards On Thu, Aug 13, 2015 at 2:56 PM, cheez <11besemjaved@seecs.edu.pk> wrote: > I understand that the current master branch of Spark uses Sort based > shuffle. > Is there a way to change that to Hash based shuffle, just for experimental > purposes by modifying the source code ? > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org > For additional commands, e-mail: dev-help@spark.apache.org > > --089e0122f08e7a3e46051d319a4f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Have a look at=C2=A0spark.shuffle.manager, You= can switch between sort and hash with this configuration.
spark.shuffle.managersortImplementation to use for sh= uffling data. There are two implementations available:sort=C2=A0and=C2=A0hash. Sort-based shuffle is more memory-effi= cient and is the default option starting in 1.2.
<= /div>

Thanks
Best Regards
=

On Thu, Aug 13, 2015 at 2:56 PM, cheez <11besemjaved@seecs.edu.pk> wrote:
I understand that the current master branch of Spark uses Sor= t based shuffle.
Is there a way to change that to Hash based shuffle, just for experimental<= br> purposes by modifying the source code ?



--
View this message in context: http://apache-spark-developers= -list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp= 13661.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.c= om.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


--089e0122f08e7a3e46051d319a4f--