spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Switch from Sort based to Hash based shuffle
Date Thu, 13 Aug 2015 13:49:25 GMT
Have a look at spark.shuffle.manager, You can switch between sort and hash
with this configuration.

spark.shuffle.managersortImplementation to use for shuffling data. There
are two implementations available:sort and hash. Sort-based shuffle is more
memory-efficient and is the default option starting in 1.2.

Thanks
Best Regards

On Thu, Aug 13, 2015 at 2:56 PM, cheez <11besemjaved@seecs.edu.pk> wrote:

> I understand that the current master branch of Spark uses Sort based
> shuffle.
> Is there a way to change that to Hash based shuffle, just for experimental
> purposes by modifying the source code ?
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message