spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corey Nolet <cjno...@gmail.com>
Subject Spark 1.2.x Yarn Auxiliary Shuffle Service
Date Tue, 27 Jan 2015 20:00:23 GMT
I've read that this is supposed to be a rather significant optimization to
the shuffle system in 1.1.0 but I'm not seeing much documentation on
enabling this in Yarn. I see github classes for it in 1.2.0 and a property
"spark.shuffle.service.enabled" in the spark-defaults.conf.

The code mentions that this is supposed to be run inside the Nodemanager so
I'm assuming it needs to be wired up in the yarn-site.xml under the
"yarn.nodemanager.aux-services" property?

Mime
View raw message