spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mateiz <>
Subject [GitHub] spark pull request: SPARK-2787: Make sort-based shuffle write file...
Date Sun, 10 Aug 2014 22:07:36 GMT
Github user mateiz commented on a diff in the pull request:
    --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
    @@ -246,8 +250,13 @@ object SparkEnv extends Logging {
    -    val shuffleManager = instantiateClass[ShuffleManager](
    -      "spark.shuffle.manager", "org.apache.spark.shuffle.hash.HashShuffleManager")
    +    // Let the user specify short names for shuffle managers
    +    val shortShuffleMgrNames = Map(
    +      "hash" -> "org.apache.spark.shuffle.hash.HashShuffleManager",
    +      "sort" -> "org.apache.spark.shuffle.sort.SortShuffleManager")
    +    val shuffleMgrName = conf.get("spark.shuffle.manager", "hash")
    --- End diff --
    I'd rather not change the configuration under the user, that would be confusing if they
later print it or look in the web UI. Instead, maybe add a SparkEnv.getShuffleManagerClass(conf:
SparkConf) that can return the real class name.
    Also I'd be fine initializing the ShuffleBlockManager after the ShuffleManager if that
works, and using isInstanceOf. That would be the cleanest.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message