spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jerryshao <...@git.apache.org>
Subject [GitHub] spark issue #19788: [SPARK-9853][Core] Optimize shuffle fetch of contiguous ...
Date Mon, 27 Nov 2017 07:16:40 GMT
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/19788
  
    @yucai I'm thinking of the necessity to add this new configuration `spark.shuffle.continuousFetch`
like you mentioned above. This PR you proposed is actually a superset of previous way, it
is compatible with original shuffle way if length = 1. The configuration here is only used
to keep compatible for external shuffle service, I think it is not so intuitive and user may
confused whether this should be enabled or not (since since conf is functionality-orientiated).
Besides do we need to guarantee forward compatible, also is there a transparent way to automatically
switch between two shuffles without configuration?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message