spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jerryshao <>
Subject [GitHub] spark issue #19788: [SPARK-9853][Core] Optimize shuffle fetch of contiguous ...
Date Mon, 27 Nov 2017 07:16:40 GMT
Github user jerryshao commented on the issue:
    @yucai I'm thinking of the necessity to add this new configuration `spark.shuffle.continuousFetch`
like you mentioned above. This PR you proposed is actually a superset of previous way, it
is compatible with original shuffle way if length = 1. The configuration here is only used
to keep compatible for external shuffle service, I think it is not so intuitive and user may
confused whether this should be enabled or not (since since conf is functionality-orientiated).
Besides do we need to guarantee forward compatible, also is there a transparent way to automatically
switch between two shuffles without configuration?


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message