spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <sai.sai.s...@gmail.com>
Subject Re: How about the fetch the shuffle data in one same machine?
Date Wed, 10 May 2017 10:14:41 GMT
There is a JIRA about this thing (
https://issues.apache.org/jira/browse/SPARK-6521). In the current Spark
shuffle fetch still leverages Netty even two executors are on the same
node, but according to the test on the JIRA, the performance is close
whether to bypass network or not. From my understanding, kernel will not
transfer data into NIC if it is just a loopback communication (please
correct me if I'm wrong).

On Wed, May 10, 2017 at 5:53 PM, raintung li <raintung.li@gmail.com> wrote:

> Hi all,
>
> Now Spark only think the executorId same that fetch local file, but for
> same IP different ExecutorId will fetch using network that actually it can
> be fetch in the local Or Loopback.
>
> Apparently fetch the local file that it is fast that can use the LVS
> cache.
>
> How do you think?
>
> Regards
> -Raintung
>

Mime
View raw message