spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: External Shuffle service over yarn
Date Fri, 26 Jun 2015 07:54:09 GMT
A second advantage is that it allows individual Executors to go into GC
pause (or even crash) and still allow other Executors to read shuffle data
and make progress, which tends to improve stability of memory-intensive
jobs.

On Thu, Jun 25, 2015 at 11:42 PM, Sandy Ryza <sandy.ryza@cloudera.com>
wrote:

> Hi Yash,
>
> One of the main advantages is that, if you turn dynamic allocation on, and
> executors are discarded, your application is still able to get at the
> shuffle data that they wrote out.
>
> -Sandy
>
> On Thu, Jun 25, 2015 at 11:08 PM, yash datta <saucam@gmail.com> wrote:
>
>> Hi devs,
>>
>> Can someone point out if there are any distinct advantages of using
>> external shuffle service over yarn (runs on node manager  as an auxiliary
>> service
>>
>> https://issues.apache.org/jira/browse/SPARK-3797)  instead of the
>> default execution in the executor containers ?
>>
>> Please also mention if you have seen any differences having used both
>> ways ?
>>
>> Thanks and Best Regards
>> Yash
>>
>> --
>> When events unfold with calm and ease
>> When the winds that blow are merely breeze
>> Learn from nature, from birds and bees
>> Live your life in love, and let joy not cease.
>>
>
>

Mime
View raw message