spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Takeshi Yamamuro <linguin....@gmail.com>
Subject Re: HDFS as Shuffle Service
Date Wed, 27 Apr 2016 03:59:11 GMT
Hi, all

See SPARK-1529 for related discussion.

// maropu

On Wed, Apr 27, 2016 at 12:27 PM, Saisai Shao <sai.sai.shao@gmail.com>
wrote:

> Quite curious about the benefits of using HDFS as shuffle service, also
> what's the problem of using current shuffle service?
>
>
> Thanks
> Saisai
>
> On Wed, Apr 27, 2016 at 4:31 AM, Timothy Chen <tnachen@gmail.com> wrote:
>
>> Are you suggesting to have shuffle service persist and fetch data with
>> hdfs, or skip shuffle service altogether and just write to hdfs?
>>
>> Tim
>>
>>
>> > On Apr 26, 2016, at 11:20 AM, Michael Gummelt <mgummelt@mesosphere.io>
>> wrote:
>> >
>> > Has there been any thought or work on this (or any other networked file
>> system)?  It would be valuable to support dynamic allocation without
>> depending on the shuffle service.
>> >
>> > --
>> > Michael Gummelt
>> > Software Engineer
>> > Mesosphere
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>


-- 
---
Takeshi Yamamuro

Mime
View raw message