spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <sai.sai.s...@gmail.com>
Subject Re: HDFS as Shuffle Service
Date Wed, 27 Apr 2016 03:27:41 GMT
Quite curious about the benefits of using HDFS as shuffle service, also
what's the problem of using current shuffle service?


Thanks
Saisai

On Wed, Apr 27, 2016 at 4:31 AM, Timothy Chen <tnachen@gmail.com> wrote:

> Are you suggesting to have shuffle service persist and fetch data with
> hdfs, or skip shuffle service altogether and just write to hdfs?
>
> Tim
>
>
> > On Apr 26, 2016, at 11:20 AM, Michael Gummelt <mgummelt@mesosphere.io>
> wrote:
> >
> > Has there been any thought or work on this (or any other networked file
> system)?  It would be valuable to support dynamic allocation without
> depending on the shuffle service.
> >
> > --
> > Michael Gummelt
> > Software Engineer
> > Mesosphere
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message