spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@alumni.cmu.edu>
Subject Re: CoHadoop Papers
Date Mon, 15 Sep 2014 22:21:53 GMT
This feature is called "block affinity groups" and it's been under
discussion for a while, but isn't fully implemented yet.  HDFS-2576 is
not a complete solution because it doesn't change the way the balancer
works, just the initial placement of blocks.  Once heterogeneous
storage management (HDFS-2832) is implemented, you will be able to get
a similar effect through using separate storages, at the cost of
fragmenting the backing store somewhat.

Of course, "co-locating related data blocks" is often bad, not good,
because it reduces the amount of parallelism a single job can exploit,
and can increase the chance of losing an entire dataset due to node
failures.  That's one reason why the current semi-random placement
strategy has lasted so long.  In other words, this is
workload-dependent.

best,
Colin

On Tue, Aug 26, 2014 at 5:20 AM, Gary Malouf <malouf.gary@gmail.com> wrote:
> It appears support for this type of control over block placement is going
> out in the next version of HDFS:
> https://issues.apache.org/jira/browse/HDFS-2576
>
>
> On Tue, Aug 26, 2014 at 7:43 AM, Gary Malouf <malouf.gary@gmail.com> wrote:
>
>> One of my colleagues has been questioning me as to why Spark/HDFS makes no
>> attempts to try to co-locate related data blocks.  He pointed to this
>> paper: http://www.vldb.org/pvldb/vol4/p575-eltabakh.pdf from 2011 on the
>> CoHadoop research and the performance improvements it yielded for
>> Map/Reduce jobs.
>>
>> Would leveraging these ideas for writing data from Spark make sense/be
>> worthwhile?
>>
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message