incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <burtona...@gmail.com>
Subject Re: Peregrine: A new map reduce framework for iterative/pipelined jobs.
Date Tue, 27 Dec 2011 11:14:58 GMT
> A key innovation here is a partitioning layout algorithm that can support
>> fast
>> many to many recovery similar to HDFS but still support partitioned
>> operation
>> with deterministic key placement.
>>
>
> Thanks for your contribution.
>
> Is here more detail info on this point?
>

yes... our design document:

http://peregrine_mapreduce.bitbucket.org/design/

I actually will probably write a paper on this...

The more I started down the partitioned filesystem approach in terms of
mapreduce the more I realized that there were some REALLY elegant
imoplementation and design issues that I did not originally appreciate ...
(so I partially got lucky).

I think this approach could be generalized to work on normal map reduce
jobs without much overhead.

-- 

Founder/CEO Spinn3r.com <http://spinn3r.com/>

Location: *San Francisco, CA*
Skype: *burtonator*

Skype-in: *(415) 871-0687*

Mime
View raw message