Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of burtonator2011@gmail.com
 designates 74.125.82.44 as permitted sender)
MIME-Version: 1.0
Sender: burtonator2011@gmail.com
In-Reply-To: 
 <CAF7KpS-rqJgXPuUjhXkODDN9XU28wVJ28T8nBJO_ZZZjSEEG3g@mail.gmail.com>
References: 
 <CAAZU44nu8PhvXd6jktcxdb0UWK_FHwrc2ek6W+2zzE1=Hkc_SA@mail.gmail.com>
 <CAF7KpS-rqJgXPuUjhXkODDN9XU28wVJ28T8nBJO_ZZZjSEEG3g@mail.gmail.com>
From: Kevin Burton <burtonator@gmail.com>
Date: Tue, 27 Dec 2011 03:14:58 -0800
Message-ID: 
 <CAAZU44=FvOT9tzxNBGfafmseyRknSVueF8GpLVWPN3RqAFm75g@mail.gmail.com>
Subject: Re: Peregrine: A new map reduce framework for iterative/pipelined
 jobs.
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=0015174940fc8a391b04b510ff5d

--0015174940fc8a391b04b510ff5d
Content-Type: text/plain; charset=ISO-8859-1

> A key innovation here is a partitioning layout algorithm that can support
>> fast
>> many to many recovery similar to HDFS but still support partitioned
>> operation
>> with deterministic key placement.
>>
>
> Thanks for your contribution.
>
> Is here more detail info on this point?
>

yes... our design document:

http://peregrine_mapreduce.bitbucket.org/design/

I actually will probably write a paper on this...

The more I started down the partitioned filesystem approach in terms of
mapreduce the more I realized that there were some REALLY elegant
imoplementation and design issues that I did not originally appreciate ...
(so I partially got lucky).

I think this approach could be generalized to work on normal map reduce
jobs without much overhead.

-- 

Founder/CEO Spinn3r.com <http://spinn3r.com/>

Location: *San Francisco, CA*
Skype: *burtonator*

Skype-in: *(415) 871-0687*

--0015174940fc8a391b04b510ff5d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"m=
argin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left=
:1ex"><div class=3D"gmail_quote"><div class=3D"im"><blockquote class=3D"gma=
il_quote" style=3D"margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,2=
04,204);padding-left:1ex">

A key innovation here is a partitioning layout algorithm that can support f=
ast<br>


many to many recovery similar to HDFS but still support partitioned operati=
on<br>with deterministic key placement.<br></blockquote></div><div><br>Than=
ks for your contribution.<br><br>Is here more detail info on this point? <b=
r>

</div></div></blockquote><div><br>yes... our design document:<br><br><a hre=
f=3D"http://peregrine_mapreduce.bitbucket.org/design/" target=3D"_blank">ht=
tp://peregrine_mapreduce.bitbucket.org/design/</a><br><br>I actually will p=
robably write a paper on this... <br>

<br>The more I started down the partitioned filesystem approach in terms of=
 mapreduce the more I realized that there were some REALLY elegant imopleme=
ntation and design issues that I did not originally appreciate ... (so I pa=
rtially got lucky).<br>

<br>I think this approach could be generalized to work on normal map reduce=
 jobs without much overhead.<br>=A0</div></div>-- <br><div><div><p style=3D=
"margin:0px 0px 12pt">Founder/CEO=A0<a href=3D"http://spinn3r.com/" target=
=3D"_blank">Spinn3r.com</a></p>

<div><p style=3D"margin:0px">Location:=A0<b>San Francisco, CA</b><br>Skype:=
=A0<b>burtonator</b></p></div></div></div><div><p style=3D"margin:0px">Skyp=
e-in:=A0<span><b><span style=3D"color:rgb(44,44,44)"><a>(415) 871-0687</a><=
/span></b></span></p>

</div><br>

--0015174940fc8a391b04b510ff5d--