Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of burtonator2011@gmail.com
 designates 74.125.82.172 as permitted sender)
MIME-Version: 1.0
Sender: burtonator2011@gmail.com
From: Kevin Burton <burtonator@gmail.com>
Date: Mon, 26 Dec 2011 22:31:39 -0800
Message-ID: 
 <CAAZU44nu8PhvXd6jktcxdb0UWK_FHwrc2ek6W+2zzE1=Hkc_SA@mail.gmail.com>
Subject: Peregrine: A new map reduce framework for iterative/pipelined jobs.
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=0016e6de007850d38304b50d0acf

--0016e6de007850d38304b50d0acf
Content-Type: text/plain; charset=ISO-8859-1

I'm pleased to announce Peregrine 0.5.0 - a new map reduce framework
optimized
for iterative and pipelined map reduce jobs.

http://peregrine_mapreduce.bitbucket.org/

This originally started off with some internal work at Spinn3r to build a
fast
and efficient Pagerank implementation.  We realized that what we wanted was
a MR
runtime optimized for this type of work which differs radically from the
traditional Hadoop design.

Peregrine implements a partitioned distributed filesystem where key/value
pairs
are routed to defined partitions.  This enables work to be joined against
previous iterations or different units of work by the same key on the same
local
system.

Peregrine is optimized for ETL jobs where the primary data storage system
is an
external database such as Cassandra, Hbase, MySQL, etc.  Jobs are then run
as a
Extract, Transform and Load stages with intermediate data being stored in
the
Peregrine FS.

We enable features such as Map/Reduce/Merge as well as some additional
functionality like ExtractMap and ReduceLoad (in ETL parlance).

A key innovation here is a partitioning layout algorithm that can support
fast
many to many recovery similar to HDFS but still support partitioned
operation
with deterministic key placement.

We've also tried to optimize for single instance performance and use modern
IO
primitives as much as possible.  This includes NOT shying away from
operating
specific features such as mlock, fadvise, fallocate, etc.

There is still a bit more work I want to do before I am ready to benchmark
it
against Hadoop.  Instead of implementing a synthetic benchmark we wanted to
get
a production ready version first which would allow people to port existing
applications and see what the before / after performance numbers looked
like in
the real world.

For more information please see:

http://peregrine_mapreduce.bitbucket.org/

As well as our design documentation:

http://peregrine_mapreduce.bitbucket.org/design/


-- 
-- 

Founder/CEO Spinn3r.com <http://spinn3r.com/>

Location: *San Francisco, CA*
Skype: *burtonator*

Skype-in: *(415) 871-0687*

--0016e6de007850d38304b50d0acf
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<br>I&#39;m pleased to announce Peregrine 0.5.0 - a new map reduce framewor=
k optimized<br>for iterative and pipelined map reduce jobs.<br><br><a href=
=3D"http://peregrine_mapreduce.bitbucket.org/">http://peregrine_mapreduce.b=
itbucket.org/</a><br>

<br>This originally started off with some internal work at Spinn3r to build=
 a fast<br>and efficient Pagerank implementation.=A0 We realized that what =
we wanted was a MR<br>runtime optimized for this type of work which differs=
 radically from the<br>

traditional Hadoop design.<br><br>Peregrine implements a partitioned distri=
buted filesystem where key/value pairs<br>are routed to defined partitions.=
=A0 This enables work to be joined against<br>previous iterations or differ=
ent units of work by the same key on the same local<br>

system.<br><br>Peregrine is optimized for ETL jobs where the primary data s=
torage system is an<br>external database such as Cassandra, Hbase, MySQL, e=
tc.=A0 Jobs are then run as a<br>Extract, Transform and Load stages with in=
termediate data being stored in the<br>

Peregrine FS.<br><br>We enable features such as Map/Reduce/Merge as well as=
 some additional<br>functionality like ExtractMap and ReduceLoad (in ETL pa=
rlance).<br><br>A key innovation here is a partitioning layout algorithm th=
at can support fast<br>

many to many recovery similar to HDFS but still support partitioned operati=
on<br>with deterministic key placement.<br><br>We&#39;ve also tried to opti=
mize for single instance performance and use modern IO<br>primitives as muc=
h as possible.=A0 This includes NOT shying away from operating<br>

specific features such as mlock, fadvise, fallocate, etc.=A0 <br><br>There =
is still a bit more work I want to do before I am ready to benchmark it<br>=
against Hadoop.=A0 Instead of implementing a synthetic benchmark we wanted =
to get<br>

a production ready version first which would allow people to port existing<=
br>applications and see what the before / after performance numbers looked =
like in<br>the real world.<br><br>For more information please see: <br>

<br><a href=3D"http://peregrine_mapreduce.bitbucket.org/">http://peregrine_=
mapreduce.bitbucket.org/</a><br><br>As well as our design documentation:<br=
><br><a href=3D"http://peregrine_mapreduce.bitbucket.org/design/">http://pe=
regrine_mapreduce.bitbucket.org/design/</a><br>

<br><br clear=3D"all"><br>-- <br>-- <br><div><div><p style=3D"margin:0px 0p=
x 12pt">Founder/CEO=A0<a href=3D"http://spinn3r.com/" target=3D"_blank">Spi=
nn3r.com</a></p><div><p style=3D"margin:0px">Location:=A0<b>San Francisco, =
CA</b><br>

Skype:=A0<b>burtonator</b></p></div></div></div><div><p style=3D"margin:0px=
">Skype-in:=A0<span><b><span style=3D"color:rgb(44,44,44)"><a>(415) 871-068=
7</a></span></b></span></p></div><br>

--0016e6de007850d38304b50d0acf--