mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Mannix <jake.man...@gmail.com>
Subject Re: Apache Giraph?
Date Fri, 16 Sep 2011 20:31:30 GMT
On Fri, Sep 16, 2011 at 1:24 PM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Well, distributed memory to me would have fetch and store operations.  Here
> we can send a message, but we can't actually fetch or store data without
> cooperation.
>

Funny you mention that - I've been considering suggesting that Giraph modify
the "sendMsg()" method contract to not be void, but return something too...

  -jake


> On Fri, Sep 16, 2011 at 4:45 AM, Grant Ingersoll <gsingers@apache.org
> >wrote:
>
> >
> > On Sep 16, 2011, at 12:27 AM, Ted Dunning wrote:
> >
> > > Actually, I don't think that these really provide a distributed memory
> > > layer.
> > >
> > > What they is multiple iterations without having to renegotiate JVM
> > launches,
> > > local memory that persists across iterations and decent message
> passing.
> > > (and of course some level of synchronization).
> > >
> > > And that is plenty for us.
> > >
> >
> > That sounds a lot like a distributed memory layer (i.e. the JVM stays up
> w/
> > it's memory) and then a msg passing layer on top of it.  It smells like
> to
> > me that it does for memory what the map-reduce + DFS abstraction did for
> > that space, i.e. it gave a base platform + API that made it easy for
> people
> > to build large scale distributed, disk-based, batch oriented systems.  We
> > need a base platform for large-scale, distributed memory-based systems so
> > that it is easy to write implementations on top of it.
> >
> >
> > > On Fri, Sep 16, 2011 at 12:14 AM, Jake Mannix <jake.mannix@gmail.com>
> > wrote:
> > >
> > >> A big "distributed memory layer" does indeed sound great, however.
> >  Spark
> > >> and Giraph both provide their own, although the former seems to lean
> > more
> > >> toward "read-only, with allowed side-effects", and very general
> purpose,
> > >> while the latter is couched in the language of graphs, and computation
> > is
> > >> specifically BSP (currently), but allows for fairly arbitrary mutation
> > (and
> > >> persisting final results back to HDFS).
> > >>
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message