hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Delip Rao <delip...@gmail.com>
Subject Re: FYI, Large-scale graph computing at Google
Date Thu, 02 Jul 2009 20:57:12 GMT
@Ed, the paper is still available at
http://www.clsp.jhu.edu/~delip/nocrawl/textgraphs09.pdf<http://www.clsp.jhu.edu/%7Edelip/nocrawl/textgraphs09.pdf>
(Looks like you accessed when our server closet had a brief power outage)

@Michal, as Ted suggests this is easy to implement if you run several
iterations. Please look for the algorithm "Label Propagation" in the above
paper for something very close to your needs.

- delip

On Thu, Jul 2, 2009 at 4:21 AM, Michal Laclavik <laclavik.ui@savba.sk>wrote:

> very interesting discussion ...
>
> we are dealing with processing social networks from email
> communication with connection to other objects extracted from the
> email.
>
> Extraction of the network works very fine on hadoop, but processing of
> the graph  it it is not that easy.
> We would like to implement spread activation algorithm over MR but it
> is quite dificult.
>
> Anyone tried something with spread activation on Hadoop?
>
>
> Michal
>
>
>
> On Thu, Jul 2, 2009 at 12:36 PM, Edward J. Yoon<edwardyoon@apache.org>
> wrote:
> > Thanks. BTW this link seems broken. Could you send me a paper? ;)
> >
> > And, We've just begun to design the Hamburg --
> > http://wiki.apache.org/hadoop/Hamburg -- any comments are welcome.
> >
> > On Wed, Jul 1, 2009 at 1:38 AM, Delip Rao<deliprao@gmail.com> wrote:
> >> We've had some success in dealing with locality problems using the
> adjacency
> >> list
> >> representation. This could be serialized using frameworks like Thrift
> >> or Protocol Buffers.
> >> For details, please see:
> >> http://www.clsp.jhu.edu/~delip/nocrawl/textgraphs09.pdf<http://www.clsp.jhu.edu/%7Edelip/nocrawl/textgraphs09.pdf>
> >>
> >> I intend to
> >> continue this line of work and will be very happy to be of any help.
> >>
> >> On Thu, Jun 25, 2009 at 8:24 PM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
> >>
> >>> I just made a wiki page -- http://wiki.apache.org/hadoop/Hambrug --
> >>> Let's discuss about the graph computing framework named Hambrug.
> >>>
> >>> On Fri, Jun 26, 2009 at 8:43 AM, Edward J. Yoon<edwardyoon@apache.org>
> >>> wrote:
> >>> > To be honest, I was thought the BigTable (HBase) for the map/reduce
> >>> > based graph/matrix operations. The main problems of performance were
> >>> > the sequential algorithm, the cost for MR job building in iterations.
> >>> > and, the locality of adjacent components. As mentioned on Pregel, If
> >>> > some algorithm requires small resources to get result, the BSP model
> >>> > based another computing framework on HDFS can be useful for us.
> >>> >
> >>> > On Fri, Jun 26, 2009 at 3:37 AM, Amandeep Khurana<amansk@gmail.com>
> >>> wrote:
> >>> >> I've been working on some graph stuff using MR as well. I'd be
more
> than
> >>> >> interested to chip in as well..
> >>> >>
> >>> >> I remember exchanging a few mails with Paolo about having an RDF
> store
> >>> over
> >>> >> HBase and developing graph algorithms over it.
> >>> >>
> >>> >>
> >>> >> Amandeep Khurana
> >>> >> Computer Science Graduate Student
> >>> >> University of California, Santa Cruz
> >>> >>
> >>> >>
> >>> >> On Thu, Jun 25, 2009 at 2:57 AM, Steve Loughran <stevel@apache.org>
> >>> wrote:
> >>> >>
> >>> >>> Edward J. Yoon wrote:
> >>> >>>
> >>> >>>> What do you think about another new computation framework
on HDFS?
> >>> >>>>
> >>> >>>> On Mon, Jun 22, 2009 at 3:50 PM, Edward J. Yoon <
> >>> edwardyoon@apache.org>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>>
> >>> >>>>>
> >>>
> http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html
> >>> >>>>> -- It sounds like Pregel seems, a computing framework
based on
> >>> dynamic
> >>> >>>>> programming for the graph operations. I guess maybe
they removed
> the
> >>> >>>>> file communications/intermediate files during iterations.
> >>> >>>>>
> >>> >>>>> Anyway, What do you think?
> >>> >>>>>
> >>> >>>>
> >>> >>> I have a colleague (paolo) who would be interested in adding
a set
> of
> >>> graph
> >>> >>> algorithms on top of the MR engine
> >>> >>>
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Best Regards, Edward J. Yoon @ NHN, corp.
> >>> > edwardyoon@apache.org
> >>> > http://blog.udanax.org
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Edward J. Yoon @ NHN, corp.
> >>> edwardyoon@apache.org
> >>> http://blog.udanax.org
> >>>
> >>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon @ NHN, corp.
> > edwardyoon@apache.org
> > http://blog.udanax.org
> >
>
>
>
> --
> S pozdravom
> Michal Laclavik
> ==
> Institute of Informatics SAS
> email: laclavik.ui@savba.sk
> web: http://laclavik.net/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message