giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hassan Eslami <hsn.esl...@gmail.com>
Subject Re: OOC on Giraph 1.1 or 1.2
Date Fri, 11 Dec 2015 06:18:47 GMT
Yes, OOC mechanism in 1.1 may not work in certain scenarios:
 - If a worker goes into long GC pauses (which is likely since a job that
is supposed to use OOC feature usually suffer from high memory pressure),
all incoming messages will be buffered before coming to Giraph runtime, and
this will cause he job to crash.
 - The rate at which a worker receives messages may be more than the rate
at which a worker spills messages to disk (specially in applications where
compute function does a little work, but generates a fair number of
messages, such as PageRank). In this case, messages keep accumulating and
the worker crashes.
 - OOC feature is not supported for input superstep for Giraph 1.0 and 1.1.
That being said, if graph size is larger than aggregate available memory,
you are likely to see the job crashes during input superstep.

OOC feature is under active development at this moment. A new OOC mechanism
is recently pushed to the code base (which basically solves the problem for
input superstep and OOC graph). However, the new OOC mechanism still have
problems for OOC messages. An entirely new approach to OOC is being
developed and soon will be available in the code base.

Best,
Hassan

On Thu, Dec 10, 2015 at 8:58 PM, Khaled Ammar <khaled.ammar@gmail.com>
wrote:

> Hi,
>
> I remember that I used OOC on Giraph 1.0, how ever it is not working on
> 1.1.
>
> Is there a known bug in the OOC feature for recent Giraph versions?
>
> --
> Thanks,
> -Khaled
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message