giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Giraph offloadPartition fails creation directory
Date Thu, 12 Sep 2013 01:37:27 GMT
Giraph does not offload partitions or messages to HDFS in the out-of-core
module. It uses local disk on the computing nodes. By defualt, it uses the
tasktracker local directory where for example the distributed cache is
stored.

Could you provide the stacktrace Giraph is spitting when failing?


On Thu, Sep 12, 2013 at 12:54 AM, Alexander Asplund
<alexasplund@gmail.com>wrote:

> Hi,
>
> I'm still trying to get Giraph to work on a graph that requires more
> memory that is available. The problem is that when the Workers try to
> offload partitions, the offloading fails. The DiskBackedPartitionStore
> fails to create the directory
> _bsp/_partitions/job-xxxx/part-vertices-xxx (roughly from recall).
>
> The input or computation will then continue for a while, which I
> believe is because it is still managing to hold everything in memory -
> but at some point it reaches the limit where there simply is no more
> heap space, and it crashes with OOM.
>
> Has anybody had this problem with giraph failing to make HDFS directories?
>



-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message