giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Asplund <alexaspl...@gmail.com>
Subject Re: Giraph offloadPartition fails creation directory
Date Thu, 12 Sep 2013 21:07:47 GMT
Actually, I take that back. It seems it does succeeded in creating
partitions - it just struggles with it sometimes. Should I be worried about
these errors if partition directories seem to be filling up?
On Sep 11, 2013 6:38 PM, "Claudio Martella" <claudio.martella@gmail.com>
wrote:

> Giraph does not offload partitions or messages to HDFS in the out-of-core
> module. It uses local disk on the computing nodes. By defualt, it uses the
> tasktracker local directory where for example the distributed cache is
> stored.
>
> Could you provide the stacktrace Giraph is spitting when failing?
>
>
> On Thu, Sep 12, 2013 at 12:54 AM, Alexander Asplund <alexasplund@gmail.com
> > wrote:
>
>> Hi,
>>
>> I'm still trying to get Giraph to work on a graph that requires more
>> memory that is available. The problem is that when the Workers try to
>> offload partitions, the offloading fails. The DiskBackedPartitionStore
>> fails to create the directory
>> _bsp/_partitions/job-xxxx/part-vertices-xxx (roughly from recall).
>>
>> The input or computation will then continue for a while, which I
>> believe is because it is still managing to hold everything in memory -
>> but at some point it reaches the limit where there simply is no more
>> heap space, and it crashes with OOM.
>>
>> Has anybody had this problem with giraph failing to make HDFS directories?
>>
>
>
>
> --
>    Claudio Martella
>    claudio.martella@gmail.com
>

Mime
View raw message