incubator-giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: writing/emitting to HDFS
Date Thu, 22 Sep 2011 14:31:22 GMT
Hi Avery,

thanks, yes it does. The question would be though how to share the
file handle between the vertices on the same node. i could open the
file on the preApplication() and close it on the postApplication() but
i would end up potentially with as many files as vertices in the
graph.

Do you have any idea on this side? Maybe share somehow the handle and a lock?

On Thu, Sep 22, 2011 at 4:07 PM, Avery Ching <aching@apache.org> wrote:
> There are some methods in Vertex (i.e. preApplication(), preSuperstep(),
> postApplication(), postSuperstep()) that can be overidden to do anything you
> like, for instance write out some data to an HDFS file.  We have an open
> issue on outputting Aggregator values that is unassigned if you'd like to
> take a look at it as well (https://issues.apache.org/jira/browse/GIRAPH-10).
>
> Hope this helps,
>
> Avery
>
> On 9/22/11 7:34 AM, Claudio Martella wrote:
>>
>> Hello list,
>>
>> I have the need to emit to HDFS once in a while some Text. This
>> doesn't happen necessarily at the end of the computation and I might
>> need to emit something more complex than just the VertexValue, so I'd
>> like more control than what the VertexWriter gives me.
>>
>> What do you suggest I might do to obtain a handler to a HDFS file (it
>> can be in parts aswell) to write to?
>> Is there any code I can start looking at?
>>
>> Thanks!
>> Claudio
>>
>
>



-- 
    Claudio Martella
    claudio.martella@gmail.com

Mime
View raw message