incubator-giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: writing/emitting to HDFS
Date Mon, 26 Sep 2011 14:04:37 GMT
Thanks for the feedback. As a matter of fact that's exactly the type
of functionality i'm looking for, with minimal infrastructure cost
though. Thanks!

On Fri, Sep 23, 2011 at 7:13 PM, Andy Schlaikjer <hazen@twitter.com> wrote:
> How about Scribing messages (and writing to HDFS) during calculation?
> Then you could perform bulk log analysis on the output with a separate
> Hadoop (or Pig) job.
>
> http://en.wikipedia.org/wiki/Scribe_(log_server)
>
> Andy
>
>
> On Thu, Sep 22, 2011 at 7:31 AM, Claudio Martella
> <claudio.martella@gmail.com> wrote:
>> Hi Avery,
>>
>> thanks, yes it does. The question would be though how to share the
>> file handle between the vertices on the same node. i could open the
>> file on the preApplication() and close it on the postApplication() but
>> i would end up potentially with as many files as vertices in the
>> graph.
>>
>> Do you have any idea on this side? Maybe share somehow the handle and a lock?
>>
>> On Thu, Sep 22, 2011 at 4:07 PM, Avery Ching <aching@apache.org> wrote:
>>> There are some methods in Vertex (i.e. preApplication(), preSuperstep(),
>>> postApplication(), postSuperstep()) that can be overidden to do anything you
>>> like, for instance write out some data to an HDFS file.  We have an open
>>> issue on outputting Aggregator values that is unassigned if you'd like to
>>> take a look at it as well (https://issues.apache.org/jira/browse/GIRAPH-10).
>>>
>>> Hope this helps,
>>>
>>> Avery
>>>
>>> On 9/22/11 7:34 AM, Claudio Martella wrote:
>>>>
>>>> Hello list,
>>>>
>>>> I have the need to emit to HDFS once in a while some Text. This
>>>> doesn't happen necessarily at the end of the computation and I might
>>>> need to emit something more complex than just the VertexValue, so I'd
>>>> like more control than what the VertexWriter gives me.
>>>>
>>>> What do you suggest I might do to obtain a handler to a HDFS file (it
>>>> can be in parts aswell) to write to?
>>>> Is there any code I can start looking at?
>>>>
>>>> Thanks!
>>>> Claudio
>>>>
>>>
>>>
>>
>>
>>
>> --
>>     Claudio Martella
>>     claudio.martella@gmail.com
>>
>



-- 
    Claudio Martella
    claudio.martella@gmail.com

Mime
View raw message