giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Schlaikjer <ha...@twitter.com>
Subject Re: writing/emitting to HDFS
Date Fri, 23 Sep 2011 17:13:26 GMT
How about Scribing messages (and writing to HDFS) during calculation?
Then you could perform bulk log analysis on the output with a separate
Hadoop (or Pig) job.

http://en.wikipedia.org/wiki/Scribe_(log_server)

Andy


On Thu, Sep 22, 2011 at 7:31 AM, Claudio Martella
<claudio.martella@gmail.com> wrote:
> Hi Avery,
>
> thanks, yes it does. The question would be though how to share the
> file handle between the vertices on the same node. i could open the
> file on the preApplication() and close it on the postApplication() but
> i would end up potentially with as many files as vertices in the
> graph.
>
> Do you have any idea on this side? Maybe share somehow the handle and a lock?
>
> On Thu, Sep 22, 2011 at 4:07 PM, Avery Ching <aching@apache.org> wrote:
>> There are some methods in Vertex (i.e. preApplication(), preSuperstep(),
>> postApplication(), postSuperstep()) that can be overidden to do anything you
>> like, for instance write out some data to an HDFS file.  We have an open
>> issue on outputting Aggregator values that is unassigned if you'd like to
>> take a look at it as well (https://issues.apache.org/jira/browse/GIRAPH-10).
>>
>> Hope this helps,
>>
>> Avery
>>
>> On 9/22/11 7:34 AM, Claudio Martella wrote:
>>>
>>> Hello list,
>>>
>>> I have the need to emit to HDFS once in a while some Text. This
>>> doesn't happen necessarily at the end of the computation and I might
>>> need to emit something more complex than just the VertexValue, so I'd
>>> like more control than what the VertexWriter gives me.
>>>
>>> What do you suggest I might do to obtain a handler to a HDFS file (it
>>> can be in parts aswell) to write to?
>>> Is there any code I can start looking at?
>>>
>>> Thanks!
>>> Claudio
>>>
>>
>>
>
>
>
> --
>     Claudio Martella
>     claudio.martella@gmail.com
>

Mime
View raw message