giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Schelter <...@apache.org>
Subject Re: debugging graph-programs
Date Wed, 19 Feb 2014 12:52:24 GMT
Hi Martin,


There is no need to install ZooKeeper to use InternalVertexRunner. It 
will also use the input and output formats specified.

InternalVertexRunner writes your input data to disk and starts ZooKeeper 
and Giraph locally in a single VM.

It is very easy to use, I suggest to look at the following class for an 
example:

org.apache.giraph.examples.ConnectedComponentsComputationTest#testToyData

Best,
Sebastian

On 02/19/2014 12:55 PM, Martin Neumann wrote:
> I saw in the documentation that InternalVertexRunner has a configuration
> field (LOCAL_ZOOKEEPER_PORT) does that mean I have to have Zookeper
> installed and configured? Also what else would I need to have installed to
> test that way?
>
> I couldn't find any good guides how to do this online so I'm thankful for
> every hint. So thanks everyone for the help so far.
>
>
>
> On Wed, Feb 19, 2014 at 10:44 AM, Lukas Nalezenec <
> lukas.nalezenec@firma.seznam.cz> wrote:
>
>>   Hi,
>> Its simple, just pass the TestGraph to InternalVertexRunner.
>>
>> If i had  lot of logic in input format, I would test as much as i could in
>> pure isolated InputFormat unit test without any framework. I would make
>> POJO mock for Record Reader and then add some Mockito mocks.
>> (InternalVertexRunner is little bit slow - it starts giraph in every unit
>> test). Then i would test the input format together with computations in
>> InternalVertexRunner in separate unit test classes.
>> I have never used InternalVertexRunner for testing input/output formats -
>> I am not sure if it possible.
>>
>> Lukas
>>
>>
>>
>>
>> On 18.2.2014 17:38, Martin Neumann wrote:
>>
>>   @ Lukas: Do you have some links where I can read how to use it to debug
>> programs?
>>
>>   My current program has a lot of logic in the EdgeInputFormat as well
>> that was why I was looking at MapReduce tools since loading the data uses
>> the same classes (as far as I can see). But I agree its a rather desperate
>> try.
>>
>>   I need a simple way to test the input format, output format and the
>> vertex program.
>> For the vertex program I currently develop it on a white bord to get the
>> interactions right then some unit test to make sure it works on a micro
>> level. I have not found a good way to test the input format (e.g. if the
>> Graph was build correctly) or how to do a good end to end test.
>>
>>   cheers Martin
>>
>>   On Tue, Feb 18, 2014 at 5:08 PM, Lukas Nalezenec <
>> lukas.nalezenec@firma.seznam.cz> wrote:
>>
>>>   Hi,
>>> How about org.apache.giraph.utils.TestGraph?
>>> I would not try using mrunit for testing Giraph.
>>>
>>> cheers
>>> Lukas
>>>
>>>
>>>
>>> On 18.2.2014 17:01, Mirko Kämpf wrote:
>>>
>>>    Hi Martin,
>>>
>>>   it depends on the details of your implementation. In principle, you do
>>> not write Mapper and Reducer
>>>   classes, which are tested by MR-Unit. You usually have to implement the
>>> application / algorithm specific
>>> logic, in e.g. the Vertex class. MR-Unit might not help that much here.
>>>
>>>   This leads to the question, if there is a comparable approach / or tool
>>> like MR Unit for Giraph?
>>>
>>>   Best wishes
>>> Mirko
>>>
>>>
>>>
>>>
>>> On Tue, Feb 18, 2014 at 3:52 PM, Martin Neumann <mneumann@spotify.com>wrote:
>>>
>>>>   How much of the Giraph 1.1.0 version is still based on MapReduce?
>>>> Would something like mrunit (http://mrunit.apache.org/) work with it?
>>>>
>>>>
>>>> On Tue, Feb 18, 2014 at 12:52 PM, Martin Neumann <mneumann@spotify.com>wrote:
>>>>
>>>>>   I'm using the Yarn version (1.1.0 trunk)  and I was not able to get
>>>>> it to work that way. But I'm no expert on that, if anyone knows a way
to
>>>>> get it done It would be great.
>>>>>
>>>>>
>>>>> On Tue, Feb 18, 2014 at 2:07 AM, Roman Shaposhnik <roman@shaposhnik.org
>>>>>> wrote:
>>>>>
>>>>>> I find running Giraph apps via hadoop's local execution mode
>>>>>> quite useful for debugging.
>>>>>>
>>>>>> Thanks,
>>>>>> Roman.
>>>>>>
>>>>>> On Mon, Feb 17, 2014 at 4:07 AM, Martin Neumann <mneumann@spotify.com>
>>>>>> wrote:
>>>>>>> Hej,
>>>>>>>
>>>>>>> I find Giraph programs quite difficult to debug (all I have is
log
>>>>>> outputs).
>>>>>>> Anyone has some hints for debugging tools or coding practices
to
>>>>>> make it
>>>>>>> simpler?
>>>>>>>
>>>>>>> I'm using Giraph 1.1.0 trunk on YARN so I can't use Map/Reduce
>>>>>> debugging
>>>>>>> aids (at least I think so, I haven't tried any)
>>>>>>>
>>>>>>> cheers
>>>>>>> Martin
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>


Mime
View raw message