giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Neumann <mneum...@spotify.com>
Subject Re: debugging graph-programs
Date Wed, 19 Feb 2014 11:55:41 GMT
I saw in the documentation that InternalVertexRunner has a configuration
field (LOCAL_ZOOKEEPER_PORT) does that mean I have to have Zookeper
installed and configured? Also what else would I need to have installed to
test that way?

I couldn't find any good guides how to do this online so I'm thankful for
every hint. So thanks everyone for the help so far.



On Wed, Feb 19, 2014 at 10:44 AM, Lukas Nalezenec <
lukas.nalezenec@firma.seznam.cz> wrote:

>  Hi,
> Its simple, just pass the TestGraph to InternalVertexRunner.
>
> If i had  lot of logic in input format, I would test as much as i could in
> pure isolated InputFormat unit test without any framework. I would make
> POJO mock for Record Reader and then add some Mockito mocks.
> (InternalVertexRunner is little bit slow - it starts giraph in every unit
> test). Then i would test the input format together with computations in
> InternalVertexRunner in separate unit test classes.
> I have never used InternalVertexRunner for testing input/output formats -
> I am not sure if it possible.
>
> Lukas
>
>
>
>
> On 18.2.2014 17:38, Martin Neumann wrote:
>
>  @ Lukas: Do you have some links where I can read how to use it to debug
> programs?
>
>  My current program has a lot of logic in the EdgeInputFormat as well
> that was why I was looking at MapReduce tools since loading the data uses
> the same classes (as far as I can see). But I agree its a rather desperate
> try.
>
>  I need a simple way to test the input format, output format and the
> vertex program.
> For the vertex program I currently develop it on a white bord to get the
> interactions right then some unit test to make sure it works on a micro
> level. I have not found a good way to test the input format (e.g. if the
> Graph was build correctly) or how to do a good end to end test.
>
>  cheers Martin
>
>  On Tue, Feb 18, 2014 at 5:08 PM, Lukas Nalezenec <
> lukas.nalezenec@firma.seznam.cz> wrote:
>
>>  Hi,
>> How about org.apache.giraph.utils.TestGraph?
>> I would not try using mrunit for testing Giraph.
>>
>> cheers
>> Lukas
>>
>>
>>
>> On 18.2.2014 17:01, Mirko Kämpf wrote:
>>
>>   Hi Martin,
>>
>>  it depends on the details of your implementation. In principle, you do
>> not write Mapper and Reducer
>>  classes, which are tested by MR-Unit. You usually have to implement the
>> application / algorithm specific
>> logic, in e.g. the Vertex class. MR-Unit might not help that much here.
>>
>>  This leads to the question, if there is a comparable approach / or tool
>> like MR Unit for Giraph?
>>
>>  Best wishes
>> Mirko
>>
>>
>>
>>
>> On Tue, Feb 18, 2014 at 3:52 PM, Martin Neumann <mneumann@spotify.com>wrote:
>>
>>>  How much of the Giraph 1.1.0 version is still based on MapReduce?
>>> Would something like mrunit (http://mrunit.apache.org/) work with it?
>>>
>>>
>>> On Tue, Feb 18, 2014 at 12:52 PM, Martin Neumann <mneumann@spotify.com>wrote:
>>>
>>>>  I'm using the Yarn version (1.1.0 trunk)  and I was not able to get
>>>> it to work that way. But I'm no expert on that, if anyone knows a way to
>>>> get it done It would be great.
>>>>
>>>>
>>>> On Tue, Feb 18, 2014 at 2:07 AM, Roman Shaposhnik <roman@shaposhnik.org
>>>> > wrote:
>>>>
>>>>> I find running Giraph apps via hadoop's local execution mode
>>>>> quite useful for debugging.
>>>>>
>>>>> Thanks,
>>>>> Roman.
>>>>>
>>>>> On Mon, Feb 17, 2014 at 4:07 AM, Martin Neumann <mneumann@spotify.com>
>>>>> wrote:
>>>>> > Hej,
>>>>> >
>>>>> > I find Giraph programs quite difficult to debug (all I have is log
>>>>> outputs).
>>>>> > Anyone has some hints for debugging tools or coding practices to
>>>>> make it
>>>>> > simpler?
>>>>> >
>>>>> > I'm using Giraph 1.1.0 trunk on YARN so I can't use Map/Reduce
>>>>> debugging
>>>>> > aids (at least I think so, I haven't tried any)
>>>>> >
>>>>> > cheers
>>>>> > Martin
>>>>>
>>>>
>>>>
>>>
>>
>>
>
>

Mime
View raw message