giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukas Nalezenec <lukas.naleze...@firma.seznam.cz>
Subject Re: debugging graph-programs
Date Wed, 19 Feb 2014 09:44:43 GMT
Hi,
Its simple, just pass the TestGraph to InternalVertexRunner.

If i had  lot of logic in input format, I would test as much as i could 
in pure isolated InputFormat unit test without any framework. I would 
make POJO mock for Record Reader and then add some Mockito mocks. 
(InternalVertexRunner is little bit slow - it starts giraph in every 
unit test). Then i would test the input format together with 
computations in InternalVertexRunner in separate unit test classes.
I have never used InternalVertexRunner for testing input/output formats 
- I am not sure if it possible.

Lukas



On 18.2.2014 17:38, Martin Neumann wrote:
> @ Lukas: Do you have some links where I can read how to use it to 
> debug programs?
>
> My current program has a lot of logic in the EdgeInputFormat as well 
> that was why I was looking at MapReduce tools since loading the data 
> uses the same classes (as far as I can see). But I agree its a rather 
> desperate try.
>
> I need a simple way to test the input format, output format and the 
> vertex program.
> For the vertex program I currently develop it on a white bord to get 
> the interactions right then some unit test to make sure it works on a 
> micro level. I have not found a good way to test the input format 
> (e.g. if the Graph was build correctly) or how to do a good end to end 
> test.
>
> cheers Martin
>
> On Tue, Feb 18, 2014 at 5:08 PM, Lukas Nalezenec 
> <lukas.nalezenec@firma.seznam.cz 
> <mailto:lukas.nalezenec@firma.seznam.cz>> wrote:
>
>     Hi,
>     How about org.apache.giraph.utils.TestGraph?
>     I would not try using mrunit for testing Giraph.
>
>     cheers
>     Lukas
>
>
>
>     On 18.2.2014 17:01, Mirko Kämpf wrote:
>>     Hi Martin,
>>
>>     it depends on the details of your implementation. In principle,
>>     you do not write Mapper and Reducer
>>     classes, which are tested by MR-Unit. You usually have to
>>     implement the application / algorithm specific
>>     logic, in e.g. the Vertex class. MR-Unit might not help that much
>>     here.
>>
>>     This leads to the question, if there is a comparable approach /
>>     or tool like MR Unit for Giraph?
>>
>>     Best wishes
>>     Mirko
>>
>>
>>
>>
>>     On Tue, Feb 18, 2014 at 3:52 PM, Martin Neumann
>>     <mneumann@spotify.com <mailto:mneumann@spotify.com>> wrote:
>>
>>         How much of the Giraph 1.1.0 version is still based on
>>         MapReduce?
>>         Would something like mrunit (http://mrunit.apache.org/) work
>>         with it?
>>
>>
>>         On Tue, Feb 18, 2014 at 12:52 PM, Martin Neumann
>>         <mneumann@spotify.com <mailto:mneumann@spotify.com>> wrote:
>>
>>             I'm using the Yarn version (1.1.0 trunk)  and I was not
>>             able to get it to work that way. But I'm no expert on
>>             that, if anyone knows a way to get it done It would be great.
>>
>>
>>             On Tue, Feb 18, 2014 at 2:07 AM, Roman Shaposhnik
>>             <roman@shaposhnik.org <mailto:roman@shaposhnik.org>> wrote:
>>
>>                 I find running Giraph apps via hadoop's local
>>                 execution mode
>>                 quite useful for debugging.
>>
>>                 Thanks,
>>                 Roman.
>>
>>                 On Mon, Feb 17, 2014 at 4:07 AM, Martin Neumann
>>                 <mneumann@spotify.com <mailto:mneumann@spotify.com>>
>>                 wrote:
>>                 > Hej,
>>                 >
>>                 > I find Giraph programs quite difficult to debug
>>                 (all I have is log outputs).
>>                 > Anyone has some hints for debugging tools or coding
>>                 practices to make it
>>                 > simpler?
>>                 >
>>                 > I'm using Giraph 1.1.0 trunk on YARN so I can't use
>>                 Map/Reduce debugging
>>                 > aids (at least I think so, I haven't tried any)
>>                 >
>>                 > cheers
>>                 > Martin
>>
>>
>>
>>
>
>


Mime
View raw message