Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EBED910546 for ; Wed, 19 Feb 2014 11:56:13 +0000 (UTC) Received: (qmail 56595 invoked by uid 500); 19 Feb 2014 11:56:13 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 56175 invoked by uid 500); 19 Feb 2014 11:56:11 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 56154 invoked by uid 99); 19 Feb 2014 11:56:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Feb 2014 11:56:09 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mneumann@spotify.com designates 209.85.220.171 as permitted sender) Received: from [209.85.220.171] (HELO mail-vc0-f171.google.com) (209.85.220.171) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Feb 2014 11:56:03 +0000 Received: by mail-vc0-f171.google.com with SMTP id le5so254464vcb.2 for ; Wed, 19 Feb 2014 03:55:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=pqamWrPxjRTjhCx+n2jm26ZOcPzjHmkRyB6tIL0Bt7I=; b=TG/QICbqWwk/PgGO0RxkugsLAwtuuzCft2KOCBXqtOczNo4k85970ptcCyRVFsYEBg a7QvTrEymRzul6Ely4aN2irOW+Oopa8ytsGdt9DBbLxwqwVa10s4aLAcjmBKJaWzerfG oupT3iMZxms2kJgU/RBtQZHI3+DnN8GyzDizdrCsVB8vCU6lPI+R63tuwtOXm8v0qnsQ gzJhr08ntHGIv9fzm0Ne4xYX+h/n7/WH2Nu6CPq9aSDJIetFgD7XxZqRP8CFcJnOhA5Z hF+a6nbhoB3iT/4HEJ4x9J4J53++9QYLDWBQVGQjrd/etHcSRxQNOzRpB6jZ/sI6mk6o h+RA== X-Gm-Message-State: ALoCoQnfL1viUAdKQ0VdkjyDolFdHlejgdPvwcL9Uns2UZ+X6ko+3n4NRNWiLNuCshBpzVHh3MSH MIME-Version: 1.0 X-Received: by 10.221.53.194 with SMTP id vr2mr8832124vcb.41.1392810941712; Wed, 19 Feb 2014 03:55:41 -0800 (PST) Received: by 10.58.216.72 with HTTP; Wed, 19 Feb 2014 03:55:41 -0800 (PST) In-Reply-To: <53047D0B.7010208@firma.seznam.cz> References: <5303857F.5090500@firma.seznam.cz> <53047D0B.7010208@firma.seznam.cz> Date: Wed, 19 Feb 2014 12:55:41 +0100 Message-ID: Subject: Re: debugging graph-programs From: Martin Neumann To: user@giraph.apache.org Content-Type: multipart/alternative; boundary=001a11333dfa6b9d6404f2c111d7 X-Virus-Checked: Checked by ClamAV on apache.org --001a11333dfa6b9d6404f2c111d7 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I saw in the documentation that InternalVertexRunner has a configuration field (LOCAL_ZOOKEEPER_PORT) does that mean I have to have Zookeper installed and configured? Also what else would I need to have installed to test that way? I couldn't find any good guides how to do this online so I'm thankful for every hint. So thanks everyone for the help so far. On Wed, Feb 19, 2014 at 10:44 AM, Lukas Nalezenec < lukas.nalezenec@firma.seznam.cz> wrote: > Hi, > Its simple, just pass the TestGraph to InternalVertexRunner. > > If i had lot of logic in input format, I would test as much as i could i= n > pure isolated InputFormat unit test without any framework. I would make > POJO mock for Record Reader and then add some Mockito mocks. > (InternalVertexRunner is little bit slow - it starts giraph in every unit > test). Then i would test the input format together with computations in > InternalVertexRunner in separate unit test classes. > I have never used InternalVertexRunner for testing input/output formats - > I am not sure if it possible. > > Lukas > > > > > On 18.2.2014 17:38, Martin Neumann wrote: > > @ Lukas: Do you have some links where I can read how to use it to debug > programs? > > My current program has a lot of logic in the EdgeInputFormat as well > that was why I was looking at MapReduce tools since loading the data uses > the same classes (as far as I can see). But I agree its a rather desperat= e > try. > > I need a simple way to test the input format, output format and the > vertex program. > For the vertex program I currently develop it on a white bord to get the > interactions right then some unit test to make sure it works on a micro > level. I have not found a good way to test the input format (e.g. if the > Graph was build correctly) or how to do a good end to end test. > > cheers Martin > > On Tue, Feb 18, 2014 at 5:08 PM, Lukas Nalezenec < > lukas.nalezenec@firma.seznam.cz> wrote: > >> Hi, >> How about org.apache.giraph.utils.TestGraph? >> I would not try using mrunit for testing Giraph. >> >> cheers >> Lukas >> >> >> >> On 18.2.2014 17:01, Mirko K=E4mpf wrote: >> >> Hi Martin, >> >> it depends on the details of your implementation. In principle, you do >> not write Mapper and Reducer >> classes, which are tested by MR-Unit. You usually have to implement the >> application / algorithm specific >> logic, in e.g. the Vertex class. MR-Unit might not help that much here. >> >> This leads to the question, if there is a comparable approach / or tool >> like MR Unit for Giraph? >> >> Best wishes >> Mirko >> >> >> >> >> On Tue, Feb 18, 2014 at 3:52 PM, Martin Neumann wr= ote: >> >>> How much of the Giraph 1.1.0 version is still based on MapReduce? >>> Would something like mrunit (http://mrunit.apache.org/) work with it? >>> >>> >>> On Tue, Feb 18, 2014 at 12:52 PM, Martin Neumann = wrote: >>> >>>> I'm using the Yarn version (1.1.0 trunk) and I was not able to get >>>> it to work that way. But I'm no expert on that, if anyone knows a way = to >>>> get it done It would be great. >>>> >>>> >>>> On Tue, Feb 18, 2014 at 2:07 AM, Roman Shaposhnik >>> > wrote: >>>> >>>>> I find running Giraph apps via hadoop's local execution mode >>>>> quite useful for debugging. >>>>> >>>>> Thanks, >>>>> Roman. >>>>> >>>>> On Mon, Feb 17, 2014 at 4:07 AM, Martin Neumann >>>>> wrote: >>>>> > Hej, >>>>> > >>>>> > I find Giraph programs quite difficult to debug (all I have is log >>>>> outputs). >>>>> > Anyone has some hints for debugging tools or coding practices to >>>>> make it >>>>> > simpler? >>>>> > >>>>> > I'm using Giraph 1.1.0 trunk on YARN so I can't use Map/Reduce >>>>> debugging >>>>> > aids (at least I think so, I haven't tried any) >>>>> > >>>>> > cheers >>>>> > Martin >>>>> >>>> >>>> >>> >> >> > > --001a11333dfa6b9d6404f2c111d7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I saw in the documentation that InternalVertexRunner = has a configuration field (LOCAL_ZOOKEEPER_PORT) does that mean I have to h= ave Zookeper installed and configured? Also what else would I need to have = installed to test that way?

I couldn't find any good guides how to do this online so I'm th= ankful for every hint. So thanks everyone for the help so far.
<= br>


On Wed, Feb 19, 2014 at 10:44 AM, Lukas Nalezenec <lukas.nal= ezenec@firma.seznam.cz> wrote:
=20 =20 =20
Hi,
Its simple, just pass the TestGraph to InternalVertexRunner.

If i had=A0 lot of logic in input format, I would test as much as i could in pure isolated InputFormat unit test without any framework. I would make POJO mock for Record Reader and then add some Mockito mocks. (InternalVertexRunner is little bit slow - it starts giraph in every unit test). Then i would test the input format together with computations in InternalVertexRunner in separate unit test classes.
I have never used InternalVertexRunner for testing input/output formats - I am not sure if it possible.

Lukas




On 18.2.2014 17:38, Martin Neumann wrote:
=20
@ Lukas: Do you have some links where I can read how to use it to debug programs?

My current program has a lot of logic in the EdgeInputFormat as well that was why I was looking at MapReduce tools since loading the data uses the same classes (as far as I can see). But I agree its a rather desperate try.

I need a simple way to test the input format, output format and the vertex program.
For the vertex program I currently develop it on a white bord to get the interactions right then some unit test to make sure it works on a micro level. I have not found a good way to test the input format (e.g. if the Graph was build correctly) or how to do a good end to end test.

cheers Martin

On Tue, Feb 18, 2014 at 5:08 PM, Lukas Nalezenec <lukas.nalezenec@firma.seznam.cz= > wrote:
Hi,
How about org.apache.giraph.utils.TestGraph?
I would not try using mrunit for testing Giraph.

cheers
Lukas



On 18.2.2014 17:01, Mirko K=E4mpf wrote:
Hi Martin,

it depends on the details of your implementation. In principle, you do not write Mapper and Reducer
classes, which are tested by MR-Unit. You usually have to implement the application / algorithm specific
logic, in e.g. the Vertex class. MR-Unit might not help that much here.

This leads to the question, if there is a comparable approach / or tool like MR Unit for Giraph?

Best wishes
Mirko




On Tue, Feb 18, 2014 at 3:52 PM, Martin Neumann &l= t;mneumann@spotif= y.com> wrote:
How much of the Giraph 1.1.0 version is still based on MapReduce?
Would something like mrunit (http://mrunit.apache.org/) work with it?


On Tue, Feb 18, 2014 at 12:52 PM, Martin Neumann <mneumann@spotify.com= > wrote:
I'm using the Yarn version (1.1.0 trunk)=A0 and I was not able to get it to work that way. But I'm no expert on that, if anyone knows a way to get it done It would be great.

On Tue, Feb 18, 2014 at 2:07 AM, Roman Shaposhnik <roman@shapo= shnik.org> wrote:
I find running Giraph apps via hadoop's local execution mode
quite useful for debugging.

Thanks,
Roman.

On Mon, Feb 17, 2014 at 4:07 AM, Martin Neumann <mneumann@spotify.com> wrote:
> Hej,
>
> I find Giraph programs quite difficult to debug (all I have is log outputs).
> Anyone has some hints for debugging tools or coding practices to make it
> simpler?
>
> I'm using Giraph 1.1.0 trunk on YARN so I can'= t use Map/Reduce debugging
> aids (at least I think so, I haven't tried any)
>
> cheers
> Martin







--001a11333dfa6b9d6404f2c111d7--