giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maja Kabiljo <>
Subject Re: InputFormat for the example SimpleMasterComputeVertex
Date Fri, 22 Feb 2013 01:12:54 GMT
I wrote up a basic info about aggregators:
Please take a look, and let me know if something needs to be changed / improved.

From: Eli Reisman <<>>
Reply-To: "<>" <<>>
Date: Thursday, February 21, 2013 4:21 PM
To: "<>" <<>>
Subject: Re: InputFormat for the example SimpleMasterComputeVertex

That sounds great to me, maybe just a mention in the wiki that the two functionalities are
tied together will help the idea click for people. Either way this will be a big help I think.

On Thu, Feb 21, 2013 at 3:24 PM, Maja Kabiljo <<>>
Eli, that's an interesting idea, we could have some class which user extends and which is
there only for aggregator registration. Sometimes we want to register some aggregators later
on during the computation, so we need to keep allowing registration from masterCompute too.

But I think for users the biggest problem is to realize that they have to extend and set MasterCompute/this
new class in order to use aggregators. Currently, if user tries to aggregate a value to unregistered
aggregator he will get an exception, but if he tries to get the value of unregistered aggregator
he will just get null. So maybe adding a warning message in that case, together with a wiki
page, might be enough? What do you think?

From: Eli Reisman <<>>
Reply-To: "<>" <<>>
Date: Thursday, February 21, 2013 10:25 AM

To: "<>" <<>>
Subject: Re: InputFormat for the example SimpleMasterComputeVertex

Thanks for the explanation, that makes sense. I would love to see a wiki page at some point,
you have so much knowledge of this piece of Giraph from all your dev work on it and have also
the additional bonus of experience running big cluster jobs using these features so you have
a lot of insight to share.

Would there be any point to a future JIRA to break out the aggregator registration from the
master compute stuff, at least from the user's view? Or is it not that confusing once you've
used them a bit?

On Thu, Feb 14, 2013 at 4:52 PM, Maja Kabiljo <<>>
Progressable exception can be caused by many different reasons (it's totally unrelated to
aggregators), and when looking at which exception it's caused by users should get better sense
about what's going on.
What you are suggesting about providing default master compute is not doable, since the part
which needs to be done there is aggregator registration. We can't know what kind of aggregators
(names and types) an application needs.
I remember I was talking about writing a short tutorial for aggregators long time ago, sorry
for not doing that, will try to get to it soon.

From: Eli Reisman <<>>
Reply-To: "<>" <<>>
Date: Thursday, February 14, 2013 2:23 PM

To: "<>" <<>>
Subject: Re: InputFormat for the example SimpleMasterComputeVertex

Other folks on the list are also having this problem with the progressable utile exception
& job failures. I don't know much about master compute usage but if it is needed to make
the aggregators work, maybe we should have a default dummy class that just handles aggregators
if no other master compute is specified? Or a wiki page? The progressable error message does
not lead us to this conclusion directly.

On Wed, Feb 13, 2013 at 3:04 AM, Maria Stylianou <<>>

I am trying to run the example SimpleMasterComputeVertex, but no matter which Input Format
and graph I give, it doesn't work. Each worker gives the error:

Caused by: java.lang.NullPointerException
        at org.apache.giraph.examples.SimpleMasterComputeVertex.compute(

This line 42 is the first line of the compute()
public void compute(Iterable<DoubleWritable> messages){

So I guess, the initialization is not done correctly, because the input file does not have
the correct format.

Any help would be appreciated,
Maria Stylianou
Intern at Telefonica, Barcelona, Spain
Master Student of European Master in Distributed Computing<>
Universitat Polit├Ęcnica de Catalunya - BarcelonaTech, Barcelona, Spain
KTH Royal Institute of Technology, Stockholm, Sweden

View raw message