incubator-giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Homan <jgho...@gmail.com>
Subject Re: Giraph Architecture bug in
Date Thu, 09 Feb 2012 19:30:22 GMT
> Since we call waitForCompletion() (which calls submit() internally) in GiraphJob#run(),
we cannot override those methods.
Why can't we just call super.waitForCompletion() and override them to
our users?  It seems there are some advantages to GiraphJob being-a
Job (easier integration with schedulers, Pig, etc.), but I'm not sure
if it outweighs the inheritance issues that may be reaised...


On Wed, Feb 8, 2012 at 9:57 PM, David Garcia <dgarcia@potomacfusion.com> wrote:
> Not tonight.  But I might be able to tomorrow.
>
> Sent from my HTC Inspire™ 4G on AT&T
>
>
> ----- Reply message -----
> From: "Avery Ching" <aching@apache.org>
> To: "giraph-user@incubator.apache.org" <giraph-user@incubator.apache.org>
> Subject: Giraph Architecture bug in
> Date: Wed, Feb 8, 2012 8:17 pm
>
>
>
> AFAIK we don't have any SOP for opening issues.  Maybe I'll take a crack
> at this one tonight if I find some time, unless you were planning to
> work on it David.
>
> Avery
>
> On 2/8/12 5:46 PM, David Garcia wrote:
>> I opened up
>>
>> * GIRAPH-144<https://issues.apache.org/jira/browse/GIRAPH-144>
>>
>>
>> I apologize if I didn't do it up according to project SOP's.  I haven't
>> had time to read it thoroughly.
>>
>> -David
>>
>>
>> On 2/8/12 7:29 PM, "David Garcia"<dgarcia@potomacfusion.com>  wrote:
>>
>>> Yeah, I'll write something up.
>>>
>>>
>>> On 2/8/12 7:26 PM, "Avery Ching"<aching@apache.org>  wrote:
>>>
>>>> Since we call waitForCompletion() (which calls submit() internally) in
>>>> GiraphJob#run(), we cannot override those methods.  A better fix would
>>>> probably be to use composition rather than inheritance (i.e.
>>>>
>>>> public class GiraphJob {
>>>>      Job internalJob;
>>>> }
>>>>
>>>> and expose the methods we would like as necessary.  There are other
>>>> methods we don't want the user to call, (i.e. setMapperClass(), etc.).
>>>> David, can you please open an issue for this?
>>>>
>>>> Avery
>>>>
>>>> On 2/8/12 5:17 PM, David Garcia wrote:
>>>>> This is a very subtle bug.  GiraphJob inherits from
>>>>> org.apache.mapreduce.Job.  However, the methods submit() and
>>>>> waitForCompletion() are not overridden.  I assumed that they were
>>>>> implemented, so when I called either one of these methods, the
>>>>> framework
>>>>> started up identity mappers/reducers.  A simple fix is to throw
>>>>> unsupported operation exceptions or to implement these methods.
>>>>> Perhaps
>>>>> this has been done already?
>>>>>
>>>>> -David
>>>>>
>>>>> On 2/7/12 7:46 PM, "David Garcia"<dgarcia@potomacfusion.com>  
wrote:
>>>>>
>>>>>> I am running into a weird error that I haven't seen yet (I suppose
>>>>>> I've
>>>>>> been lucky).  I see the following in the logging:
>>>>>>
>>>>>> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>>>>>> library for your platform... using builtin-java classes where
>>>>>> applicable
>>>>>>
>>>>>>
>>>>>> In the job definition, the property "mapreduce.map.class" is not
even
>>>>>> defined.  For Giraph, this is usually set to
>>>>>> "mapreduce.map.class=org.apache.giraph.graph.GraphMapper"
>>>>>>
>>>>>> I'm building my project with hadoop 0.20.204.
>>>>>>
>>>>>> When I build the GiraphProject myself (and run my own tests with
the
>>>>>> projects dependencies), I have no problems.  The main difference
is
>>>>>> that
>>>>>> I'm using a Giraph dependency in my work project.  All input is
>>>>>> welcome.
>>>>>> Thx!!
>>>>>>
>>>>>> -David
>>>>>>
>

Mime
View raw message