giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Giraph Architecture bug in
Date Thu, 09 Feb 2012 20:20:40 GMT
We can I suppose, but there are quite a few methods that are there and 
it's a little messy.  Can we discuss on

https://issues.apache.org/jira/browse/GIRAPH-144

I have already submitted a patch that handles this with composition, but 
exposes the internal job object when necessary (i.e. 
FileInputFormat.addInputPath).  This should give us the benefits of 
composition while allowing us to still call convenience methods on the 
Job object.

Avery

On 2/9/12 11:30 AM, Jakob Homan wrote:
>> Since we call waitForCompletion() (which calls submit() internally) in GiraphJob#run(),
we cannot override those methods.
> Why can't we just call super.waitForCompletion() and override them to
> our users?  It seems there are some advantages to GiraphJob being-a
> Job (easier integration with schedulers, Pig, etc.), but I'm not sure
> if it outweighs the inheritance issues that may be reaised...
>
>
> On Wed, Feb 8, 2012 at 9:57 PM, David Garcia<dgarcia@potomacfusion.com>  wrote:
>> Not tonight.  But I might be able to tomorrow.
>>
>> Sent from my HTC Inspireā„¢ 4G on AT&T
>>
>>
>> ----- Reply message -----
>> From: "Avery Ching"<aching@apache.org>
>> To: "giraph-user@incubator.apache.org"<giraph-user@incubator.apache.org>
>> Subject: Giraph Architecture bug in
>> Date: Wed, Feb 8, 2012 8:17 pm
>>
>>
>>
>> AFAIK we don't have any SOP for opening issues.  Maybe I'll take a crack
>> at this one tonight if I find some time, unless you were planning to
>> work on it David.
>>
>> Avery
>>
>> On 2/8/12 5:46 PM, David Garcia wrote:
>>> I opened up
>>>
>>> * GIRAPH-144<https://issues.apache.org/jira/browse/GIRAPH-144>
>>>
>>>
>>> I apologize if I didn't do it up according to project SOP's.  I haven't
>>> had time to read it thoroughly.
>>>
>>> -David
>>>
>>>
>>> On 2/8/12 7:29 PM, "David Garcia"<dgarcia@potomacfusion.com>   wrote:
>>>
>>>> Yeah, I'll write something up.
>>>>
>>>>
>>>> On 2/8/12 7:26 PM, "Avery Ching"<aching@apache.org>   wrote:
>>>>
>>>>> Since we call waitForCompletion() (which calls submit() internally) in
>>>>> GiraphJob#run(), we cannot override those methods.  A better fix would
>>>>> probably be to use composition rather than inheritance (i.e.
>>>>>
>>>>> public class GiraphJob {
>>>>>        Job internalJob;
>>>>> }
>>>>>
>>>>> and expose the methods we would like as necessary.  There are other
>>>>> methods we don't want the user to call, (i.e. setMapperClass(), etc.).
>>>>> David, can you please open an issue for this?
>>>>>
>>>>> Avery
>>>>>
>>>>> On 2/8/12 5:17 PM, David Garcia wrote:
>>>>>> This is a very subtle bug.  GiraphJob inherits from
>>>>>> org.apache.mapreduce.Job.  However, the methods submit() and
>>>>>> waitForCompletion() are not overridden.  I assumed that they were
>>>>>> implemented, so when I called either one of these methods, the
>>>>>> framework
>>>>>> started up identity mappers/reducers.  A simple fix is to throw
>>>>>> unsupported operation exceptions or to implement these methods.
>>>>>> Perhaps
>>>>>> this has been done already?
>>>>>>
>>>>>> -David
>>>>>>
>>>>>> On 2/7/12 7:46 PM, "David Garcia"<dgarcia@potomacfusion.com>
   wrote:
>>>>>>
>>>>>>> I am running into a weird error that I haven't seen yet (I suppose
>>>>>>> I've
>>>>>>> been lucky).  I see the following in the logging:
>>>>>>>
>>>>>>> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>>>>>>> library for your platform... using builtin-java classes where
>>>>>>> applicable
>>>>>>>
>>>>>>>
>>>>>>> In the job definition, the property "mapreduce.map.class" is
not even
>>>>>>> defined.  For Giraph, this is usually set to
>>>>>>> "mapreduce.map.class=org.apache.giraph.graph.GraphMapper"
>>>>>>>
>>>>>>> I'm building my project with hadoop 0.20.204.
>>>>>>>
>>>>>>> When I build the GiraphProject myself (and run my own tests with
the
>>>>>>> projects dependencies), I have no problems.  The main difference
is
>>>>>>> that
>>>>>>> I'm using a Giraph dependency in my work project.  All input
is
>>>>>>> welcome.
>>>>>>> Thx!!
>>>>>>>
>>>>>>> -David
>>>>>>>


Mime
View raw message