incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Koontz <ekoo...@hiro-tan.org>
Subject Re: hadoop version profiles
Date Thu, 22 Mar 2012 01:13:55 GMT
Thanks Hyunsik and Avery for your comments!

I've added a JIRA here: 
https://issues.apache.org/jira/browse/GIRAPH-168  to present a patch 
that adds two new profiles:

hadoop_0.20.203
hadoop_trunk

hadoop_0.20.203 is the default (i.e. same as if you do not give a -P 
argument).

I tried "mvn clean verify && mvn -Phadoop_0.20.203 clean verify && mvn 
-Phadoop_trunk clean verify" and this worked with the GIRAPH-168 patch 
applied.

As I said on GIRAPH-168, I could not get hadoop_non_secure or 
hadoop_facebook to compile.

I agree that munge is hacky - in my patch I tried to modify our use of 
munge a bit to make it simpler: I removed the usages of HADOOP (e.g. 
"/*if_not[HADOOP]"). I felt like this usage is misleading, because it 
might make you think that Giraph can be compiled without Hadoop support 
at all, which is not what is meant by "/*if_not[HADOOP]": in fact, what 
it actually means is that the Hadoop to be used is the default version 
(Hadoop 0.205.0).

-Eugene




On 3/21/12 3:46 AM, Hyunsik Choi wrote:
> This approach looks good.
> +1
>
> On Wed, Mar 21, 2012 at 3:53 PM, Avery Ching<aching@apache.org>  wrote:
>
>> I agree with this approach, although munge is kinda hacky.  It is easy
>> though. =)
>>
>> Avery
>>
>>
>> On 3/20/12 5:52 PM, Eugene Koontz wrote:
>>
>>> Hi Giraphers,
>>>     I think it might be good to look at how we can add support for new
>>> hadoop versions. Currently we have hadoop_facebook (
>>> https://issues.apache.org/**jira/browse/GIRAPH-14<https://issues.apache.org/jira/browse/GIRAPH-14>).
>>> I am considering adding new ones such as hadoop_0.24. Looking at the code,
>>> it seems that the main hadoop variation between the stock hadoop used
>>> (0.203.0) versus facebook has to do with the new security-related APIs in
>>> the latter that is, fortunately, also available in hadoop 0.23 and 0.24.
>>> So, hopefully we can make use of the existing work that Avery has done for
>>> hadoop_facebook and apply it to other hadoop versions. Therefore I would
>>> propose that:
>>>
>>> 1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java and
>>> a few other places, where we are currently checking for HADOOP_FACEBOOK and
>>> HADOOP.
>>> 2. we make a new profile called hadoop_secure, which, as with
>>> hadoop_facebook, will use the above munge flag.
>>> 3. we make a new profile "hadoop_0.20.203" for the existing default
>>> hadoop and make it the default profile (activeByDefault=true). This will
>>> makes it easier to handle the differences in the hadoop library dependency
>>> set that have happened between 0.20.203 and hadoop trunk.
>>>
>>> Please see https://github.com/ekoontz/**giraph/tree/security-profile<https://github.com/ekoontz/giraph/tree/security-profile>for
my branch that implements the above.
>>>
>>> Thanks,
>>>
>>> -Eugene
>>>
>>


Mime
View raw message