incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Koontz <>
Subject Re: hadoop version profiles
Date Thu, 22 Mar 2012 01:13:55 GMT
Thanks Hyunsik and Avery for your comments!

I've added a JIRA here:  to present a patch 
that adds two new profiles:


hadoop_0.20.203 is the default (i.e. same as if you do not give a -P 

I tried "mvn clean verify && mvn -Phadoop_0.20.203 clean verify && mvn 
-Phadoop_trunk clean verify" and this worked with the GIRAPH-168 patch 

As I said on GIRAPH-168, I could not get hadoop_non_secure or 
hadoop_facebook to compile.

I agree that munge is hacky - in my patch I tried to modify our use of 
munge a bit to make it simpler: I removed the usages of HADOOP (e.g. 
"/*if_not[HADOOP]"). I felt like this usage is misleading, because it 
might make you think that Giraph can be compiled without Hadoop support 
at all, which is not what is meant by "/*if_not[HADOOP]": in fact, what 
it actually means is that the Hadoop to be used is the default version 
(Hadoop 0.205.0).


On 3/21/12 3:46 AM, Hyunsik Choi wrote:
> This approach looks good.
> +1
> On Wed, Mar 21, 2012 at 3:53 PM, Avery Ching<>  wrote:
>> I agree with this approach, although munge is kinda hacky.  It is easy
>> though. =)
>> Avery
>> On 3/20/12 5:52 PM, Eugene Koontz wrote:
>>> Hi Giraphers,
>>>     I think it might be good to look at how we can add support for new
>>> hadoop versions. Currently we have hadoop_facebook (
>>> I am considering adding new ones such as hadoop_0.24. Looking at the code,
>>> it seems that the main hadoop variation between the stock hadoop used
>>> (0.203.0) versus facebook has to do with the new security-related APIs in
>>> the latter that is, fortunately, also available in hadoop 0.23 and 0.24.
>>> So, hopefully we can make use of the existing work that Avery has done for
>>> hadoop_facebook and apply it to other hadoop versions. Therefore I would
>>> propose that:
>>> 1. a new munge flag HADOOP_SECURE to be used in and
>>> a few other places, where we are currently checking for HADOOP_FACEBOOK and
>>> 2. we make a new profile called hadoop_secure, which, as with
>>> hadoop_facebook, will use the above munge flag.
>>> 3. we make a new profile "hadoop_0.20.203" for the existing default
>>> hadoop and make it the default profile (activeByDefault=true). This will
>>> makes it easier to handle the differences in the hadoop library dependency
>>> set that have happened between 0.20.203 and hadoop trunk.
>>> Please see**giraph/tree/security-profile<>for
my branch that implements the above.
>>> Thanks,
>>> -Eugene

View raw message