incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Koontz <ekoo...@hiro-tan.org>
Subject hadoop version profiles
Date Wed, 21 Mar 2012 00:52:55 GMT
Hi Giraphers,
     I think it might be good to look at how we can add support for new 
hadoop versions. Currently we have hadoop_facebook 
(https://issues.apache.org/jira/browse/GIRAPH-14). I am considering 
adding new ones such as hadoop_0.24. Looking at the code, it seems that 
the main hadoop variation between the stock hadoop used (0.203.0) versus 
facebook has to do with the new security-related APIs in the latter that 
is, fortunately, also available in hadoop 0.23 and 0.24. So, hopefully 
we can make use of the existing work that Avery has done for 
hadoop_facebook and apply it to other hadoop versions. Therefore I would 
propose that:

1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java 
and a few other places, where we are currently checking for 
HADOOP_FACEBOOK and HADOOP.
2. we make a new profile called hadoop_secure, which, as with 
hadoop_facebook, will use the above munge flag.
3. we make a new profile "hadoop_0.20.203" for the existing default 
hadoop and make it the default profile (activeByDefault=true). This will 
makes it easier to handle the differences in the hadoop library 
dependency set that have happened between 0.20.203 and hadoop trunk.

Please see https://github.com/ekoontz/giraph/tree/security-profile for 
my branch that implements the above.

Thanks,

-Eugene

Mime
View raw message