hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Gates <ga...@yahoo-inc.com>
Subject Re: hadoop15 & hadoop14 both in lib
Date Sat, 16 Feb 2008 00:01:55 GMT
A few answers to your questions.

The hadoopX.jar files in pig's lib directory are not the standard hadoop 
jars.  They differ in two ways.  First, we recreate a hadoop jar that 
rolls in all the jars needed to compile with hadoop.  This is somewhere 
around 15 jars.  Second, we have a small hack we add for historical 
reasons.  We need to resolve both of those issues.  Once we do we can 
use stock hadoop jars instead of carrying along our own.

The reason for having multiple versions is to support compilation 
against multiple versions of hadoop.  I'm not sure what the use of 
hadoop14 is since we can't compile against it anymore.  But once we've 
tested a build against hadoop16 (coming soon), we'll need a library for 
that.  And we will be able to build against either 15 or 16.


Benjamin Francisoud wrote:
> Stefan Groschupf a écrit :
>>> Also, I think the Pig project should follow the common practice and 
>>> NOT rename third-party libraries, i.e. in this case to keep the 
>>> original name of hadoop-0.15.0.jar (if indeed it was that Hadoop 
>>> release).
>> 100 % agreed. What would be great!
>> What would be a perfect solution would be using 
>> http://ant.apache.org/ivy/.
>> However as I understand it this required that the hadoop developers 
>> publish there releases into a repository.
>> However not sure if hadoop developers are willing to do that. It 
>> would help quite a lot for many other projects as well.
>> Stefan
> Even if I think ivy is great pig has so few libs (4 if I exclude 
> hadoop14) that I think a "classical" lib folder holding jars (with 
> version numbers in the jars name) could be enough...
> http://svn.apache.org/repos/asf/incubator/pig/trunk/lib/
> my 2 cents

View raw message