hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-924) Make Pig work with multiple versions of Hadoop
Date Wed, 19 Aug 2009 19:02:14 GMT

    [ https://issues.apache.org/jira/browse/PIG-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745166#action_12745166

Todd Lipcon commented on PIG-924:

bq. If existing deployments need a single pig.jar without a hadoop dependency, it might be
possible to create a new target (pig-all) that would create a statically bundled jar; but
I think the default behavior should be to not bundle, build all the shims, and use whatever
hadoop is on the path.

+1 for making the default to *not* bundle hadoop inside pig.jar, and adding another non-default
target for those people who might want it.

bq. The current patch is written as is so that it can be applied to trunk, enabling people
to compile statically, and only require a change to the ant build files to switch to a dynamic
compile later on (after 0.4, probably)

>From the packager's perspective, I'd love if this change could get in for 0.4. If it doesn't,
we'll end up applying the patch ourselves for packaging purposes - we need to have the hadoop
dependency be on the user's installed hadoop, not on whatever happened to get bundled into

> Make Pig work with multiple versions of Hadoop
> ----------------------------------------------
>                 Key: PIG-924
>                 URL: https://issues.apache.org/jira/browse/PIG-924
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Dmitriy V. Ryaboy
>         Attachments: pig_924.2.patch, pig_924.3.patch, pig_924.patch
> The current Pig build scripts package hadoop and other dependencies into the pig.jar
> This means that if users upgrade Hadoop, they also need to upgrade Pig.
> Pig has relatively few dependencies on Hadoop interfaces that changed between 18, 19,
and 20.  It is possibly to write a dynamic shim that allows Pig to use the correct calls for
any of the above versions of Hadoop. Unfortunately, the building process precludes us from
the ability to do this at runtime, and forces an unnecessary Pig rebuild even if dynamic shims
are created.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message