pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-924) Make Pig work with multiple versions of Hadoop
Date Tue, 18 Aug 2009 00:47:14 GMT

    [ https://issues.apache.org/jira/browse/PIG-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744307#action_12744307

Dmitriy V. Ryaboy commented on PIG-924:

Thanks for looking, Todd -- most of those changes, like the factor of 0.9, deprecation, excluding
HBase test, etc, are consistent with the 0.20 patch posted to PIG-660 . 
Moving junit.hadoop.conf is critical -- there are comments about this in 660 -- without it,
resetting hadoop.version doesn't actually work, as some of the information from a previous
build sticks around.

I'll fix the whitespace; this wasn't a final patch, more of a proof of concept.  Point being
this could work, but it can't, because Hadoop is bundled in the jar. I am looking for comments
from the core developer team regarding the possibility of un-bundling.

> Make Pig work with multiple versions of Hadoop
> ----------------------------------------------
>                 Key: PIG-924
>                 URL: https://issues.apache.org/jira/browse/PIG-924
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Dmitriy V. Ryaboy
>         Attachments: pig_924.patch
> The current Pig build scripts package hadoop and other dependencies into the pig.jar
> This means that if users upgrade Hadoop, they also need to upgrade Pig.
> Pig has relatively few dependencies on Hadoop interfaces that changed between 18, 19,
and 20.  It is possibly to write a dynamic shim that allows Pig to use the correct calls for
any of the above versions of Hadoop. Unfortunately, the building process precludes us from
the ability to do this at runtime, and forces an unnecessary Pig rebuild even if dynamic shims
are created.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message