hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-924) Make Pig work with multiple versions of Hadoop
Date Tue, 18 Aug 2009 17:56:14 GMT

    [ https://issues.apache.org/jira/browse/PIG-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744620#action_12744620

Todd Lipcon commented on PIG-924:

Few more comments I missed on the first pass through:

- A few of the shim methods appear unused:
  - fileSystemDeleteOnExit
  - inputFormatValidateInput
  - setTmpFiles

- Is the inner MiniDFSCluster class used? I think this is replaced by the MiniDFSClusterShim
if I understand it correctly.

- Still seems to be some unrelated changes to build.xml - the javac.deprecation for example
- If we are now excluding TestHBaseStorage on all platforms, we should get rid of the two
lines above it that exclude it only on windows - it's redundant and confusing.


> Make Pig work with multiple versions of Hadoop
> ----------------------------------------------
>                 Key: PIG-924
>                 URL: https://issues.apache.org/jira/browse/PIG-924
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Dmitriy V. Ryaboy
>         Attachments: pig_924.2.patch, pig_924.patch
> The current Pig build scripts package hadoop and other dependencies into the pig.jar
> This means that if users upgrade Hadoop, they also need to upgrade Pig.
> Pig has relatively few dependencies on Hadoop interfaces that changed between 18, 19,
and 20.  It is possibly to write a dynamic shim that allows Pig to use the correct calls for
any of the above versions of Hadoop. Unfortunately, the building process precludes us from
the ability to do this at runtime, and forces an unnecessary Pig rebuild even if dynamic shims
are created.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message