pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5253) Pig Hadoop 3 support
Date Fri, 19 Jan 2018 20:54:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332889#comment-16332889

Rohini Palaniswamy commented on PIG-5253:

Will review the code later. Few comments below based on the patch description.

bq. No breaking API change was introduced in Hadoop 3, the current code compiles with Hadoop
  If this is the case, do we really need a jar compiled against hadoop 3. Can't the hadoop
2 compiled pig.jar be directly used against the Hadoop 3 cluster?

bq.  I decided to write the config into all of these files in MiniCluster.java (into hadoop-site.xml
too to stay compatible with Hadoop 2)
 hadoop-site.xml was a Hadoop 1.x and pre-YARN thing.  We can totally get rid of the hadoop-site.xml
in MiniCluster.java as we don't support Hadoop 1.x anymore

bq. HadoopShims: code is identical with Hadoop 2, not sure if we need shims any more. I think
we should move it to the src instead.
bq. build.xml: new target to package both hadoop2 and hadoop3 - not sure that this is needed,
if we move shims, the I think we don't need this target
  We should keep the shims and not move it to src. If a future release comes with incompatibility
we will have to move it back again 
there. If there is no change required for hadoop 2, we can just point it to compile from hadoop
2 shims directory.

> Pig Hadoop 3 support
> --------------------
>                 Key: PIG-5253
>                 URL: https://issues.apache.org/jira/browse/PIG-5253
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>            Priority: Major
>             Fix For: 0.18.0

This message was sent by Atlassian JIRA

View raw message