pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5253) Pig Hadoop 3 support
Date Fri, 19 Jan 2018 20:48:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332881#comment-16332881

Rohini Palaniswamy commented on PIG-5253:

Copying [~nkollar]'s comments from review board here

This is an initial patch that adds Hadoop 3 support to Pig in addition to Hadoop 2.

Major modifications:
 * No breaking API change was introduced in Hadoop 3, the current code compiles with Hadoop
 * hadoopversion property tells which mode the tests should run, the default is hadoop 2
 * Hadoop 3 introduced a security fix, only whitelisted environment variables are passed to
 * In Hadoop 3 hadoop-site.xml is deprecated, and is replaced by core-site.xml, hdfs-site.xml
and mapred-site.xml. I decided to write the config into all of these files in MiniCluster.java
(into hadoop-site.xml too to stay compatible with Hadoop 2) for the shake of simplicity, we
might want to have different files for Hadoop 2 and separate the properties for Hadoop 3.
 * TestErrorHandling.java: small format change in error message, modified the assert so it
works on both on Hadoop 2 and Hadoop 3
 * HadoopShims: code is identical with Hadoop 2, not sure if we need shims any more. I think
we should move it to the src instead.
 * Split properties into 3 files: common properties, Hadoop 2 and Hadoop 3 specific properties
 * ivy.xml: new config for Hadoop 3
 * build.xml: new target to package both hadoop2 and hadoop3 - not sure that this is needed,
if we move shims, the I think we don't need this target
 * HBase unit test fails on Hadoop 3 (as per https://hbase.apache.org/book.html HBase 1.x
is not tested against Hadoop 2)

> Pig Hadoop 3 support
> --------------------
>                 Key: PIG-5253
>                 URL: https://issues.apache.org/jira/browse/PIG-5253
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>            Priority: Major
>             Fix For: 0.18.0

This message was sent by Atlassian JIRA

View raw message