pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Szita <sz...@cloudera.com>
Subject Re: Review Request 54056: Drop Hadoop 1.x support in Pig 0.17
Date Thu, 15 Dec 2016 13:10:58 GMT


> On Dec. 13, 2016, 6:22 p.m., Rohini Palaniswamy wrote:
> > ivy/libraries.properties, lines 75-76
> > <https://reviews.apache.org/r/54056/diff/3/?file=1577650#file1577650line75>
> >
> >     Having two versions of netty will actually create conflict. Hadoop folks were
talking about this in relation to Hadoop 2.8 and Spark. I did not pay much attention then
though. Why do we need both? Thought Hadoop 2.7 still uses netty 3

Lot of unit tests failed after upgrading to Hadoop 2.7.3. I found that besides netty 3, netty-all
4 is used in this version of Hadoop (https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-project/pom.xml#L572)
After referencing it, the test issue is gone.


I suppose it doesn't create a conflict because there was a package name change between netty
3->4, see here:
http://netty.io/wiki/new-and-noteworthy-in-4.0.html#wiki-h2-1


- Adam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54056/#review158925
-----------------------------------------------------------


On Dec. 15, 2016, 1:02 p.m., Adam Szita wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54056/
> -----------------------------------------------------------
> 
> (Updated Dec. 15, 2016, 1:02 p.m.)
> 
> 
> Review request for pig, Daniel Dai and Rohini Palaniswamy.
> 
> 
> Bugs: PIG-4923
>     https://issues.apache.org/jira/browse/PIG-4923
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Drop Hadoop 1.x support in Pig 0.17
> 
> 
> Diffs
> -----
> 
>   .gitignore 4ff5f7e3146fd575b477a910fbf827a78fddff95 
>   BUILDING.md ccee21cc50cb8fa1ceaf70e66fe2feaedf4b2263 
>   bin/pig 81f142681101024761025822aeba9fae23f0a975 
>   bin/pig.py 5ff1d37f8e3d2410c2dacfe180ecd77407c275e3 
>   build.xml 49f887b306835852168529fe9d3a721f804c8b8c 
>   contrib/piggybank/java/build.xml a647e71932a9a066574751b7a7167f0be033d1c6 
>   contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/IndexedStorage.java
47c6105eb407c2260523f53a5e73a4ae18698259 
>   ivy.xml bae1a8180d37ce86ff01cfccd6864e277f64f90b 
>   ivy/ant-contrib-1.0b3.jar PRE-CREATION 
>   ivy/libraries.properties 3a819a510fe87721bc086129a4aa10e3f36e31e3 
>   shims/src/hadoop2/org/apache/pig/backend/hadoop/executionengine/shims/HadoopShims.java
PRE-CREATION 
>   shims/src/hadoop20/org/apache/pig/backend/hadoop/PigATSClient.java 07c5f4989d7113b00d3472b7295e5729e8b775e1

>   shims/src/hadoop20/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigMapBase.java
8bdfa2cea5575cac9f262dafbfa997c897e92e89 
>   shims/src/hadoop20/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigMapReduce.java
50d3b1b50db86ce871db50ce09a735d1a52b7889 
>   shims/src/hadoop20/org/apache/pig/backend/hadoop/executionengine/shims/HadoopShims.java
515ce660a1c738984ae63bca3ccfd3dd162a2c71 
>   shims/src/hadoop20/org/apache/pig/backend/hadoop20/PigJobControl.java 07c984101ccad1df97f9afa5896f29dfedc2150f

>   shims/src/hadoop23/org/apache/hadoop/mapred/DowngradeHelper.java b3a17722c5417c894e298d5693b4616ecee4df39

>   shims/src/hadoop23/org/apache/pig/backend/hadoop/PigATSClient.java  
>   shims/src/hadoop23/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigMapBase.java
 
>   shims/src/hadoop23/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigMapReduce.java
 
>   shims/src/hadoop23/org/apache/pig/backend/hadoop/executionengine/shims/HadoopShims.java
8fbf33fc793986f9b165a61db30baae1be18e0f2 
>   shims/src/hadoop23/org/apache/pig/backend/hadoop23/PigJobControl.java 6439611ae1b983266a9832bd54e653862bdc9564

>   shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 2ceeaad53a15bedd2c8d53452c349fb1fe9cbc9d

>   shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 505c4727caaf849ff20e301222fa67a20655768d

>   shims/test/hadoop23/org/apache/pig/test/MiniCluster.java  
>   shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java  
>   src/docs/src/documentation/content/xdocs/start.xml 8de0933c41736909a273ec782c96e87798df9701

>   src/docs/src/documentation/content/xdocs/tabs.xml f2f34eba132f92b48bc88e5ac80005aa7aeea825

>   src/org/apache/pig/backend/hadoop/accumulo/AbstractAccumuloStorage.java ee04950d102130c0c07be1880fb8ad267856e815

>   src/org/apache/pig/backend/hadoop/accumulo/Utils.java ccbcaaf4baa563aee34038b9fb8856bd92e43755

>   src/org/apache/pig/backend/hadoop/datastorage/ConfigurationUtil.java 26fc5d59616f1a2d418525b08e2823295dae37f7

>   src/org/apache/pig/backend/hadoop/executionengine/Launcher.java cc85991bd24bae02b5499d74b726ea83b50a9044

>   src/org/apache/pig/backend/hadoop/executionengine/fetch/FetchLauncher.java 019fc87c771e7a13eedb6374d2b09431056ac3d1

>   src/org/apache/pig/backend/hadoop/executionengine/fetch/FetchPOStoreImpl.java 2a57435e0724c30c7b3f18b31ca6e2506e45d558

>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
4966f57481d03a38e7adb5021890e7517ac13b35 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java 193905ffec003aaf0329dc0733a020ec5fa83174

>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java
632806fbdb517b87b87971833cffbf2a6d0fcf67 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigInputFormat.java
e6a3d65c7f603e718310beec8e71dcd2cd01fdf2 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigOutputCommitter.java
83f3fbbef1f538bffd0a9439cdb7bcaf5ffe5f45 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java fb401b8afadc175e4ff8b90f81714b2c3c074f23

>   src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java 13ab055e37a084a168ee7a29009e13e34d878134

>   src/org/apache/pig/builtin/HiveUDFBase.java 1c87fdb124cdbe105d2116bb6d1a6d730feaa79e

>   src/org/apache/pig/builtin/OrcStorage.java 5e05dd2f8709a6414b04f9cb459eaca435bed762

>   src/org/apache/pig/builtin/PigStorage.java 730946f779569d45564c414273fc38b2b930e753

>   src/org/apache/pig/builtin/TextLoader.java 8083b6391eef2398555a2bc2f86ae4ccf26f8ed7

>   src/org/apache/pig/impl/io/PigFile.java d46e10b401322c3abf81563825b54b8561adf3ff 
>   src/org/apache/pig/impl/util/JarManager.java d62923497fd085becd23e98a1631456a7eeac58b

>   src/org/apache/pig/impl/util/Utils.java 0f037df9779964239fe4507b69d3c9842be0eb32 
>   src/org/apache/pig/tools/pigstats/PigStatsUtil.java 542cc2ebc0f40c4114566af46b28ae78c7a0efb1

>   src/org/apache/pig/tools/pigstats/mapreduce/MRJobStats.java c8b9128200389882048ad5a05f970ec549a2de8a

>   src/org/apache/pig/tools/pigstats/mapreduce/MRPigStatsUtil.java 4b5927511d1c0c155a6bbaf9cc5eed350ab0f9a3

>   test/e2e/pig/build.xml 2c197adcb0ba12ff611981b1e4e8511c2b95d303 
>   test/excluded-tests-20 f453aa84c2d8af551da425478124b8f9e8f99fab 
>   test/excluded-tests-23  
>   test/org/apache/pig/TestLoadStoreFuncLifeCycle.java f944476c501ed8e7868808983015c87649475abb

>   test/org/apache/pig/parser/TestQueryParserUtils.java a6fb3919d28cc2df98a86c8d3d5a1beabf7b6063

>   test/org/apache/pig/test/TestBZip.java 5a51d206a58418ae5a55c1949a4f08e1557fca23 
>   test/org/apache/pig/test/TestJobControlCompiler.java 37d6542c7335f2fd3e8111e558da9f535e99b485

>   test/org/apache/pig/test/TestLoaderStorerShipCacheFiles.java 5e273ab5dda75f8dd92f3d40cb2582e21c5972d5

>   test/org/apache/pig/test/TestPigRunner.java 13ed4688c3fa26e024f16fa40bcc68fc97a83340

>   test/org/apache/pig/test/TestPigStatsMR.java e429612529534a6c313b4850850e3d786fa53997

>   test/org/apache/pig/test/Util.java 67af8e232210e977b4b999cd2de851425c1aebd2 
>   test/perf/pigmix/bin/generate_data.sh cc216d6f2e9a78975a77e85e9abffe5786bdd9e8 
>   test/perf/pigmix/build.xml 8fe2d1550c8f90e6402152eb1148d1957db6485c 
> 
> Diff: https://reviews.apache.org/r/54056/diff/
> 
> 
> Testing
> -------
> 
> Ran unit tests, made sure e2e and pigmix tests run
> 
> 
> Thanks,
> 
> Adam Szita
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message