pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nandor Kollar <nkol...@cloudera.com>
Subject Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Date Mon, 19 Jun 2017 09:43:59 GMT


> On June 15, 2017, 10:18 a.m., Adam Szita wrote:
> > build.xml
> > Lines 795 (patched)
> > <https://reviews.apache.org/r/59530/diff/5/?file=1750170#file1750170line796>
> >
> >     This will leave the jar inside build directory having Spark2 classes only. We
should fix that, so there is no discrepancy between pig jars generated.

Deleting the jar from build directory as discussed offline.


- Nandor


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177990
-----------------------------------------------------------


On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
> 
> (Updated June 19, 2017, 9:41 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -----
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262

>   src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION

>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION

>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c

>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION

>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
b58415e7e18ca4cf1331beef06e9214600a51424 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
130c8b9a747b176ce2b649ca6d5260527595fb76 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
d1c43b1e06adc4c9fe45a83b81103333402e3756 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
e003bbd95763b2d189ff9ec540c89abe52592420 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
00d29b44848546ed16dde2baa8c61b36939971b2 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
baabfa090323e3bef087e259ce19df2e4c34dd63 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
3166fdc31745c013380492e089c83f3e853a3e6e 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6

>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe

>   test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641

> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/6/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message