pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Szita <sz...@cloudera.com>
Subject Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Date Wed, 24 May 2017 21:46:23 GMT


> On May 24, 2017, 9:21 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Lines 251 (patched)
> > <https://reviews.apache.org/r/59530/diff/1/?file=1731615#file1731615line252>
> >
> >     Can just load one property file for spark2, leaving the spark 1.6 version in
libraries.properties

AFAIK property values in ant are immutable, so once we set e.g. spark.version=1.6.1 by loading
the default property file and then we load the other propery file for spark2 I think it will
stay unchanged and we cannot override it for anything else e.g. spark.version=2.1.1
Also this way it looks clearer I think.


- Adam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review175998
-----------------------------------------------------------


On May 24, 2017, 4:21 p.m., Nandor Kollar wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
> 
> (Updated May 24, 2017, 4:21 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -----
> 
>   build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
>   ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
>   ivy/spark1.properties PRE-CREATION 
>   ivy/spark2.properties PRE-CREATION 
>   shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
PRE-CREATION 
>   shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
PRE-CREATION 
>   shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
PRE-CREATION 
>   shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
PRE-CREATION 
>   shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
PRE-CREATION 
>   shims/src/spark16/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION

>   shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
PRE-CREATION 
>   shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
PRE-CREATION 
>   shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
PRE-CREATION 
>   shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
PRE-CREATION 
>   shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
PRE-CREATION 
>   shims/src/spark21/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION

>   src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262

>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078

>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
b58415e7e18ca4cf1331beef06e9214600a51424 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
d1c43b1e06adc4c9fe45a83b81103333402e3756 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
e003bbd95763b2d189ff9ec540c89abe52592420 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
00d29b44848546ed16dde2baa8c61b36939971b2 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
baabfa090323e3bef087e259ce19df2e4c34dd63 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
3166fdc31745c013380492e089c83f3e853a3e6e 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6

> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message