hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Na Yang" <ny...@maprtech.com>
Subject Re: Review Request 25176: HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch]
Date Wed, 03 Sep 2014 04:45:35 GMT


> On Aug. 29, 2014, 5:30 p.m., Brock Noland wrote:
> > Hi Na,
> > 
> > Thank you very much for the patch! I have one high level question:
> > 
> > It appears we created the union_remove_spark* files because we wanted to add an
additional property to the union_remove .q file? Meaning what is the delta beween union_remove_spark_1.q
and union_remove_?
> > 
> > Cheers!
> 
> Na Yang wrote:
>     Hi Brock,
>     
>     That is correct. the union_remove_spark* files include an extra config property hive.merge.sparkfile
comparing to the corresponding union_remove_* files. Except that extra config property, all
other queries in the union_remove_spark* file are same as the queries in the union_remove_*
file. 
>     
>     The hive.merge.sparkfile value is set according to the hive.merge.mapfile and hive.merge.mapredfile
properity values in the orginal union_remove_* file. Regarding to the test result, we expect
to see the same data are returned from the union_remove_spark* queries and the corresponding
union_remove_* queries.
>     
>     Thanks,
>     Na
> 
> Brock Noland wrote:
>     Hi,
>     
>     Thank you very much for the information! I think instead of adding the new union_remove_spark
tests we should just add the hive.merge.sparkfile property to the union_remove q files. The
extra property won't impact the existng tests other than an extra line of outpit.
>     
>     If instead we'd like to keep the hive_remove_spark* properties then we'd need to
add a check to QTestUtil that does not run spark files for MR:
>     
>     https://github.com/apache/hive/blob/trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L431
>     
>     as the tests are currently running for both spark and MR. As such, I think the first
solution (just add the property to the existing tests) makes sense.
>     
>     Thoughts?
> 
> Na Yang wrote:
>     Hi Brock,
>     
>     Thank you for your suggestion. I also prefer the first solution. Let me modify the
existing union_remove q files and re-genenrate the .q.out files for both MR and Spark.
>     
>     Thanks,
>     Na
> 
> Brock Noland wrote:
>     Awesome, thanks!!

Hi Brock,

I have uploaded a new patch according to our previously discussion by modifying the existing
union_remove q files. Can you please review the new changes?

Thanks & Regards,
Na


- Na


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25176/#review51889
-----------------------------------------------------------


On Aug. 29, 2014, 8:59 p.m., Na Yang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25176/
> -----------------------------------------------------------
> 
> (Updated Aug. 29, 2014, 8:59 p.m.)
> 
> 
> Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-7870
>     https://issues.apache.org/jira/browse/HIVE-7870
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch]
> 
> The cause of this problem is during spark/tez task generation, the union file sink operator
are cloned to two new filesink operator. The linkedfilesinkdesc info for those new filesink
operators are missing. In addition, the two new filesink operators also need to be linked
together.   
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties 88ef4f8 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 5ddc16d

>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 379a39c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 76fc290 
>   ql/src/test/queries/clientpositive/union_remove_1.q c87b3fe 
>   ql/src/test/queries/clientpositive/union_remove_10.q 6701952 
>   ql/src/test/queries/clientpositive/union_remove_11.q 4b2fa42 
>   ql/src/test/queries/clientpositive/union_remove_12.q 69d0d0a 
>   ql/src/test/queries/clientpositive/union_remove_13.q 7605f0e 
>   ql/src/test/queries/clientpositive/union_remove_14.q a4fdfc8 
>   ql/src/test/queries/clientpositive/union_remove_15.q e3c937b 
>   ql/src/test/queries/clientpositive/union_remove_16.q 537078b 
>   ql/src/test/queries/clientpositive/union_remove_17.q d70f3d3 
>   ql/src/test/queries/clientpositive/union_remove_18.q 6352bc3 
>   ql/src/test/queries/clientpositive/union_remove_19.q 8c45953 
>   ql/src/test/queries/clientpositive/union_remove_2.q 83cd288 
>   ql/src/test/queries/clientpositive/union_remove_20.q f80f7c1 
>   ql/src/test/queries/clientpositive/union_remove_21.q 8963c25 
>   ql/src/test/queries/clientpositive/union_remove_22.q b0c1ccd 
>   ql/src/test/queries/clientpositive/union_remove_23.q a1b989a 
>   ql/src/test/queries/clientpositive/union_remove_24.q ec561e0 
>   ql/src/test/queries/clientpositive/union_remove_25.q 76c1ff5 
>   ql/src/test/queries/clientpositive/union_remove_3.q 9617f73 
>   ql/src/test/queries/clientpositive/union_remove_4.q cae323b 
>   ql/src/test/queries/clientpositive/union_remove_5.q 5df84e1 
>   ql/src/test/queries/clientpositive/union_remove_6.q bfce26d 
>   ql/src/test/queries/clientpositive/union_remove_7.q 3a95674 
>   ql/src/test/queries/clientpositive/union_remove_8.q a83a43e 
>   ql/src/test/queries/clientpositive/union_remove_9.q e71f6dd 
>   ql/src/test/results/clientpositive/spark/union10.q.out 20c681e 
>   ql/src/test/results/clientpositive/spark/union18.q.out 3f37a0a 
>   ql/src/test/results/clientpositive/spark/union19.q.out 6922fcd 
>   ql/src/test/results/clientpositive/spark/union28.q.out 8bd5218 
>   ql/src/test/results/clientpositive/spark/union29.q.out b9546ef 
>   ql/src/test/results/clientpositive/spark/union3.q.out 3ae6536 
>   ql/src/test/results/clientpositive/spark/union30.q.out 12717a1 
>   ql/src/test/results/clientpositive/spark/union33.q.out b89757f 
>   ql/src/test/results/clientpositive/spark/union4.q.out 6341cd9 
>   ql/src/test/results/clientpositive/spark/union6.q.out 263d9f4 
>   ql/src/test/results/clientpositive/spark/union_remove_10.q.out 927a15d 
>   ql/src/test/results/clientpositive/spark/union_remove_11.q.out 96651e1 
>   ql/src/test/results/clientpositive/spark/union_remove_16.q.out 0954ae4 
>   ql/src/test/results/clientpositive/spark/union_remove_4.q.out cc46dda 
>   ql/src/test/results/clientpositive/spark/union_remove_5.q.out f6cdeb3 
>   ql/src/test/results/clientpositive/spark/union_remove_9.q.out 1f0260c 
> 
> Diff: https://reviews.apache.org/r/25176/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Na Yang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message