hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pengcheng xiong" <pxi...@hortonworks.com>
Subject Re: Review Request 32549: HiveOnTez: Union followed by Multi-GB followed by Multi-insert loses data
Date Sun, 12 Apr 2015 23:27:45 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32549/
-----------------------------------------------------------

(Updated April 12, 2015, 11:27 p.m.)


Review request for hive, Gunther Hagleitner and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
-------

In q.test environment with src table, execute the following query: 
{code}
CREATE TABLE DEST1(key STRING, value STRING) STORED AS TEXTFILE;

CREATE TABLE DEST2(key STRING, val1 STRING, val2 STRING) STORED AS TEXTFILE;

FROM (select 'tst1' as key, cast(count(1) as string) as value from src s1
                         UNION all 
      select s2.key as key, s2.value as value from src s2) unionsrc
INSERT OVERWRITE TABLE DEST1 SELECT unionsrc.key, COUNT(DISTINCT SUBSTR(unionsrc.value,5))
GROUP BY unionsrc.key
INSERT OVERWRITE TABLE DEST2 SELECT unionsrc.key, unionsrc.value, COUNT(DISTINCT SUBSTR(unionsrc.value,5))

GROUP BY unionsrc.key, unionsrc.value;

select * from DEST1;
select * from DEST2;
{code}

DEST1 and DEST2 should both have 310 rows. However, DEST2 only has 1 row "tst1    500    
1"


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java b45c782 
  itests/src/test/resources/testconfiguration.properties 1502d80 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java 90616ad 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 4dcdf91 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 0990894 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java 08fd61e 
  ql/src/test/queries/clientpositive/explainuser_2.q 03264ca 
  ql/src/test/queries/clientpositive/tez_union_multiinsert.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez/explainuser_2.q.out 61ebe1a 
  ql/src/test/results/clientpositive/tez/tez_union_multiinsert.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/32549/diff/


Testing
-------


Thanks,

pengcheng xiong


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message