hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navis Ryu" <>
Subject Review Request 30345: Problem with subquery using windowing functions
Date Wed, 28 Jan 2015 01:03:10 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for hive.

Bugs: HIVE-9228

Repository: hive-git


The following query with window functions failed. The internal query works fine.

select col1, col2, col3 from (select col1,col2, col3, count(case when col4=1 then 1 end )
over (partition by col1, col2) as col5, row_number() over (partition by col1, col2 order by
col4) as col6 from tab1) t;

HIVE generates an execution plan with 2 jobs. 
1. The first job is to basically calculate window function for col5.  
2. The second job is to calculate window function for col6 and output.

The plan says the first job outputs the columns (col1, col2, col3, col4) to a tmp file since
only these columns are used in later stage. While, the PTF operator for the first job outputs
(_wcol0, col1, col2, col3, col4) with _wcol0 as the result of the window function even it's
not used. 

In the second job, the map operator still reads the 4 columns (col1, col2, col3, col4) from
the temp file using the plan. That causes the exception.


  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ 57ce849 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ 469dc9f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ c05b2aa 
  ql/src/test/queries/clientpositive/windowing_windowspec.q 6d8ce67 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 020fdff 
  ql/src/test/results/clientpositive/subquery_notin.q.out 599a61e 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 06d5708 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out a814849 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 1e3c43c 
  ql/src/test/results/clientpositive/windowing_windowspec.q.out 00af6b8 




Navis Ryu

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message