hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Vary <>
Subject Re: Review Request 58203: HIVE-16345 BeeLineDriver should be able to run qtest files which are using default database tables
Date Thu, 06 Apr 2017 09:25:26 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated April 6, 2017, 9:25 a.m.)

Review request for hive, Aihua Xu, Zoltan Haindrich, Yongzhi Chen, and Barna Zsombor Klara.


Instead of the needsContinuation method refactor moved the getCommands to BeeLine, so we can
reuse the whole file parsing algorithm in the tests.
Modified the test classes to use the BeeLine.getCommands method
Added a warning message, when use database command is used in the test script

Bugs: HIVE-16345

Repository: hive-git


The goal of the change is to run qtest files which contain queries on tables created by the
init scripts.
It adds the possibility to rewrite the src table references to default.src

This patch contains the following changes:
- Added new parameter to the driver, to control weather the rewrite the table names or not
(test.rewrite.source.tables) - default is true
- Made QTestUtil.getSrcTables() available for QFile class
- Run the QFile not with "!run testfile.q", but reading the file, and assembling the commands
- enable us to parse the queries, and provide better feedback about the failing queries
- QFile rewrites the source tables, if it is required
- Used 9 qtest files from the CliDriver, and added them to BeeLine tests
- Added new filters, and removed redundant ones - I was able to remove every QFile specific
filter, and corresponding setter methods as well
- Moved QFile classes to org.apache.hive.beeline package, so it can use package private methods
from BeeLine, and Commands
- Refactored needsContinuation method in BeeLine, so it can be called from a static context
as well

And one important change is:
- In Utilities.setMapRedWork, change the INPUT_NAME value in the conf to a mapreduce task
specific value. This one is used by the IOContextMap to cache the IOContext objects. Using
the same value for every mapred task prevented them to run in the same JVM. The test were
running sequencially, but failed randomly in parallel

Diffs (updated)

  beeline/src/java/org/apache/hive/beeline/ 11526a7 
  beeline/src/java/org/apache/hive/beeline/ 2578728 
  itests/src/test/resources/ 7a70c9c 
  itests/util/src/main/java/org/apache/hadoop/hive/cli/control/ 0d63f5d

  itests/util/src/main/java/org/apache/hadoop/hive/ql/ 2abf252 
  itests/util/src/main/java/org/apache/hive/beeline/qfile/ ae5a349 
  itests/util/src/main/java/org/apache/hive/beeline/qfile/ 760fde6

  itests/util/src/main/java/org/apache/hive/beeline/qfile/ fcd50ec 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ 79955e9 
  ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 385f9b7 
  ql/src/test/results/clientpositive/beeline/escape_comments.q.out abc0fee 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_12.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_13.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_16.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out PRE-CREATION 




Run the test multiple times with the various combinations of the following parameters:
- test.rewrite.source.tables - runs with true, or without it, fails when set to false
- junit.parallel.threads - runs with 1, or without this parameter


Peter Vary

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message