hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yongzhi Chen <yc...@cloudera.com>
Subject Re: Review Request 58203: HIVE-16345 BeeLineDriver should be able to run qtest files which are using default database tables
Date Wed, 05 Apr 2017 21:34:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58203/#review171166
-----------------------------------------------------------




itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java
Lines 130 (patched)
<https://reviews.apache.org/r/58203/#comment244045>

    How do you handle the case command has comment following ';' and new command start after
;  ? Do these cases matters?
    For example:
    show tables; --comment
    
    show tables; select * from
    src;
    
    The beeline.Commands class has code similar to getCommands:
    handleMultiLineCmd, logic in execute
    Could you figure out a way to use the some of the code there?



itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java
Lines 160 (patched)
<https://reviews.apache.org/r/58203/#comment244048>

    Is that possible the table belong to other database?
    For example:
    use foo;
    select * from tableinfoo;



itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java
Line 92 (original), 90 (patched)
<https://reviews.apache.org/r/58203/#comment244047>

    Why we need to replace the tablename with default.tablename? Could you just add use default
?


- Yongzhi Chen


On April 5, 2017, 10:35 a.m., Peter Vary wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58203/
> -----------------------------------------------------------
> 
> (Updated April 5, 2017, 10:35 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Zoltan Haindrich, Yongzhi Chen, and Barna Zsombor
Klara.
> 
> 
> Bugs: HIVE-16345
>     https://issues.apache.org/jira/browse/HIVE-16345
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> The goal of the change is to run qtest files which contain queries on tables created
by the init scripts.
> It adds the possibility to rewrite the src table references to default.src
> 
> This patch contains the following changes:
> - Added new parameter to the driver, to control weather the rewrite the table names or
not (test.rewrite.source.tables) - default is true
> - Made QTestUtil.getSrcTables() available for QFile class
> - Run the QFile not with "!run testfile.q", but reading the file, and assembling the
commands - enable us to parse the queries, and provide better feedback about the failing queries
> - QFile rewrites the source tables, if it is required
> - Used 9 qtest files from the CliDriver, and added them to BeeLine tests
> - Added new filters, and removed redundant ones - I was able to remove every QFile specific
filter, and corresponding setter methods as well
> - Moved QFile classes to org.apache.hive.beeline package, so it can use package private
methods from BeeLine, and Commands
> - Refactored needsContinuation method in BeeLine, so it can be called from a static context
as well
> 
> And one important change is:
> - In Utilities.setMapRedWork, change the INPUT_NAME value in the conf to a mapreduce
task specific value. This one is used by the IOContextMap to cache the IOContext objects.
Using the same value for every mapred task prevented them to run in the same JVM. The test
were running sequencially, but failed randomly in parallel
> 
> 
> Diffs
> -----
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 11526a7 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
0d63f5d 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2abf252 
>   itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java ae5a349 
>   itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java 760fde6

>   itests/util/src/main/java/org/apache/hive/beeline/qfile/package-info.java fcd50ec 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 79955e9 
>   ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 385f9b7 
>   ql/src/test/results/clientpositive/beeline/escape_comments.q.out abc0fee 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_11.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_12.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_13.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_16.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/58203/diff/1/
> 
> 
> Testing
> -------
> 
> Run the test multiple times with the various combinations of the following parameters:
> - test.rewrite.source.tables - runs with true, or without it, fails when set to false
> - junit.parallel.threads - runs with 1, or without this parameter
> 
> 
> Thanks,
> 
> Peter Vary
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message