hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang (JIRA)" <>
Subject [jira] Commented: (HIVE-1359) Unit test should be shim-aware
Date Fri, 21 May 2010 19:42:16 GMT


Ning Zhang commented on HIVE-1359:

The proposal is to change the unit test directory structure to the following:

ql/src/test/queries/ clientpositive/common/  -- all .q files that should work on all Hadoop
versions and whose outputs are the same
                                                      /shim/0.17 -- all .q files that only
works on 0.17 and have different output than other Hadoop versions.
                                                                /0.18 -- ditto, could be a
symlink to 0.17
                                                                /0.17 -- ditto coulbe a symlink
to 0.17
                                                                / 0.20 -- .q files only works
on hadoop 0.20 or output files are different from other hadoop versions.
ql/src/test/results/clientpositive/common/ -- .out files in the corresponding queries directory.
                                                    /shim/0.17 -- corresponding out file directories

The QTestUtil should be changed to be shim-aware: in addition to test common/*.q files, it
also need to test .q files in the specific hadoop version. 

For committers, if there are multiple distinct set of test files (say 0.17 and 0.20), they
need to test both versions before committing. 

  $ ant test -Dhadoop.version= ...
  $ ant test -Dhadoop.version=0.20.0 ...

For developers, if a new .q test is added and it is not shim-aware (i.e., does not depends
on Hadoop version), it should be put into common only. If it is shim-aware (say having different
behavior in Hadoop 0.20), it should be put in ALL shim subdirectories shim/0.17--0.20, but
NOT common. The reason is that
common is tested for all hadoop versions and if it is put there different results will be

> Unit test should be shim-aware
> ------------------------------
>                 Key: HIVE-1359
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Ning Zhang
>            Assignee: Ning Zhang
> Some features in Hive only works for certain Hadoop versions through shim. However the
unit test structure is not shim-aware in that there is only one set of queries and expected
outputs for all Hadoop versions. This may not be sufficient when we will have different output
for different Hadoop versions. One example is CombineHiveInputFormat wich is only available
from Hadoop 0.20. The plan using CombineHiveInputFormat and HiveInputFormat may be different.
Another example is archival partitions (HAR) which is also only available from 0.20. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message