hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shuaishuai Nie (JIRA)" <>
Subject [jira] [Commented] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests
Date Sun, 23 Jun 2013 18:10:20 GMT


Shuaishuai Nie commented on HIVE-4448:

Hi []
Thanks for the comments.
You are right, since HiveConf.ConfVars.METASTOREWAREHOUSE is a path on hdfs, so the "convertPathsFromWindowsToHdfs"
not only change the file separator, it also remove the drive name in Windows path. Here is
an example for HiveConf.ConfVars.METASTOREWAREHOUSE before and after the conversion:
before: "pfile://E:\HIVE-4448\hive-trunk\build\ql/test/data/warehouse" (this path will cause
confusion and "E:\HIVE-4448\hive-trunk\build\ql" will be removes as authority of the path.
We need the /HIVE-4448/hive-trunk/build/ql folder structure when creating the warehouse folder)
after conversion: "pfile:///HIVE-4448/hive-trunk/build/ql/test/data/warehouse" (valid hdfs
path which will create the /test/data/warehouse folder in the correct location on windows
by only use the proper file separators, the driver name in Windows path like E:\ will still
cause confusion in HDFS path.
In addition, the reason I use the function "convertPathsFromWindowsToHdfs" is because the
same function is in the class QTestUtil. This insure the same behavior of path conversion
in all unit tests (a few unit test does not use QTestUtil which make the fix like this easier
to apply to all of the tests)
> Fix metastore warehouse incorrect location on Windows in unit tests
> -------------------------------------------------------------------
>                 Key: HIVE-4448
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Testing Infrastructure
>    Affects Versions: 0.11.0
>         Environment: Windows
>            Reporter: Shuaishuai Nie
>            Assignee: Shuaishuai Nie
>         Attachments: HIVE-4448.1.patch
> Unit test cases which not using QTestUtil will pass incompatible Windows path of METASTOREWAREHOUSE
to HiveConf which result in creating the /test/data/warehouse folder in the wrong location
in Windows. This folder will not be deleted at the beginning of the unit test and the content
will cause failure of unit tests if run the same test case repeatedly. The root cause of this
problem is for path like this "pfile://C:\hive\build\ql/test/data/warehouse", the "C:\hive\build\"
part will be parsed as authority of the path and removed from the path string. The patch will
fix this problem and make the unit test result consistent between Windows and Linux.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message