hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-3519) partition to directory comparison in CombineHiveInputFormat needs to accept partitions dir without scheme
Date Fri, 05 Oct 2012 01:45:47 GMT

    [ https://issues.apache.org/jira/browse/HIVE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469962#comment-13469962
] 

Thejas M Nair commented on HIVE-3519:
-------------------------------------

The previous patch exposed another problem with related code on windows. TestSymlinkTextInputFormat
throws the following exception -

{code}

java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.doGetWorksFromPath(HiveFileFormatUtils.java:420)
        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:349)
        at org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:187)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at junit.framework.TestCase.runTest(TestCase.java:154)
        at junit.framework.TestCase.runBare(TestCase.java:127)
        at junit.framework.TestResult$1.protect(TestResult.java:106)
        at junit.framework.TestResult.runProtected(TestResult.java:124)
        at junit.framework.TestResult.run(TestResult.java:109)
        at junit.framework.TestCase.run(TestCase.java:118)
        at junit.framework.TestSuite.runTest(TestSuite.java:208)
        at junit.framework.TestSuite.run(TestSuite.java:203)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
PREHOOK: query: drop table text_symlink_text

{code}

This is because in HiveFileFormatUtils.populateNewPartitionDesc it was looking up a path whose
string representation has an additional "/" before the drive letter. While the strings that
were keys in the Map that did not have the "/C".
Eg - It was doing lookup on map- {C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1=[text_symlink_text],
C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1=[text_symlink_text]}
with
with file:/C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
with /C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1


                
> partition to directory comparison in CombineHiveInputFormat needs to accept partitions
dir without scheme
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-3519
>                 URL: https://issues.apache.org/jira/browse/HIVE-3519
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Thejas M Nair
>            Assignee: Thejas M Nair
>             Fix For: 0.10.0
>
>         Attachments: HIVE-3519.1.patch
>
>
> TestSymlinkTextInputFormat.testCombine throws following exception. The test case is just
printing out the stacktrace when that happens instead of failing.
> {code}
> java.io.IOException: cannot find dir = file:/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
in pathToPartitionInfo: [/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1,
/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1]
>         at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:288)
>         at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:256)
>         at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:289)
>         at org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:186)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at junit.framework.TestCase.runTest(TestCase.java:168)
>         at junit.framework.TestCase.runBare(TestCase.java:134)
>         at junit.framework.TestResult$1.protect(TestResult.java:110)
>         at junit.framework.TestResult.runProtected(TestResult.java:128)
>         at junit.framework.TestResult.run(TestResult.java:113)
>         at junit.framework.TestCase.run(TestCase.java:124)
>         at junit.framework.TestSuite.runTest(TestSuite.java:232)
>         at junit.framework.TestSuite.run(TestSuite.java:227)
>         at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
>         at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
>         at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message