hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5070) Need to implement listLocatedStatus() in ProxyFileSystem for 0.23 shim
Date Thu, 19 Sep 2013 23:21:52 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772453#comment-13772453
] 

Jason Dere commented on HIVE-5070:
----------------------------------

Just looking at this a bit closer, and it appears that listLocatedStatus() was added in hadoop-0.22.0.
 It's only hadoop-2.1.1 that MapReduce has been using this method which is breaking the unit
tests.  So really I think it would be ok for this method to be overridden in 20/20S/23 - I
think the original patch v1 should be sufficient, sorry for the extra work Shanyu! Does anyone
else agree that the v1 patch should be fine to fix this issue, without having to resort to
a new shim method?
                
> Need to implement listLocatedStatus() in ProxyFileSystem for 0.23 shim
> ----------------------------------------------------------------------
>
>                 Key: HIVE-5070
>                 URL: https://issues.apache.org/jira/browse/HIVE-5070
>             Project: Hive
>          Issue Type: Bug
>          Components: CLI
>    Affects Versions: 0.12.0
>            Reporter: shanyu zhao
>             Fix For: 0.13.0
>
>         Attachments: HIVE-5070.patch.txt, HIVE-5070-v2.patch, HIVE-5070-v3.patch
>
>
> MAPREDUCE-1981 introduced a new API for FileSystem - listLocatedStatus. It is used in
Hadoop's FileInputFormat.getSplits(). Hive's ProxyFileSystem class needs to implement this
API in order to make Hive unit test work.
> Otherwise, you'll see these exceptions when running TestCliDriver test case, e.g. results
of running allcolref_in_udf.q:
> [junit] Running org.apache.hadoop.hive.cli.TestCliDriver
>     [junit] Begin query: allcolref_in_udf.q
>     [junit] java.lang.IllegalArgumentException: Wrong FS: pfile:/GitHub/Monarch/project/hive-monarch/build/ql/test/data/warehouse/src,
expected: file:///
>     [junit] 	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)
>     [junit] 	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:69)
>     [junit] 	at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:375)
>     [junit] 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1482)
>     [junit] 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1522)
>     [junit] 	at org.apache.hadoop.fs.FileSystem$4.<init>(FileSystem.java:1798)
>     [junit] 	at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1797)
>     [junit] 	at org.apache.hadoop.fs.ChecksumFileSystem.listLocatedStatus(ChecksumFileSystem.java:579)
>     [junit] 	at org.apache.hadoop.fs.FilterFileSystem.listLocatedStatus(FilterFileSystem.java:235)
>     [junit] 	at org.apache.hadoop.fs.FilterFileSystem.listLocatedStatus(FilterFileSystem.java:235)
>     [junit] 	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
>     [junit] 	at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:217)
>     [junit] 	at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:69)
>     [junit] 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:385)
>     [junit] 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:351)
>     [junit] 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:389)
>     [junit] 	at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:503)
>     [junit] 	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:495)
>     [junit] 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:390)
>     [junit] 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>     [junit] 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>     [junit] 	at java.security.AccessController.doPrivileged(Native Method)
>     [junit] 	at javax.security.auth.Subject.doAs(Subject.java:396)
>     [junit] 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1481)
>     [junit] 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>     [junit] 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
>     [junit] 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:552)
>     [junit] 	at java.security.AccessController.doPrivileged(Native Method)
>     [junit] 	at javax.security.auth.Subject.doAs(Subject.java:396)
>     [junit] 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1481)
>     [junit] 	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:552)
>     [junit] 	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:543)
>     [junit] 	at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:448)
>     [junit] 	at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:688)
>     [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     [junit] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     [junit] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     [junit] 	at java.lang.reflect.Method.invoke(Method.java:597)
>     [junit] 	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message