hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10003) HarFileSystem.listLocatedStatus() fails
Date Mon, 30 Sep 2013 21:22:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782256#comment-13782256

Jason Dere commented on HADOOP-10003:

So I'm not a bit unsure as to why the test case is failing in the HadoopQA - I've run this
locally on both MacOS and CentOS VM, and the test passes for me.

If HarFileSystem doesn't override listLocatedStatus, the new unit test errors out with the
error shown below.  It looks like FilterFileSystem.listLocatedStatus() simply pushing the
call down the its underlying filesystem, but since this is a hadoop archive the underlying
filesystem won't know about the files in the archive, it needs to look at the index/data files
in the archive to be able to get information about the files in the archive. This fix to HarFileSystem
originally was in MAPREDUCE-1981 (patch v5), but seems to be missing in subsequent versions
of the patch.

Running org.apache.hadoop.fs.TestHarFileSystemBasics
Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.869 sec <<< FAILURE!
- in org.apache.hadoop.fs.TestHarFileSystemBasics
testListLocatedStatus(org.apache.hadoop.fs.TestHarFileSystemBasics)  Time elapsed: 0.403 sec
 <<< ERROR!
java.io.FileNotFoundException: File dir1 does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:386)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1482)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1522)
	at org.apache.hadoop.fs.FileSystem$4.<init>(FileSystem.java:1678)
	at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1677)
	at org.apache.hadoop.fs.ChecksumFileSystem.listLocatedStatus(ChecksumFileSystem.java:579)
	at org.apache.hadoop.fs.FilterFileSystem.listLocatedStatus(FilterFileSystem.java:235)
	at org.apache.hadoop.fs.TestHarFileSystemBasics.testListLocatedStatus(TestHarFileSystemBasics.java:255)

> HarFileSystem.listLocatedStatus() fails
> ---------------------------------------
>                 Key: HADOOP-10003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10003
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.1.1-beta
>            Reporter: Jason Dere
>         Attachments: HADOOP-10003.1.patch, HADOOP-10003.2.patch, HADOOP-10003.3.patch
> It looks like HarFileSystem.listLocatedStatus() doesn't work properly because it is inheriting
FilterFileSystem's implementation.  This is causing archive unit tests to fail in Hive when
using hadoop 2.1.1.
> If HarFileSystem overrides listLocatedStatus() to use FileSystem's implementation, the
Hive unit tests pass.

This message was sent by Atlassian JIRA

View raw message