hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5958) Use JDK 1.6 File APIs in DF.java wherever possible
Date Thu, 22 Oct 2009 21:04:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768900#action_12768900

Aaron Kimball commented on HADOOP-5958:

Hm. A casual inspection shows that uses of DF do not seem to depend on superclass behavior.
I concur that we're probably safe with just changing its interface outright and flagging the
change as incompatible in JIRA.

As for your other question... not so easy.

{{File}} provides a {{listRoots()}} method which doesn't exactly do the same thing. A quick
test shows that it just returns {{"/"}} for a file named {{/home/aaron/foo}}, even though
I have {{/home}} on a separate partition than {{/}}. 

The {{getMounts()}} method is used in {{LocalFileSystem.reportChecksumFailure()}} to move
files to a {{bad_files}} directory on the same device as their original storage. Replacing
{{getMounts()}} naively with {{listRoots()}} could result in actual IO transfer, vs. just
metadata updates in the local fs.

In Java 7, the {{java.nio.file.FileSystem}} class (and friends like {{Path}}) will actually
make all this trivial. It'll even let you change and inspect file ownership and permissions.
But we're not there yet. ::sigh::

So we can't eliminate getMounts() without changing the interface. Fortunately, it seems as
though this is only used in one method, so maybe changing LFS.reportChecksumFailure() is the
right answer?

Here's an alternative strategy:

* Make {{o.a.h.fs.DF}} an interface
* The current DF implementation is renamed {{ShellDF}}
* Add another implementation named {{JavaDF}} which uses {{File}} for what it can.
** Throws {{IOException}} for things like {{getMounts()}}, etc.
* The {{NativeFileSystem}} specifically instantiates a {{ShellDF}} for the situation where
it needs mount-point info (inside reportChecksumFailure)
* All other uses of DF change to JavaDF

> Use JDK 1.6 File APIs in DF.java wherever possible
> --------------------------------------------------
>                 Key: HADOOP-5958
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5958
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Devaraj Das
>             Fix For: 0.21.0
> JDK 1.6 has File APIs like File.getFreeSpace() which should be used instead of spawning
a command process for getting the various disk/partition related attributes. This would avoid
spikes in memory consumption by tasks when things like LocalDirAllocator is used for creating
paths on the filesystem.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message