hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-6678) Propose some changes to FileContext
Date Sat, 03 Apr 2010 00:14:27 GMT
Propose some changes to FileContext
-----------------------------------

                 Key: HADOOP-6678
                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Hairong Kuang
             Fix For: 0.21.0, 0.22.0


# Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not
to have the whole listing in the memory, benefit more from the iterative listing added in
HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
# Remove methods isFile(Path), isDirectory(Path), and exists.
All these methods are implemented by calling getFileStatus(Path).But most users are not aware
of this. They would write code as below: 
{code}
  FileContext fc = ..;
  if (fc.exists(path)) {
    if (fc.isFile(path)) {
     ...
    } else {
    ...
    }
  }
{code}
The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we
often see that the number of getFileStatus calls is multiple times of the open calls. If we
remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus
first, it is more likely that they will write more efficient code as follow:
{code}
  FileContext fc = ...;
  FileStatus fstatus = fc.getFileStatus(path);
  if (fstatus.isFile() {
    ...
  } else {
    ...
  }
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message