hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brad Childs <...@redhat.com>
Subject DistributedFileSystem.listStatus() - Why does it do partial listings then assemble?
Date Thu, 02 May 2013 16:18:22 GMT
Could someone explain why the DistributedFileSystem's listStatus() method does a piecemeal
assembly of a directory listing within the method?

Is there a locking issue? What if an element is added to the the directory during the operation?
 What if elements are removed?

It would make sense to me that the FileSystem class listStatus() method returned an Iterator
allowing only partial fetching/chatter as needed.  But I dont understand why you'd want to
assemble a giant array of the listing chunk by chunk.

Here's the source of the listStatus() method, and I've linked the entire class below.


  public FileStatus[] listStatus(Path p) throws IOException {
    String src = getPathName(p);
    // fetch the first batch of entries in the directory
    DirectoryListing thisListing = dfs.listPaths(
        src, HdfsFileStatus.EMPTY_NAME);
    if (thisListing == null) { // the directory does not exist
      return null;
    HdfsFileStatus[] partialListing = thisListing.getPartialListing();
    if (!thisListing.hasMore()) { // got all entries of the directory
      FileStatus[] stats = new FileStatus[partialListing.length];
      for (int i = 0; i < partialListing.length; i++) {
        stats[i] = makeQualified(partialListing[i], p);
      return stats;
    // The directory size is too big that it needs to fetch more
    // estimate the total number of entries in the directory
    int totalNumEntries = 
      partialListing.length + thisListing.getRemainingEntries();
    ArrayList<FileStatus> listing = 
      new ArrayList<FileStatus>(totalNumEntries);
    // add the first batch of entries to the array list
    for (HdfsFileStatus fileStatus : partialListing) {
      listing.add(makeQualified(fileStatus, p));

    // now fetch more entries
    do {
      thisListing = dfs.listPaths(src, thisListing.getLastName());
      if (thisListing == null) {
        return null; // the directory is deleted
      partialListing = thisListing.getPartialListing();
      for (HdfsFileStatus fileStatus : partialListing) {
        listing.add(makeQualified(fileStatus, p));
    } while (thisListing.hasMore());

    return listing.toArray(new FileStatus[listing.size()]);





View raw message