Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates
 209.85.217.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <975535892.8541494.1367511502391.JavaMail.root@redhat.com>
References: <770152377.8528073.1367510470358.JavaMail.root@redhat.com>
 <975535892.8541494.1367511502391.JavaMail.root@redhat.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 2 May 2013 09:28:05 -0700
Message-ID: 
 <CADY20s7XdUsO3N+ftBEC0D8UJab48sdrvL+HX7FyW7Uf0MG1dQ@mail.gmail.com>
Subject: Re: DistributedFileSystem.listStatus() - Why does it do partial
 listings then assemble?
To: hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=14dae93d8f564761b404dbbeb91b

--14dae93d8f564761b404dbbeb91b
Content-Type: text/plain; charset=ISO-8859-1

Hi Brad,

The reasoning is that the NameNode locking is somewhat coarse grained. In
older versions of Hadoop, before it worked this way, we found that listing
large directories (eg with 100k+ files) could end up holding the namenode's
lock for a quite long period of time and starve other clients.

Additionally, I believe there is a second API that does the "on-demand"
fetching of the next set of files from the listing as well, no?

As for the consistency argument, you're correct that you may have a
non-atomic view of the directory contents, but I can't think of any
applications where this would be problematic.

-Todd

On Thu, May 2, 2013 at 9:18 AM, Brad Childs <bdc@redhat.com> wrote:

> Could someone explain why the DistributedFileSystem's listStatus() method
> does a piecemeal assembly of a directory listing within the method?
>
> Is there a locking issue? What if an element is added to the the directory
> during the operation?  What if elements are removed?
>
> It would make sense to me that the FileSystem class listStatus() method
> returned an Iterator allowing only partial fetching/chatter as needed.  But
> I dont understand why you'd want to assemble a giant array of the listing
> chunk by chunk.
>
>
> Here's the source of the listStatus() method, and I've linked the entire
> class below.
>
>
> ---------------------------------
>
>   public FileStatus[] listStatus(Path p) throws IOException {
>     String src = getPathName(p);
>
>     // fetch the first batch of entries in the directory
>     DirectoryListing thisListing = dfs.listPaths(
>         src, HdfsFileStatus.EMPTY_NAME);
>
>     if (thisListing == null) { // the directory does not exist
>       return null;
>     }
>
>     HdfsFileStatus[] partialListing = thisListing.getPartialListing();
>     if (!thisListing.hasMore()) { // got all entries of the directory
>       FileStatus[] stats = new FileStatus[partialListing.length];
>       for (int i = 0; i < partialListing.length; i++) {
>         stats[i] = makeQualified(partialListing[i], p);
>       }
>       statistics.incrementReadOps(1);
>       return stats;
>     }
>
>     // The directory size is too big that it needs to fetch more
>     // estimate the total number of entries in the directory
>     int totalNumEntries =
>       partialListing.length + thisListing.getRemainingEntries();
>     ArrayList<FileStatus> listing =
>       new ArrayList<FileStatus>(totalNumEntries);
>     // add the first batch of entries to the array list
>     for (HdfsFileStatus fileStatus : partialListing) {
>       listing.add(makeQualified(fileStatus, p));
>     }
>     statistics.incrementLargeReadOps(1);
>
>     // now fetch more entries
>     do {
>       thisListing = dfs.listPaths(src, thisListing.getLastName());
>
>       if (thisListing == null) {
>         return null; // the directory is deleted
>       }
>
>       partialListing = thisListing.getPartialListing();
>       for (HdfsFileStatus fileStatus : partialListing) {
>         listing.add(makeQualified(fileStatus, p));
>       }
>       statistics.incrementLargeReadOps(1);
>     } while (thisListing.hasMore());
>
>     return listing.toArray(new FileStatus[listing.size()]);
>   }
>
> --------------------------------------------
>
>
>
>
>
> Ref:
>
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4/src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java
> http://docs.oracle.com/javase/6/docs/api/java/util/Iterator.html
>
>
> thanks!
>
> -bc
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--14dae93d8f564761b404dbbeb91b--