Return-Path: X-Original-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4F07711F88 for ; Wed, 20 Aug 2014 18:02:29 +0000 (UTC) Received: (qmail 35488 invoked by uid 500); 20 Aug 2014 18:02:28 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 35379 invoked by uid 500); 20 Aug 2014 18:02:28 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 35296 invoked by uid 99); 20 Aug 2014 18:02:28 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Aug 2014 18:02:28 +0000 Date: Wed, 20 Aug 2014 18:02:28 +0000 (UTC) From: "Kihwal Lee (JIRA)" To: hdfs-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HDFS-6889) Provide an iterator-based listing API for FileSystem MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Kihwal Lee created HDFS-6889: -------------------------------- Summary: Provide an iterator-based listing API for FileSystem Key: HDFS-6889 URL: https://issues.apache.org/jira/browse/HDFS-6889 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee Iterator based listing methods already exist in {{FileContext}} for both simple listing and listing with locations. However, {{FileSystem}} lacks the former. From what I understand, it wasn't added to {{FileSystem}} because it was believed to be phased out soon. Since {{FileSystem}} is very well alive today and new features are getting added frequently, I propose adding an iterator based {{listStatus}} method. As for the name of the new method, we can use the same name used in {{FileContext}} : {{listStatusIterator()}}. It will be particularly useful when listing giant directories. Without this, the client has to build up a huge data structure and hold it in memory. We've seen client JVMs running out of memory because of this. Once this change is made, we can modify FsShell, etc. in followup jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)