From hdfs-dev-return-1068-apmail-hadoop-hdfs-dev-archive=hadoop.apache.org@hadoop.apache.org Tue May 04 17:07:19 2010 Return-Path: Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: (qmail 68206 invoked from network); 4 May 2010 17:07:19 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 4 May 2010 17:07:19 -0000 Received: (qmail 19181 invoked by uid 500); 4 May 2010 17:07:19 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 19077 invoked by uid 500); 4 May 2010 17:07:18 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 19069 invoked by uid 99); 4 May 2010 17:07:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 17:07:17 +0000 X-ASF-Spam-Status: No, hits=-1386.1 required=10.0 tests=ALL_TRUSTED,AWL X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 17:07:16 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o44H6uZe013097 for ; Tue, 4 May 2010 17:06:56 GMT Message-ID: <20060738.41721272992816553.JavaMail.jira@thor> Date: Tue, 4 May 2010 13:06:56 -0400 (EDT) From: "Koji Noguchi (JIRA)" To: hdfs-dev@hadoop.apache.org Subject: [jira] Resolved: (HDFS-157) dfs client -ls/-lsr outofmemory when one directory contained 2 million files. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved HDFS-157. ------------------------------- Resolution: Duplicate Hairong just opened a better Jira. HADOOP:6732 Thanks! > dfs client -ls/-lsr outofmemory when one directory contained 2 million files. > ----------------------------------------------------------------------------- > > Key: HDFS-157 > URL: https://issues.apache.org/jira/browse/HDFS-157 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Koji Noguchi > Priority: Minor > > Heapsize was set to 1G. > It'll be nice if dfs client doesn't require that much memory when listing the directory. > Exception in thread "IPC Client connection to namenode/11.11.11.111:1111" java.lang.OutOfMemoryError: GC overhead limit exceeded > at java.util.regex.Pattern.compile(Pattern.java:846) > at java.lang.String.replace(String.java:2208) > at org.apache.hadoop.fs.Path.normalizePath(Path.java:147) > at org.apache.hadoop.fs.Path.initialize(Path.java:137) > at org.apache.hadoop.fs.Path.(Path.java:126) > at org.apache.hadoop.dfs.DFSFileInfo.readFields(DFSFileInfo.java:141) > at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:230) > at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:166) > at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:214) > at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:61) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:273) > Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded > at java.util.Arrays.copyOfRange(Arrays.java:3209) > at java.lang.String.(String.java:216) > at java.lang.StringBuffer.toString(StringBuffer.java:585) > at java.net.URI.toString(URI.java:1907) > at java.net.URI.(URI.java:732) > at org.apache.hadoop.fs.Path.initialize(Path.java:137) > at org.apache.hadoop.fs.Path.(Path.java:126) > at org.apache.hadoop.fs.Path.makeQualified(Path.java:296) > at org.apache.hadoop.dfs.DfsPath.(DfsPath.java:35) > at org.apache.hadoop.dfs.DistributedFileSystem.listPaths(DistributedFileSystem.java:181) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:405) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:423) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:423) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:423) > at org.apache.hadoop.fs.FsShell.ls(FsShell.java:399) > at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1054) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:1244) > at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:1333) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.