hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Srinivas <sur...@hortonworks.com>
Subject Re: listing a 530k files directory
Date Fri, 30 May 2014 13:03:01 GMT
Listing such a directory should not be a big problem. Can you cut and paste the command output.

Which release are you using?

Sent from phone

> On May 30, 2014, at 5:49 AM, Guido Serra <zeph@fsfe.org> wrote:
> already tried, didn't work (24cores at 100% and a-lot-memory, stilll ... "GC overhead
limit exceed")
> thanks anyhow
>> On 05/30/2014 02:43 PM, bharath vissapragada wrote:
>> Hi Guido,
>> You can set client side heap in HADOOP_OPTS variable before running the ls command.
>> export HADOOP_OPTS="-Xmx3g"; hadoop fs -ls /
>> - Bharath
>>> On Fri, May 30, 2014 at 5:22 PM, Guido Serra <zeph@fsfe.org> wrote:
>>> Hi,
>>> do you have an idea on how to look at the content of a 530k-files HDFS folder?
>>> (yes, I know it is a bad idea to have such setup, but that’s the status and
I’d like to debug it)
>>> and the only tool that doesn’t go out of memory is "hdfs dfs -count folder/“
>>> -ls goes out of memory, -count with the folder/* goes out of memory …
>>> I’d like at least at the first 10 file names, see the size, maybe open one
>>> thanks,
>>> G.

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message