hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guido Serra <z...@fsfe.org>
Subject Re: listing a 530k files directory
Date Fri, 30 May 2014 12:49:35 GMT
already tried, didn't work (24cores at 100% and a-lot-memory, stilll ... 
"GC overhead limit exceed")

thanks anyhow

On 05/30/2014 02:43 PM, bharath vissapragada wrote:
> Hi Guido,
>
> You can set client side heap in HADOOP_OPTS variable before running 
> the ls command.
>
> export HADOOP_OPTS="-Xmx3g"; hadoop fs -ls /
>
> - Bharath
>
>
> On Fri, May 30, 2014 at 5:22 PM, Guido Serra <zeph@fsfe.org 
> <mailto:zeph@fsfe.org>> wrote:
>
>     Hi,
>     do you have an idea on how to look at the content of a 530k-files
>     HDFS folder?
>     (yes, I know it is a bad idea to have such setup, but that’s the
>     status and I’d like to debug it)
>     and the only tool that doesn’t go out of memory is "hdfs dfs
>     -count folder/“
>
>     -ls goes out of memory, -count with the folder/* goes out of memory …
>     I’d like at least at the first 10 file names, see the size, maybe
>     open one
>
>     thanks,
>     G.
>
>


Mime
View raw message