hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bharath vissapragada <bharathvissapragada1...@gmail.com>
Subject Re: listing a 530k files directory
Date Fri, 30 May 2014 12:43:57 GMT
Hi Guido,

You can set client side heap in HADOOP_OPTS variable before running the ls
command.

export HADOOP_OPTS="-Xmx3g"; hadoop fs -ls /

- Bharath


On Fri, May 30, 2014 at 5:22 PM, Guido Serra <zeph@fsfe.org> wrote:

> Hi,
> do you have an idea on how to look at the content of a 530k-files HDFS
> folder?
> (yes, I know it is a bad idea to have such setup, but that’s the status
> and I’d like to debug it)
> and the only tool that doesn’t go out of memory is "hdfs dfs -count
> folder/“
>
> -ls goes out of memory, -count with the folder/* goes out of memory …
> I’d like at least at the first 10 file names, see the size, maybe open one
>
> thanks,
> G.

Mime
View raw message