Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAK3hZ7Q+3mth+xY=RBP=0QqTxFeeFwJ83Ge5viX1KND+Mniofg@mail.gmail.com>
References: <5B121937-A4AF-4982-98A3-F2FAA92EDFF1@fsfe.org>
 <CAK3hZ7Q+3mth+xY=RBP=0QqTxFeeFwJ83Ge5viX1KND+Mniofg@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Sat, 31 May 2014 00:12:59 +0530
Message-ID: 
 <CAOcnVr10e_3MGOJbbnX9FuyGenyeTuu4VrrD9qgF-dkBHPhGzw@mail.gmail.com>
Subject: Re: listing a 530k files directory
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

The HADOOP_OPTS gets overriden by HADOOP_CLIENT_OPTS for FsShell
utilities. The right way to extend is to use HADOOP_CLIENT_OPTS
instead, for FsShell and other client applications such as "hadoop
fs"/"hdfs dfs"/"hadoop jar", etc..

On Fri, May 30, 2014 at 6:13 PM, bharath vissapragada
<bharathvissapragada1990@gmail.com> wrote:
> Hi Guido,
>
> You can set client side heap in HADOOP_OPTS variable before running the l=
s
> command.
>
> export HADOOP_OPTS=3D"-Xmx3g"; hadoop fs -ls /
>
> - Bharath
>
>
> On Fri, May 30, 2014 at 5:22 PM, Guido Serra <zeph@fsfe.org> wrote:
>>
>> Hi,
>> do you have an idea on how to look at the content of a 530k-files HDFS
>> folder?
>> (yes, I know it is a bad idea to have such setup, but that=E2=80=99s the=
 status
>> and I=E2=80=99d like to debug it)
>> and the only tool that doesn=E2=80=99t go out of memory is "hdfs dfs -co=
unt
>> folder/=E2=80=9C
>>
>> -ls goes out of memory, -count with the folder/* goes out of memory =E2=
=80=A6
>> I=E2=80=99d like at least at the first 10 file names, see the size, mayb=
e open one
>>
>> thanks,
>> G.
>
>


--=20
Harsh J