Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of
 bharathvissapragada1990@gmail.com designates 209.85.192.54 as permitted
 sender)
MIME-Version: 1.0
In-Reply-To: <5B121937-A4AF-4982-98A3-F2FAA92EDFF1@fsfe.org>
References: <5B121937-A4AF-4982-98A3-F2FAA92EDFF1@fsfe.org>
From: bharath vissapragada <bharathvissapragada1990@gmail.com>
Date: Fri, 30 May 2014 18:13:57 +0530
Message-ID: 
 <CAK3hZ7Q+3mth+xY=RBP=0QqTxFeeFwJ83Ge5viX1KND+Mniofg@mail.gmail.com>
Subject: Re: listing a 530k files directory
To: User <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11c13db259fbda04fa9d673a

--001a11c13db259fbda04fa9d673a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Guido,

You can set client side heap in HADOOP_OPTS variable before running the ls
command.

export HADOOP_OPTS=3D"-Xmx3g"; hadoop fs -ls /

- Bharath


On Fri, May 30, 2014 at 5:22 PM, Guido Serra <zeph@fsfe.org> wrote:

> Hi,
> do you have an idea on how to look at the content of a 530k-files HDFS
> folder?
> (yes, I know it is a bad idea to have such setup, but that=E2=80=99s the =
status
> and I=E2=80=99d like to debug it)
> and the only tool that doesn=E2=80=99t go out of memory is "hdfs dfs -cou=
nt
> folder/=E2=80=9C
>
> -ls goes out of memory, -count with the folder/* goes out of memory =E2=
=80=A6
> I=E2=80=99d like at least at the first 10 file names, see the size, maybe=
 open one
>
> thanks,
> G.

--001a11c13db259fbda04fa9d673a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Guido,<div><br></div><div>You can set client side heap =
in HADOOP_OPTS variable before running the ls command.</div><div><br></div>=
<div><span style=3D"color:rgb(0,0,0);font-family:arial,sans-serif;font-size=
:13px">export=C2=A0</span><span style=3D"font-family:arial,sans-serif;font-=
size:13px">HADOOP_OPTS</span><span style=3D"color:rgb(0,0,0);font-family:ar=
ial,sans-serif;font-size:13px">=3D&quot;-Xmx3g&quot;; hadoop fs -ls /</span=
><br>


</div><div><span style=3D"color:rgb(0,0,0);font-family:arial,sans-serif;fon=
t-size:13px"><br></span></div><div><span style=3D"color:rgb(0,0,0);font-fam=
ily:arial,sans-serif;font-size:13px">- Bharath</span></div><div class=3D"gm=
ail_extra">


<br><br><div class=3D"gmail_quote">
On Fri, May 30, 2014 at 5:22 PM, Guido Serra <span dir=3D"ltr">&lt;<a href=
=3D"mailto:zeph@fsfe.org" target=3D"_blank">zeph@fsfe.org</a>&gt;</span> wr=
ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex=
;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style=
:solid;padding-left:1ex">


Hi,<br>
do you have an idea on how to look at the content of a 530k-files HDFS fold=
er?<br>
(yes, I know it is a bad idea to have such setup, but that=E2=80=99s the st=
atus and I=E2=80=99d like to debug it)<br>
and the only tool that doesn=E2=80=99t go out of memory is &quot;hdfs dfs -=
count folder/=E2=80=9C<br>
<br>
-ls goes out of memory, -count with the folder/* goes out of memory =E2=80=
=A6<br>
I=E2=80=99d like at least at the first 10 file names, see the size, maybe o=
pen one<br>
<br>
thanks,<br>
G.</blockquote></div><br></div></div>

--001a11c13db259fbda04fa9d673a--