Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.213.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <869970D71E26D7498BDAC4E1CA92226B86E1FEDD@MBX021-E3-NJ-2.exch021.domain.local>
References: 
 <869970D71E26D7498BDAC4E1CA92226B86E1FEDD@MBX021-E3-NJ-2.exch021.domain.local>
Date: Mon, 27 Jan 2014 20:41:51 +0530
Message-ID: 
 <CAOcnVr2zd523Yf6S3DdnRsjKua-09irYU5WCNS8H=PgEChHj0A@mail.gmail.com>
Subject: Re: HDFS open file limit
From: Harsh J <harsh@cloudera.com>
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11c1ca12a1f9a904f0f5204f

--001a11c1ca12a1f9a904f0f5204f
Content-Type: text/plain; charset=ISO-8859-1

Hi John,

There is a concurrent connections limit on the DNs that's set to a default
of 4k max parallel threaded connections for reading or writing blocks. This
is also expandable via configuration but usually the default value suffices
even for pretty large operations given the replicas help spread read load
around.

Beyond this you will mostly just run into configurable OS limitations.
On Jan 26, 2014 11:03 PM, "John Lilley" <john.lilley@redpoint.net> wrote:

>  I have an application that wants to open a large set of files in HDFS
> simultaneously.  Are there hard or practical limits to what can be opened
> at once by a single process?  By the entire cluster in aggregate?
>
> Thanks
>
> John
>
>
>
>
>

--001a11c1ca12a1f9a904f0f5204f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">Hi John,</p>
<p dir=3D"ltr">There is a concurrent connections limit on the DNs that&#39;=
s set to a default of 4k max parallel threaded connections for reading or w=
riting blocks. This is also expandable via configuration but usually the de=
fault value suffices even for pretty large operations given the replicas he=
lp spread read load around.</p>

<p dir=3D"ltr">Beyond this you will mostly just run into configurable OS li=
mitations.</p>
<div class=3D"gmail_quote">On Jan 26, 2014 11:03 PM, &quot;John Lilley&quot=
; &lt;<a href=3D"mailto:john.lilley@redpoint.net">john.lilley@redpoint.net<=
/a>&gt; wrote:<br type=3D"attribution"><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


<div lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div>
<p class=3D"MsoNormal">I have an application that wants to open a large set=
 of files in HDFS simultaneously.=A0 Are there hard or practical limits to =
what can be opened at once by a single process?=A0 By the entire cluster in=
 aggregate?<u></u><u></u></p>

<p class=3D"MsoNormal">Thanks<u></u><u></u></p>
<p class=3D"MsoNormal">John<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
</div>
</div>

</blockquote></div>

--001a11c1ca12a1f9a904f0f5204f--