Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1E1F410105 for ; Mon, 27 Jan 2014 11:35:32 +0000 (UTC) Received: (qmail 86028 invoked by uid 500); 27 Jan 2014 11:35:23 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 85672 invoked by uid 500); 27 Jan 2014 11:35:21 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 85664 invoked by uid 99); 27 Jan 2014 11:35:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jan 2014 11:35:20 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sudhakara.st@gmail.com designates 209.85.219.50 as permitted sender) Received: from [209.85.219.50] (HELO mail-oa0-f50.google.com) (209.85.219.50) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jan 2014 11:35:16 +0000 Received: by mail-oa0-f50.google.com with SMTP id n16so6496273oag.37 for ; Mon, 27 Jan 2014 03:34:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=BnpZacXPODgrc7q1rno7gB/GyyzjIgjXz+daZiHw72Q=; b=g/5LTsDmoRsaSc/la7eVqy7cTOvN8c5d2X2wV5KZell3eV8dgDf7w3ucupQ6gEcnid do3XY9Xhc4bDdzkM8a+guYKz3nVyiO70iqr2A6yXzvh60gafOK+092XwMGtOjv7kPS5K 9K1QDvGmHEYc7S6vbKh3ug9KtOU8eFnf8YPtkatMwHmbvVzS0QypvwkaS1VVF+xzH1/E WSxNwIZWjqwnLASTZhRJpjT/sWAAkEVS+UM9/islGmR8d1ah94y/I4lmt5y5Ft6InYGS dvlHojaHscNZ6mlRURedz5w4bIJstvicbLrGbR9A34S0JgGzt+B6Vy4ag/4DVB6M4XPq AdfQ== MIME-Version: 1.0 X-Received: by 10.182.241.8 with SMTP id we8mr61292obc.62.1390822495370; Mon, 27 Jan 2014 03:34:55 -0800 (PST) Received: by 10.76.188.195 with HTTP; Mon, 27 Jan 2014 03:34:55 -0800 (PST) In-Reply-To: References: <869970D71E26D7498BDAC4E1CA92226B86E1FEDD@MBX021-E3-NJ-2.exch021.domain.local> Date: Mon, 27 Jan 2014 17:04:55 +0530 Message-ID: Subject: Re: HDFS open file limit From: sudhakara st To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c322c0c8428104f0f21813 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c322c0c8428104f0f21813 Content-Type: text/plain; charset=ISO-8859-1 There is no open file limitation for HDFS. The 'Too many open file' limit is for OS file system. Increase *system-wide maximum number of open files, Per-User/Group/Process file descriptor limits.* On Mon, Jan 27, 2014 at 1:52 AM, Bertrand Dechoux wrote: > At least for each machine, there is the *ulimit *that need to be verified. > > Regards > > Bertrand > > Bertrand Dechoux > > > On Sun, Jan 26, 2014 at 6:32 PM, John Lilley wrote: > >> I have an application that wants to open a large set of files in HDFS >> simultaneously. Are there hard or practical limits to what can be opened >> at once by a single process? By the entire cluster in aggregate? >> >> Thanks >> >> John >> >> >> >> >> > > -- Regards, ...Sudhakara.st --001a11c322c0c8428104f0f21813 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
There is no open file limitation for HDFS. The '= Too many open file'=A0 limit is for OS file system. Increase = system-wide maximum number of open files,=A0 Per-User/Group/Process file de= scriptor limits.


On Mon,= Jan 27, 2014 at 1:52 AM, Bertrand Dechoux <dechouxb@gmail.com> wrote:
At least for each= machine, there is the ulimit that need to be verifie= d.

Regards

Bertrand

Bertrand Dechoux


On Sun, Jan 26, 2014 at 6:32 PM, John Li= lley <john.lilley@redpoint.net> wrote:

I have an application that wants to open a large set= of files in HDFS simultaneously.=A0 Are there hard or practical limits to = what can be opened at once by a single process?=A0 By the entire cluster in= aggregate?

Thanks

John

=A0

=A0





--
=A0 = =A0 =A0=A0
Regards,
...Sudhakara.st
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0
--001a11c322c0c8428104f0f21813--