Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1056920783.1208388442535.JavaMail.jira@brutus>
Date: Wed, 16 Apr 2008 16:27:22 -0700 (PDT)
From: "Bryan Duxbury (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Updated: (HBASE-24) Scaling: Too many open file handles to
 datanodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-24?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Duxbury updated HBASE-24:
-------------------------------

    Fix Version/s:     (was: 0.2.0)

This is really more of a DFS issue, and there's an admittedly suboptimal solution to be had in increasing the max number of open file handles at the OS level. As such, we're going to hold off on solving this issue until after 0.2.

> Scaling: Too many open file handles to datanodes
> ------------------------------------------------
>
>                 Key: HBASE-24
>                 URL: https://issues.apache.org/jira/browse/HBASE-24
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: regionserver
>            Reporter: stack
>            Priority: Critical
>
> We've been here before (HADOOP-2341).
> Today the rapleaf gave me an lsof listing from a regionserver.  Had thousands of open sockets to datanodes all in ESTABLISHED and CLOSE_WAIT state.  On average they seem to have about ten file descriptors/sockets open per region (They have 3 column families IIRC.  Per family, can have between 1-5 or so mapfiles open per family -- 3 is max... but compacting we open a new one, etc.).
> They have thousands of regions.   400 regions -- ~100G, which is not that much -- takes about 4k open file handles.
> If they want a regionserver to server a decent disk worths -- 300-400G -- then thats maybe 1600 regions... 16k file handles.  If more than just 3 column families..... then we are in danger of blowing out limits if they are 32k.
> We've been here before with HADOOP-2341.
> A dfsclient that used non-blocking i/o would help applications like hbase (The datanode doesn't have this problem as bad -- CLOSE_WAIT on regionserver side, the bulk of the open fds in the rapleaf log, don't have a corresponding open resource on datanode end).
> Could also just open mapfiles as needed, but that'd kill our random read performance and its bad enough already.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.