hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Xcievers Load
Date Thu, 23 Sep 2010 17:10:33 GMT
On Thu, Sep 23, 2010 at 7:05 AM, Michael Segel
<michael_segel@hotmail.com> wrote:
>
>
> 4000 xcievers is a lot.
>
> I'm wondering if there's a correlation between the number of xcievers and ulimit -n.
Should they be configured on a 1 to 1 ratio?
>

2:1 ratio of file descriptors to xceivers. 4000 xceivers is quite
normal on a heavily loaded HBase cluster in my experience.

The cost is the RAM of the extra threads, but there's not much you can
do about it, given the current design of the datanode.

-Todd

> -Mike
>
>> Date: Thu, 23 Sep 2010 08:04:40 -0400
>> Subject: Xcievers Load
>> From: marnandze@gmail.com
>> To: user@hbase.apache.org; common-user@hadoop.apache.org
>>
>> Hi,
>>   We have a job that writes many small files (using MultipleOutputFormat)
>> and its exceeding the 4000 xcievers that we have configured. What is the
>> effect on the cluster of increasing this count to some higher number?
>> Many thanks,
>>    Martin
>>
>> PD: Hbase is also running on the cluster.
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message