hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhaval Y Shah <prince_mithi...@yahoo.co.in>
Subject Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
Date Sat, 28 Apr 2012 04:02:13 GMT

Cool thanks Joey... What I also noticed is that the number of threads spawned
by the Thrift server also goes to > 16000 when the number of file
descriptors is going to > 32000.. Probably we are not re-using threads in a
correct way and we are missing the option to limit the number of threads
(which I guess is fixed for 0.94 as per what I read in a jira ticket)


Joey Echeverria-2 wrote:
> 
> I don't have the code in front of me, but I bet we're not reusing the
> Configuration object when getting the HTable handle. I'll take a look at
> when the regression went in this evening. 
> 
> -Joey
> 
> Sent from my iPhone
> 
> On Apr 27, 2012, at 14:02, Dhaval Y Shah <prince_mithibai@yahoo.co.in>
> wrote:
> 
>> 
>> We have an app written in Ruby which uses HBase as the backing store.. It
>> uses 
>> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4
>> distro
>> until 
>> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3
>> (which
>> is 
>> the latest stable CDH release at this point) and in a matter of hours all
>> Thrift 
>> servers went down..
>> 
>> Upon further investigation I realized that it was hitting the limit on
>> the
>> number 
>> of allowed file descriptors (which is pretty high at 32k).. This problem
>> occurs if 
>> I use thrift in any configuration (hsha, framed transport, threadpool)
>> except the 
>> nonblocking mode.. Digging further I realized a couple of things:
>> 1. Even with light load (1-2 processes hitting the thrift server in quick 
>> succession), thrift is spinning up new threads and each of the threads is 
>> maintaining a socket connection to zookeeper.. In a matter on minutes
>> (with
>> this 
>> load test), thrift has > 32k open connections with > 8k threads having
>> connection 
>> to zookeeper which do not seem to die even after a day..
>> 2. The logs show approx 3-4 open connections (presumably for each
>> thread):
>> java    53588 hbase 4135r  FIFO                0,6             177426
>> pipe
>> java    53588 hbase 4136w  FIFO                0,6             177426
>> pipe
>> java    53588 hbase 4137r  0000               0,11         0   177427
>> eventpoll
>> java    53588 hbase 4138u  IPv4             177428                TCP 
>> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
>> ED)
>> 
>> CDH3B4 with the exact same configurations and the exact same setup works
>> fine but 
>> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
>> because 
>> of the low throughput and single threaded nature..
>> 
>> Any help is greatly appreciated..
>> 
>> -- 
>> View this message in context:
>> http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33761191.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message