hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
Date Fri, 27 Apr 2012 18:41:43 GMT
I don't have the code in front of me, but I bet we're not reusing the Configuration object
when getting the HTable handle. I'll take a look at when the regression went in this evening.


-Joey

Sent from my iPhone

On Apr 27, 2012, at 14:02, Dhaval Y Shah <prince_mithibai@yahoo.co.in> wrote:

> 
> We have an app written in Ruby which uses HBase as the backing store.. It
> uses 
> Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
> until 
> now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which
> is 
> the latest stable CDH release at this point) and in a matter of hours all
> Thrift 
> servers went down..
> 
> Upon further investigation I realized that it was hitting the limit on the
> number 
> of allowed file descriptors (which is pretty high at 32k).. This problem
> occurs if 
> I use thrift in any configuration (hsha, framed transport, threadpool)
> except the 
> nonblocking mode.. Digging further I realized a couple of things:
> 1. Even with light load (1-2 processes hitting the thrift server in quick 
> succession), thrift is spinning up new threads and each of the threads is 
> maintaining a socket connection to zookeeper.. In a matter on minutes (with
> this 
> load test), thrift has > 32k open connections with > 8k threads having
> connection 
> to zookeeper which do not seem to die even after a day..
> 2. The logs show approx 3-4 open connections (presumably for each thread):
> java    53588 hbase 4135r  FIFO                0,6             177426 pipe
> java    53588 hbase 4136w  FIFO                0,6             177426 pipe
> java    53588 hbase 4137r  0000               0,11         0   177427
> eventpoll
> java    53588 hbase 4138u  IPv4             177428                TCP 
> njhaddev05:49729->njhaddev01:2181 (ESTABLISH
> ED)
> 
> CDH3B4 with the exact same configurations and the exact same setup works
> fine but 
> CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option
> because 
> of the low throughput and single threaded nature..
> 
> Any help is greatly appreciated..
> 
> -- 
> View this message in context: http://old.nabble.com/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-connections-to-Zookeeper-tp33760461p33760461.html
> Sent from the HBase User mailing list archive at Nabble.com.
> 

Mime
View raw message