hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhaval Shah <prince_mithi...@yahoo.co.in>
Subject HBase Thrift for CDH3U3 leaking file descriptors/socket connections to Zookeeper
Date Fri, 27 Apr 2012 16:04:29 GMT
We have an app written in Ruby which uses HBase as the backing store.. It uses 
Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro 
until 
now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which is 
the latest stable CDH release at this point) and in a matter of hours all Thrift 
servers went down..

Upon further investigation I realized that it was hitting the limit on the 
number 
of allowed file descriptors (which is pretty high at 32k).. This problem occurs 
if 
I use thrift in any configuration (hsha, framed transport, threadpool) except 
the 
nonblocking mode.. Digging further I realized a couple of things:
1. Even with light load (1-2 processes hitting the thrift server in quick 
succession), thrift is spinning up new threads and each of the threads is 
maintaining a socket connection to zookeeper.. In a matter on minutes (with this 
load test), thrift has > 32k open connections with > 8k threads having 
connection 
to zookeeper which do not seem to die even after a day..
2. The logs show approx 3-4 open connections (presumably for each thread):
java    53588 hbase 4135r  FIFO                0,6             177426 pipe
java    53588 hbase 4136w  FIFO                0,6             177426 pipe
java    53588 hbase 4137r  0000               0,11         0   177427 eventpoll
java    53588 hbase 4138u  IPv4             177428                TCP 
njhaddev05:49729->njhaddev01:2181 (ESTABLISH
ED)

CDH3B4 with the exact same configurations and the exact same setup works fine 
but 
CDH3U3 does not.. Using Thrift in nonblocking mode isn't really an option 
because 
of the low throughput and single threaded nature..

Can someone help please?


Mime
View raw message