hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: zk connection leak with TableInput/OutputFormat (CDH3b4, 0.90.1)
Date Wed, 23 Mar 2011 21:22:09 GMT
I assume you passed true as second parameter to deleteConnection().

On Wed, Mar 23, 2011 at 1:54 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:

> Hi,
>
> I am experiencing severe connection leak in my MR client that uses
> Hbase as input/output . Every job that uses TableInputFormat leaks 1
> zookeeper connection per run as evidenced by netstat.
>
> I understand that the way HTable manages connections now is it creates
> a new HBase (and also Zookeeper) connection per each instance of
> Configuration it is initialized with. By looking at the code of the
> TableInputFormat class, i see that it creates HTable in the front end
> during configuration (of course, it probably needs to use it to
> determine region splits).
>
> Since i have to configure each job individually, i must create a new
> instance of Configuration. Thus, i am not able to use shared HBase
> connections (which i would prefer to, but there seems to be no way now
> to do that).
>
> So... after i run an instance of MR job, the hbase connection seems to
> be leaked. It also leaks zk connection , which is a problem since
> zookeeper instances have limits on how many connections can be made
> from the same IP and eventually the client is not able to create any
> new HTables anymore since it can't establish any new zookeeper
> connections.
>
> I tried to do explicit cleanup by calling
> HConnectionManager.deleteConnection (Configuration) passing in the
> configuration that i used to create MR job. Doesn't seem to work.
>
> So.. Is there a way to run MR job with TableInputFormat without
> leaking a connection? I am pretty sure i am not creating any HTables
> in the client side. Or is it a bug? I spent several days now
> investigation an issue but i am still not able to come up with a
> workaround against zookeeper connection leaks in HBase MR jobs.
>
> thank you very much.
> -Dmitriy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message