Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of dlieu.7@gmail.com designates
 209.85.212.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wib6xTIKLnyxu/4DQg+ebVrWFsjuPtZGWRsg3BMvGvhwxCUBQJtLS5sv5I5C/wGWL8
         +su8M1f63M7sUsU2JcwigHPNC3kJweIfmY0iuIR+09vurZrCDXdY6TM80NYvKMy2V9rn
         PDRiODhjsTv/V2dkfq1bWtZgLMJDpZ9KSyfnk=
MIME-Version: 1.0
In-Reply-To: <AANLkTinoWPC3xM9H90UrePtubPEuOGdgDXL+7U2HoSO1@mail.gmail.com>
References: <AANLkTik3g-qgx4sF_CMEWivZSBxXmkuf9EZ2pxTqAhY9@mail.gmail.com>
	<AANLkTinoWPC3xM9H90UrePtubPEuOGdgDXL+7U2HoSO1@mail.gmail.com>
Date: Wed, 23 Mar 2011 14:24:59 -0700
Message-ID: <AANLkTikvx4on3FLx7L2Z__T9Y-jEVHYtHySEw_Ftng5p@mail.gmail.com>
Subject: Re: zk connection leak with TableInput/OutputFormat (CDH3b4, 0.90.1)
From: Dmitriy Lyubimov <dlieu.7@gmail.com>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

yes i am passing destroyProxy=true. But according to the code, it
doesn't affect closing zookeeper connection (it should be closed
anyway) but i  have +1 zk connection each time i run the MR job still.

-d

On Wed, Mar 23, 2011 at 2:22 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> I assume you passed true as second parameter to deleteConnection().
>
> On Wed, Mar 23, 2011 at 1:54 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:
>
>> Hi,
>>
>> I am experiencing severe connection leak in my MR client that uses
>> Hbase as input/output . Every job that uses TableInputFormat leaks 1
>> zookeeper connection per run as evidenced by netstat.
>>
>> I understand that the way HTable manages connections now is it creates
>> a new HBase (and also Zookeeper) connection per each instance of
>> Configuration it is initialized with. By looking at the code of the
>> TableInputFormat class, i see that it creates HTable in the front end
>> during configuration (of course, it probably needs to use it to
>> determine region splits).
>>
>> Since i have to configure each job individually, i must create a new
>> instance of Configuration. Thus, i am not able to use shared HBase
>> connections (which i would prefer to, but there seems to be no way now
>> to do that).
>>
>> So... after i run an instance of MR job, the hbase connection seems to
>> be leaked. It also leaks zk connection , which is a problem since
>> zookeeper instances have limits on how many connections can be made
>> from the same IP and eventually the client is not able to create any
>> new HTables anymore since it can't establish any new zookeeper
>> connections.
>>
>> I tried to do explicit cleanup by calling
>> HConnectionManager.deleteConnection (Configuration) passing in the
>> configuration that i used to create MR job. Doesn't seem to work.
>>
>> So.. Is there a way to run MR job with TableInputFormat without
>> leaking a connection? I am pretty sure i am not creating any HTables
>> in the client side. Or is it a bug? I spent several days now
>> investigation an issue but i am still not able to come up with a
>> workaround against zookeeper connection leaks in HBase MR jobs.
>>
>> thank you very much.
>> -Dmitriy
>>
>