Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 209.85.213.174 as permitted sender)
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: 
 <CABNOTEVneH7NidyqDA1JK+0VSqKdwPBmPButucGdxFJQhe1-mA@mail.gmail.com>
References: 
 <CABNOTEVMvVTfx__nXhShzdziuvRW6OLJ2+B59g1Ls2eZ1=3rZA@mail.gmail.com>
	<CADcMMgFbf_qWFq23R8x2mvrcsa5cXG6ZKFC_efyFf_3DhCgcTA@mail.gmail.com>
	<CABNOTEVwwm4LB9X3hhONXtfWved6VAgNcpFW3Yuz6t+9PnPLCQ@mail.gmail.com>
	<CADcMMgHpwi-xR9_BAskBpa0y9oCNUmc_fs=E335xoev70-MXGQ@mail.gmail.com>
	<CABNOTEVneH7NidyqDA1JK+0VSqKdwPBmPButucGdxFJQhe1-mA@mail.gmail.com>
Date: Fri, 12 Dec 2014 17:03:35 -0800
Message-ID: 
 <CADcMMgG9ns3so6KGBbaJyLc=CSuBb89=Hedp8bDqwUz1mv_Veg@mail.gmail.com>
Subject: Re: HConnectionManager leaks with zookeeper conection oo many
 connections from /my.tomcat.server.com - max is 60
From: Stack <stack@duboce.net>
To: Hbase-User <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c3b40c30dafa050a0e945f

--001a11c3b40c30dafa050a0e945f
Content-Type: text/plain; charset=UTF-8

On Fri, Dec 12, 2014 at 11:45 AM, Serega Sheypak <serega.sheypak@gmail.com>
wrote:
>
> i have 10K doPost/doGet requests per second.
>

How many concurrent threads going on in your tomcat?  Is the Connection
shared amongst all threads?  Perhaps you have > 60 concurrent connections
running at a time and each Connection has its own zk instance.  Thread dump?

Post all the code?

I tried multithreaded with 100 threads but still can't repro (see code
below).  Next would be to get zk to dump concurrent connection count but
lets see what your answers above are first.

St.Ack


public class TestConnnection {

  static class Putter extends Thread {

    Configuration c;

    Putter(final Configuration c, final String i) {

      super(i);

      this.c = c;

    }


    @Override

    public void run() {

      try {

      for (int i = 0; i < 100; i++) {

        HConnection connection = HConnectionManager.createConnection(this.c
);

        HTableInterface table = connection.getTable(TableName.valueOf(
"table1"));

        byte [] cf = Bytes.toBytes("t");

        try {

          byte [] bytes = Bytes.toBytes(i);

          Put p = new Put(bytes);

          p.add(cf, cf, bytes);

          table.put(p);

        } finally {

          table.close();

          connection.close();

        }

        System.out.println(getName() + ", i=" + i);

      }

      } catch (Throwable t) {

        throw new RuntimeException(t);

      }

    }

  };


  public static void main(String[] args) throws IOException {

    Configuration config = HBaseConfiguration.create();

    for (int i = 0; i < 100; i++) {

      Thread t = new Putter(config, "" + i);

      t.start();

    }

  }

}


> Servlet is NOT single-threaded. each doPost/doGet
> invokes these lines (encapsulated in DAO):
>
>  16       HConnection connection =
> HConnectionManager.createConnection(config);
>  17       HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
>
> and
>
> 24       } finally {
>  25         table.close();
>  26         connection.close();
>  27       }
>
> I assumed that this static construction
>
> 16       HConnection connection =
> HConnectionManager.createConnection(config);
>
> correctly handles multi-threaded access somewhere deep inside.
> Right now I don't understand what do I do wrong.
>
> Try to wrap each your request in Runnable to emulate multi-threaded
> pressure on ZK. Your code is linear and mine is not, it's concurrent.
>
> Thanks....
>
>
>
>
> 2014-12-12 22:28 GMT+03:00 Stack <stack@duboce.net>:
> >
> > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted your
> > code adding in a put for each cycle.  I ran loop 1000 times and no
> > complaint from zk.
> >
> > Tell me more (Is servlet doing single-threaded model?  A single
> > Configuration is being used or new ones are being created per servlet
> > invocation?
> >
> > Below is code and output.
> >
> > (For better perf, cache the connection)
> > St.Ack
> >
> >
> >   1 package org.apache.hadoop.hbase;
> >   2
> >   3 import java.io.IOException;
> >   4
> >   5 import org.apache.hadoop.conf.Configuration;
> >   6 import org.apache.hadoop.hbase.client.HConnection;
> >   7 import org.apache.hadoop.hbase.client.HConnectionManager;
> >   8 import org.apache.hadoop.hbase.client.HTableInterface;
> >   9 import org.apache.hadoop.hbase.client.Put;
> >  10 import org.apache.hadoop.hbase.util.Bytes;
> >  11
> >  12 public class TestConnnection {
> >  13   public static void main(String[] args) throws IOException {
> >  14     Configuration config = HBaseConfiguration.create();
> >  15     for (int i = 0; i < 1000; i++) {
> >  16       HConnection connection =
> > HConnectionManager.createConnection(config);
> >  17       HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> >  18       byte [] cf = Bytes.toBytes("t");
> >  19       try {
> >  20         byte [] bytes = Bytes.toBytes(i);
> >  21         Put p = new Put(bytes);
> >  22         p.add(cf, cf, bytes);
> >  23         table.put(p);
> >  24       } finally {
> >  25         table.close();
> >  26         connection.close();
> >  27       }
> >  28       System.out.println("i=" + i);
> >  29     }
> >  30   }
> >  31 }
> >
> >
> > ....
> > 2014-12-12 11:26:10,397 INFO  [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,397 INFO  [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,398 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,398 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,401 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,406 INFO  [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083a
> > 2014-12-12 11:26:10,407 INFO  [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083a closed
> > 2014-12-12 11:26:10,407 INFO  [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=997
> > 2014-12-12 11:26:10,511 INFO  [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,511 INFO  [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,512 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using
> SASL
> > (unknown error)
> > 2014-12-12 11:26:10,512 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > 2014-12-12 11:26:10,515 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, negotiated
> > timeout = 40000
> > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,521 INFO  [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083b
> > 2014-12-12 11:26:10,521 INFO  [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083b closed
> > 2014-12-12 11:26:10,521 INFO  [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=998
> > 2014-12-12 11:26:10,627 INFO  [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,627 INFO  [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,628 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,629 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,631 INFO  [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,638 INFO  [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083c
> > 2014-12-12 11:26:10,639 INFO  [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083c closed
> > 2014-12-12 11:26:10,639 INFO  [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=999
> >
> >
> >
> >
> > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> serega.sheypak@gmail.com
> > >
> > wrote:
> >
> > > Hi, I'm using CDH 5.2, 0.98
> > > I don't know how to use it correctly. I've just used this sample:
> > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > >
> > >  HConnection connection = HConnectionManager.createConnection(config);
> > >  HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> > >  try {
> > >    // Use the table as needed, for a single operation and a single
> thread
> > >  } finally {
> > >    table.close();
> > >    connection.close();
> > >  }
> > >
> > >
> > > Functional testing OK, performance fails :((((
> > >
> > >
> > >
> > > 2014-12-12 19:32 GMT+03:00 Stack <stack@duboce.net>:
> > > >
> > > > Does zk count go up on each request to the servlet? Which version of
> > > hbase
> > > > so can try on this end?  Do you have some client-side log? Better if
> > you
> > > > cache the connection rather than make it each time since setup is
> > costly
> > > > but lets fix first problem first.
> > > > St.Ack
> > > >
> > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > serega.sheypak@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > Looks like it's leaking, all my zookeepers complains that there is
> > too
> > > > many
> > > > > connections from servlet hosts.
> > > > > Typical line from lZK log:
> > > > >
> > > > > oo many connections from /my.tomcat.server.com  - max is 60
> > > > >
> > > > >
> > > > > Here is code sample
> > > > >
> > > > > public class BaseConnection {
> > > > >
> > > > >     private static final Logger LOG =
> > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > >
> > > > >     protected void close(HConnection connection){
> > > > >         try{
> > > > >             if(connection == null){
> > > > >                 return;
> > > > >             }
> > > > >             connection.close();
> > > > >         }
> > > > >         catch (Exception e){
> > > > >             LOG.warn("Error while closing HConnection", e);
> > > > >         }
> > > > >     }
> > > > >
> > > > >     protected void close(HTableInterface hTable){
> > > > >         try{
> > > > >             if(hTable == null){
> > > > >                 return;
> > > > >             }
> > > > >             hTable.close();
> > > > >         }
> > > > >         catch (Exception e){
> > > > >             LOG.warn("Error while closing HTable", e);
> > > > >         }
> > > > >     }
> > > > > }
> > > > >
> > > > > sample PUT code from subclass:
> > > > >
> > > > > public void put(List<MyBean> entries){
> > > > >     HConnection hConnection = null;
> > > > >     HTableInterface hTable = null;
> > > > >     try {
> > > > >         List<Put> puts = new ArrayList<Put>(entries.size());
> > > > >         for(MyBean myBean : entries){
> > > > >             puts.add(new MyBeanSerDe().createPut(myBean));
> > > > >         }
> > > > >         hConnection =
> > > HConnectionManager.createConnection(configuration);
> > > > >         hTable = hConnection.getTable(NAME_B);
> > > > >         hTable.put(puts);
> > > > >
> > > > >     }catch (Exception e){
> > > > >         LOG.error("Error while doing bulk put", e);
> > > > >     }finally{
> > > > >         close(hTable);
> > > > >         close(hConnection);
> > > > >     }
> > > > > }
> > > > >
> > > >
> > >
> >
>

--001a11c3b40c30dafa050a0e945f--