hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serega Sheypak <serega.shey...@gmail.com>
Subject Re: HConnectionManager leaks with zookeeper conection oo many connections from /my.tomcat.server.com - max is 60
Date Mon, 15 Dec 2014 19:48:20 GMT
was: 200-300 ms per request
now: <80 ms request

request=full trip from servlet to HBase and back to response.

2014-12-15 22:40 GMT+03:00 lars hofhansl <larsh@apache.org>:
>
> Excellent! Should be quite a bit faster too.
> -- Lars
>       From: Serega Sheypak <serega.sheypak@gmail.com>
>  To: user <user@hbase.apache.org>
> Cc: lars hofhansl <larsh@apache.org>
>  Sent: Monday, December 15, 2014 5:57 AM
>  Subject: Re: HConnectionManager leaks with zookeeper conection oo many
> connections from /my.tomcat.server.com - max is 60
>
> Hi, the problem is gone.
> I did what you say :)
> Thanks!
>
>
>
> 2014-12-13 22:38 GMT+03:00 Serega Sheypak <serega.sheypak@gmail.com>:
>
> > Great, I'll refactor the code. and report back
> >
> > 2014-12-13 22:36 GMT+03:00 Stack <stack@duboce.net>:
> >>
> >> On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak <
> >> serega.sheypak@gmail.com>
> >> wrote:
> >> >
> >> > So the idea is
> >> > 1. instantiate HConnection using HConnectionManager once
> >> > 2. Create HTable instance for each Servlet.doPost and close after
> >> operation
> >> > is done.
> >> > Is that correct?
> >> >
> >>
> >>
> >> Yes.
> >>
> >>
> >>
> >> >
> >> > Do region locations cached in this case?
> >> > Are ZK connections to region/ZK reused?
> >> >
> >>
> >> Yes
> >>
> >>
> >> > Can I "harm" put/get data because of Servlet concurrency? Each
> >> connection
> >> > could be used in many threads fro different HTable instances for the
> >> same
> >> > HBase table?
> >> >
> >> >
> >> Its a bug if the above has issues.
> >>
> >> St.Ack
> >>
> >>
> >>
> >> >
> >> > 2014-12-13 22:21 GMT+03:00 lars hofhansl <larsh@apache.org>:
> >> > >
> >> > > Note also that the createConnection part is somewhat expensive
> >> (creates a
> >> > > new thread pool for use with Puts, also does a ZK lookup, etc).If
> >> > possible
> >> > > create the connection ahead of time and only get/close an HTable per
> >> > > request/thread.
> >> > > -- Lars
> >> > >      From: Serega Sheypak <serega.sheypak@gmail.com>
> >> > >  To: user <user@hbase.apache.org>
> >> > >  Sent: Friday, December 12, 2014 11:45 AM
> >> > >  Subject: Re: HConnectionManager leaks with zookeeper conection oo
> >> many
> >> > > connections from /my.tomcat.server.com - max is 60
> >> > >
> >> > > i have 10K doPost/doGet requests per second.
> >> > > Servlet is NOT single-threaded. each doPost/doGet
> >> > > invokes these lines (encapsulated in DAO):
> >> > >
> >> > >  16      HConnection connection =
> >> > > HConnectionManager.createConnection(config);
> >> > >  17      HTableInterface table =
> >> > > connection.getTable(TableName.valueOf("table1"));
> >> > >
> >> > > and
> >> > >
> >> > > 24      } finally {
> >> > >  25        table.close();
> >> > >  26        connection.close();
> >> > >  27      }
> >> > >
> >> > > I assumed that this static construction
> >> > >
> >> > > 16      HConnection connection =
> >> > > HConnectionManager.createConnection(config);
> >> > >
> >> > > correctly handles multi-threaded access somewhere deep inside.
> >> > > Right now I don't understand what do I do wrong.
> >> > >
> >> > > Try to wrap each your request in Runnable to emulate multi-threaded
> >> > > pressure on ZK. Your code is linear and mine is not, it's
> concurrent.
> >> > >
> >> > > Thanks....
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > 2014-12-12 22:28 GMT+03:00 Stack <stack@duboce.net>:
> >> > > >
> >> > > > I cannot reproduce. I stood up a cdh5.2 server and then
> copy/pasted
> >> > your
> >> > > > code adding in a put for each cycle.  I ran loop 1000 times and
no
> >> > > > complaint from zk.
> >> > > >
> >> > > > Tell me more (Is servlet doing single-threaded model?  A single
> >> > > > Configuration is being used or new ones are being created per
> >> servlet
> >> > > > invocation?
> >> > > >
> >> > > > Below is code and output.
> >> > > >
> >> > > > (For better perf, cache the connection)
> >> > > > St.Ack
> >> > > >
> >> > > >
> >> > > >  1 package org.apache.hadoop.hbase;
> >> > > >  2
> >> > > >  3 import java.io.IOException;
> >> > > >  4
> >> > > >  5 import org.apache.hadoop.conf.Configuration;
> >> > > >  6 import org.apache.hadoop.hbase.client.HConnection;
> >> > > >  7 import org.apache.hadoop.hbase.client.HConnectionManager;
> >> > > >  8 import org.apache.hadoop.hbase.client.HTableInterface;
> >> > > >  9 import org.apache.hadoop.hbase.client.Put;
> >> > > >  10 import org.apache.hadoop.hbase.util.Bytes;
> >> > > >  11
> >> > > >  12 public class TestConnnection {
> >> > > >  13  public static void main(String[] args) throws IOException
{
> >> > > >  14    Configuration config = HBaseConfiguration.create();
> >> > > >  15    for (int i = 0; i < 1000; i++) {
> >> > > >  16      HConnection connection =
> >> > > > HConnectionManager.createConnection(config);
> >> > > >  17      HTableInterface table =
> >> > > > connection.getTable(TableName.valueOf("table1"));
> >> > > >  18      byte [] cf = Bytes.toBytes("t");
> >> > > >  19      try {
> >> > > >  20        byte [] bytes = Bytes.toBytes(i);
> >> > > >  21        Put p = new Put(bytes);
> >> > > >  22        p.add(cf, cf, bytes);
> >> > > >  23        table.put(p);
> >> > > >  24      } finally {
> >> > > >  25        table.close();
> >> > > >  26        connection.close();
> >> > > >  27      }
> >> > > >  28      System.out.println("i=" + i);
> >> > > >  29    }
> >> > > >  30  }
> >> > > >  31 }
> >> > > >
> >> > > >
> >> > > > ....
> >> > > > 2014-12-12 11:26:10,397 INFO  [main]
> zookeeper.RecoverableZooKeeper:
> >> > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> >> > > > ensemble=localhost:2181
> >> > > > 2014-12-12 11:26:10,397 INFO  [main] zookeeper.ZooKeeper:
> Initiating
> >> > > client
> >> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> >> > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181,
> >> baseZNode=/hbase
> >> > > > 2014-12-12 11:26:10,398 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Opening socket connection to server
> localhost/
> >> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
> >> (unknown
> >> > > > error)
> >> > > > 2014-12-12 11:26:10,398 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
> >> > > > 127.0.0.1:2181, initiating session
> >> > > > 2014-12-12 11:26:10,401 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Session establishment complete on server
> >> > localhost/
> >> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated
> timeout =
> >> > > 40000
> >> > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
> >> > Finished
> >> > > > with small scan at {ENCODED => 1588230740, NAME =>
> 'hbase:meta,,1',
> >> > > > STARTKEY => '', ENDKEY => ''}
> >> > > > 2014-12-12 11:26:10,406 INFO  [main]
> >> > > > client.HConnectionManager$HConnectionImplementation: Closing
> >> zookeeper
> >> > > > sessionid=0x14a3fe0664b083a
> >> > > > 2014-12-12 11:26:10,407 INFO  [main] zookeeper.ZooKeeper: Session:
> >> > > > 0x14a3fe0664b083a closed
> >> > > > 2014-12-12 11:26:10,407 INFO  [main-EventThread]
> >> zookeeper.ClientCnxn:
> >> > > > EventThread shut down
> >> > > > i=997
> >> > > > 2014-12-12 11:26:10,511 INFO  [main]
> zookeeper.RecoverableZooKeeper:
> >> > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> >> > > > ensemble=localhost:2181
> >> > > > 2014-12-12 11:26:10,511 INFO  [main] zookeeper.ZooKeeper:
> Initiating
> >> > > client
> >> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> >> > > > watcher=hconnection-0x22a7106, quorum=localhost:2181,
> >> baseZNode=/hbase
> >> > > > 2014-12-12 11:26:10,512 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Opening socket connection to server
> >> > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate
> >> using
> >> > > SASL
> >> > > > (unknown error)
> >> > > > 2014-12-12 11:26:10,512 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Socket connection established to
> >> > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> >> > > > 2014-12-12 11:26:10,515 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Session establishment complete on server
> >> > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
> >> > negotiated
> >> > > > timeout = 40000
> >> > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
> >> > Finished
> >> > > > with small scan at {ENCODED => 1588230740, NAME =>
> 'hbase:meta,,1',
> >> > > > STARTKEY => '', ENDKEY => ''}
> >> > > > 2014-12-12 11:26:10,521 INFO  [main]
> >> > > > client.HConnectionManager$HConnectionImplementation: Closing
> >> zookeeper
> >> > > > sessionid=0x14a3fe0664b083b
> >> > > > 2014-12-12 11:26:10,521 INFO  [main] zookeeper.ZooKeeper: Session:
> >> > > > 0x14a3fe0664b083b closed
> >> > > > 2014-12-12 11:26:10,521 INFO  [main-EventThread]
> >> zookeeper.ClientCnxn:
> >> > > > EventThread shut down
> >> > > > i=998
> >> > > > 2014-12-12 11:26:10,627 INFO  [main]
> zookeeper.RecoverableZooKeeper:
> >> > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> >> > > > ensemble=localhost:2181
> >> > > > 2014-12-12 11:26:10,627 INFO  [main] zookeeper.ZooKeeper:
> Initiating
> >> > > client
> >> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> >> > > > watcher=hconnection-0x3822f407, quorum=localhost:2181,
> >> baseZNode=/hbase
> >> > > > 2014-12-12 11:26:10,628 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Opening socket connection to server
> localhost/
> >> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
> >> (unknown
> >> > > > error)
> >> > > > 2014-12-12 11:26:10,629 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
> >> > > > 127.0.0.1:2181, initiating session
> >> > > > 2014-12-12 11:26:10,631 INFO  [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Session establishment complete on server
> >> > localhost/
> >> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated
> timeout =
> >> > > 40000
> >> > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
> >> > Finished
> >> > > > with small scan at {ENCODED => 1588230740, NAME =>
> 'hbase:meta,,1',
> >> > > > STARTKEY => '', ENDKEY => ''}
> >> > > > 2014-12-12 11:26:10,638 INFO  [main]
> >> > > > client.HConnectionManager$HConnectionImplementation: Closing
> >> zookeeper
> >> > > > sessionid=0x14a3fe0664b083c
> >> > > > 2014-12-12 11:26:10,639 INFO  [main] zookeeper.ZooKeeper: Session:
> >> > > > 0x14a3fe0664b083c closed
> >> > > > 2014-12-12 11:26:10,639 INFO  [main-EventThread]
> >> zookeeper.ClientCnxn:
> >> > > > EventThread shut down
> >> > > > i=999
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> >> > > serega.sheypak@gmail.com
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi, I'm using CDH 5.2, 0.98
> >> > > > > I don't know how to use it correctly. I've just used this
> sample:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> >> > > > >
> >> > > > >  HConnection connection =
> >> > HConnectionManager.createConnection(config);
> >> > > > >  HTableInterface table =
> >> > > > connection.getTable(TableName.valueOf("table1"));
> >> > > > >  try {
> >> > > > >    // Use the table as needed, for a single operation and
a
> single
> >> > > thread
> >> > > > >  } finally {
> >> > > > >    table.close();
> >> > > > >    connection.close();
> >> > > > >  }
> >> > > > >
> >> > > > >
> >> > > > > Functional testing OK, performance fails :((((
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > 2014-12-12 19:32 GMT+03:00 Stack <stack@duboce.net>:
> >> > > > > >
> >> > > > > > Does zk count go up on each request to the servlet?
Which
> >> version
> >> > of
> >> > > > > hbase
> >> > > > > > so can try on this end?  Do you have some client-side
log?
> >> Better
> >> > if
> >> > > > you
> >> > > > > > cache the connection rather than make it each time
since setup
> >> is
> >> > > > costly
> >> > > > > > but lets fix first problem first.
> >> > > > > > St.Ack
> >> > > > > >
> >> > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> >> > > > > serega.sheypak@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi, I'm using HConnectionManager from java servlet
> >> > > > > > > Looks like it's leaking, all my zookeepers complains
that
> >> there
> >> > is
> >> > > > too
> >> > > > > > many
> >> > > > > > > connections from servlet hosts.
> >> > > > > > > Typical line from lZK log:
> >> > > > > > >
> >> > > > > > > oo many connections from /my.tomcat.server.com
 - max is 60
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Here is code sample
> >> > > > > > >
> >> > > > > > > public class BaseConnection {
> >> > > > > > >
> >> > > > > > >    private static final Logger LOG =
> >> > > > > > > LoggerFactory.getLogger(BaseConnection.class);
> >> > > > > > >
> >> > > > > > >    protected void close(HConnection connection){
> >> > > > > > >        try{
> >> > > > > > >            if(connection == null){
> >> > > > > > >                return;
> >> > > > > > >            }
> >> > > > > > >            connection.close();
> >> > > > > > >        }
> >> > > > > > >        catch (Exception e){
> >> > > > > > >            LOG.warn("Error while closing HConnection",
e);
> >> > > > > > >        }
> >> > > > > > >    }
> >> > > > > > >
> >> > > > > > >    protected void close(HTableInterface hTable){
> >> > > > > > >        try{
> >> > > > > > >            if(hTable == null){
> >> > > > > > >                return;
> >> > > > > > >            }
> >> > > > > > >            hTable.close();
> >> > > > > > >        }
> >> > > > > > >        catch (Exception e){
> >> > > > > > >            LOG.warn("Error while closing HTable",
e);
> >> > > > > > >        }
> >> > > > > > >    }
> >> > > > > > > }
> >> > > > > > >
> >> > > > > > > sample PUT code from subclass:
> >> > > > > > >
> >> > > > > > > public void put(List<MyBean> entries){
> >> > > > > > >    HConnection hConnection = null;
> >> > > > > > >    HTableInterface hTable = null;
> >> > > > > > >    try {
> >> > > > > > >        List<Put> puts = new ArrayList<Put>(entries.size());
> >> > > > > > >        for(MyBean myBean : entries){
> >> > > > > > >            puts.add(new MyBeanSerDe().createPut(myBean));
> >> > > > > > >        }
> >> > > > > > >        hConnection =
> >> > > > > HConnectionManager.createConnection(configuration);
> >> > > > > > >        hTable = hConnection.getTable(NAME_B);
> >> > > > > > >        hTable.put(puts);
> >> > > > > > >
> >> > > > > > >    }catch (Exception e){
> >> > > > > > >        LOG.error("Error while doing bulk put",
e);
> >> > > > > > >    }finally{
> >> > > > > > >        close(hTable);
> >> > > > > > >        close(hConnection);
> >> > > > > > >    }
> >> > > > > > > }
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message