hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Test fail randomly
Date Mon, 12 Oct 2009 16:03:36 GMT
What I wrote after that is what you must do to make sure you close all
connections. Currently there's a limit of one ZK connection per client
JVM so you don't want to close it if you are using another HTable,
this is why you have to use the public methods of HConnectionManager
if you really want to make sure that you close all connections.
Otherwise it'll close when the JVM is gone.

J-D

On Mon, Oct 12, 2009 at 11:56 AM, Something Something
<luckyguy2050@yahoo.com> wrote:
>>> Closing the HTable itself doesn't close the connections...
>
> Interesting... I must confess I didn't know this.
>
> So in addition to closing the table is there anything else we need to do to close connection?
 Or is the opening/closing of connections done automatically by HBase?
>
> Please let me know.  Thanks.
>
>
>
> --- On Fri, 10/9/09, Jean-Daniel Cryans <jdcryans@apache.org> wrote:
>
> From: Jean-Daniel Cryans <jdcryans@apache.org>
> Subject: Re: Test fail randomly
> To: hbase-user@hadoop.apache.org
> Date: Friday, October 9, 2009, 11:16 AM
>
> Julien,
>
> Closing the HTable itself doesn't close the connections, it just
> flushes anything that may still be in the write buffer (isn't your
> case). To close all connections, call
> HConnectionManager.deleteAllConnections(true).
>
> A client connection to Zookeeper is made by the Master, the
> RegionServer, the HBase clients and in version 0.20.0 by the
> RegionHistorian. Some of those even carry more than 1 connection so 5
> connections is too low.
>
> WRT the "Exception 2" thing, I tried and got it some times. This is an
> expected behavior, at least it's by design. If you delete and put too
> fast, you may be able to have both operations at the same timestamp in
> which case the delete wins... so your row appears to be deleted. If
> you need a higher level of precision, you have to provide your own
> timestamps or put a sleep of some milliseconds after the deletes.
>
> Also, in your code you delete 10 times the same row. Maybe you already
> know that, but deleting the row key deletes the whole row with all the
> data you inserted so you only need to delete once.
>
> J-D
>
> On Fri, Oct 9, 2009 at 1:06 PM, Julien Ruchaud
> <julien.ruchaud@codelutin.com> wrote:
>> I am continue test, I have set
>> base.zookeeper.property.maxClientCnxns = 5 and close htable each get,
>> put and delete, I have got the same error. is it normal ?
>>
>> Julien Ruchaud
>>
>> Le Fri, 9 Oct 2009 17:48:37 +0200,
>> Julien Ruchaud <julien.ruchaud@codelutin.com> a écrit :
>>
>>> J-D,
>>>
>>> Yes, it is the only test. I am downloading here
>>> http://mir2.ovh.net/ftp.apache.org/dist/hadoop/hbase/ the 0.20.0
>>> release.
>>>
>>> Julien
>>>
>>> Le Fri, 9 Oct 2009 11:47:45 -0400,
>>> Jean-Daniel Cryans <jdcryans@apache.org> a écrit :
>>>
>>> > Julien,
>>> >
>>> > Is it the only test you are running? If so, you shouldn't have to
>>> > close after each operation but only at the end of the test. Are you
>>> > sure this is the official 0.20.0 release?
>>> >
>>> > J-D
>>> >
>>> > On Fri, Oct 9, 2009 at 11:35 AM, Julien Ruchaud
>>> > <julien.ruchaud@codelutin.com> wrote:
>>> > > Hi,
>>> > >
>>> > > If hbase.zookeeper.property.maxClientCnxns increase to 300, the
>>> > > test pass. So I test the second solution to close connection each
>>> > > put, get and delete, the test pass too. How do you know when to
>>> > > close connection ? each get, delete or put ? is it better to
>>> > > instanciate a new HTable each time ?
>>> > >
>>> > > Thanks
>>> > >
>>> > > Julien Ruchaud
>>> > >
>>> > > Le Fri, 9 Oct 2009 11:07:55 -0400,
>>> > > Jean-Daniel Cryans <jdcryans@apache.org> a écrit :
>>> > >
>>> > >> Julien,
>>> > >>
>>> > >> This is a Zookeeper error that can be related to the number of
>>> > >> connections a ZK server can handle. To verify, set
>>> > >> hbase.zookeeper.property.maxClientCnxns to something a lot bigger
>>> > >> than the default 30 and restart HBase.
>>> > >>
>>> > >> This could happen because  you run a lot of unit tests
>>> > >> sequentially without closing the connections.
>>> > >>
>>> > >> J-D
>>> > >>
>>> > >> On Fri, Oct 9, 2009 at 10:53 AM, Julien Ruchaud
>>> > >> <julien.ruchaud@codelutin.com> wrote:
>>> > >> > Hi,
>>> > >> >
>>> > >> > I am a newbie on hbase. Currently I am testing hbase 0.20.0,
>>> > >> > but I don't understand why my test don't run correctly
>>> > >> > sometimes :
>>> > >> >
>>> > >> >    public void testLoop() throws Exception {
>>> > >> >        HTable hTable = new HTable(new HBaseConfiguration(),
>>> > >> > Bytes.toBytes("Table")); byte[] rowId = Bytes.toBytes("Test");
>>> > >> >
>>> > >> >        for (int j = 0; j < 10; j++) {
>>> > >> >            Put put = new Put(rowId);
>>> > >> >            put.add(Bytes.toBytes("data"), Bytes.toBytes("1:"
+
>>> > >> > j), Bytes.toBytes(j)); hTable.put(put);
>>> > >> >        }
>>> > >> >
>>> > >> >        for (int j = 0; j < 10; j++) {
>>> > >> >            Get get = new Get(rowId);
>>> > >> >            Result row = hTable.get(get);
>>> > >> >            if(row.getValue(Bytes.toBytes("data"),
>>> > >> > Bytes.toBytes("1:" + j)) == null) { throw new Exception("Error
>>> > >> > 1"); }
>>> > >> >        }
>>> > >> >
>>> > >> >        for (int j = 0; j < 10; j++) {
>>> > >> >                Delete delete = new Delete(rowId);
>>> > >> >                hTable.delete(delete);
>>> > >> >        }
>>> > >> >
>>> > >> >        for (int j = 0; j < 10; j++) {
>>> > >> >            Put put = new Put(rowId);
>>> > >> >            put.add(Bytes.toBytes("data"), Bytes.toBytes("2:"
+
>>> > >> > j), Bytes.toBytes(j)); hTable.put(put);
>>> > >> >        }
>>> > >> >
>>> > >> >        for (int j = 0; j < 10; j++) {
>>> > >> >            Get get = new Get(rowId);
>>> > >> >            Result row = hTable.get(get);
>>> > >> >            if(row.getValue(Bytes.toBytes("data"),
>>> > >> > Bytes.toBytes("2:" + j)) == null) { throw new Exception("Error
>>> > >> > 2"); }
>>> > >> >        }
>>> > >> >    }
>>> > >> >
>>> > >> > In fact I have got exception "Error 2" with a exception in
>>> > >> > hbase log :
>>> > >> >
>>> > >> > 2009-10-09 16:39:34,829 WARN
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn: Exception causing
>>> > >> > close of session 0x124393fc5390046 due to java.io.IOException:
>>> > >> > Read error 2009-10-09 16:39:34,829 DEBUG
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn: IOException stack
>>> > >> > trace java.io.IOException: Read error at
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:366)
>>> > >> > at
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:220)
>>> > >> > 2009-10-09 16:39:34,830 INFO
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn: closing
>>> > >> > session:0x124393fc5390046 NIOServerCnxn:
>>> > >> > java.nio.channels.SocketChannel[connected local=/127.0.0.1:2181
>>> > >> > remote=/127.0.0.1:48870] 2009-10-09 16:39:34,830 DEBUG
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn: ignoring exception
>>> > >> > during input shutdown java.net.SocketException: Transport
>>> > >> > endpoint is not connected at
>>> > >> > sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at
>>> > >> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>> > >> > at
>>> > >> > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>> > >> > at
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:841)
>>> > >> > at
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:492)
>>> > >> > at
>>> > >> > org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:220)
>>> > >> >
>>> > >> > Thanks in advance for your input
>>> > >> >
>>> > >> > Julien Ruchaud
>>> > >> >
>>> > >
>>
>
>
>
>

Mime
View raw message