Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0A721840A for ; Tue, 12 Jan 2016 09:50:02 +0000 (UTC) Received: (qmail 65328 invoked by uid 500); 12 Jan 2016 09:50:01 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 65216 invoked by uid 500); 12 Jan 2016 09:50:00 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 65193 invoked by uid 99); 12 Jan 2016 09:50:00 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2016 09:50:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 57D35C395B for ; Tue, 12 Jan 2016 09:49:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.099 X-Spam-Level: X-Spam-Status: No, score=-0.099 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, MIME_QP_LONG_LINE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id drwMVxJaW3IF for ; Tue, 12 Jan 2016 09:49:51 +0000 (UTC) Received: from mail-pf0-f182.google.com (mail-pf0-f182.google.com [209.85.192.182]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 119C0439C4 for ; Tue, 12 Jan 2016 09:49:51 +0000 (UTC) Received: by mail-pf0-f182.google.com with SMTP id e65so60547398pfe.0 for ; Tue, 12 Jan 2016 01:49:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:mime-version:subject :message-id:date:references:in-reply-to:to; bh=x+vzC2YWexEm518an56CzVwcvWw3/IKXgQQjoZycEf0=; b=sSfxTwsx4iNMefP+qnz6CT7pli9ddQspoIXobCkc1Opjzth5s6ZVzDidRUn+5G/h6Y fMnfSHsIO4M8xmVoOwJKqPExfI+iXqVbMUkoQPppCkkBBKoeGQog90NUaScVw1z0hz8/ VPmY1Edbodjq9h+6PvS2s6tiAupKYCduQF13ey51Al4bdqbjQJrhoIq/k0u6eLlhYCjZ mXvSnb81P8oMTJEUq18B2xEBx7hzsl5cVlpdGnhXVNQrLSpg8DVji6bKMjl8SVSoqf9B BY6PWcVZFoQCdrRAzcNZS8cmpIE4sXgRpZLG4tQEbm5Utq5bU4a7HdGzI4bXsHA8j902 e9EA== X-Received: by 10.98.16.139 with SMTP id 11mr33332227pfq.128.1452592183797; Tue, 12 Jan 2016 01:49:43 -0800 (PST) Received: from [192.168.0.14] (c-50-150-102-33.hsd1.ca.comcast.net. [50.150.102.33]) by smtp.gmail.com with ESMTPSA id q8sm189748863pap.45.2016.01.12.01.49.43 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 12 Jan 2016 01:49:43 -0800 (PST) From: Ted Yu Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: Hbase thrift server issues Message-Id: <6E08D815-EADC-438A-8011-5ADF34E80EBA@gmail.com> Date: Tue, 12 Jan 2016 01:49:42 -0800 References: In-Reply-To: To: user@hbase.apache.org X-Mailer: iPhone Mail (13A344) You can get the source code from 0.98 branch of: https://github.com/apache/hbase 0.98.17 hasn't been released yet.=20 RC would come out this month.=20 If you cannot wait, you can build locally and deploy.=20 Cheers > On Jan 11, 2016, at 11:30 PM, Rajeshkumar J w= rote: >=20 > Hi Ted, >=20 > As mentioned in that HBASE-14926 it is fixed in HBase 0.98.17. I have > googled it for both its source code and binary but I didn't find one. can > you guide me where I can find Hbase 0.98.17 source code or binary package >=20 > Thanks >=20 >> On Tue, Jan 12, 2016 at 12:57 AM, Ted Yu wrote: >>=20 >> I took a look at the stack trace you posted. >>=20 >> Some thrift worker was in the state shown in HBASE-14926 >>=20 >> I wonder if the fix for HBASE-14926 would help in your case. >>=20 >> Cheers >>=20 >> On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J < >> rajeshkumarit8292@gmail.com> >> wrote: >>=20 >>> Hi Ted, >>>=20 >>> Plz ignore my previous mail as the log is taken when hbase is working >>> fine. Please find the logs below when Hbase didn't return any records >>>=20 >>> http://pastebin.com/APYjiGSP >>>=20 >>> Thanks >>>=20 >>>> On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu wrote: >>>>=20 >>>> When you search for 'java jstack tutorial' using google, you would >> find a >>>> lot of tutorials. >>>>=20 >>>> You can use 'ps aux' command to search for the following class: >>>> org.apache.hadoop.hbase.thrift.ThriftServer >>>>=20 >>>> Apply jstack on the corresponding process id. >>>>=20 >>>> Cheers >>>>=20 >>>> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J < >>>> rajeshkumarit8292@gmail.com> >>>> wrote: >>>>=20 >>>>> Hi Ted, >>>>>=20 >>>>> I don't know how how to take stacktrace using jstack. But i tried >>>> using >>>>> some commands but it fails. can you help me in this? >>>>>=20 >>>>> Thanks >>>>>=20 >>>>> On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu >> wrote: >>>>>=20 >>>>>> What I meant was to capture stack trace of the thrift server (using >>>>> jstack >>>>>> command) when your query is running. >>>>>>=20 >>>>>> This would give us some more clue as to the state thrift server was >>> in >>>>> when >>>>>> the problem happened. >>>>>>=20 >>>>>> Cheers >>>>>>=20 >>>>>> On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J < >>>>>> rajeshkumarit8292@gmail.com> >>>>>> wrote: >>>>>>=20 >>>>>>> Hi Ted, >>>>>>>=20 >>>>>>> This is what I found in the Hbase thrift server logs >>>>>>>=20 >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:java.library.path=3D;C:\Hadoop\bin >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:java.io.tmpdir=3DC:\Windows\TEMP\ >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:java.compiler=3D >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:os.name=3DWindows Server 2012 >>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:os.arch=3Damd64 >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:os.version=3D6.2 >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:user.name=3Dxxxx >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:user.home=3DC:\ >>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>> Client >>>>>>> environment:user.dir=3DC:\HBase\bin >>>>>>> 2016-01-09 06:27:43,142 INFO [thrift-worker-0] >>> zookeeper.ZooKeeper: >>>>>>> Initiating client connection, >>>>>>> connectString=3Dhostname:2181,hostname:2181,hostname:2181 >>>>>>> sessionTimeout=3D90000 watcher=3Dhconnection-0x45e35460x0, >>>>>>> quorum=3Dhostname:2181,hostname:2181,hostname:2181, >> baseZNode=3D/hbase >>>>>>> 2016-01-09 06:27:43,339 INFO >>>>> [thrift-worker-0-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Opening socket connection to server >>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using >>> SASL >>>>>>> (unknown error) >>>>>>> 2016-01-09 06:27:43,342 INFO >>>>> [thrift-worker-0-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Socket connection established to >>>>>>> hostname/xx.xx.x.xx:2181, initiating session >>>>>>> 2016-01-09 06:27:43,365 INFO >>>>> [thrift-worker-0-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Session establishment complete on server >>>>>>> hostname/xx.xx.x.xx:2181, sessionid =3D 0x1522613893e0004, >> negotiated >>>>>> timeout >>>>>>> =3D 40000 >>>>>>> 2016-01-09 07:14:10,607 INFO >>>>>>> [hostname,60000,1452337815006_splitLogManager__ChoreService_1] >>>>>>> master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager >>> Timeout >>>>>>> Monitor missed its start time >>>>>>> 2016-01-09 07:16:43,511 INFO >>>>>>> [B.defaultRpcServer.handler=3D8,queue=3D2,port=3D60000] >> master.HMaster: >>>>>>> Client=3DAdministrator/null create 'chk2', {NAME =3D> 'fa', >>>>>> DATA_BLOCK_ENCODING >>>>>>> =3D> 'NONE', BLOOMFILTER =3D> 'ROW', REPLICATION_SCOPE =3D> '0', >> VERSIONS >>>> =3D> >>>>>> '1', >>>>>>> COMPRESSION =3D> 'NONE', MIN_VERSIONS =3D> '0', TTL =3D> 'FOREVER', >>>>>>> KEEP_DELETED_CELLS =3D> 'FALSE', BLOCKSIZE =3D> '65536', IN_MEMORY =3D= > >>>>>> 'false', >>>>>>> BLOCKCACHE =3D> 'true'} >>>>>>> 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1] >>>>>>> procedure2.ProcedureExecutor: Rolledback procedure >>>> CreateTableProcedure >>>>>>> (table=3Dchk2) user=3DAdministrator (auth:SIMPLE) id=3D2 >> state=3DROLLEDBACK >>>>>>> exec-time=3D462msec >>>>> exception=3Dorg.apache.hadoop.hbase.TableExistsException: >>>>>>> chk2 >>>>>>> 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1] >>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>> zookeeper >>>>>>> sessionid=3D0x1522613893e0004 >>>>>>> 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1] >>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed >>>>>>> 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread] >>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>> 2016-01-09 18:40:29,469 INFO [thrift-worker-1] >>>>>>> zookeeper.RecoverableZooKeeper: Process >>>>> identifier=3Dhconnection-0x591287f8 >>>>>>> connecting to ZooKeeper >>>>>> ensemble=3Dhostname:2181,hostname:2181,hostname:2181 >>>>>>> 2016-01-09 18:40:29,482 INFO [thrift-worker-1] >>> zookeeper.ZooKeeper: >>>>>>> Initiating client connection, >>>>>>> connectString=3Dhostname:2181,hostname:2181,hostname:2181 >>>>>>> sessionTimeout=3D90000 watcher=3Dhconnection-0x591287f80x0, >>>>>>> quorum=3Dhostname:2181,hostname:2181,hostname:2181, >> baseZNode=3D/hbase >>>>>>> 2016-01-09 18:40:29,553 INFO >>>>> [thrift-worker-1-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Opening socket connection to server >>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using >>> SASL >>>>>>> (unknown error) >>>>>>> 2016-01-09 18:40:29,559 INFO >>>>> [thrift-worker-1-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Socket connection established to >>>>>>> hostname/xx.xx.x.xx:2181, initiating session >>>>>>> 2016-01-09 18:40:29,569 INFO >>>>> [thrift-worker-1-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Session establishment complete on server >>>>>>> hostname/xx.xx.x.xx:2181, sessionid =3D 0x35226137e300008, >> negotiated >>>>>> timeout >>>>>>> =3D 40000 >>>>>>> 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1] >>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>> zookeeper >>>>>>> sessionid=3D0x35226137e300008 >>>>>>> 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1] >>>>>>> zookeeper.ZooKeeper: Session: 0x35226137e300008 closed >>>>>>> 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread] >>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>> 2016-01-09 20:05:42,469 INFO [thrift-worker-13] >>>>>>> zookeeper.RecoverableZooKeeper: Process >>>>> identifier=3Dhconnection-0x6305dbeb >>>>>>> connecting to ZooKeeper >>>>>> ensemble=3Dhostname:2181,hostname:2181,hostname:2181 >>>>>>> 2016-01-09 20:05:42,476 INFO [thrift-worker-13] >>> zookeeper.ZooKeeper: >>>>>>> Initiating client connection, >>>>>>> connectString=3Dhostname:2181,hostname:2181,hostname:2181 >>>>>>> sessionTimeout=3D90000 watcher=3Dhconnection-0x6305dbeb0x0, >>>>>>> quorum=3Dhostname:2181,hostname:2181,hostname:2181, >> baseZNode=3D/hbase >>>>>>> 2016-01-09 20:05:42,507 INFO >>>>>> [thrift-worker-13-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Opening socket connection to server >>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using >>> SASL >>>>>>> (unknown error) >>>>>>> 2016-01-09 20:05:42,510 INFO >>>>>> [thrift-worker-13-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Socket connection established to >>>>>>> hostname/xx.xx.x.xx:2181, initiating session >>>>>>> 2016-01-09 20:05:42,708 INFO >>>>>> [thrift-worker-13-SendThread(hostname:2181)] >>>>>>> zookeeper.ClientCnxn: Session establishment complete on server >>>>>>> hostname/xx.xx.x.xx:2181, sessionid =3D 0x1522613893e0005, >> negotiated >>>>>> timeout >>>>>>> =3D 40000 >>>>>>> 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1] >>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>> zookeeper >>>>>>> sessionid=3D0x1522613893e0005 >>>>>>> 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1] >>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed >>>>>>>=20 >>>>>>>=20 >>>>>>> Thanks, >>>>>>> Rajeshkumar J >>>>>>>=20 >>>>>>> On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu >>>> wrote: >>>>>>>=20 >>>>>>>> Can you take a stack trace of the thrift server and pastebin >> the >>>>> trace >>>>>> ? >>>>>>>>=20 >>>>>>>> Thanks >>>>>>>>=20 >>>>>>>> On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J < >>>>>>>> rajeshkumarit8292@gmail.com> >>>>>>>> wrote: >>>>>>>>=20 >>>>>>>>> Hi, >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> I am connecting via Hbase thrift server to access records in >>>> Hbase >>>>>> and >>>>>>> I >>>>>>>> am >>>>>>>>> doing this in C# and i am using range scan to get records >> from >>>> the >>>>>>> table. >>>>>>>>> This seems to be working for few hours but after few hours it >>>>> didn't >>>>>>>>> returned any records and then i searched hbase logs i found >>> this >>>>>>>>>=20 >>>>>>>>> 2016-01-04 00:44:54,880 INFO [ConnectionCleaner] >>>>>>>>> client.HConnectionManager$HConnectionImplementation: Closing >>>>>> zookeeper >>>>>>>>> sessionid=3D0x3520b1721650006 >>>>>>>>> 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] >>>>>> zookeeper.ZooKeeper: >>>>>>>>> Session: 0x3520b1721650006 closed >>>>>>>>> 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread] >>>>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>>>>=20 >>>>>>>>> And I am using hbase 0.98.13 >>>>>>>>>=20 >>>>>>>>> I have asked this question before and some user suggested me >> to >>>>>> upgrade >>>>>>>> to >>>>>>>>> new version so i have upgraded HBase to 1.1.2. But now it >> also >>> it >>>>>>> didn't >>>>>>>>> return any records but different log generated as shown below >>>>>>>>>=20 >>>>>>>>> 2016-01-09 20:05:42,708 INFO >>>>>> [thrift-worker-13-SendThread(xxxx:2181)] >>>>>>>>> zookeeper.ClientCnxn: Session establishment complete on >> server >>>>>>>>> xxxx/xx.xx.x.xx:2181, sessionid =3D 0x1522613893e0005, >> negotiated >>>>>>> timeout =3D >>>>>>>>> 40000 >>>>>>>>> 2016-01-09 20:15:45,541 INFO >> [ConnectionCache_ChoreService_1] >>>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing >>>>> zookeeper >>>>>>>>> sessionid=3D0x1522613893e0005 >>>>>>>>> 2016-01-09 20:15:45,567 INFO >> [ConnectionCache_ChoreService_1] >>>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed >>>>>>>>> 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread] >>>>>>>>> zookeeper.ClientCnxn: EventThread shut down >>>>>>>>>=20 >>>>>>>>> Can any one help me in this? >>>>>>>>>=20 >>>>>>>>> Regards, >>>>>>>>> Rajeshkumar J >>=20