Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAHu1D0hXUVPH07ek=xdf+KM_teUgtcFf7s=7e9etv=1nguWMHg@mail.gmail.com>
References: 
 <CAHu1D0g8mtbDq1EAjLz9w1oNNr+R2WdwNyjqhaGn98uxHG=ing@mail.gmail.com>
	<CALte62zHzTd6Aw-duceyRAPpXHz65HAG=vbOes1fbKY5Jo0RTg@mail.gmail.com>
	<CAHu1D0hoo=MNsxdKRFDDOqO6p0p7koqT4rJDQ8iCT9n0TXObRQ@mail.gmail.com>
	<CALte62xbJua2ueT9he+nU25AJOCf145oRnKF+S4T3ghPbcbtxw@mail.gmail.com>
	<CAHu1D0hXUVPH07ek=xdf+KM_teUgtcFf7s=7e9etv=1nguWMHg@mail.gmail.com>
Date: Sun, 10 Jan 2016 21:49:55 -0800
Message-ID: 
 <CALte62wjZeVTBxD+PMqv313m-p8BAd025m_yB0StyBrVu4iE=A@mail.gmail.com>
Subject: Re: Hbase thrift server issues
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c032720acda3405290881b7

--94eb2c032720acda3405290881b7
Content-Type: text/plain; charset=UTF-8

When you search for 'java jstack tutorial' using google, you would find a
lot of tutorials.

You can use 'ps aux' command to search for the following class:
org.apache.hadoop.hbase.thrift.ThriftServer

Apply jstack on the corresponding process id.

Cheers

On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <rajeshkumarit8292@gmail.com>
wrote:

> Hi Ted,
>
>    I don't know how how to take stacktrace using jstack. But i tried using
> some commands but it fails. can you help me in this?
>
> Thanks
>
> On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > What I meant was to capture stack trace of the thrift server (using
> jstack
> > command) when your query is running.
> >
> > This would give us some more clue as to the state thrift server was in
> when
> > the problem happened.
> >
> > Cheers
> >
> > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com>
> > wrote:
> >
> > > Hi Ted,
> > >
> > >    This is what I found in the Hbase thrift server logs
> > >
> > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:java.library.path=;C:\Hadoop\bin
> > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:java.compiler=<NA>
> > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:os.name=Windows Server 2012
> > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:os.arch=amd64
> > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:os.version=6.2
> > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:user.name=xxxx
> > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:user.home=C:\
> > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:user.dir=C:\HBase\bin
> > > 2016-01-09 06:27:43,142 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Initiating client connection,
> > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > 2016-01-09 06:27:43,339 INFO
> [thrift-worker-0-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > (unknown error)
> > > 2016-01-09 06:27:43,342 INFO
> [thrift-worker-0-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > hostname/xx.xx.x.xx:2181, initiating session
> > > 2016-01-09 06:27:43,365 INFO
> [thrift-worker-0-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > timeout
> > > = 40000
> > > 2016-01-09 07:14:10,607 INFO
> > >  [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > > Monitor missed its start time
> > > 2016-01-09 07:16:43,511 INFO
> > >  [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > DATA_BLOCK_ENCODING
> > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
> > '1',
> > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> > 'false',
> > > BLOCKCACHE => 'true'}
> > > 2016-01-09 07:16:44,121 INFO  [ProcedureExecutorThread-1]
> > > procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
> > > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > > exec-time=462msec
> exception=org.apache.hadoop.hbase.TableExistsException:
> > > chk2
> > > 2016-01-09 07:49:55,574 INFO  [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x1522613893e0004
> > > 2016-01-09 07:49:55,629 INFO  [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > > 2016-01-09 07:49:55,661 INFO  [thrift-worker-0-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > > 2016-01-09 18:40:29,469 INFO  [thrift-worker-1]
> > > zookeeper.RecoverableZooKeeper: Process
> identifier=hconnection-0x591287f8
> > > connecting to ZooKeeper
> > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > 2016-01-09 18:40:29,482 INFO  [thrift-worker-1] zookeeper.ZooKeeper:
> > > Initiating client connection,
> > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > 2016-01-09 18:40:29,553 INFO
> [thrift-worker-1-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > (unknown error)
> > > 2016-01-09 18:40:29,559 INFO
> [thrift-worker-1-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > hostname/xx.xx.x.xx:2181, initiating session
> > > 2016-01-09 18:40:29,569 INFO
> [thrift-worker-1-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> > timeout
> > > = 40000
> > > 2016-01-09 18:54:25,558 INFO  [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x35226137e300008
> > > 2016-01-09 18:54:25,566 INFO  [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > > 2016-01-09 18:54:25,578 INFO  [thrift-worker-1-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > > 2016-01-09 20:05:42,469 INFO  [thrift-worker-13]
> > > zookeeper.RecoverableZooKeeper: Process
> identifier=hconnection-0x6305dbeb
> > > connecting to ZooKeeper
> > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > 2016-01-09 20:05:42,476 INFO  [thrift-worker-13] zookeeper.ZooKeeper:
> > > Initiating client connection,
> > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > 2016-01-09 20:05:42,507 INFO
> > [thrift-worker-13-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > (unknown error)
> > > 2016-01-09 20:05:42,510 INFO
> > [thrift-worker-13-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > hostname/xx.xx.x.xx:2181, initiating session
> > > 2016-01-09 20:05:42,708 INFO
> > [thrift-worker-13-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > timeout
> > > = 40000
> > > 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x1522613893e0005
> > > 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > >
> > >
> > > Thanks,
> > > Rajeshkumar J
> > >
> > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Can you take a stack trace of the thrift server and pastebin the
> trace
> > ?
> > > >
> > > > Thanks
> > > >
> > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > I am connecting via Hbase thrift server to access records in Hbase
> > and
> > > I
> > > > am
> > > > > doing this in C# and i am using range scan to get records from the
> > > table.
> > > > > This seems to be working for few hours but after few hours it
> didn't
> > > > > returned any records and then i searched hbase logs i found this
> > > > >
> > > > > 2016-01-04 00:44:54,880 INFO  [ConnectionCleaner]
> > > > > client.HConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > sessionid=0x3520b1721650006
> > > > > 2016-01-04 00:44:54,891 INFO  [ConnectionCleaner]
> > zookeeper.ZooKeeper:
> > > > > Session: 0x3520b1721650006 closed
> > > > > 2016-01-04 00:44:54,891 INFO  [thrift-worker-0-EventThread]
> > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > >
> > > > > And I am using hbase 0.98.13
> > > > >
> > > > > I have asked this question before and some user suggested me to
> > upgrade
> > > > to
> > > > > new version so i have upgraded HBase to 1.1.2. But now it also it
> > > didn't
> > > > > return any records but different log generated as shown below
> > > > >
> > > > > 2016-01-09 20:05:42,708 INFO
> > [thrift-worker-13-SendThread(xxxx:2181)]
> > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > timeout =
> > > > > 40000
> > > > > 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> > > > > client.ConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > > sessionid=0x1522613893e0005
> > > > > 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > 2016-01-09 20:15:45,567 INFO  [thrift-worker-13-EventThread]
> > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > >
> > > > > Can any one help me in this?
> > > > >
> > > > > Regards,
> > > > > Rajeshkumar J
> > > > >
> > > >
> > >
> >
>

--94eb2c032720acda3405290881b7--