Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CALte62wjZeVTBxD+PMqv313m-p8BAd025m_yB0StyBrVu4iE=A@mail.gmail.com>
References: 
 <CAHu1D0g8mtbDq1EAjLz9w1oNNr+R2WdwNyjqhaGn98uxHG=ing@mail.gmail.com>
	<CALte62zHzTd6Aw-duceyRAPpXHz65HAG=vbOes1fbKY5Jo0RTg@mail.gmail.com>
	<CAHu1D0hoo=MNsxdKRFDDOqO6p0p7koqT4rJDQ8iCT9n0TXObRQ@mail.gmail.com>
	<CALte62xbJua2ueT9he+nU25AJOCf145oRnKF+S4T3ghPbcbtxw@mail.gmail.com>
	<CAHu1D0hXUVPH07ek=xdf+KM_teUgtcFf7s=7e9etv=1nguWMHg@mail.gmail.com>
	<CALte62wjZeVTBxD+PMqv313m-p8BAd025m_yB0StyBrVu4iE=A@mail.gmail.com>
Date: Mon, 11 Jan 2016 18:41:32 +0530
Message-ID: 
 <CAHu1D0jxZOkEdT+dv1O2c1EaeLrxq+sqNHGLq9oPO-YgWyY+Hw@mail.gmail.com>
Subject: Re: Hbase thrift server issues
From: Rajeshkumar J <rajeshkumarit8292@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a11c2b3f009f24205290ead88

--001a11c2b3f009f24205290ead88
Content-Type: text/plain; charset=UTF-8

Hi Ted,

   Plz ignore my previous mail as the log is taken when hbase is working
fine. Please find the logs below when Hbase didn't return any records

http://pastebin.com/APYjiGSP

Thanks

On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> When you search for 'java jstack tutorial' using google, you would find a
> lot of tutorials.
>
> You can use 'ps aux' command to search for the following class:
> org.apache.hadoop.hbase.thrift.ThriftServer
>
> Apply jstack on the corresponding process id.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> >    I don't know how how to take stacktrace using jstack. But i tried
> using
> > some commands but it fails. can you help me in this?
> >
> > Thanks
> >
> > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > What I meant was to capture stack trace of the thrift server (using
> > jstack
> > > command) when your query is running.
> > >
> > > This would give us some more clue as to the state thrift server was in
> > when
> > > the problem happened.
> > >
> > > Cheers
> > >
> > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > >    This is what I found in the Hbase thrift server logs
> > > >
> > > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.library.path=;C:\Hadoop\bin
> > > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.compiler=<NA>
> > > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.name=Windows Server 2012
> > > > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.arch=amd64
> > > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.version=6.2
> > > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.name=xxxx
> > > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.home=C:\
> > > > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.dir=C:\HBase\bin
> > > > 2016-01-09 06:27:43,142 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 06:27:43,339 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 06:27:43,342 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 06:27:43,365 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 07:14:10,607 INFO
> > > >  [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > > > Monitor missed its start time
> > > > 2016-01-09 07:16:43,511 INFO
> > > >  [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > > DATA_BLOCK_ENCODING
> > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS
> =>
> > > '1',
> > > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> > > 'false',
> > > > BLOCKCACHE => 'true'}
> > > > 2016-01-09 07:16:44,121 INFO  [ProcedureExecutorThread-1]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> CreateTableProcedure
> > > > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > > > exec-time=462msec
> > exception=org.apache.hadoop.hbase.TableExistsException:
> > > > chk2
> > > > 2016-01-09 07:49:55,574 INFO  [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0004
> > > > 2016-01-09 07:49:55,629 INFO  [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > > > 2016-01-09 07:49:55,661 INFO  [thrift-worker-0-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > > 2016-01-09 18:40:29,469 INFO  [thrift-worker-1]
> > > > zookeeper.RecoverableZooKeeper: Process
> > identifier=hconnection-0x591287f8
> > > > connecting to ZooKeeper
> > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > 2016-01-09 18:40:29,482 INFO  [thrift-worker-1] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 18:40:29,553 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 18:40:29,559 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 18:40:29,569 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 18:54:25,558 INFO  [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x35226137e300008
> > > > 2016-01-09 18:54:25,566 INFO  [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > > > 2016-01-09 18:54:25,578 INFO  [thrift-worker-1-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > > 2016-01-09 20:05:42,469 INFO  [thrift-worker-13]
> > > > zookeeper.RecoverableZooKeeper: Process
> > identifier=hconnection-0x6305dbeb
> > > > connecting to ZooKeeper
> > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > 2016-01-09 20:05:42,476 INFO  [thrift-worker-13] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 20:05:42,507 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 20:05:42,510 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 20:05:42,708 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0005
> > > > 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > >
> > > >
> > > > Thanks,
> > > > Rajeshkumar J
> > > >
> > > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > >
> > > > > Can you take a stack trace of the thrift server and pastebin the
> > trace
> > > ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > > > rajeshkumarit8292@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > I am connecting via Hbase thrift server to access records in
> Hbase
> > > and
> > > > I
> > > > > am
> > > > > > doing this in C# and i am using range scan to get records from
> the
> > > > table.
> > > > > > This seems to be working for few hours but after few hours it
> > didn't
> > > > > > returned any records and then i searched hbase logs i found this
> > > > > >
> > > > > > 2016-01-04 00:44:54,880 INFO  [ConnectionCleaner]
> > > > > > client.HConnectionManager$HConnectionImplementation: Closing
> > > zookeeper
> > > > > > sessionid=0x3520b1721650006
> > > > > > 2016-01-04 00:44:54,891 INFO  [ConnectionCleaner]
> > > zookeeper.ZooKeeper:
> > > > > > Session: 0x3520b1721650006 closed
> > > > > > 2016-01-04 00:44:54,891 INFO  [thrift-worker-0-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > >
> > > > > > And I am using hbase 0.98.13
> > > > > >
> > > > > > I have asked this question before and some user suggested me to
> > > upgrade
> > > > > to
> > > > > > new version so i have upgraded HBase to 1.1.2. But now it also it
> > > > didn't
> > > > > > return any records but different log generated as shown below
> > > > > >
> > > > > > 2016-01-09 20:05:42,708 INFO
> > > [thrift-worker-13-SendThread(xxxx:2181)]
> > > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > > timeout =
> > > > > > 40000
> > > > > > 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > > sessionid=0x1522613893e0005
> > > > > > 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > > 2016-01-09 20:15:45,567 INFO  [thrift-worker-13-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > >
> > > > > > Can any one help me in this?
> > > > > >
> > > > > > Regards,
> > > > > > Rajeshkumar J
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a11c2b3f009f24205290ead88--