hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajeshkumar J <rajeshkumarit8...@gmail.com>
Subject Re: Hbase thrift server issues
Date Mon, 11 Jan 2016 05:43:30 GMT
Hi Ted,

   I don't know how how to take stacktrace using jstack. But i tried using
some commands but it fails. can you help me in this?

Thanks

On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> What I meant was to capture stack trace of the thrift server (using jstack
> command) when your query is running.
>
> This would give us some more clue as to the state thrift server was in when
> the problem happened.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> >    This is what I found in the Hbase thrift server logs
> >
> > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.library.path=;C:\Hadoop\bin
> > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.io.tmpdir=C:\Windows\TEMP\
> > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.compiler=<NA>
> > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.name=Windows Server 2012
> > 2016-01-09 06:27:43,139 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.arch=amd64
> > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.version=6.2
> > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.name=xxxx
> > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.home=C:\
> > 2016-01-09 06:27:43,140 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.dir=C:\HBase\bin
> > 2016-01-09 06:27:43,142 INFO  [thrift-worker-0] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:2181,hostname:2181,hostname:2181
> > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > 2016-01-09 06:27:43,339 INFO  [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > (unknown error)
> > 2016-01-09 06:27:43,342 INFO  [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > hostname/xx.xx.x.xx:2181, initiating session
> > 2016-01-09 06:27:43,365 INFO  [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> timeout
> > = 40000
> > 2016-01-09 07:14:10,607 INFO
> >  [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > Monitor missed its start time
> > 2016-01-09 07:16:43,511 INFO
> >  [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > Client=Administrator/null create 'chk2', {NAME => 'fa',
> DATA_BLOCK_ENCODING
> > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
> '1',
> > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> > BLOCKCACHE => 'true'}
> > 2016-01-09 07:16:44,121 INFO  [ProcedureExecutorThread-1]
> > procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
> > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > exec-time=462msec exception=org.apache.hadoop.hbase.TableExistsException:
> > chk2
> > 2016-01-09 07:49:55,574 INFO  [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x1522613893e0004
> > 2016-01-09 07:49:55,629 INFO  [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > 2016-01-09 07:49:55,661 INFO  [thrift-worker-0-EventThread]
> > zookeeper.ClientCnxn: EventThread shut down
> > 2016-01-09 18:40:29,469 INFO  [thrift-worker-1]
> > zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x591287f8
> > connecting to ZooKeeper
> ensemble=hostname:2181,hostname:2181,hostname:2181
> > 2016-01-09 18:40:29,482 INFO  [thrift-worker-1] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:2181,hostname:2181,hostname:2181
> > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > 2016-01-09 18:40:29,553 INFO  [thrift-worker-1-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > (unknown error)
> > 2016-01-09 18:40:29,559 INFO  [thrift-worker-1-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > hostname/xx.xx.x.xx:2181, initiating session
> > 2016-01-09 18:40:29,569 INFO  [thrift-worker-1-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> timeout
> > = 40000
> > 2016-01-09 18:54:25,558 INFO  [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x35226137e300008
> > 2016-01-09 18:54:25,566 INFO  [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > 2016-01-09 18:54:25,578 INFO  [thrift-worker-1-EventThread]
> > zookeeper.ClientCnxn: EventThread shut down
> > 2016-01-09 20:05:42,469 INFO  [thrift-worker-13]
> > zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6305dbeb
> > connecting to ZooKeeper
> ensemble=hostname:2181,hostname:2181,hostname:2181
> > 2016-01-09 20:05:42,476 INFO  [thrift-worker-13] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:2181,hostname:2181,hostname:2181
> > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > 2016-01-09 20:05:42,507 INFO
> [thrift-worker-13-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > (unknown error)
> > 2016-01-09 20:05:42,510 INFO
> [thrift-worker-13-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > hostname/xx.xx.x.xx:2181, initiating session
> > 2016-01-09 20:05:42,708 INFO
> [thrift-worker-13-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> timeout
> > = 40000
> > 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x1522613893e0005
> > 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> >
> >
> > Thanks,
> > Rajeshkumar J
> >
> > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Can you take a stack trace of the thrift server and pastebin the trace
> ?
> > >
> > > Thanks
> > >
> > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > I am connecting via Hbase thrift server to access records in Hbase
> and
> > I
> > > am
> > > > doing this in C# and i am using range scan to get records from the
> > table.
> > > > This seems to be working for few hours but after few hours it didn't
> > > > returned any records and then i searched hbase logs i found this
> > > >
> > > > 2016-01-04 00:44:54,880 INFO  [ConnectionCleaner]
> > > > client.HConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > sessionid=0x3520b1721650006
> > > > 2016-01-04 00:44:54,891 INFO  [ConnectionCleaner]
> zookeeper.ZooKeeper:
> > > > Session: 0x3520b1721650006 closed
> > > > 2016-01-04 00:44:54,891 INFO  [thrift-worker-0-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > >
> > > > And I am using hbase 0.98.13
> > > >
> > > > I have asked this question before and some user suggested me to
> upgrade
> > > to
> > > > new version so i have upgraded HBase to 1.1.2. But now it also it
> > didn't
> > > > return any records but different log generated as shown below
> > > >
> > > > 2016-01-09 20:05:42,708 INFO
> [thrift-worker-13-SendThread(xxxx:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > timeout =
> > > > 40000
> > > > 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0005
> > > > 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > 2016-01-09 20:15:45,567 INFO  [thrift-worker-13-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > >
> > > > Can any one help me in this?
> > > >
> > > > Regards,
> > > > Rajeshkumar J
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message