hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Hbase thrift server issues
Date Tue, 12 Jan 2016 09:49:42 GMT
You can get the source code from 0.98 branch of:
https://github.com/apache/hbase

0.98.17 hasn't been released yet. 
RC would come out this month. 

If you cannot wait, you can build locally and deploy. 

Cheers

> On Jan 11, 2016, at 11:30 PM, Rajeshkumar J <rajeshkumarit8292@gmail.com> wrote:
> 
> Hi Ted,
> 
>  As mentioned in that HBASE-14926 it is fixed in HBase 0.98.17. I have
> googled it for both its source code and binary but I didn't find one. can
> you guide me where I can find Hbase 0.98.17 source code or binary package
> 
> Thanks
> 
>> On Tue, Jan 12, 2016 at 12:57 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> 
>> I took a look at the stack trace you posted.
>> 
>> Some thrift worker was in the state shown in HBASE-14926
>> 
>> I wonder if the fix for HBASE-14926 would help in your case.
>> 
>> Cheers
>> 
>> On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J <
>> rajeshkumarit8292@gmail.com>
>> wrote:
>> 
>>> Hi Ted,
>>> 
>>>   Plz ignore my previous mail as the log is taken when hbase is working
>>> fine. Please find the logs below when Hbase didn't return any records
>>> 
>>> http://pastebin.com/APYjiGSP
>>> 
>>> Thanks
>>> 
>>>> On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>> 
>>>> When you search for 'java jstack tutorial' using google, you would
>> find a
>>>> lot of tutorials.
>>>> 
>>>> You can use 'ps aux' command to search for the following class:
>>>> org.apache.hadoop.hbase.thrift.ThriftServer
>>>> 
>>>> Apply jstack on the corresponding process id.
>>>> 
>>>> Cheers
>>>> 
>>>> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
>>>> rajeshkumarit8292@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi Ted,
>>>>> 
>>>>>   I don't know how how to take stacktrace using jstack. But i tried
>>>> using
>>>>> some commands but it fails. can you help me in this?
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yuzhihong@gmail.com>
>> wrote:
>>>>> 
>>>>>> What I meant was to capture stack trace of the thrift server (using
>>>>> jstack
>>>>>> command) when your query is running.
>>>>>> 
>>>>>> This would give us some more clue as to the state thrift server was
>>> in
>>>>> when
>>>>>> the problem happened.
>>>>>> 
>>>>>> Cheers
>>>>>> 
>>>>>> On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
>>>>>> rajeshkumarit8292@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi Ted,
>>>>>>> 
>>>>>>>   This is what I found in the Hbase thrift server logs
>>>>>>> 
>>>>>>> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:java.library.path=;C:\Hadoop\bin
>>>>>>> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:java.io.tmpdir=C:\Windows\TEMP\
>>>>>>> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:java.compiler=<NA>
>>>>>>> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:os.name=Windows Server 2012
>>>>>>> 2016-01-09 06:27:43,139 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:os.arch=amd64
>>>>>>> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:os.version=6.2
>>>>>>> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:user.name=xxxx
>>>>>>> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:user.home=C:\
>>>>>>> 2016-01-09 06:27:43,140 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:user.dir=C:\HBase\bin
>>>>>>> 2016-01-09 06:27:43,142 INFO  [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>>> Initiating client connection,
>>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181
>>>>>>> sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
>>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181,
>> baseZNode=/hbase
>>>>>>> 2016-01-09 06:27:43,339 INFO
>>>>> [thrift-worker-0-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Opening socket connection to server
>>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
>>> SASL
>>>>>>> (unknown error)
>>>>>>> 2016-01-09 06:27:43,342 INFO
>>>>> [thrift-worker-0-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Socket connection established to
>>>>>>> hostname/xx.xx.x.xx:2181, initiating session
>>>>>>> 2016-01-09 06:27:43,365 INFO
>>>>> [thrift-worker-0-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Session establishment complete on server
>>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004,
>> negotiated
>>>>>> timeout
>>>>>>> = 40000
>>>>>>> 2016-01-09 07:14:10,607 INFO
>>>>>>> [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
>>>>>>> master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager
>>> Timeout
>>>>>>> Monitor missed its start time
>>>>>>> 2016-01-09 07:16:43,511 INFO
>>>>>>> [B.defaultRpcServer.handler=8,queue=2,port=60000]
>> master.HMaster:
>>>>>>> Client=Administrator/null create 'chk2', {NAME => 'fa',
>>>>>> DATA_BLOCK_ENCODING
>>>>>>> => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE =>
'0',
>> VERSIONS
>>>> =>
>>>>>> '1',
>>>>>>> COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
>>>>>>> KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY
=>
>>>>>> 'false',
>>>>>>> BLOCKCACHE => 'true'}
>>>>>>> 2016-01-09 07:16:44,121 INFO  [ProcedureExecutorThread-1]
>>>>>>> procedure2.ProcedureExecutor: Rolledback procedure
>>>> CreateTableProcedure
>>>>>>> (table=chk2) user=Administrator (auth:SIMPLE) id=2
>> state=ROLLEDBACK
>>>>>>> exec-time=462msec
>>>>> exception=org.apache.hadoop.hbase.TableExistsException:
>>>>>>> chk2
>>>>>>> 2016-01-09 07:49:55,574 INFO  [ConnectionCache_ChoreService_1]
>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>> zookeeper
>>>>>>> sessionid=0x1522613893e0004
>>>>>>> 2016-01-09 07:49:55,629 INFO  [ConnectionCache_ChoreService_1]
>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
>>>>>>> 2016-01-09 07:49:55,661 INFO  [thrift-worker-0-EventThread]
>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>> 2016-01-09 18:40:29,469 INFO  [thrift-worker-1]
>>>>>>> zookeeper.RecoverableZooKeeper: Process
>>>>> identifier=hconnection-0x591287f8
>>>>>>> connecting to ZooKeeper
>>>>>> ensemble=hostname:2181,hostname:2181,hostname:2181
>>>>>>> 2016-01-09 18:40:29,482 INFO  [thrift-worker-1]
>>> zookeeper.ZooKeeper:
>>>>>>> Initiating client connection,
>>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181
>>>>>>> sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
>>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181,
>> baseZNode=/hbase
>>>>>>> 2016-01-09 18:40:29,553 INFO
>>>>> [thrift-worker-1-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Opening socket connection to server
>>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
>>> SASL
>>>>>>> (unknown error)
>>>>>>> 2016-01-09 18:40:29,559 INFO
>>>>> [thrift-worker-1-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Socket connection established to
>>>>>>> hostname/xx.xx.x.xx:2181, initiating session
>>>>>>> 2016-01-09 18:40:29,569 INFO
>>>>> [thrift-worker-1-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Session establishment complete on server
>>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008,
>> negotiated
>>>>>> timeout
>>>>>>> = 40000
>>>>>>> 2016-01-09 18:54:25,558 INFO  [ConnectionCache_ChoreService_1]
>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>> zookeeper
>>>>>>> sessionid=0x35226137e300008
>>>>>>> 2016-01-09 18:54:25,566 INFO  [ConnectionCache_ChoreService_1]
>>>>>>> zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
>>>>>>> 2016-01-09 18:54:25,578 INFO  [thrift-worker-1-EventThread]
>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>> 2016-01-09 20:05:42,469 INFO  [thrift-worker-13]
>>>>>>> zookeeper.RecoverableZooKeeper: Process
>>>>> identifier=hconnection-0x6305dbeb
>>>>>>> connecting to ZooKeeper
>>>>>> ensemble=hostname:2181,hostname:2181,hostname:2181
>>>>>>> 2016-01-09 20:05:42,476 INFO  [thrift-worker-13]
>>> zookeeper.ZooKeeper:
>>>>>>> Initiating client connection,
>>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181
>>>>>>> sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
>>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181,
>> baseZNode=/hbase
>>>>>>> 2016-01-09 20:05:42,507 INFO
>>>>>> [thrift-worker-13-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Opening socket connection to server
>>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
>>> SASL
>>>>>>> (unknown error)
>>>>>>> 2016-01-09 20:05:42,510 INFO
>>>>>> [thrift-worker-13-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Socket connection established to
>>>>>>> hostname/xx.xx.x.xx:2181, initiating session
>>>>>>> 2016-01-09 20:05:42,708 INFO
>>>>>> [thrift-worker-13-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Session establishment complete on server
>>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005,
>> negotiated
>>>>>> timeout
>>>>>>> = 40000
>>>>>>> 2016-01-09 20:15:45,541 INFO  [ConnectionCache_ChoreService_1]
>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>> zookeeper
>>>>>>> sessionid=0x1522613893e0005
>>>>>>> 2016-01-09 20:15:45,567 INFO  [ConnectionCache_ChoreService_1]
>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Rajeshkumar J
>>>>>>> 
>>>>>>> On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yuzhihong@gmail.com>
>>>> wrote:
>>>>>>> 
>>>>>>>> Can you take a stack trace of the thrift server and pastebin
>> the
>>>>> trace
>>>>>> ?
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
>>>>>>>> rajeshkumarit8292@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I am connecting via Hbase thrift server to access records
in
>>>> Hbase
>>>>>> and
>>>>>>> I
>>>>>>>> am
>>>>>>>>> doing this in C# and i am using range scan to get records
>> from
>>>> the
>>>>>>> table.
>>>>>>>>> This seems to be working for few hours but after few
hours it
>>>>> didn't
>>>>>>>>> returned any records and then i searched hbase logs i
found
>>> this
>>>>>>>>> 
>>>>>>>>> 2016-01-04 00:44:54,880 INFO  [ConnectionCleaner]
>>>>>>>>> client.HConnectionManager$HConnectionImplementation:
Closing
>>>>>> zookeeper
>>>>>>>>> sessionid=0x3520b1721650006
>>>>>>>>> 2016-01-04 00:44:54,891 INFO  [ConnectionCleaner]
>>>>>> zookeeper.ZooKeeper:
>>>>>>>>> Session: 0x3520b1721650006 closed
>>>>>>>>> 2016-01-04 00:44:54,891 INFO  [thrift-worker-0-EventThread]
>>>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>>>> 
>>>>>>>>> And I am using hbase 0.98.13
>>>>>>>>> 
>>>>>>>>> I have asked this question before and some user suggested
me
>> to
>>>>>> upgrade
>>>>>>>> to
>>>>>>>>> new version so i have upgraded HBase to 1.1.2. But now
it
>> also
>>> it
>>>>>>> didn't
>>>>>>>>> return any records but different log generated as shown
below
>>>>>>>>> 
>>>>>>>>> 2016-01-09 20:05:42,708 INFO
>>>>>> [thrift-worker-13-SendThread(xxxx:2181)]
>>>>>>>>> zookeeper.ClientCnxn: Session establishment complete
on
>> server
>>>>>>>>> xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005,
>> negotiated
>>>>>>> timeout =
>>>>>>>>> 40000
>>>>>>>>> 2016-01-09 20:15:45,541 INFO
>> [ConnectionCache_ChoreService_1]
>>>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>>>> zookeeper
>>>>>>>>> sessionid=0x1522613893e0005
>>>>>>>>> 2016-01-09 20:15:45,567 INFO
>> [ConnectionCache_ChoreService_1]
>>>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
>>>>>>>>> 2016-01-09 20:15:45,567 INFO  [thrift-worker-13-EventThread]
>>>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>>>> 
>>>>>>>>> Can any one help me in this?
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Rajeshkumar J
>> 

Mime
View raw message