hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: HBase 0.90.0 cannot be put more data after running hours
Date Fri, 28 Jan 2011 04:02:44 GMT
Thats a lookup on the .META. table.  Is the region hosting .META. OK?
Anything in its logs?  Do your clients share a Configuration instance
or do you make a new one of these each time you make an HTable?   Your
threaded client is running on single machine?  Can we see full stack
trace?  Are all 8+ threads blocked in same location waiting on call to
return?

St.Ack



On Wed, Jan 26, 2011 at 9:19 AM, Schubert Zhang <zsongbo@gmail.com> wrote:
> The "Thread-Opr0" the client thread to put data into hbase, it is waiting.
>
> "Thread-Opr0-EventThread" daemon prio=10 tid=0x00002aaafc7a8000 nid=0xe08
> waiting on condition [0x000000004383f000]
>   java.lang.Thread.State: WAITING (parking)
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for  <0x00002aaab632ae50> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>        at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>        at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> "Thread-Opr0-SendThread(nd1-rack2-cloud:2181)" daemon prio=10
> tid=0x00002aaafc7a6800 nid=0xe07 runnable [0x000000004373e000]
>   java.lang.Thread.State: RUNNABLE
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>        - locked <0x00002aaab6304410> (a sun.nio.ch.Util$1)
>        - locked <0x00002aaab6304428> (a
> java.util.Collections$UnmodifiableSet)
>        - locked <0x00002aaab632abd0> (a sun.nio.ch.EPollSelectorImpl)
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>        at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>
> "Thread-Opr0" prio=10 tid=0x00002aab0402a000 nid=0xdf2 in Object.wait()
> [0x000000004262d000]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab04302d0> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Call)
>        at java.lang.Object.wait(Object.java:485)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:739)
>        - locked <0x00002aaab04302d0> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Call)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>        at $Proxy0.getClosestRowBefore(Unknown Source)
>        at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:517)
>        at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:515)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1000)
>        at
> org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:514)
>        at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133)
>        at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:645)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:699)
>        - locked <0x00002aaab6294660> (a java.lang.Object)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1114)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234)
>        at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
>        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
>        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
>        at com.bigdata.bench.hbase.HBaseWriter$Operator.operateTo(Unknown
> Source)
>        at com.bigdata.bench.hbase.HBaseWriter$Operator.run(Unknown Source)
>
> On Thu, Jan 27, 2011 at 12:06 AM, Schubert Zhang <zsongbo@gmail.com> wrote:
>
>> Even though cannot put more data into table, I can read the existing data.
>>
>> And I stop and re-start the HBase, still cannot put more data.
>>
>> hbase(main):031:0> status 'simple'
>> 8 live servers
>>     nd5-rack2-cloud:60020 1296057544120
>>         requests=0, regions=32, usedHeap=130, maxHeap=8973
>>     nd8-rack2-cloud:60020 1296057544350
>>         requests=0, regions=31, usedHeap=128, maxHeap=8983
>>     nd2-rack2-cloud:60020 1296057543346
>>         requests=0, regions=32, usedHeap=130, maxHeap=8973
>>     nd3-rack2-cloud:60020 1296057544224
>>         requests=0, regions=32, usedHeap=133, maxHeap=8973
>>     nd6-rack2-cloud:60020 1296057544482
>>         requests=0, regions=32, usedHeap=130, maxHeap=8983
>>     nd9-rack2-cloud:60020 1296057544565
>>         requests=174, regions=32, usedHeap=180, maxHeap=8983
>>     nd7-rack2-cloud:60020 1296057544617
>>         requests=0, regions=32, usedHeap=126, maxHeap=8983
>>     nd4-rack2-cloud:60020 1296057544138
>>         requests=0, regions=32, usedHeap=126, maxHeap=8973
>> 0 dead servers
>> Aggregate load: 174, regions: 255
>>
>>
>> On Wed, Jan 26, 2011 at 11:58 PM, Schubert Zhang <zsongbo@gmail.com>wrote:
>>
>>> I am using 0.90.0 (8 RS + 1Master)
>>> and the HDFS is CDH3b3
>>>
>>> During the first hours of running, I puts many (tens of millions entites,
>>> each about 200 bytes), it worked well.
>>>
>>> But then, the client cannot put more data.
>>>
>>> I checked all log files of hbase, no abnormal is found, I will continue to
>>> check this issue.
>>>
>>> It seems related to ZooKeeper......
>>>
>>
>>
>

Mime
View raw message