hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vamshi Krishna <vamshi2...@gmail.com>
Subject Re: Will hbase automatically distribute the data across region servers or NOT..??
Date Fri, 23 Aug 2013 13:38:58 GMT
Thanks for the clarifications.
I am using hbase-0.94.10 and zookeepr-3.4.5
But I am running into different issues .
I set  hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
rows in to hbase table. During the insertion after some time, suddenly
master is going down. I don't know what is the reason for such peculiar
behavior.
I found in master log below content and not able to make out what exactly
the mistake. Please somebody help.

master-log:

2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
Cannot transit it to OFFLINE.
    at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
    at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service threads
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: exiting
2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 1 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
vamshi,60000,1377263788019-BalancerChore exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
Stopping infoServer
2013-08-23 18:56:36,873 INFO
org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
vamshi,60000,1377263788019-CatalogJanitor exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 0 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
SelectChannelConnector@0.0.0.0:60010
2013-08-23 18:56:36,874 INFO
org.apache.hadoop.hbase.master.cleaner.LogCleaner:
master-vamshi,60000,1377263788019.oldLogCleaner exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to set watcher on znode
(/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
    at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
    at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
    at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
    at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,876 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
state=OFFLINE, ts=1377264396802, server=null
2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x140ab519b0f0000 Unable to get data of znode
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
    at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
    at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
    at
org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
    at
org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
    at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
    at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
state=OFFLINE, ts=1377264396862, server=null
2013-08-23 18:56:36,877 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
node into OFFLINE state before completing assignment but failed to do so
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
state=OFFLINE, ts=1377264396813, server=null
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,882 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,906 DEBUG
org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
vamshi,60000,1377263788019.timeoutMonitor exiting
2013-08-23 18:56:36,906 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
vamshi,60000,1377263788019.timerUpdater exiting
2013-08-23 18:56:36,907 INFO
org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
2013-08-23 18:56:36,910 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
region=05e30711673614f6b41a364c76f3f05f
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
destination server is vamshi,60020,1377263789273
2013-08-23 18:56:36,911 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
plan was found (or we are ignoring an existing plan) for
scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
so generated a random one;
hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
servers
2013-08-23 18:56:36,911 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is
missing. Is this process shutting down?
2013-08-23 18:56:36,912 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/hbase/unassigned/d476f8442ce31de90b60080b74daf47f
2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
Sleeping 2000ms before retry #1...
2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
0x140ab519b0f0000 closed
2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
HMaster main thread exiting
2013-08-23 18:56:36,914 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)



My hbase-site.xml :

<configuration>
    <property>
        <name>hbase.rootdir</name>

    <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
    </property>

    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.master</name>
        <value>vamshi</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>


   <property>
        <name>hbase.hregion.max.filesize</name>
        <value>10485760</value>
    </property>



    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>vamshi</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
    </property>

 <property>
    <name>hbase.zookeeper.property.maxClientCnxns</name>
    <value>1024</value>
  </property>

 <property>
    <name>hbase.coprocessor.user.region.classes</name>
    <value>com.bil.coproc.ColumnAggregationEndpoint</value>
  </property>
</configuration>




On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zhoushuaifeng@gmail.com> wrote:

> Hi,
> You may should check if the compact is on. If data size in a region is max
> than the limition, region will split and balance after a major
> compaction(Usually occur automatically).
> You can manually by run the compact operaction by the shell commond:
> compact <tableName>, or major_compact <tableName>
>
>
>
>
> Frank Chow




-- 
*Regards*
*
Vamshi Krishna
*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message