hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vamshi Krishna <vamshi2...@gmail.com>
Subject Re: Will hbase automatically distribute the data across region servers or NOT..??
Date Fri, 23 Aug 2013 14:21:52 GMT
No that is 10MB itself. Just to observe the region splitting with respect
to the amount of data i am inserting in to hbase.
So, here i am inserting 40-50mb data and fixing that property to 10mb and
checking the region splitting happening.
But the intersting thing is regions got split BUT they are not being
distributed across other servers.
Whatever regions formed from the created tables on machine-1, all of them
are residing on the same machine-1 not being moved to other machine.




On Fri, Aug 23, 2013 at 7:40 PM, Dhaval Shah <prince_mithibai@yahoo.co.in>wrote:

> Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small.
> Did you mean 10GB?
>
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Vamshi Krishna <vamshi2105@gmail.com>
> To: user@hbase.apache.org; zhoushuaifeng <zhoushuaifeng@gmail.com>
> Sent: Friday, 23 August 2013 9:38 AM
> Subject: Re: Will hbase automatically distribute the data across region
> servers or NOT..??
>
>
> Thanks for the clarifications.
> I am using hbase-0.94.10 and zookeepr-3.4.5
> But I am running into different issues .
> I set  hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
> rows in to hbase table. During the insertion after some time, suddenly
> master is going down. I don't know what is the reason for such peculiar
> behavior.
> I found in master log below content and not able to make out what exactly
> the mistake. Please somebody help.
>
> master-log:
>
> 2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
>     at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
>     at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>     at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
> 2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60000
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60000: exiting
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60000: exiting
> 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server listener on 60000
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 2 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 1 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
> vamshi,60000,1377263788019-BalancerChore exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
> Stopping infoServer
> 2013-08-23 18:56:36,873 INFO
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
> master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
> vamshi,60000,1377263788019-CatalogJanitor exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> Server handler 0 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
> SelectChannelConnector@0.0.0.0:60010
> 2013-08-23 18:56:36,874 INFO
> org.apache.hadoop.hbase.master.cleaner.LogCleaner:
> master-vamshi,60000,1377263788019.oldLogCleaner exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60000: exiting
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> 2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60000: exiting
> 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60000: exiting
> 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> master:60000-0x140ab519b0f0000 Unable to set watcher on znode
> (/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
> java.lang.InterruptedException
>     at java.lang.Object.wait(Native Method)
>     at java.lang.Object.wait(Object.java:485)
>     at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
>     at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
>     at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
>     at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
>     at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
>     at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
>     at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>     at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,876 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
> state=OFFLINE, ts=1377264396802, server=null
> 2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> master:60000-0x140ab519b0f0000 Unable to get data of znode
> /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> java.lang.InterruptedException
>     at java.lang.Object.wait(Native Method)
>     at java.lang.Object.wait(Object.java:485)
>     at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
>     at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
>     at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
>     at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
>     at
>
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
>     at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
>     at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>     at java.lang.Thread.run(Thread.java:662)
> 2013-08-23 18:56:36,877 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> state=OFFLINE, ts=1377264396862, server=null
> 2013-08-23 18:56:36,877 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to create/force
> node into OFFLINE state before completing assignment but failed to do so
> for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> state=OFFLINE, ts=1377264396813, server=null
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
> region=d476f8442ce31de90b60080b74daf47f
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> destination server is vamshi,60020,1377263789273
> 2013-08-23 18:56:36,882 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> plan was found (or we are ignoring an existing plan) for
>
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> so generated a random one;
>
> hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
> src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
> servers
> 2013-08-23 18:56:36,882 ERROR
> org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> [ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor is
> missing. Is this process shutting down?
> 2013-08-23 18:56:36,906 DEBUG
> org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
> org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
> 2013-08-23 18:56:36,906 INFO
> org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
> vamshi,60000,1377263788019.timeoutMonitor exiting
> 2013-08-23 18:56:36,906 INFO
> org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
> vamshi,60000,1377263788019.timerUpdater exiting
> 2013-08-23 18:56:36,907 INFO
> org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
> vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
> 2013-08-23 18:56:36,910 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=vamshi_RS,60020,1377263792053,
> region=05e30711673614f6b41a364c76f3f05f
> 2013-08-23 18:56:36,911 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> destination server is vamshi,60020,1377263789273
> 2013-08-23 18:56:36,911 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> plan was found (or we are ignoring an existing plan) for
>
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> so generated a random one;
>
> hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
> src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1) available
> servers
> 2013-08-23 18:56:36,911 ERROR
> org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> [ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor is
> missing. Is this process shutting down?
> 2013-08-23 18:56:36,912 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
> ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> 2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
> Sleeping 2000ms before retry #1...
> 2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x140ab519b0f0000 closed
> 2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
> HMaster main thread exiting
> 2013-08-23 18:56:36,914 ERROR
> org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> java.lang.RuntimeException: HMaster Aborted
>     at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
>     at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>     at
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>     at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
>
>
>
> My hbase-site.xml :
>
> <configuration>
>     <property>
>         <name>hbase.rootdir</name>
>
>     <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
>     </property>
>
>     <property>
>         <name>hbase.cluster.distributed</name>
>         <value>true</value>
>     </property>
>     <property>
>         <name>hbase.master</name>
>         <value>vamshi</value>
>     </property>
>     <property>
>         <name>hbase.zookeeper.property.clientPort</name>
>         <value>2181</value>
>     </property>
>
>
>    <property>
>         <name>hbase.hregion.max.filesize</name>
>         <value>10485760</value>
>     </property>
>
>
>
>     <property>
>         <name>hbase.zookeeper.quorum</name>
>         <value>vamshi</value>
>     </property>
>     <property>
>         <name>hbase.zookeeper.property.dataDir</name>
>         <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
>     </property>
>
> <property>
>     <name>hbase.zookeeper.property.maxClientCnxns</name>
>     <value>1024</value>
>   </property>
>
> <property>
>     <name>hbase.coprocessor.user.region.classes</name>
>     <value>com.bil.coproc.ColumnAggregationEndpoint</value>
>   </property>
> </configuration>
>
>
>
>
> On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zhoushuaifeng@gmail.com>
> wrote:
>
> > Hi,
> > You may should check if the compact is on. If data size in a region is
> max
> > than the limition, region will split and balance after a major
> > compaction(Usually occur automatically).
> > You can manually by run the compact operaction by the shell commond:
> > compact <tableName>, or major_compact <tableName>
> >
> >
> >
> >
> > Frank Chow
>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
>



-- 
*Regards*
*
Vamshi Krishna
*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message