hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Axiak <m...@axiak.net>
Subject Re: Will hbase automatically distribute the data across region servers or NOT..??
Date Fri, 23 Aug 2013 14:23:52 GMT
Is your balancer enabled?
On Aug 23, 2013 10:22 AM, "Vamshi Krishna" <vamshi2105@gmail.com> wrote:

> No that is 10MB itself. Just to observe the region splitting with respect
> to the amount of data i am inserting in to hbase.
> So, here i am inserting 40-50mb data and fixing that property to 10mb and
> checking the region splitting happening.
> But the intersting thing is regions got split BUT they are not being
> distributed across other servers.
> Whatever regions formed from the created tables on machine-1, all of them
> are residing on the same machine-1 not being moved to other machine.
>
>
>
>
> On Fri, Aug 23, 2013 at 7:40 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
> >wrote:
>
> > Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small.
> > Did you mean 10GB?
> >
> >
> > Regards,
> > Dhaval
> >
> >
> > ________________________________
> > From: Vamshi Krishna <vamshi2105@gmail.com>
> > To: user@hbase.apache.org; zhoushuaifeng <zhoushuaifeng@gmail.com>
> > Sent: Friday, 23 August 2013 9:38 AM
> > Subject: Re: Will hbase automatically distribute the data across region
> > servers or NOT..??
> >
> >
> > Thanks for the clarifications.
> > I am using hbase-0.94.10 and zookeepr-3.4.5
> > But I am running into different issues .
> > I set  hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
> > rows in to hbase table. During the insertion after some time, suddenly
> > master is going down. I don't know what is the reason for such peculiar
> > behavior.
> > I found in master log below content and not able to make out what exactly
> > the mistake. Please somebody help.
> >
> > master-log:
> >
> > 2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Master server abort: loaded coprocessors are: []
> > 2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Unexpected state :
> >
> >
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> > state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273
> ..
> > Cannot transit it to OFFLINE.
> > java.lang.IllegalStateException: Unexpected state :
> >
> >
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> > state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273
> ..
> > Cannot transit it to OFFLINE.
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> >     at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >     at java.lang.Thread.run(Thread.java:662)
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.hbase.master.HMaster:
> > Aborting
> > 2013-08-23 18:56:36,867 DEBUG org.apache.hadoop.hbase.master.HMaster:
> > Stopping service threads
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > server on 60000
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 0 on 60000: exiting
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 5 on 60000: exiting
> > 2013-08-23 18:56:36,867 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 3 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > IPC Server listener on 60000
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> > Server handler 2 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> > Server handler 1 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster$2:
> > vamshi,60000,1377263788019-BalancerChore exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.hbase.master.HMaster:
> > Stopping infoServer
> > 2013-08-23 18:56:36,873 INFO
> > org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
> > master-vamshi,60000,1377263788019.archivedHFileCleaner exiting
> > 2013-08-23 18:56:36,873 INFO
> org.apache.hadoop.hbase.master.CatalogJanitor:
> > vamshi,60000,1377263788019-CatalogJanitor exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
> > Server handler 0 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 9 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.mortbay.log: Stopped
> > SelectChannelConnector@0.0.0.0:60010
> > 2013-08-23 18:56:36,874 INFO
> > org.apache.hadoop.hbase.master.cleaner.LogCleaner:
> > master-vamshi,60000,1377263788019.oldLogCleaner exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 1 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 7 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 6 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 8 on 60000: exiting
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > IPC Server Responder
> > 2013-08-23 18:56:36,876 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> > IPC Server Responder
> > 2013-08-23 18:56:36,874 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 2 on 60000: exiting
> > 2013-08-23 18:56:36,873 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 4 on 60000: exiting
> > 2013-08-23 18:56:36,877 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > master:60000-0x140ab519b0f0000 Unable to set watcher on znode
> > (/hbase/unassigned/05e30711673614f6b41a364c76f3f05f)
> > java.lang.InterruptedException
> >     at java.lang.Object.wait(Native Method)
> >     at java.lang.Object.wait(Object.java:485)
> >     at
> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> >     at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> >     at
> >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
> >     at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
> >     at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:271)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> >     at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >     at java.lang.Thread.run(Thread.java:662)
> > 2013-08-23 18:56:36,876 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> create/force
> > node into OFFLINE state before completing assignment but failed to do so
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x08b8,1377264147374.39794b7deea3203fc260756f5038d6f8.
> > state=OFFLINE, ts=1377264396802, server=null
> > 2013-08-23 18:56:36,876 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > master:60000-0x140ab519b0f0000 Unable to get data of znode
> > /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> > java.lang.InterruptedException
> >     at java.lang.Object.wait(Native Method)
> >     at java.lang.Object.wait(Object.java:485)
> >     at
> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> >     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
> >     at
> >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
> >     at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
> >     at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataNoWatch(ZKAssign.java:904)
> >     at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:283)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1905)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> >     at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >     at java.lang.Thread.run(Thread.java:662)
> > 2013-08-23 18:56:36,877 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> create/force
> > node into OFFLINE state before completing assignment but failed to do so
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> > state=OFFLINE, ts=1377264396862, server=null
> > 2013-08-23 18:56:36,877 WARN
> > org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> create/force
> > node into OFFLINE state before completing assignment but failed to do so
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> > state=OFFLINE, ts=1377264396813, server=null
> > 2013-08-23 18:56:36,882 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_FAILED_OPEN,
> server=vamshi_RS,60020,1377263792053,
> > region=d476f8442ce31de90b60080b74daf47f
> > 2013-08-23 18:56:36,882 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> > destination server is vamshi,60020,1377263789273
> > 2013-08-23 18:56:36,882 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> > plan was found (or we are ignoring an existing plan) for
> >
> >
> scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.
> > so generated a random one;
> >
> >
> hri=scores,\x00\x00\x00\x00\x00\x17\xC0i,1377264302391.d476f8442ce31de90b60080b74daf47f.,
> > src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1)
> available
> > servers
> > 2013-08-23 18:56:36,882 ERROR
> > org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> > [ClosedRegionHandler-vamshi,60000,1377263788019-38] because the executor
> is
> > missing. Is this process shutting down?
> > 2013-08-23 18:56:36,906 DEBUG
> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
> > org.apache.hadoop.hbase.catalog.CatalogTracker@451415c8
> > 2013-08-23 18:56:36,906 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
> > vamshi,60000,1377263788019.timeoutMonitor exiting
> > 2013-08-23 18:56:36,906 INFO
> > org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
> > vamshi,60000,1377263788019.timerUpdater exiting
> > 2013-08-23 18:56:36,907 INFO
> > org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
> > vamshi,60000,1377263788019.splitLogManagerTimeoutMonitor exiting
> > 2013-08-23 18:56:36,910 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_FAILED_OPEN,
> server=vamshi_RS,60020,1377263792053,
> > region=05e30711673614f6b41a364c76f3f05f
> > 2013-08-23 18:56:36,911 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
> > for
> >
> >
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> > destination server is vamshi,60020,1377263789273
> > 2013-08-23 18:56:36,911 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: No previous transition
> > plan was found (or we are ignoring an existing plan) for
> >
> >
> scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.
> > so generated a random one;
> >
> >
> hri=scores,\x00\x00\x00\x00\x00\x10\xC1\xF4,1377264146360.05e30711673614f6b41a364c76f3f05f.,
> > src=, dest=vamshi,60020,1377263789273; 2 (online=2, available=1)
> available
> > servers
> > 2013-08-23 18:56:36,911 ERROR
> > org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
> > [ClosedRegionHandler-vamshi,60000,1377263788019-39] because the executor
> is
> > missing. Is this process shutting down?
> > 2013-08-23 18:56:36,912 WARN
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient
> > ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for
> > /hbase/unassigned/d476f8442ce31de90b60080b74daf47f
> > 2013-08-23 18:56:36,912 INFO org.apache.hadoop.hbase.util.RetryCounter:
> > Sleeping 2000ms before retry #1...
> > 2013-08-23 18:56:36,914 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x140ab519b0f0000 closed
> > 2013-08-23 18:56:36,914 INFO org.apache.hadoop.hbase.master.HMaster:
> > HMaster main thread exiting
> > 2013-08-23 18:56:36,914 ERROR
> > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> > java.lang.RuntimeException: HMaster Aborted
> >     at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
> >     at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
> >     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >     at
> >
> >
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> >     at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)
> >
> >
> >
> > My hbase-site.xml :
> >
> > <configuration>
> >     <property>
> >         <name>hbase.rootdir</name>
> >
> >     <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
> >     </property>
> >
> >     <property>
> >         <name>hbase.cluster.distributed</name>
> >         <value>true</value>
> >     </property>
> >     <property>
> >         <name>hbase.master</name>
> >         <value>vamshi</value>
> >     </property>
> >     <property>
> >         <name>hbase.zookeeper.property.clientPort</name>
> >         <value>2181</value>
> >     </property>
> >
> >
> >    <property>
> >         <name>hbase.hregion.max.filesize</name>
> >         <value>10485760</value>
> >     </property>
> >
> >
> >
> >     <property>
> >         <name>hbase.zookeeper.quorum</name>
> >         <value>vamshi</value>
> >     </property>
> >     <property>
> >         <name>hbase.zookeeper.property.dataDir</name>
> >         <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
> >     </property>
> >
> > <property>
> >     <name>hbase.zookeeper.property.maxClientCnxns</name>
> >     <value>1024</value>
> >   </property>
> >
> > <property>
> >     <name>hbase.coprocessor.user.region.classes</name>
> >     <value>com.bil.coproc.ColumnAggregationEndpoint</value>
> >   </property>
> > </configuration>
> >
> >
> >
> >
> > On Fri, Aug 23, 2013 at 7:00 PM, Frank Chow <zhoushuaifeng@gmail.com>
> > wrote:
> >
> > > Hi,
> > > You may should check if the compact is on. If data size in a region is
> > max
> > > than the limition, region will split and balance after a major
> > > compaction(Usually occur automatically).
> > > You can manually by run the compact operaction by the shell commond:
> > > compact <tableName>, or major_compact <tableName>
> > >
> > >
> > >
> > >
> > > Frank Chow
> >
> >
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
> >
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message