hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Genady " <gena...@exelate.com>
Subject RE: Hbase 0.19 failed to start: exceeds the limit of concurrent xcievers 3000
Date Wed, 28 Jan 2009 16:54:34 GMT
Jean-Daniel,

 

Eventually a setting the xceivers to 3500 and Hadoop heap size to 1.5GB have
brought the HBase server alive, but MR task now working 10 times slower, it
seems that mostly because of region server is overloaded(by CPU), as far as
I understand the main cause is too big number of regions on each region
server(~1400), splitting a disk space on the server 1Tb / 100( number you
suggested) make 10Gb, is it sounds a reasonable file size for region from
your experience?

 

Gennady

 

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-Daniel Cryans
Sent: Wednesday, January 28, 2009 5:42 PM
To: hbase-user@hadoop.apache.org
Subject: Re: Hbase 0.19 failed to start: exceeds the limit of concurrent
xcievers 3000

 

Genady,

 

Answers inline.

 

J-D

 

On Wed, Jan 28, 2009 at 10:33 AM, Genady <genadyg@exelate.com> wrote:

 

>  Thanks for your quick response Jean-Daniel,

> 

> 

> 

> By setting a bigger heap size you've probably meant a Hadoop heap size,
the

> problem is that it start to cause an original problem, first Hadoop heap
was

> 2Gb and it caused

> 

Try 1.5GB

 

 

> "java.lang.OutOfMemoryError: unable to create new native
thread<http://www.egilh.com/blog/archive/2006/06/09/2811.aspx>",

> probably as result of increasing xceivers thread number,  so I set number

> Hadoop heap to 1Gb and xceivers to 3400, but on some level region servers

> are stop to respond again, now it's 481 regions on the each region server(

> assuming that each server contains a different regions).

> 

 

You have 481 regions on each region server?

 

 

> I'll try to increase a max file size, by the way it will have an influence

> on the new regions or only old one?

> 

 

Only the new ones, unless you recreate your table.

 

 

> 

> 

> By the way is there some limitation of regions for server to handle? ( CPU

> 2Hz x4, RAM 8GB).

> 

 

Depends on your schema, with 2 families it's easy to handle but the low

number of nodes makes it that your region servers are overloaded. Our load

average is more like 80-100 regions per node but we have 15 of them.

 

 

> 

> 

> Regarding Hbase schema, we have only one table that have a two column

> families.

> 

> 

> 

> Thanks again,

> 

> 

> 

> Gennady

> 

> 

> 

> 

> 

> *-----Original Message-----

> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of

> Jean-Daniel Cryans

> Sent: Wednesday, January 28, 2009 4:02 PM

> To: hbase-user@hadoop.apache.org

> Subject: Re: Hbase 0.19 failed to start: exceeds the limit of concurrent

> xcievers 3000*

> 

> * *

> 

> *Genady,*

> 

> * *

> 

> *Some comments.*

> 

> * *

> 

> *Try a bigger heap size, something like 2GB.*

> 

> * *

> 

> *Set the handler count to 4, that thing eats a lot of memory.*

> 

> * *

> 

> *430 regions for 3 nodes is really a lot and HBase currently opens a lot

> of*

> 

> *files. Try increasing the max file size of your tables so that it takes*

> 

> *longer to split and therefore have less regions. Search the list on how

> to*

> 

> *do that.*

> 

> * *

> 

> *Also do you happen to have a lot of families in your tables?*

> 

> * *

> 

> *Thx,*

> 

> * *

> 

> *J-D*

> 

> * *

> 

> *On Wed, Jan 28, 2009 at 8:53 AM, Genady <genadyg@exelate.com> wrote:*

> 

> * *

> 

> *> Hi,*

> 

> *> *

> 

> *> *

> 

> *> *

> 

> *> It seems that HBase 0.19 on Hadoop 0.19 fail to start because of

> exceeding*

> 

> *> limit of concurrent xceivers( in hadoop datanode logs), which is

> currently*

> 

> *> 3000, setting more than 3000 xceivers is causing JVM out of memory*

> 

> *> exception, is there is something wrong with configuration parameters of

> *

> 

> *> cluster( three nodes, 430 regions,Hadoop heap size is default - 1GB)?*

> 

> *> Additional parameters in hbase configuration are:*

> 

> *> *

> 

> *> dfs.datanode.handler.count = 6,*

> 

> *> *

> 

> *> dfs.datanode.socket.write.timeout=0*

> 

> *> *

> 

> *> *

> 

> *> *

> 

> *> java.io.IOException: xceiverCount 3001 exceeds the limit of concurrent*

> 

> *> xcievers 3000*

> 

> *> *

> 

> *>        at*

> 

> *>

>
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:87)

> *

> 

> *> *

> 

> *>        at java.lang.Thread.run(Thread.java:619)*

> 

> *> *

> 

> *> *

> 

> *> *

> 

> *> Any help is very appreciated,*

> 

> *> *

> 

> *> Genady*

> 

> *> *

> 

> *> *

> 

> *> *

> 

> *> *

> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message