hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Hbase 0.19 failed to start: exceeds the limit of concurrent xcievers 3000
Date Wed, 28 Jan 2009 17:07:14 GMT
Genady,

There seems to be other stuff going on on your cluster, is there any
swapping?

Also, setting a very high number for the file size isn't something I
experienced. The higher we went is 512MB (the default is 258MB). Using this
number would already drop your number of regions by half, that may be
enough. You could also buy more machines ;)

J-D

On Wed, Jan 28, 2009 at 11:54 AM, Genady <genadyg@exelate.com> wrote:

> Jean-Daniel,
>
>
>
> Eventually a setting the xceivers to 3500 and Hadoop heap size to 1.5GB
> have
> brought the HBase server alive, but MR task now working 10 times slower, it
> seems that mostly because of region server is overloaded(by CPU), as far as
> I understand the main cause is too big number of regions on each region
> server(~1400), splitting a disk space on the server 1Tb / 100( number you
> suggested) make 10Gb, is it sounds a reasonable file size for region from
> your experience?
>
>
>
> Gennady
>
>
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Wednesday, January 28, 2009 5:42 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Hbase 0.19 failed to start: exceeds the limit of concurrent
> xcievers 3000
>
>
>
> Genady,
>
>
>
> Answers inline.
>
>
>
> J-D
>
>
>
> On Wed, Jan 28, 2009 at 10:33 AM, Genady <genadyg@exelate.com> wrote:
>
>
>
> >  Thanks for your quick response Jean-Daniel,
>
> >
>
> >
>
> >
>
> > By setting a bigger heap size you've probably meant a Hadoop heap size,
> the
>
> > problem is that it start to cause an original problem, first Hadoop heap
> was
>
> > 2Gb and it caused
>
> >
>
> Try 1.5GB
>
>
>
>
>
> > "java.lang.OutOfMemoryError: unable to create new native
> thread<http://www.egilh.com/blog/archive/2006/06/09/2811.aspx>",
>
> > probably as result of increasing xceivers thread number,  so I set number
>
> > Hadoop heap to 1Gb and xceivers to 3400, but on some level region servers
>
> > are stop to respond again, now it's 481 regions on the each region
> server(
>
> > assuming that each server contains a different regions).
>
> >
>
>
>
> You have 481 regions on each region server?
>
>
>
>
>
> > I'll try to increase a max file size, by the way it will have an
> influence
>
> > on the new regions or only old one?
>
> >
>
>
>
> Only the new ones, unless you recreate your table.
>
>
>
>
>
> >
>
> >
>
> > By the way is there some limitation of regions for server to handle? (
> CPU
>
> > 2Hz x4, RAM 8GB).
>
> >
>
>
>
> Depends on your schema, with 2 families it's easy to handle but the low
>
> number of nodes makes it that your region servers are overloaded. Our load
>
> average is more like 80-100 regions per node but we have 15 of them.
>
>
>
>
>
> >
>
> >
>
> > Regarding Hbase schema, we have only one table that have a two column
>
> > families.
>
> >
>
> >
>
> >
>
> > Thanks again,
>
> >
>
> >
>
> >
>
> > Gennady
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > *-----Original Message-----
>
> > From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
>
> > Jean-Daniel Cryans
>
> > Sent: Wednesday, January 28, 2009 4:02 PM
>
> > To: hbase-user@hadoop.apache.org
>
> > Subject: Re: Hbase 0.19 failed to start: exceeds the limit of concurrent
>
> > xcievers 3000*
>
> >
>
> > * *
>
> >
>
> > *Genady,*
>
> >
>
> > * *
>
> >
>
> > *Some comments.*
>
> >
>
> > * *
>
> >
>
> > *Try a bigger heap size, something like 2GB.*
>
> >
>
> > * *
>
> >
>
> > *Set the handler count to 4, that thing eats a lot of memory.*
>
> >
>
> > * *
>
> >
>
> > *430 regions for 3 nodes is really a lot and HBase currently opens a lot
>
> > of*
>
> >
>
> > *files. Try increasing the max file size of your tables so that it takes*
>
> >
>
> > *longer to split and therefore have less regions. Search the list on how
>
> > to*
>
> >
>
> > *do that.*
>
> >
>
> > * *
>
> >
>
> > *Also do you happen to have a lot of families in your tables?*
>
> >
>
> > * *
>
> >
>
> > *Thx,*
>
> >
>
> > * *
>
> >
>
> > *J-D*
>
> >
>
> > * *
>
> >
>
> > *On Wed, Jan 28, 2009 at 8:53 AM, Genady <genadyg@exelate.com> wrote:*
>
> >
>
> > * *
>
> >
>
> > *> Hi,*
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> It seems that HBase 0.19 on Hadoop 0.19 fail to start because of
>
> > exceeding*
>
> >
>
> > *> limit of concurrent xceivers( in hadoop datanode logs), which is
>
> > currently*
>
> >
>
> > *> 3000, setting more than 3000 xceivers is causing JVM out of memory*
>
> >
>
> > *> exception, is there is something wrong with configuration parameters
> of
>
> > *
>
> >
>
> > *> cluster( three nodes, 430 regions,Hadoop heap size is default - 1GB)?*
>
> >
>
> > *> Additional parameters in hbase configuration are:*
>
> >
>
> > *> *
>
> >
>
> > *> dfs.datanode.handler.count = 6,*
>
> >
>
> > *> *
>
> >
>
> > *> dfs.datanode.socket.write.timeout=0*
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> java.io.IOException: xceiverCount 3001 exceeds the limit of
> concurrent*
>
> >
>
> > *> xcievers 3000*
>
> >
>
> > *> *
>
> >
>
> > *>        at*
>
> >
>
> > *>
>
> >
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:87)
>
> > *
>
> >
>
> > *> *
>
> >
>
> > *>        at java.lang.Thread.run(Thread.java:619)*
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> Any help is very appreciated,*
>
> >
>
> > *> *
>
> >
>
> > *> Genady*
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
> > *> *
>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message