hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Whitney Sorenson <wsoren...@hubspot.com>
Subject Re: Heavy Writes Block Reads to RegionServer
Date Wed, 16 May 2012 16:10:11 GMT
Thanks for the response Ron,

If I'm reading that right, the maximum concurrent RPC requests that an
HBase region server can serve is 10, by default, and adjustable via
this property?

This would have some interesting implications for us IRT to MapReduce jobs.

On Tue, May 15, 2012 at 8:02 PM, Buckley,Ron <buckleyr@oclc.org> wrote:
> Bryan,
> We ran into that same condition here last week doing pretty much the
> same thing. Maybe you're hitting it too.
> We found that the region server wasn't blocked all the time, but when it
> was blocked  there was a associated log message ("INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC
> Server handler 1 on 9009' on region
> WorldcatXmlFragments,7233395,1335296580834.c7ecef95084b8babca50f405afbb1
> 77f.: memstore size 256.1m is >= than blocking 256.0m size") in our
> logs.     We had the same IPC Server Info that you described too.
> It turned out, when that region would block, that the mappers were
> taking all the rpc listener slots into the region server (visible from
> the region server directly in the "Show Active RPC Calls"). Since the
> mappers had all the slots, our gets for other tables would wait just to
> get into the region server.
> The rpc handler count is configurable, see:
> http://hbase.apache.org/book/config.files.html#hbase.regionserver.handle
> r.count
> We upped our value for that from it's default of 10 to 50 (more than the
> number of mappers we were running) and the problem went away.
> I'm sure there's an art to setting that value, we think 50 will work
> well for us. YMMV.
> -----Original Message-----
> From: Bryan Beaudreault [mailto:bbeaudreault@hubspot.com]
> Sent: Tuesday, May 15, 2012 5:39 PM
> To: user@hbase.apache.org
> Subject: Heavy Writes Block Reads to RegionServer
> We are running a job that does heavy writes into a new table.  The table
> is
> not pre-split so it has 1 region.  I know this is not recommended; we
> were
> doing it partially to test this particular case.
> Here's what we're seeing:
>   1. Reads are entirely blocked.  No reads to any region on that server
>   make it through.
>   2. Writes are insanely slow.  Some writes appear to be taking over 10
>   minutes.
>   3. All of the box's resources are quiet:  Around < 20% CPU usage,
> plenty
>   of memory to spare, iostat looked normal
>   4. ngrep showed only writes coming through.  no reads
>   5. The logs showed lots of WARN org.apache.hadoop.ipc.HBaseServer:
>   Server Responder, call
>   multi(org.apache.hadoop.hbase.client.MultiAction@714cd947) from
> output error
> Any ideas what's up?  Is there some sort of global lock that might halt
> reads during heavy writes?  Anything else we can look for during this?
> We
> can rerun the job to reproduce this, as this is a test cluster which can
> afford to be brought down.

View raw message