hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Beaudreault <bbeaudrea...@hubspot.com>
Subject Heavy Writes Block Reads to RegionServer
Date Tue, 15 May 2012 21:38:58 GMT
We are running a job that does heavy writes into a new table.  The table is
not pre-split so it has 1 region.  I know this is not recommended; we were
doing it partially to test this particular case.

Here's what we're seeing:

   1. Reads are entirely blocked.  No reads to any region on that server
   make it through.
   2. Writes are insanely slow.  Some writes appear to be taking over 10
   3. All of the box's resources are quiet:  Around < 20% CPU usage, plenty
   of memory to spare, iostat looked normal
   4. ngrep showed only writes coming through.  no reads
   5. The logs showed lots of WARN org.apache.hadoop.ipc.HBaseServer: IPC
   Server Responder, call
   multi(org.apache.hadoop.hbase.client.MultiAction@714cd947) from output error

Any ideas what's up?  Is there some sort of global lock that might halt
reads during heavy writes?  Anything else we can look for during this?  We
can rerun the job to reproduce this, as this is a test cluster which can
afford to be brought down.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message