hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Greene <mgre...@hubspot.com>
Subject Region server instability with large number of deletes
Date Wed, 12 Dec 2012 19:16:50 GMT
Hi All,

I'm running HBase 0.92.1-cdh4.1.2.

I'm seeing a region server get into a state where they constantly emit
LeaseExceptions and ClosedChannelException from the IPC handler under load
that is several standard deviations above what we normally see.

One thing that's different about this cluster is we do a lot of deletes
against it. Ultimately all the data in one of our tables is deleted.

When the region server was in this state, the application was doing a lot
of scans, single gets, and then a series of batch deletes. While this cycle
was churning at a higher than average rate then what I normally see, it is
being done in a single thread. There are of course other processes reading
and writing to the cluster while this was going on but the observed load
from those were nothing out of the norm.

So I'm wondering if the large number of deletes via batch is a usage
profile that isn't optimized for hbase?

Other info:
-We just migrated to cdh4.1.2 a few days ago
-Currently do not have major compactions running
-Saw the same behavior in cdh3
-CPU and iowait were low during the time I observed these errors happening.

Thanks in advance,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message