hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dejan Menges <dejan.men...@gmail.com>
Subject Strange issue when DataNode goes down
Date Mon, 16 Mar 2015 13:40:51 GMT
Hi All,

We have a strange issue with HBase performance (overall cluster
performance) in case one of datanodes in the cluster unexpectedly goes
down.

So scenario is like follows:
- Cluster works fine, it's stable.
- One DataNode unexpectedly goes down (PSU issue, network issue, anything)
- Whole HBase cluster goes down (performance becomes so bad that we have to
restart all RegionServers to get it back to life).

Most funny and latest issue that happened was that we added new node to the
cluster (having 8 x 4T SATA disks) and we left just DataNode running on it
to give it couple of days to get some data. At some point in time, due to
hardware issue, server rebooted (twice during three hours) in moment when
it had maybe 5% of data it would have in a couple of days. Nothing else
beside DataNode was running, and once it went down, it affected literary
everything, and restarting RegionServers in the end fixed it.

We are using HBase 0.98.0 with Hadoop 2.4.0

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message