hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Strange issue when DataNode goes down
Date Mon, 16 Mar 2015 13:54:49 GMT
Can you post some redacted log files from the period after the data node
failed, up to the restart?

-- 
Sean
On Mar 16, 2015 8:41 AM, "Dejan Menges" <dejan.menges@gmail.com> wrote:

> Hi All,
>
> We have a strange issue with HBase performance (overall cluster
> performance) in case one of datanodes in the cluster unexpectedly goes
> down.
>
> So scenario is like follows:
> - Cluster works fine, it's stable.
> - One DataNode unexpectedly goes down (PSU issue, network issue, anything)
> - Whole HBase cluster goes down (performance becomes so bad that we have to
> restart all RegionServers to get it back to life).
>
> Most funny and latest issue that happened was that we added new node to the
> cluster (having 8 x 4T SATA disks) and we left just DataNode running on it
> to give it couple of days to get some data. At some point in time, due to
> hardware issue, server rebooted (twice during three hours) in moment when
> it had maybe 5% of data it would have in a couple of days. Nothing else
> beside DataNode was running, and once it went down, it affected literary
> everything, and restarting RegionServers in the end fixed it.
>
> We are using HBase 0.98.0 with Hadoop 2.4.0
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message