Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of dejan.menges@gmail.com
 designates 209.85.215.48 as permitted sender)
MIME-Version: 1.0
References: 
 <CAEf6Z5+bsh7AN8-36zGOwzhdb3pG4ueLyMpDHEZo3s45P9Zh_g@mail.gmail.com>
 <CALPvCiAeHnV=cBFtC8nwpVYHDc5UutZBr34k-WHpNgPLS+Wi-Q@mail.gmail.com>
In-Reply-To: 
 <CALPvCiAeHnV=cBFtC8nwpVYHDc5UutZBr34k-WHpNgPLS+Wi-Q@mail.gmail.com>
From: Dejan Menges <dejan.menges@gmail.com>
Date: Mon, 13 Apr 2015 18:08:38 +0000
Message-ID: 
 <CAEf6Z5+VidWFOASvxN8_=nsdO4jkhgB97S_87TF5zxKh-3MUmg@mail.gmail.com>
Subject: Re: How server gets into failed servers list?
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11341aeee0d46905139f00d2

--001a11341aeee0d46905139f00d2
Content-Type: text/plain; charset=UTF-8

Hi Esteban,

Thanks for pointing to that, will try to collect all logs tomorrow and to
take deeper look and post here specific errors. Yes, good news are that all
logs are preserved.

Thanks a lot,
Dejan

On Mon, Apr 13, 2015 at 8:01 PM Esteban Gutierrez <esteban@cloudera.com>
wrote:

> Hi Dejan,
>
> Do you have the logs from any of those failed region servers? Usually in
> case of a critical failure the RS will shutdown itself or if the RS "hangs"
> for a long time and the master will start processing the expiration of that
> RS and reject the RS if it tries to reconnect with a YouAreDeadException.
> The HBase master and RS logs for sure will tell us.
>
> thanks,
> esteban.
>
>
> --
> Cloudera, Inc.
>
>
> On Mon, Apr 13, 2015 at 1:11 AM, Dejan Menges <dejan.menges@gmail.com>
> wrote:
>
> > Hi,
> >
> > We had some issues recently with HDFS - hardware issue with one of the
> > nodes, nodes died, HDFS recovered, but we figured out that something is
> > wrong with HBase. Checking HMaster log, we saw that bunch of our region
> > servers got to the famous failed servers list, and it was going on and on
> > until we restarted every one of them.
> >
> > Are we doing something wrong? Is it possible somehow to tune this out,
> once
> > the server is in this list to forget about it or something?
> >
> > Main question - how HMaster decides at all that server should be in the
> > failed server list, and what does this means exactly?
> >
> > Was looking into HBase book, googling, but beside some generic answers
> > wasn't able to find anything more internal.
> >
> > Thanks in advance!
> >
>

--001a11341aeee0d46905139f00d2--