Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CAKYcnAvNmzR56Hbupb_GK4BZv+qcrOx4ayYDdd0+5OCV096H0Q@mail.gmail.com>
References: <CAGgZ7qBG=qVa3wbhrs=jTsp0KKcZVnKiVKmwgcXmePt+HJ9W6A@mail.gmail.com>
 <CAKYcnAvNmzR56Hbupb_GK4BZv+qcrOx4ayYDdd0+5OCV096H0Q@mail.gmail.com>
From: Alexander Ilyin <alexander@weborama.com>
Date: Mon, 17 Oct 2016 17:56:59 +0200
Message-ID: <CAGgZ7qBZprCpf6iD5-LpeWbHUKiE8Bz6N3YwsCS8f5Gqy+30kQ@mail.gmail.com>
Subject: Re: HBase restart without region reassigning
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=94eb2c05460454d514053f11a081
archived-at: Mon, 17 Oct 2016 15:57:09 -0000

--94eb2c05460454d514053f11a081
Content-Type: text/plain; charset=UTF-8

Hi Dima,

These are instances in the cloud and we're using Consul for name
resolution. Regarding network settings, your question is a bit broad...
Which settings would you recommend to check first?

On Mon, Oct 17, 2016 at 5:28 PM, Dima Spivak <dimaspivak@apache.org> wrote:

> Hey Alexander,
>
> Could something be amiss in your network settings? Seeing phantom datanodes
> could be tripping things up. Are these physical machines or instances in
> the cloud?
>
> On Monday, October 17, 2016, Alexander Ilyin <alexander@weborama.com>
> wrote:
>
> > Hi,
> >
> > We have a 7-node HBase cluster (version 1.1.2) and we change some of its
> > settings from time to time which requires a restart. The problem is that
> > every time after the restart load balancer reassigns the regions making
> > data locality low.
> >
> > To address this issue we tried the settings described here:
> > https://issues.apache.org/jira/browse/HBASE-6389,
> > "hbase.master.wait.on.regionservers.interval" in particular. We tried it
> > two times in slightly different ways but neither of them worked. First
> time
> > we did a rolling restart (master, then each of datanodes) and we saw 14
> > datanodes instead of 7 in Master UI. Half of them had the regions on it
> > while the other half was empty. We restarted master only and we got 7
> empty
> > datanodes in Master UI. After that we rollbacked the setting.
> >
> > Second time we restarted master and datanodes at the same time but master
> > failed to read meta table, moved it to a different datanode and
> reassigned
> > the regions again.
> >
> > Please advise on how to use hbase.master.wait.on.regionservers.*
> settings
> > properly. Launching major compactions for all the tables after each
> config
> > change seems to be an overkill. Attaching Master server logs with
> relevant
> > lines for two attempts mentioned above.
> >
> > Thanks in advance.
> >
>
>
> --
> -Dima
>

--94eb2c05460454d514053f11a081--