Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jgray@facebook.com designates
 66.220.144.136 as permitted sender)
From: Jonathan Gray <jgray@facebook.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: RE: Sanity date time check when a region server joins the cluster
Thread-Topic: Sanity date time check when a region server joins the cluster
Thread-Index: AQHLdsEuAV6geKgzy0KhC0EENrL0mJNXCweA//+dTrA=
Date: Thu, 28 Oct 2010 18:08:13 +0000
Message-ID: 
 <5A76F6CE309AD049AAF9A039A39242820F0B9BF1@sc-mbx04.TheFacebook.com>
References: <4CC9AB49.2080506@qualtrics.com>
 <AANLkTimtjgwPeX8taLkxWNNFLX6g3HGPCqjoxvQRYOsp@mail.gmail.com>
In-Reply-To: <AANLkTimtjgwPeX8taLkxWNNFLX6g3HGPCqjoxvQRYOsp@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I was discussing this exact issue this morning.  Ran into a problem where m=
aster was timing out a region in transition because the RS was 5 minutes be=
hind the master.

I like the idea of the RS sending it's timestamp on startup and if it is ou=
tside a certain threshold, the master throws it a ClockOutOfSync-like excep=
tion and the RS goes down.

Please do file a jira, Jeff.  Or let me know and I can do it.

JG

> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-
> Daniel Cryans
> Sent: Thursday, October 28, 2010 10:00 AM
> To: user@hbase.apache.org
> Subject: Re: Sanity date time check when a region server joins the
> cluster
>=20
> That could be done easily when the server checks in by looking at the
> given start code. In ServerManager we already do:
>=20
>     HServerInfo info =3D new HServerInfo(serverInfo);
>     checkIsDead(info.getServerName(), "STARTUP");
>     checkAlreadySameHostPort(info);
>     recordNewServer(info, false, null);
>=20
> A new check in there would fit nicely. Can you open a jira Jeff?
>=20
> Thx!
>=20
> J-D
>=20
> On Thu, Oct 28, 2010 at 9:56 AM, Jeff Whiting <jeffw@qualtrics.com>
> wrote:
> > We recently had a problem where one of our machines in the cluster
> had a
> > time that was 6 hours behind the other ones (ntp was supposed to be
> setup on
> > that machine but wasn't). =A0We subsequently restarted our cluster and
> the
> > '-ROOT-' table was assigned to that machine. =A0The problem was that
> when it
> > tried to update the value (info:server) for who was holding the
> '.META.'
> > table the value wasn't updating and stayed set as the previous
> machine. I'm
> > pretty sure the problem was the timestamp for the new server was
> older than
> > the timestamp for the previous server preventing the value from
> updating
> > correctly. =A0Having the incorrect info:server in the ROOT table
> basically
> > made the cluster unusable.
> >
> > So my question is, would it make sense to have a sanity time check
> when a
> > region server joins the cluster? =A0Basically when the region server
> joins it
> > would sent its current time and the master would check that time
> against its
> > current time and if difference is too large then it would prevent the
> region
> > server from joining. =A0I know this is basic server configuration stuff
> but
> > because of human error these things happen and seem like they can
> cause
> > major problems for the cluster if the servers times aren't
> synchronized.
> >
> > ~Jeff
> >
> > --
> >
> > Jeff Whiting
> > Qualtrics Senior Software Engineer
> > jeffw@qualtrics.com
> >
> >