Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
From: jeff saremi <jeffsaremi@hotmail.com>
To: hbase-user <user@hbase.apache.org>, "dev@hbase.apache.org"
	<dev@hbase.apache.org>
Subject: Re: What is Dead Region Servers and how to clear them up?
Thread-Topic: What is Dead Region Servers and how to clear them up?
Thread-Index: AQHS1KdN5y0CjJuNS0au+NbkeUqhZaID2KezgAAC3ACAAAKQpoABb6LNgAGIdESAAAL+AIAAEiX1
Date: Fri, 26 May 2017 18:03:55 +0000
Message-ID: <MWHPR14MB12932EAC4CCA7FB940058160C1FC0@MWHPR14MB1293.namprd14.prod.outlook.com>
References: <MWHPR14MB1293E3E44094B496B15DBA74C1FE0@MWHPR14MB1293.namprd14.prod.outlook.com>
 <MWHPR14MB129358FE9998730E272C9547C1FE0@MWHPR14MB1293.namprd14.prod.outlook.com>
 <CALte62yKEGHk5uw6Ad-M4=GTfCWa=DSvUVCqh8Zh06+wcA8ecg@mail.gmail.com>
 <MWHPR14MB129384D4B706F8073E015BBAC1FE0@MWHPR14MB1293.namprd14.prod.outlook.com>
 <MWHPR14MB1293EE06144BFCAC8EF4611BC1FF0@MWHPR14MB1293.namprd14.prod.outlook.com>
 <MWHPR14MB1293B463803EB4B4B87916F2C1FC0@MWHPR14MB1293.namprd14.prod.outlook.com>,<CAKYcnAtN+jzsq8G5n6zrMgSVxPzS0GYi1zDdP4eEw0BebnMLMg@mail.gmail.com>
In-Reply-To: <CAKYcnAtN+jzsq8G5n6zrMgSVxPzS0GYi1zDdP4eEw0BebnMLMg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [mpcZ4h0LVePnQjq34VG0aaodGJL6EOUZB54fx3NslULf4ayBA8WXsg==]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1;CO1NAM04HT017;24:gds4yM0pcxEblgiQ6ry5h/wunfmUyL844JJcDD4a1s6guUFFV2RyAO+OK6jKacDncH4Mm9hYXhHi6HSRAbC5qvLeT2TW6wtLAD5+FlcRp78=;7:OwkzENSMGY8li8szrwyS9GcwEaZOYWFkO04nudR9ugdwq5eeF2gBwCG/mhUweymZgJdliEgNLX48ePOjs8BD3jSNRuqWA8+ICKrdRj8ClRRocitqGLfhx7Gjk4mjQJdMqPE9t3vjOVEtb7zgWTfOD/+W11KeAgCz8Qx8h85MFqlRIED//SAATjvUWPdpUN87pfrofYvEArzvjKtsKpVKz4XHkhi947LDFMjz/XIXHAd9ey6k6+uJfMrNZcxkjpRrfEkpPtvEbDFmjTmmIp9kruHBP164cd6qSFzHDew5LWUzgTUgs+uqV8ygliQq8Lqb
x-incomingheadercount: 45
x-eopattributedmessage: 0
x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(7070007)(98901004);DIR:OUT;SFP:1901;SCL:1;SRVR:CO1NAM04HT017;H:MWHPR14MB1293.namprd14.prod.outlook.com;FPR:;SPF:None;LANG:en;
x-ms-traffictypediagnostic: CO1NAM04HT017:
x-ms-office365-filtering-correlation-id: 92e288f8-8294-4ad7-71f0-08d4a461937e
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322274)(1603101448)(1601125374)(1701031045);SRVR:CO1NAM04HT017;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700061)(100105000095)(100000701061)(100105300095)(100000702061)(100105100095)(444000031);SRVR:CO1NAM04HT017;BCL:0;PCL:0;RULEID:(100000800061)(100110000095)(100000801061)(100110300095)(100000802061)(100110100095)(100000803061)(100110400095)(100000804061)(100110200095)(100000805054)(100110500095);SRVR:CO1NAM04HT017;
x-forefront-prvs: 031996B7EF
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_MWHPR14MB12932EAC4CCA7FB940058160C1FC0MWHPR14MB1293namp_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 26 May 2017 18:03:55.9032
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1NAM04HT017
archived-at: Fri, 26 May 2017 18:04:10 -0000

--_000_MWHPR14MB12932EAC4CCA7FB940058160C1FC0MWHPR14MB1293namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Thank you for the GFY answer

And i guess to figure out how to fix these I can always go through the HBas=
e source code.


________________________________
From: Dima Spivak <dimaspivak@apache.org>
Sent: Friday, May 26, 2017 9:58:00 AM
To: hbase-user
Subject: Re: What is Dead Region Servers and how to clear them up?

Sending this back to the user mailing list.

RegionServers can die for many reasons. Looking at your RegionServer log
files should give hints as to why it's happening.


-Dima

On Fri, May 26, 2017 at 9:48 AM, jeff saremi <jeffsaremi@hotmail.com> wrote=
:

> I had posted this to the user mailing list and I have not got any direct
> answer to my question.
>
> Where do dead RS's come from and how can they be cleaned up? Someone in
> the midst of developers should know this.
>
> thanks
>
> Jeff
>
> ________________________________
> From: jeff saremi <jeffsaremi@hotmail.com>
> Sent: Thursday, May 25, 2017 10:23:17 AM
> To: user@hbase.apache.org
> Subject: Re: What is Dead Region Servers and how to clear them up?
>
> I'm still looking to get hints on how to remove the dead regions. thanks
>
> ________________________________
> From: jeff saremi <jeffsaremi@hotmail.com>
> Sent: Wednesday, May 24, 2017 12:27:06 PM
> To: user@hbase.apache.org
> Subject: Re: What is Dead Region Servers and how to clear them up?
>
> i'm trying to eliminate the dead region servers.
>
> ________________________________
> From: Ted Yu <yuzhihong@gmail.com>
> Sent: Wednesday, May 24, 2017 12:17:40 PM
> To: user@hbase.apache.org
> Subject: Re: What is Dead Region Servers and how to clear them up?
>
> bq. running hbck (many times
>
> Can you describe the specific inconsistencies you were trying to resolve =
?
> Depending on the inconsistencies, advice can be given on the best known
> hbck command arguments to use.
>
> Feel free to pastebin master log if needed.
>
> On Wed, May 24, 2017 at 12:10 PM, jeff saremi <jeffsaremi@hotmail.com>
> wrote:
>
> > these are the things I have done so far:
> >
> >
> > - restarting master (few times)
> >
> > - running hbck (many times; this tool does not seem to be doing anythin=
g
> > at all)
> >
> > - checking the list of region servers in ZK (none of the dead ones are
> > listed here)
> >
> > - checking the WALs under <hbase_hdfs>/WALs. Out of 11 dead ones only 3
> > are listed here with "-splitting" at the end of their names and they
> > contain one single file like: 1493846660401..meta.1493922323600.meta
> >
> >
> >
> >
> > ________________________________
> > From: jeff saremi <jeffsaremi@hotmail.com>
> > Sent: Wednesday, May 24, 2017 9:04:11 AM
> > To: user@hbase.apache.org
> > Subject: What is Dead Region Servers and how to clear them up?
> >
> > Apparently having dead region servers is so common that a section of th=
e
> > master console is dedicated to that?
> > How can we clean this up (preferably in an automated fashion)? Why isn'=
t
> > this being done by HBase automatically?
> >
> >
> > thanks
> >
>

--_000_MWHPR14MB12932EAC4CCA7FB940058160C1FC0MWHPR14MB1293namp_--