Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Content-type: text/plain; charset=utf-8
MIME-version: 1.0 (Mac OS X Mail 8.2 \(2098\))
Subject: Re: HBase strange behaviour
From: Akmal Abbasov <akmal.abbasov@icloud.com>
In-reply-to: 
 <CALte62ym6nYOQ-WnDU82z8wtUD_0RR_r0LsE3=E2Lv3UN7Kx1Q@mail.gmail.com>
Date: Mon, 06 Jul 2015 17:49:54 +0200
Content-transfer-encoding: quoted-printable
Message-id: <4D642ED6-D32F-4B94-AF28-BC9AFA5DDBB2@icloud.com>
References: <3B59357E-5CD8-4918-97A7-0F65142A2067@icloud.com>
 <CALte62ym6nYOQ-WnDU82z8wtUD_0RR_r0LsE3=E2Lv3UN7Kx1Q@mail.gmail.com>
To: user@hbase.apache.org

> What error(s) did you get when trying to restart the region server ? =
Have
> you checked its log files ?
it was a VM, and I was not able to access it any more, I can=E2=80=99t =
login to it. Restarting several times didn=E2=80=99t helped.


> Can you check master log around this time ? If there was region in
> transition, balancer wouldn't balance.
I have a lot of this=20
2015-07-06 15:15:39,918 INFO  [snapshot-log-cleaner-cache-refresher] =
util.FSVisitor: No logs under =
directory:hdfs://test/hbase/.hbase-snapshot/table1-snapshot-31.05.2015_18.=
14/WALs
2015-07-06 15:15:39,918 INFO  [snapshot-log-cleaner-cache-refresher] =
util.FSVisitor: No logs under =
directory:hdfs://test/hbase/.hbase-snapshot/table1-snapshot-31.05.2015_19.=
14/WALs
2015-07-06 15:15:39,921 INFO  [snapshot-log-cleaner-cache-refresher] =
util.FSVisitor: No logs under =
directory:hdfs://test/hbase/.hbase-snapshot/table1-snapshot-31.05.2015_20.=
13/WALs
2015-07-06 15:15:39,925 INFO  [snapshot-log-cleaner-cache-refresher] =
util.FSVisitor: No logs under =
directory:hdfs://test/hbase/.hbase-snapshot/table1-snapshot-31.05.2015_21.=
14/WALs
2015-07-06 15:15:39,926 INFO  [snapshot-log-cleaner-cache-refresher] =
util.FSVisitor: No logs under =
directory:hdfs://test/hbase/.hbase-snapshot/table1-snapshot-31.05.2015_22.=
14/WALs
2015-07-06 15:15:39,927 INFO  [snapshot-log-cleaner-cache-refresher] =
util.FSVisitor: No logs under =
directory:hdfs://test/hbase/.hbase-snapshot/table1-snapshot-31.05.2015_23.=
14/WALs
2015-07-06 15:15:39,928 INFO  [snapshot-log-cleaner-cache-refresher] =
util.FSVisitor: No logs under =
directory:hdfs://test/hbase/.hbase-snapshot/testsnap/WALs
2015-07-06 15:15:47,324 INFO  [FifoRpcScheduler.handler1-thread-18] =
master.HMaster: Client=3Dhadoop//10.32.0.140 set balanceSwitch=3Dfalse
2015-07-06 15:23:31,265 DEBUG [master:hbase-m2:60000.oldLogCleaner] =
master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: =
hbase-rs1%2C60020%2C1436189457794.1436190023718
2015-07-06 15:23:31,504 DEBUG [master:hbase-m2:60000.oldLogCleaner] =
master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: =
hbase-rs1%2C60020%2C1436189457794.1436193624562
2015-07-06 15:32:49,382 INFO  [FifoRpcScheduler.handler1-thread-14] =
master.HMaster: Client=3Dhadoop//10.32.0.156 set balanceSwitch=3Dfalse
2015-07-06 15:32:56,936 INFO  [FifoRpcScheduler.handler1-thread-1] =
master.HMaster: Client=3Dhadoop//10.32.0.156 set balanceSwitch=3Dfalse

Thank you.

> On 06 Jul 2015, at 17:37, Ted Yu <yuzhihong@gmail.com> wrote:
>=20
> bq. I had to delete and recreate it
>=20
> What error(s) did you get when trying to restart the region server ? =
Have
> you checked its log files ?
>=20
> bq. start balancer manually, but it returned false
>=20
> Can you check master log around this time ? If there was region in
> transition, balancer wouldn't balance.
>=20
> Cheers
>=20
> On Mon, Jul 6, 2015 at 8:29 AM, Akmal Abbasov =
<akmal.abbasov@icloud.com>
> wrote:
>=20
>> Hi all,
>> I have a strange behaviour in my HBase cluster. I have 5 rs and 2 =
masters.
>> One of the rs stopped working, restart didn=E2=80=99t worked, and I =
had to delete
>> and recreate it.
>> But when this rs have stopped, the cluster also stopped functioning.
>> There were a lot of inconsistencies. When I recreated the rs with =
disks of
>> the previous one, cluster started working.
>> But now, only 3 rs host the regions, other 2 have 0 regions.
>> I=E2=80=99ve tried to start balancer manually, but it returned false?
>> Any idea?
>>=20
>> I am using hbase hbase-0.98.7-hadoop2.
>> Thank you.
>>=20
>> Kind regards,
>> Akmal Abbasov
>>=20
>>=20