Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 27 Oct 2016 01:15:58 +0000 (UTC)
From: "Guanghao Zhang (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13015408.1477487257000.95720.1477530958923@Atlassian.JIRA>
In-Reply-To: <JIRA.13015408.1477487257000@Atlassian.JIRA>
References: <JIRA.13015408.1477487257000@Atlassian.JIRA> <JIRA.13015408.1477487257763@arcas>
Subject: [jira] [Comment Edited] (HBASE-16947) Some improvements for
 DumpReplicationQueues tool
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 27 Oct 2016 01:16:01 -0000


    [ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610262#comment-15610262 ] 

Guanghao Zhang edited comment on HBASE-16947 at 10/27/16 1:15 AM:
------------------------------------------------------------------

A example for our cluster. Skip the host:port info.
{code}
Found 4 deleted queues, run hbck -fixReplication in order to remove the deleted replication queues
    "host","port",1476784747260/70-"host","port",,1476779010928
    "host","port",1476784747260/70-"host","port",1470421093464-"host","port",1476773708939-"host","port",1476779010928
    "host","port",1476784763605/70-"host","port",1470418208092-"host","port",1476773709589
    "host","port",1476784763605/70-"host","port",1476773709589
Found 2 dead regionservers, restart one regionserver to transfer the queues of dead regionservers
    "host","port",1476784747260
    "host","port",1476784763605
Dumping all peers's number of WALs in replication queue
    PeerId: 80 , sizeOfLogQueue: 1352
    PeerId: 30 , sizeOfLogQueue: 202
    PeerId: 20 , sizeOfLogQueue: 202
    PeerId: 10 , sizeOfLogQueue: 202
    PeerId: 70 , sizeOfLogQueue: 31724
    PeerId: 17000 , sizeOfLogQueue: 775
    PeerId: 50 , sizeOfLogQueue: 4490
    Total size of WALs on HDFS: 61.6G
    ERROR: There are 120 WALs not found!!!
{code}


was (Author: zghaobac):
A example for our cluster. Skip the host:port info.
{format}
Found 4 deleted queues, run hbck -fixReplication in order to remove the deleted replication queues
    "host","port",1476784747260/70-"host","port",,1476779010928
    "host","port",1476784747260/70-"host","port",1470421093464-"host","port",1476773708939-"host","port",1476779010928
    "host","port",1476784763605/70-"host","port",1470418208092-"host","port",1476773709589
    "host","port",1476784763605/70-"host","port",1476773709589
Found 2 dead regionservers, restart one regionserver to transfer the queues of dead regionservers
    "host","port",1476784747260
    "host","port",1476784763605
Dumping all peers's number of WALs in replication queue
    PeerId: 80 , sizeOfLogQueue: 1352
    PeerId: 30 , sizeOfLogQueue: 202
    PeerId: 20 , sizeOfLogQueue: 202
    PeerId: 10 , sizeOfLogQueue: 202
    PeerId: 70 , sizeOfLogQueue: 31724
    PeerId: 17000 , sizeOfLogQueue: 775
    PeerId: 50 , sizeOfLogQueue: 4490
    Total size of WALs on HDFS: 61.6G
    ERROR: There are 120 WALs not found!!!
{format}

> Some improvements for DumpReplicationQueues tool
> ------------------------------------------------
>
>                 Key: HBASE-16947
>                 URL: https://issues.apache.org/jira/browse/HBASE-16947
>             Project: HBase
>          Issue Type: Improvement
>          Components: Replication
>    Affects Versions: 2.0.0, 1.4.0
>            Reporter: Guanghao Zhang
>            Assignee: Guanghao Zhang
>         Attachments: HBASE-16947.patch
>
>
> Recently we met too many replication WALs problem in our production cluster. We need the DumpReplicationQueues tool to analyze the replication queues info in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some improvements for it.
> 1. Show the dead regionservers under replication/rs znode. When there are too many WALs under znode, it can't be atomic transferred to new rs znode. So the dead rs znode will be leaved on zookeeper.
> 2. Make a summary about all the queues that belong to peer has been deleted. 
> 3. Aggregate all regionservers' size of replication queue. Now the regionserver report ReplicationLoad to master, but there were not a aggregate metrics for replication.
> 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not Found) need more time to dig.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)