hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11654) WAL Splitting dirs are not deleted after replay.
Date Sun, 03 Aug 2014 07:47:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083900#comment-14083900
] 

Hudson commented on HBASE-11654:
--------------------------------

SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #409 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/409/])
HBASE-11654 WAL Splitting dirs are not deleted after replay (Victor Xu) (apurtell: rev c5da1cfafa9e5881c96a0e8d3a1532e714d4198c)
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java


> WAL Splitting dirs are not deleted after replay.
> ------------------------------------------------
>
>                 Key: HBASE-11654
>                 URL: https://issues.apache.org/jira/browse/HBASE-11654
>             Project: HBase
>          Issue Type: Bug
>          Components: master, wal
>    Affects Versions: 0.98.4
>            Reporter: Victor Xu
>            Assignee: Victor Xu
>             Fix For: 0.98.5
>
>         Attachments: HBASE-11654.patch
>
>
> I build a small cluster (20 nodes, several hundred regions) with hbase-0.98.4. And I
found some splitting directories in /hbase/WALs/ today, which is very strange because those
logs should have been replayed and deleted. Even though the ZK nodes of the dead RS had been
deleted, these splitting directories still can cause a serious trouble for cluster restart.
It resplitted and replayed all the splitting directories every time I restart my cluster,
and cost a huge amount of time. Can't imagine what could happened if it's a cluster with hundreds
of nodes and tens of thousands of regions.
> Found 56 items
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:21 /hbase/WALs/hdpdev1.cm6.tbsite.net,60020,1406714828440-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev1.cm6.tbsite.net,60020,1406716991836-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:16 /hbase/WALs/hdpdev1.cm6.tbsite.net,60020,1406778815585
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:26 /hbase/WALs/hdpdev10.cm6.tbsite.net,60020,1406526862752-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev10.cm6.tbsite.net,60020,1406716933471-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:02 /hbase/WALs/hdpdev10.cm6.tbsite.net,60020,1406778815536
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:26 /hbase/WALs/hdpdev11.cm6.tbsite.net,60020,1406526862802-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev11.cm6.tbsite.net,60020,1406716992986-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:14 /hbase/WALs/hdpdev11.cm6.tbsite.net,60020,1406778815552
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:15 /hbase/WALs/hdpdev12.cm6.tbsite.net,60020,1406526862752-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev12.cm6.tbsite.net,60020,1406716992874-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:20 /hbase/WALs/hdpdev12.cm6.tbsite.net,60020,1406778816074
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:27 /hbase/WALs/hdpdev13.cm6.tbsite.net,60020,1406526862832-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev13.cm6.tbsite.net,60020,1406716992753-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:15 /hbase/WALs/hdpdev13.cm6.tbsite.net,60020,1406857929773
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:26 /hbase/WALs/hdpdev14.cm6.tbsite.net,60020,1406526862736-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev14.cm6.tbsite.net,60020,1406716992923-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:25 /hbase/WALs/hdpdev14.cm6.tbsite.net,60020,1406778815595
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:19 /hbase/WALs/hdpdev15.cm6.tbsite.net,60020,1406526862821-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev15.cm6.tbsite.net,60020,1406716993082-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:15 /hbase/WALs/hdpdev15.cm6.tbsite.net,60020,1406778815578
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:21 /hbase/WALs/hdpdev16.cm6.tbsite.net,60020,1406526862816-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev16.cm6.tbsite.net,60020,1406716992787-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:15 /hbase/WALs/hdpdev16.cm6.tbsite.net,60020,1406778816006
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev17.cm6.tbsite.net,60020,1406716992814-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:26 /hbase/WALs/hdpdev17.cm6.tbsite.net,60020,1406778815579
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev18.cm6.tbsite.net,60020,1406716993051-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:24 /hbase/WALs/hdpdev18.cm6.tbsite.net,60020,1406778815587
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:23 /hbase/WALs/hdpdev19.cm6.tbsite.net,60020,1406526862720-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev19.cm6.tbsite.net,60020,1406716992736-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:22 /hbase/WALs/hdpdev19.cm6.tbsite.net,60020,1406865567732
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:21 /hbase/WALs/hdpdev2.cm6.tbsite.net,60020,1406778815846
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:27 /hbase/WALs/hdpdev20.cm6.tbsite.net,60020,1406714346484-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev20.cm6.tbsite.net,60020,1406716991741-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:27 /hbase/WALs/hdpdev20.cm6.tbsite.net,60020,1406778815555
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:25 /hbase/WALs/hdpdev3.cm6.tbsite.net,60020,1406714830504-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev3.cm6.tbsite.net,60020,1406716992137-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:24 /hbase/WALs/hdpdev3.cm6.tbsite.net,60020,1406778815585
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:21 /hbase/WALs/hdpdev4.cm6.tbsite.net,60020,1406714829881-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev4.cm6.tbsite.net,60020,1406716992118-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:13 /hbase/WALs/hdpdev4.cm6.tbsite.net,60020,1406864942962
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:10 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406526862790-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:22 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406715762598-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406716991309-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-31 15:39 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406778815529-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:13 /hbase/WALs/hdpdev5.cm6.tbsite.net,60020,1406941782379
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev6.cm6.tbsite.net,60020,1406716992903-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:19 /hbase/WALs/hdpdev6.cm6.tbsite.net,60020,1406778815530
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:24 /hbase/WALs/hdpdev7.cm6.tbsite.net,60020,1406526862796-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev7.cm6.tbsite.net,60020,1406716993002-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:24 /hbase/WALs/hdpdev7.cm6.tbsite.net,60020,1406778815785
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev8.cm6.tbsite.net,60020,1406716991377-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:21 /hbase/WALs/hdpdev8.cm6.tbsite.net,60020,1406778815557
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:28 /hbase/WALs/hdpdev9.cm6.tbsite.net,60020,1406716099285-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-07-30 18:43 /hbase/WALs/hdpdev9.cm6.tbsite.net,60020,1406716991336-splitting
> drwxr-xr-x   - hadoop hadoop          0 2014-08-02 19:10 /hbase/WALs/hdpdev9.cm6.tbsite.net,60020,1406778815554



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message