hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Latham (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories
Date Tue, 26 Sep 2017 17:10:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181154#comment-16181154
] 

Dave Latham commented on HBASE-14247:
-------------------------------------

Thanks, [~zghaobac].  It looks like the latest patch would address the concern.  There's an
implicit dependency between the clocks assigning the file modification time and checking time
checking the ZK queues.  But the TimeToLiveCleaner would seem to provide plenty of a safety
margin for unsynced clocks.

I didn't review the rest, aside from happening to notice a misspelling of SEPERATE vs SEPARATE
(just in case you care).

> Separate the old WALs into different regionserver directories
> -------------------------------------------------------------
>
>                 Key: HBASE-14247
>                 URL: https://issues.apache.org/jira/browse/HBASE-14247
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: Liu Shaohui
>            Assignee: Guanghao Zhang
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HBASE-14247.master.001.patch, HBASE-14247.master.002.patch, HBASE-14247.master.003.patch,
HBASE-14247-v001.diff, HBASE-14247-v002.diff, HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single directory of oldWALs.
In big clusters, because of long TTL of WAL or disabled replications, the number of files
under oldWALs may reach the max-directory-items limit of HDFS, which will make the hbase cluster
crashed.
> {quote}
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded: limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different  directories according to
the server name of the WAL.
> Suggestions are welcomed~ Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message