Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2AFE1200D0F for ; Fri, 29 Sep 2017 15:32:10 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 295511609D1; Fri, 29 Sep 2017 13:32:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7030E1609C5 for ; Fri, 29 Sep 2017 15:32:09 +0200 (CEST) Received: (qmail 51311 invoked by uid 500); 29 Sep 2017 13:32:08 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 51300 invoked by uid 99); 29 Sep 2017 13:32:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Sep 2017 13:32:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 28CCA1A2491 for ; Fri, 29 Sep 2017 13:32:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Sl-vCv-Ycfsw for ; Fri, 29 Sep 2017 13:32:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 785805F367 for ; Fri, 29 Sep 2017 13:32:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 2C2EAE2568 for ; Fri, 29 Sep 2017 13:32:02 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 441E9242D1 for ; Fri, 29 Sep 2017 13:32:01 +0000 (UTC) Date: Fri, 29 Sep 2017 13:32:01 +0000 (UTC) From: "Guanghao Zhang (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 29 Sep 2017 13:32:10 -0000 [ https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185829#comment-16185829 ] Guanghao Zhang commented on HBASE-14247: ---------------------------------------- Pint [~davelatham] [~stack] [~Apache9] [~anoop.hbase] for reviewing. Thanks. > Separate the old WALs into different regionserver directories > ------------------------------------------------------------- > > Key: HBASE-14247 > URL: https://issues.apache.org/jira/browse/HBASE-14247 > Project: HBase > Issue Type: Improvement > Components: wal > Reporter: Liu Shaohui > Assignee: Guanghao Zhang > Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-14247.master.001.patch, HBASE-14247.master.002.patch, HBASE-14247.master.003.patch, HBASE-14247.master.004.patch, HBASE-14247-v001.diff, HBASE-14247-v002.diff, HBASE-14247-v003.diff > > > Currently all old WALs of regionservers are achieved into the single directory of oldWALs. In big clusters, because of long TTL of WAL or disabled replications, the number of files under oldWALs may reach the max-directory-items limit of HDFS, which will make the hbase cluster crashed. > {quote} > Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded: limit=1048576 items=1048576 > {quote} > A simple solution is to separate the old WALs into different directories according to the server name of the WAL. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029)