Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 66275200D24 for ; Tue, 19 Sep 2017 04:44:13 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 64CB61609DB; Tue, 19 Sep 2017 02:44:13 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A8E861609DE for ; Tue, 19 Sep 2017 04:44:12 +0200 (CEST) Received: (qmail 49386 invoked by uid 500); 19 Sep 2017 02:44:11 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 49327 invoked by uid 99); 19 Sep 2017 02:44:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Sep 2017 02:44:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 487171A2EFA for ; Tue, 19 Sep 2017 02:44:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id wgq_vQoKTOUb for ; Tue, 19 Sep 2017 02:44:10 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 696E7610F0 for ; Tue, 19 Sep 2017 02:44:09 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8A50AE0ECE for ; Tue, 19 Sep 2017 02:44:08 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id A8C0624167 for ; Tue, 19 Sep 2017 02:44:03 +0000 (UTC) Date: Tue, 19 Sep 2017 02:44:03 +0000 (UTC) From: "Guanghao Zhang (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 19 Sep 2017 02:44:13 -0000 [ https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-14247: ----------------------------------- Attachment: HBASE-14247.master.002.patch > Separate the old WALs into different regionserver directories > ------------------------------------------------------------- > > Key: HBASE-14247 > URL: https://issues.apache.org/jira/browse/HBASE-14247 > Project: HBase > Issue Type: Improvement > Components: wal > Reporter: Liu Shaohui > Assignee: Guanghao Zhang > Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-14247.master.001.patch, HBASE-14247.master.002.patch, HBASE-14247-v001.diff, HBASE-14247-v002.diff, HBASE-14247-v003.diff > > > Currently all old WALs of regionservers are achieved into the single directory of oldWALs. In big clusters, because of long TTL of WAL or disabled replications, the number of files under oldWALs may reach the max-directory-items limit of HDFS, which will make the hbase cluster crashed. > {quote} > Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded: limit=1048576 items=1048576 > {quote} > A simple solution is to separate the old WALs into different directories according to the server name of the WAL. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.4.14#64029)