Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8902B10BDF for ; Fri, 15 Nov 2013 06:37:32 +0000 (UTC) Received: (qmail 24685 invoked by uid 500); 15 Nov 2013 06:37:29 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 24648 invoked by uid 500); 15 Nov 2013 06:37:27 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 24618 invoked by uid 99); 15 Nov 2013 06:37:26 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Nov 2013 06:37:26 +0000 Date: Fri, 15 Nov 2013 06:37:25 +0000 (UTC) From: "Uma Maheswara Rao G (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-5517) Lower the default maximum number of blocks per file MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823344#comment-13823344 ] Uma Maheswara Rao G commented on HDFS-5517: ------------------------------------------- +1 for lowering the value. Thanks Aaron for proposing this change. Recently we also faced this issue. When file having 2L blocks itself NN is behaving something abnormal. For every block addition we do update blocks, so this big array will be persisted every time incrementally I think. This will be overhead really, if allow to write more and more blocks for same file with the current code But in practice I am not sure what will be the scenario to configure less block sizes and more blocks for same file. But we saw this in testing cluster. > Lower the default maximum number of blocks per file > --------------------------------------------------- > > Key: HDFS-5517 > URL: https://issues.apache.org/jira/browse/HDFS-5517 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode > Affects Versions: 2.2.0 > Reporter: Aaron T. Myers > Assignee: Aaron T. Myers > > We introduced the maximum number of blocks per file in HDFS-4305, but we set the default to 1MM. In practice this limit is so high as to never be hit, whereas we know that an individual file with 10s of thousands of blocks can cause problems. We should lower the default value, in my opinion to 10k. -- This message was sent by Atlassian JIRA (v6.1#6144)