Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Sun, 19 Aug 2012 09:15:38 +1100 (NCT)
From: "Andy Isaacson (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <1566226621.27374.1345328138149.JavaMail.jiratomcat@arcas>
In-Reply-To: 
 <849442626.32526.1321391518745.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HDFS-2554) Add separate metrics for missing
 blocks with desired replication level 1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andy Isaacson updated HDFS-2554:
--------------------------------

    Attachment: hdfs-2554.txt

Attach patch implementing new statistics.

One downside to this patch is that we iterate the entire block map under a writer lock every 3 seconds.  Benchmarks show this taking 85 ms on average for 200,000 blocks (Sun JDK 1.6.0_20 64-bit on Xeon E5540 2.53 GHz).  Still gathering data on a range of DFS sizes. 
                
> Add separate metrics for missing blocks with desired replication level 1
> ------------------------------------------------------------------------
>
>                 Key: HDFS-2554
>                 URL: https://issues.apache.org/jira/browse/HDFS-2554
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Andy Isaacson
>            Priority: Minor
>         Attachments: hdfs-2554.txt
>
>
> Some users use replication level set to 1 for datasets which are unimportant and can be lost with no worry (eg the output of terasort tests). But other data on the cluster is important and should not be lost. It would be useful to separate the metric for missing blocks by the desired replication level of those blocks, so that one could ignore missing blocks at repl 1 while still alerting on missing blocks with higher desired replication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira