Return-Path: 
 <hdfs-issues-return-21697-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 524296D72
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Tue, 28 Jun 2011 23:13:50 +0000 (UTC)
Received: (qmail 60910 invoked by uid 500); 28 Jun 2011 23:13:50 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 60891 invoked by uid 500); 28 Jun 2011 23:13:49 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 60883 invoked by uid 99); 28 Jun 2011 23:13:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 23:13:49 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 23:13:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F37E4139D2
	for <hdfs-issues@hadoop.apache.org>; Tue, 28 Jun 2011 23:13:28 +0000 (UTC)
Date: Tue, 28 Jun 2011 23:13:28 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <656366663.517.1309302808517.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <898460685.24076.1305754367345.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HDFS-1955) HDFS-1826 made FSImage.doUpgrade()
 too fault-tolerant
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HDFS-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056878#comment-13056878 ] 

Daryn Sharp commented on HDFS-1955:
-----------------------------------

+FSEditLog+
{code}
  public synchronized void errorOccurred(StorageDirectory sd)
      throws IOException {
    if (editStreams == null) {
{code}

Should this call {{getNumEditStreams()}}, or maybe even better yet {{isOpen()}} which returns {{false}} if there are no edit streams?

+FSImage+
{code}
    storage.reportErrorsOnDirectories(errorSDs);
    if (!errorSDs.isEmpty()) {
{code}

Would it make sense to move the call to {{reportErrorOnDirectories}} inside the {{if}}?  Other callers of the method tend to not unconditionally call the method.

This isn't strictly related to your change, but is a question/observation while tracing the code.  I'm not a java threading expert, but is there a race condition here?

{code}
  private void waitForThreads(List<Thread> threads) {
    for (Thread thread : threads) {
      while (thread.isAlive()) {
        try {
          thread.join();
        } catch (InterruptedException iex) {
          LOG.error("Caught exception while waiting for thread " +
                    thread.getName() + " to finish. Retrying join");
        }        
      }
    }
  }
{code}

Can {{isAlive}} return {{false}} because the thread already terminated before {{waitForThreads}} is invoked?  I ask because won't the thread be left in limbo?  In which case, should the {{while}} be a {{do-while}}?

> HDFS-1826 made FSImage.doUpgrade() too fault-tolerant
> -----------------------------------------------------
>
>                 Key: HDFS-1955
>                 URL: https://issues.apache.org/jira/browse/HDFS-1955
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hdfs-1955_1.patch, hdfs-1955_1.patch
>
>
> Prior to HDFS-1826, doUpgrade() would fail if any of the storage directories failed to successfully write the new fsimage or edits files.
> Now it appears to "succeed" even if some or all of the individual directories fail.
> There is some discussion about whether doUpgrade() should have some fault tolerance, but for now make it fail on any single storage directory failure, as before.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira