Return-Path: X-Original-To: apmail-hadoop-hdfs-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6ECB8DC08 for ; Thu, 6 Sep 2012 06:58:04 +0000 (UTC) Received: (qmail 65159 invoked by uid 500); 6 Sep 2012 06:58:04 -0000 Delivered-To: apmail-hadoop-hdfs-commits-archive@hadoop.apache.org Received: (qmail 65131 invoked by uid 500); 6 Sep 2012 06:58:04 -0000 Mailing-List: contact hdfs-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-commits@hadoop.apache.org Received: (qmail 65121 invoked by uid 99); 6 Sep 2012 06:58:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Sep 2012 06:58:03 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Sep 2012 06:58:02 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 52F0E238896F; Thu, 6 Sep 2012 06:57:19 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1381481 - in /hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs: ./ src/main/java/org/apache/hadoop/hdfs/qjournal/client/ src/main/java/org/apache/hadoop/hdfs/server/namenode/ src/test/java/org/apache/hadoop/hdfs/qjournal/cl... Date: Thu, 06 Sep 2012 06:57:18 -0000 To: hdfs-commits@hadoop.apache.org From: todd@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20120906065719.52F0E238896F@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: todd Date: Thu Sep 6 06:57:18 2012 New Revision: 1381481 URL: http://svn.apache.org/viewvc?rev=1381481&view=rev Log: HDFS-3891. Make selectInputStreams throw IOE instead of RTE. Contributed by Todd Lipcon. Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-3077.txt hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-3077.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-3077.txt?rev=1381481&r1=1381480&r2=1381481&view=diff ============================================================================== --- hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-3077.txt (original) +++ hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-3077.txt Thu Sep 6 06:57:18 2012 @@ -44,3 +44,5 @@ HDFS-3869. Expose non-file journal manag HDFS-3884. Journal format() should reset cached values (todd) HDFS-3870. Add metrics to JournalNode (todd) + +HDFS-3891. Make selectInputStreams throw IOE instead of RTE (todd) Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java?rev=1381481&r1=1381480&r2=1381481&view=diff ============================================================================== --- hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java (original) +++ hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java Thu Sep 6 06:57:18 2012 @@ -158,9 +158,11 @@ class AsyncLoggerSet { timeoutMs); } catch (InterruptedException e) { Thread.currentThread().interrupt(); - throw new IOException("Interrupted waiting for quorum results"); + throw new IOException("Interrupted waiting " + timeoutMs + "ms for a " + + "quorum of nodes to respond."); } catch (TimeoutException e) { - throw new IOException("Timed out waiting " + timeoutMs + " for write quorum"); + throw new IOException("Timed out waiting " + timeoutMs + "ms for a " + + "quorum of nodes to respond."); } if (q.countSuccesses() < majority) { Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java?rev=1381481&r1=1381480&r2=1381481&view=diff ============================================================================== --- hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java (original) +++ hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java Thu Sep 6 06:57:18 2012 @@ -394,17 +394,12 @@ public class QuorumJournalManager implem @Override public void selectInputStreams(Collection streams, - long fromTxnId, boolean inProgressOk) { + long fromTxnId, boolean inProgressOk) throws IOException { QuorumCall q = loggers.getEditLogManifest(fromTxnId); - Map resps; - try { - resps = loggers.waitForWriteQuorum(q, selectInputStreamsTimeoutMs); - } catch (IOException ioe) { - // TODO: can we do better here? - throw new RuntimeException(ioe); - } + Map resps = + loggers.waitForWriteQuorum(q, selectInputStreamsTimeoutMs); LOG.debug("selectInputStream manifests:\n" + Joiner.on("\n").withKeyValueSeparator(": ").join(resps)); Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java?rev=1381481&r1=1381480&r2=1381481&view=diff ============================================================================== --- hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java (original) +++ hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java Thu Sep 6 06:57:18 2012 @@ -242,15 +242,8 @@ public class FileJournalManager implemen @Override synchronized public void selectInputStreams( Collection streams, long fromTxId, - boolean inProgressOk) { - List elfs; - try { - elfs = matchEditLogs(sd.getCurrentDir()); - } catch (IOException e) { - LOG.error("error listing files in " + this + ". " + - "Skipping all edit logs in this directory.", e); - return; - } + boolean inProgressOk) throws IOException { + List elfs = matchEditLogs(sd.getCurrentDir()); LOG.debug(this + ": selecting input streams starting at " + fromTxId + (inProgressOk ? " (inProgress ok) " : " (excluding inProgress) ") + "from among " + elfs.size() + " candidate file(s)"); Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java?rev=1381481&r1=1381480&r2=1381481&view=diff ============================================================================== --- hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java (original) +++ hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java Thu Sep 6 06:57:18 2012 @@ -65,9 +65,11 @@ public interface JournalManager extends * @param inProgressOk whether or not in-progress streams should be returned * * @return a list of streams + * @throws IOException if the underlying storage has an error or is otherwise + * inaccessible */ void selectInputStreams(Collection streams, - long fromTxnId, boolean inProgressOk); + long fromTxnId, boolean inProgressOk) throws IOException; /** * Set the amount of memory that this stream should use to buffer edits Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java?rev=1381481&r1=1381480&r2=1381481&view=diff ============================================================================== --- hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java (original) +++ hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java Thu Sep 6 06:57:18 2012 @@ -247,7 +247,12 @@ public class JournalSet implements Journ LOG.info("Skipping jas " + jas + " since it's disabled"); continue; } - jas.getManager().selectInputStreams(allStreams, fromTxId, inProgressOk); + try { + jas.getManager().selectInputStreams(allStreams, fromTxId, inProgressOk); + } catch (IOException ioe) { + LOG.warn("Unable to determine input streams from " + jas.getManager() + + ". Skipping.", ioe); + } } chainAndMakeRedundantStreams(streams, allStreams, fromTxId, inProgressOk); } Modified: hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java?rev=1381481&r1=1381480&r2=1381481&view=diff ============================================================================== --- hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java (original) +++ hadoop/common/branches/HDFS-3077/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java Thu Sep 6 06:57:18 2012 @@ -212,6 +212,26 @@ public class TestQuorumJournalManager { } /** + * Regression test for HDFS-3891: selectInputStreams should throw + * an exception when a majority of journalnodes have crashed. + */ + @Test + public void testSelectInputStreamsMajorityDown() throws Exception { + // Shut down all of the JNs. + cluster.shutdown(); + + List streams = Lists.newArrayList(); + try { + qjm.selectInputStreams(streams, 0, false); + fail("Did not throw IOE"); + } catch (QuorumException ioe) { + GenericTestUtils.assertExceptionContains( + "Got too many exceptions", ioe); + assertTrue(streams.isEmpty()); + } + } + + /** * Test the case where the NN crashes after starting a new segment * on all nodes, but before writing the first transaction to it. */