Return-Path: 
 <hdfs-issues-return-145700-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A583618802
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Mon, 23 Nov 2015 02:17:11 +0000 (UTC)
Received: (qmail 66961 invoked by uid 500); 23 Nov 2015 02:17:11 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 66891 invoked by uid 500); 23 Nov 2015 02:17:11 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 66877 invoked by uid 99); 23 Nov 2015 02:17:11 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2015 02:17:11 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id ED99A2C1F58
	for <hdfs-issues@hadoop.apache.org>; Mon, 23 Nov 2015 02:17:10 +0000 (UTC)
Date: Mon, 23 Nov 2015 02:17:10 +0000 (UTC)
From: "Xiao Chen (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12913040.1447466295000.154548.1448245030970@Atlassian.JIRA>
In-Reply-To: <JIRA.12913040.1447466295000@Atlassian.JIRA>
References: <JIRA.12913040.1447466295000@Atlassian.JIRA>
 <JIRA.12913040.1447466295562@arcas>
Subject: [jira] [Commented] (HDFS-9429) Tests in TestDFSAdminWithHA
 intermittently fail with EOFException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HDFS-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021383#comment-15021383 ] 

Xiao Chen commented on HDFS-9429:
---------------------------------

Attached patch to reproduce the failure to a same stack trace but with a different type of exception. As mentioned above, EOFE needs to be very exact to reproduce. I think this reproduce patch is sufficient to prove that a {{waitActive}}-ish method is needed.

The reproduced failure is caused by JN rpc server starting later than the rpc call inside the said stack trace. Un-commenting the {{journalCluster.waitActive();}} in {{MiniQJMHACluster#MiniQJMHACluster}} at line 101 will make the unit test pass, due to the introduced {{waitActive}}.

Below is a sample failure stack trace using the attached patch.
{noformat}
java.io.IOException: Timed out waiting for response from loggers
	at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:229)
	at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:916)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:180)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1067)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:370)
	at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:228)
	at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1005)
	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:482)
	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
	at org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster.<init>(MiniQJMHACluster.java:111)
	at org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster.<init>(MiniQJMHACluster.java:37)
	at org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster$Builder.build(MiniQJMHACluster.java:65)
	at org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.setUpHaCluster(TestDFSAdminWithHA.java:84)
	at org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testMetaSave(TestDFSAdminWithHA.java:205)
{noformat}

Please kindly review patch 1. Thanks.

> Tests in TestDFSAdminWithHA intermittently fail with EOFException
> -----------------------------------------------------------------
>
>                 Key: HDFS-9429
>                 URL: https://issues.apache.org/jira/browse/HDFS-9429
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: HDFS
>            Reporter: Xiao Chen
>            Assignee: Xiao Chen
>         Attachments: HDFS-9429.001.patch, HDFS-9429.reproduce
>
>
> I have seen this fail a handful of times for {{testMetaSave}}, but from my understanding this is from {{setUpHaCluster}} so theoretically it could fail for any cases in the class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)