Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:return-path:x-originalarrivaltime;
	b=PWjZtyTrkSwFFNmZ0pKndcJiN3ixZMypNkMTcdwkFU2AZ8KPNaLARIkRXjAPisn0
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C8CEEE.50ADD35D"
Subject: All datanodes getting marked as dead
Date: Sun, 15 Jun 2008 06:47:10 -0700
Message-ID: 
 <6BEB21C567740F4A83F93D3BC8C9FE6D02104D89@SNV-EXVS03.ds.corp.yahoo.com>
Thread-Topic: All datanodes getting marked as dead
Thread-Index: AcjO7k9HKJshnm9jT4S+mP8oeCzHMQ==
From: "Murali Krishna" <muralik@yahoo-inc.com>
To: <core-user@hadoop.apache.org>

------_=_NextPart_001_01C8CEEE.50ADD35D
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

            I was running some M/R job on a 90+ node cluster. While the
job was running the entire data nodes seems to have become dead. Only
major error I saw in the name node log is 'java.io.IOException: Too many
open files'. The job might try to open thousands of file.

            After some time, there are lot of exceptions saying 'could
only be replicated to 0 nodes instead of 1'. So looks like all the data
nodes are not responding now; job has failed since it couldn't write. I
can see the following in the data nodes logs:

            2008-06-15 02:38:28,477 WARN org.apache.hadoop.dfs.DataNode:
java.net.SocketTimeoutException: timed out waiting for rpc response

        at org.apache.hadoop.ipc.Client.call(Client.java:484)

        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)

        at org.apache.hadoop.dfs.$Proxy0.sendHeartbeat(Unknown Source)

=20

All processes (datanodes + namenodes) are still running..(dfs health
status page shows all nodes as dead)

=20

Some questions:

*         Is this kind of behavior expected when name node runs out of
file handles?

*         Why the data nodes are not able to send the heart beat (is it
related to name node not having enough handles?)

*         What happens to the data in the hdfs when all the data nodes
fail to send the heart beat and name node is in this state?

*         Is the solution is to just increase the number of file handles
and restart the cluster?=20

=20

Thanks,

Murali


------_=_NextPart_001_01C8CEEE.50ADD35D--