Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 91BE9107ED for ; Mon, 2 Dec 2013 15:46:09 +0000 (UTC) Received: (qmail 43537 invoked by uid 500); 2 Dec 2013 15:44:30 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 43431 invoked by uid 500); 2 Dec 2013 15:44:27 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 43395 invoked by uid 99); 2 Dec 2013 15:44:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Dec 2013 15:44:24 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of siddharth.tiwari@live.com designates 65.55.90.93 as permitted sender) Received: from [65.55.90.93] (HELO snt0-omc2-s18.snt0.hotmail.com) (65.55.90.93) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Dec 2013 15:44:19 +0000 Received: from SNT148-W19 ([65.55.90.72]) by snt0-omc2-s18.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 2 Dec 2013 07:43:58 -0800 X-TMN: [uUmnJGAjOdfIKW3Un9ml77mAlZ4DVP8C] X-Originating-Email: [siddharth.tiwari@live.com] Message-ID: Content-Type: multipart/alternative; boundary="_1cd20624-147d-4094-a060-0459d578c491_" From: Siddharth Tiwari To: USers Hadoop Subject: RE: Strange error on Datanodes Date: Mon, 2 Dec 2013 15:43:58 +0000 Importance: Normal In-Reply-To: References: , MIME-Version: 1.0 X-OriginalArrivalTime: 02 Dec 2013 15:43:58.0808 (UTC) FILETIME=[50EA7D80:01CEEF75] X-Virus-Checked: Checked by ClamAV on apache.org --_1cd20624-147d-4094-a060-0459d578c491_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Jeet I have a cluster of size 25=2C 4 Admin nodes and 21 datanodes.2 NN 2 JT 3 Z= ookeepers and 3 QJNs if you could help me in understanding what kind of logs you want I will pro= vide it to you. Do you need hdfs-site.xml=2C core-site.xml and mapred-site.= xmls ? *------------------------* =0A= Cheers !!! =0A= Siddharth Tiwari =0A= Have a refreshing day !!! "Every duty is holy=2C and devotion to duty is the highest form of worship = of God.=94=20 =0A= "Maybe other people will try to limit me but I don't limit myself" Date: Mon=2C 2 Dec 2013 21:09:03 +0530 Subject: Re: Strange error on Datanodes From: jeetuyadav200890@gmail.com To: user@hadoop.apache.org Hi=2C Can you share some more logs from Data nodes? could you please also share t= he conf and cluster size? RegardsJitendra =20 =0A= On Mon=2C Dec 2=2C 2013 at 8:49 PM=2C Siddharth Tiwari wrote: =0A= =0A= =0A= =0A= Hi team I see following errors on datanodes. What is the reason for this and how ca= n it will be resolved:-=0A= 2013-12-02 13:11:36=2C441 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputS= tream ResponseProcessor exception for block BP-1854340821-10.238.9.151-138= 5733655875:blk_-2927699636194035560_63092=0A= java.net.SocketTimeoutException: 65000 millis timeout while waiting for cha= nnel to be ready for read. ch : java.nio.channels.SocketChannel[connected l= ocal=3D/10.238.10.43:54040 remote=3D/10.238.10.43:50010]=0A= at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java= :165)=0A= at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156= )=0A= at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129= )=0A= at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117= )=0A= at java.io.FilterInputStream.read(FilterInputStream.java:83)=0A= at java.io.FilterInputStream.read(FilterInputStream.java:83)=0A= at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUti= l.java:169)=0A= at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(Pip= elineAck.java:114)=0A= at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.r= un(DFSOutputStream.java:694)=0A= 2013-12-02 13:12:06=2C572 INFO org.apache.hadoop.mapred.TaskLogsTruncater: = Initializing logs' truncater with mapRetainSize=3D-1 and reduceRetainSize= =3D-1=0A= 2013-12-02 13:12:06=2C581 ERROR org.apache.hadoop.security.UserGroupInforma= tion: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOEx= ception: All datanodes 10.238.10.43:50010 are bad. Aborting...=0A= 2013-12-02 13:12:06=2C581 WARN org.apache.hadoop.mapred.Child: Error runnin= g child=0A= java.io.IOException: All datanodes 10.238.10.43:50010 are bad. Aborting...= =0A= at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForApp= endOrRecovery(DFSOutputStream.java:959)=0A= at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeErro= r(DFSOutputStream.java:779)=0A= at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream= .java:448)=0A= 2013-12-02 13:12:06=2C583 INFO org.apache.hadoop.mapred.Task: Runnning clea= nup for the task *------------------------* =0A= Cheers !!! =0A= Siddharth Tiwari =0A= Have a refreshing day !!! "Every duty is holy=2C and devotion to duty is the highest form of worship = of God.=94=20 =0A= =0A= "Maybe other people will try to limit me but I don't limit myself" =0A= =0A= = --_1cd20624-147d-4094-a060-0459d578c491_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
Hi Jeet

I have a cluster of s= ize 25=2C 4 Admin nodes and 21 datanodes.
2 NN 2 JT 3 Zookeepers and 3 QJNs
=
if you could help me in understanding what kind = of logs you want I will provide it to you. Do you need hdfs-site.xml=2C cor= e-site.xml and mapred-site.xmls ?


*------------------------*
=0A= = Cheers !!!
=0A= = Siddharth Tiwari=0A= Have a r= efreshing day !!!"Every duty is holy=2C and devotion to duty is the highest form of wors= hip of God.=94
=0A= "Maybe other people will try to limit me but I do= n't limit myself"


Date: Mon=2C 2 Dec 2013 21:09:03= +0530
Subject: Re: Strange error on Datanodes
From: jeetuyadav200890= @gmail.com
To: user@hadoop.apache.org

Hi=2C
=
Can you share some more logs from Data nodes? could you plea= se also share the conf and cluster size?

Regards
Jitendra  =3B
=0A=

On Mon=2C Dec 2=2C 2013 at 8:49 PM=2C= Siddharth Tiwari <=3Bsiddharth.tiwari@live.com>=3B wr= ote:
=0A= =0A= =0A= =0A=
Hi team

I see following errors on datanodes. What = is the reason for this and how can it will be resolved:-
=0A= 

201=
3-12-02 13:11:36=2C441 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStre=
am ResponseProcessor exception  for block BP-1854340821-10.238.9.151-138573=
3655875:blk_-2927699636194035560_63092=0A=
java.net.SocketTimeoutException: 65000 millis timeout while waiting for cha=
nnel to be ready for read. ch : java.nio.channels.SocketChannel[connected l=
ocal=3D/10.238.10.4=
3:54040 remote=3D/10.238.10.43:50010]=0A=
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java=
:165)=0A=
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156=
)=0A=
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129=
)=0A=
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117=
)=0A=
	at java.io.FilterInputStream.read(FilterInputStream.java:83)=0A=
	at java.io.FilterInputStream.read(FilterInputStream.java:83)=0A=
	at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUti=
l.java:169)=0A=
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(Pip=
elineAck.java:114)=0A=
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.r=
un(DFSOutputStream.java:694)=0A=
2013-12-02 13:12:06=2C572 INFO org.apache.hadoop.mapred.TaskLogsTruncater: =
Initializing logs' truncater with mapRetainSize=3D-1 and reduceRetainSize=
=3D-1=0A=
2013-12-02 13:12:06=2C581 ERROR org.apache.hadoop.security.UserGroupInforma=
tion: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOEx=
ception: All datanodes 10.238.10.43:50010 are bad. Aborting...=0A=
2013-12-02 13:12:06=2C581 WARN org.apache.hadoop.mapred.Child: Error runnin=
g child=0A=
java.io.IOException: All datanodes 10.238.10.43:50010 are bad. Aborting...=0A=
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForApp=
endOrRecovery(DFSOutputStream.java:959)=0A=
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeErro=
r(DFSOutputStream.java:779)=0A=
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream=
.java:448)=0A=
2013-12-02 13:12:06=2C583 INFO org.apache.hadoop.mapred.Task: Runnning clea=
nup for the task


*------------= ------------*
=0A= Cheers != !!
=0A= Siddharth T= iwariHave a refreshing d= ay !!!
"Every duty is holy=2C and = devotion to duty is the highest form of worship of God.=94
= =0A= =0A= "Maybe othe= r people will try to limit me but I don't limit myself"
=0A=
=0A=

= --_1cd20624-147d-4094-a060-0459d578c491_--