hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yusaku Sako <yus...@hortonworks.com>
Subject Re: DataNode logs have exceptions - DataXceiver error processing unknown operation
Date Fri, 26 Jun 2015 00:09:27 GMT
Hi Rajesh,

Are you running Ambari?  If so, this is benign and can be ignored.
Ambari "pings" the DataNode by making a socket connection once a minute to make sure it's
up and running.  Otherwise, it will trigger an alert.
Unfortunately, there's no known way to "ping" the DataNode with a valid payload to not cause
this log (or at least when the devs implemented this in Ambari, there didn't seem to be one).

Yusaku

From: Rajesh Kartha <kartha02@gmail.com<mailto:kartha02@gmail.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Thursday, June 25, 2015 4:57 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: DataNode logs have exceptions - DataXceiver error processing unknown operation

Hello,

I am using a Hadoop 2.7.1 build and noticed a constant flow of exceptions every 60 seconds
in the DataNode log files:

2015-06-25 13:02:36,292 ERROR datanode.DataNode (DataXceiver.java:run(278)) - bdavm063.svl.ibm.com:50010:DataXceiver
error processing unknown operation  src: /127.0.0.1:54415<http://127.0.0.1:54415> dst:
/127.0.0.1:50010<http://127.0.0.1:50010>
java.io.EOFException
        at java.io.DataInputStream.readShort(DataInputStream.java:315)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
        at java.lang.Thread.run(Thread.java:745)
2015-06-25 13:03:34,328 ERROR datanode.DataNode (DataXceiver.java:run(278)) - bdavm063.svl.ibm.com:50010:DataXceiver
error processing unknown operation  src: /127.0.0.1:54441<http://127.0.0.1:54441> dst:
/127.0.0.1:50010<http://127.0.0.1:50010>
java.io.EOFException
        at java.io.DataInputStream.readShort(DataInputStream.java:315)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
        at java.lang.Thread.run(Thread.java:745)
2015-06-25 13:05:36,081 ERROR datanode.DataNode (DataXceiver.java:run(278)) - bdavm063.svl.ibm.com:50010:DataXceiver
error processing unknown operation  src: /127.0.0.1:54477<http://127.0.0.1:54477> dst:
/127.0.0.1:50010<http://127.0.0.1:50010>
java.io.EOFException
        at java.io.DataInputStream.readShort(DataInputStream.java:315)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
        at java.lang.Thread.run(Thread.java:745)



I checked:
-  ulimit to be -  open files                      (-n) 32768
- dfs.datanode.max.transfer.threads to be 16384

While HDFS seem to work without issues, the logs are filled with these.

Any thoughts/ideas on resolving this is greatly appreciated.

Regards,
Rajesh


Mime
View raw message