Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1897018132.1216072953320.JavaMail.jira@brutus>
Date: Mon, 14 Jul 2008 15:02:33 -0700 (PDT)
From: "Jim Huang (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Issue Comment Edited: (HADOOP-3758) Excessive exceptions in
 HDFS namenode log file
In-Reply-To: <1118303887.1216071391878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613460#action_12613460 ] 

jimhuang edited comment on HADOOP-3758 at 7/14/08 3:01 PM:
------------------------------------------------------------

In the span of 11 seconds, there were 103,367 exceptions generated for only 10 unique nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.

{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 8020, call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256, 2150426691690, 0, 0) from A.B.C.D:43226: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 8020, call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888, 2151120783483, 0, 0) from A.B.C.E:56503: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}


      was (Author: jimhuang):
    In the span of 11 seconds, there were 103,367 exceptions generated for only 10 unique nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.

{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 8020, call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256, 2150426691690, 0, 0) from 76.1
3.169.226:43226: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 8020, call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888, 2151120783483, 0, 0) from 76.1
3.169.218:56503: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}


> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
>                 Key: HADOOP-3758
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3758
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.1
>            Reporter: Jim Huang
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.  
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.