hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Huang (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-3758) Excessive exceptions in HDFS namenode log file
Date Mon, 14 Jul 2008 22:02:33 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613460#action_12613460
] 

jimhuang edited comment on HADOOP-3758 at 7/14/08 3:01 PM:
------------------------------------------------------------

In the span of 11 seconds, there were 103,367 exceptions generated for only 10 unique nodes
that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.

{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 8020,
call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256, 2150426691690, 0, 0) from A.B.C.D:43226:
error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported:
-11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported:
-11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 8020,
call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888, 2151120783483, 0, 0) from A.B.C.E:56503:
error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported:
-11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported:
-11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}



      was (Author: jimhuang):
    In the span of 11 seconds, there were 103,367 exceptions generated for only 10 unique
nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.

{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 8020,
call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256, 2150426691690, 0, 0) from 76.1
3.169.226:43226: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version
of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported:
-11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 8020,
call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888, 2151120783483, 0, 0) from 76.1
3.169.218:56503: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version
of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported:
-11. Expecting = -13.
        at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
        at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
        at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}


  
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
>                 Key: HADOOP-3758
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3758
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.17.1
>            Reporter: Jim Huang
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.  
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire
partition space, in this case close to 700GB log file was generated on the namenode.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message