hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lujie (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-14216) NullPointerException happens in NamenodeWebHdfs
Date Sat, 19 Jan 2019 06:57:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

lujie updated HDFS-14216:
-------------------------
    Description: 
 workload
{code:java}
curl -i -X PUT -T $HOMEPARH/test.txt "http://hadoop1:9870/webhdfs/v1/input?op=CREATE&excludedatanodes=hadoop2"
{code}
the method
{code:java}
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(String
excludeDatanodes){

    HashSet<Node> excludes = new HashSet<Node>();
    if (excludeDatanodes != null) {
       for (String host : StringUtils
         .getTrimmedStringCollection(excludeDatanodes)) {
         int idx = host.indexOf(":");
       if (idx != -1) { 
            excludes.add(bm.getDatanodeManager().getDatanodeByXferAddr(
           host.substring(0, idx), Integer.parseInt(host.substring(idx + 1))));
       } else {
          excludes.add(bm.getDatanodeManager().getDatanodeByHost(host));//line280
       }
      }
    }

}
{code}
when datanode(e.g.hadoop2) just crashed before line280, then  bm.getDatanodeManager().getDatanodeByHost(host)
will return null, *_excludes_* containes *{color:#ff0000}null{color}*. while *_excludes_*
are used later, NPE happens:
{code:java}
java.lang.NullPointerException
at org.apache.hadoop.net.NodeBase.getPath(NodeBase.java:113)
at org.apache.hadoop.net.NetworkTopology.countNumOfAvailableNodes(NetworkTopology.java:672)
at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:533)
at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:491)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(NamenodeWebHdfsMethods.java:323)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.redirectURI(NamenodeWebHdfsMethods.java:384)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.put(NamenodeWebHdfsMethods.java:652)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:600)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:597)
at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:73)
at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
{code}
 

  was:
 workload
{code:java}
curl -i -X PUT -T $HOMEPARH/test.txt "http://hadoop1:9870/webhdfs/v1/input?op=CREATE&excludedatanodes=hadoop2"
{code}
the method
{code:java}
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(String
excludeDatanodes){

    HashSet<Node> excludes = new HashSet<Node>();
    if (excludeDatanodes != null) {
       for (String host : StringUtils
         .getTrimmedStringCollection(excludeDatanodes)) {
         int idx = host.indexOf(":");
       if (idx != -1) { 
            excludes.add(bm.getDatanodeManager().getDatanodeByXferAddr(
           host.substring(0, idx), Integer.parseInt(host.substring(idx + 1))));
       } else {
          excludes.add(bm.getDatanodeManager().getDatanodeByHost(host));//line280
       }
      }
    }

}
{code}
when host(e.g.hadoop2) just crash before line280, then  bm.getDatanodeManager().getDatanodeByHost(host)
will return null, *_excludes_* containes *{color:#ff0000}null{color}*. while *_excludes_*
are used later, NPE happens:
{code:java}
java.lang.NullPointerException
at org.apache.hadoop.net.NodeBase.getPath(NodeBase.java:113)
at org.apache.hadoop.net.NetworkTopology.countNumOfAvailableNodes(NetworkTopology.java:672)
at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:533)
at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:491)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(NamenodeWebHdfsMethods.java:323)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.redirectURI(NamenodeWebHdfsMethods.java:384)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.put(NamenodeWebHdfsMethods.java:652)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:600)
at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:597)
at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:73)
at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
{code}
 


> NullPointerException happens in NamenodeWebHdfs
> -----------------------------------------------
>
>                 Key: HDFS-14216
>                 URL: https://issues.apache.org/jira/browse/HDFS-14216
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: lujie
>            Priority: Critical
>         Attachments: HDFS-14216_1.patch, hadoop-hires-namenode-hadoop11.log
>
>
>  workload
> {code:java}
> curl -i -X PUT -T $HOMEPARH/test.txt "http://hadoop1:9870/webhdfs/v1/input?op=CREATE&excludedatanodes=hadoop2"
> {code}
> the method
> {code:java}
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(String
excludeDatanodes){
>     HashSet<Node> excludes = new HashSet<Node>();
>     if (excludeDatanodes != null) {
>        for (String host : StringUtils
>          .getTrimmedStringCollection(excludeDatanodes)) {
>          int idx = host.indexOf(":");
>        if (idx != -1) { 
>             excludes.add(bm.getDatanodeManager().getDatanodeByXferAddr(
>            host.substring(0, idx), Integer.parseInt(host.substring(idx + 1))));
>        } else {
>           excludes.add(bm.getDatanodeManager().getDatanodeByHost(host));//line280
>        }
>       }
>     }
> }
> {code}
> when datanode(e.g.hadoop2) just crashed before line280, then  bm.getDatanodeManager().getDatanodeByHost(host)
will return null, *_excludes_* containes *{color:#ff0000}null{color}*. while *_excludes_*
are used later, NPE happens:
> {code:java}
> java.lang.NullPointerException
> at org.apache.hadoop.net.NodeBase.getPath(NodeBase.java:113)
> at org.apache.hadoop.net.NetworkTopology.countNumOfAvailableNodes(NetworkTopology.java:672)
> at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:533)
> at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:491)
> at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.chooseDatanode(NamenodeWebHdfsMethods.java:323)
> at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.redirectURI(NamenodeWebHdfsMethods.java:384)
> at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.put(NamenodeWebHdfsMethods.java:652)
> at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:600)
> at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:597)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:73)
> at org.apache.hadoop.ipc.ExternalCall.run(ExternalCall.java:30)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830)
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message