Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3BD58105DD for ; Fri, 31 Jan 2014 13:16:09 +0000 (UTC) Received: (qmail 33082 invoked by uid 500); 31 Jan 2014 13:16:00 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 32531 invoked by uid 500); 31 Jan 2014 13:16:00 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 32524 invoked by uid 99); 31 Jan 2014 13:15:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jan 2014 13:15:58 +0000 X-ASF-Spam-Status: No, hits=3.8 required=5.0 tests=HTML_MESSAGE,HTTP_ESCAPED_HOST,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of stutiawasthi@hcl.com designates 203.105.185.23 as permitted sender) Received: from [203.105.185.23] (HELO GWS05.hcl.com) (203.105.185.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jan 2014 13:15:53 +0000 X-IronPort-AV: E=Sophos;i="4.95,757,1384281000"; d="scan'208,217";a="54296425" Received: from unknown (HELO NDA-CORP-HT03.CORP.HCL.IN) ([10.248.64.35]) by GWS05.hcl.com with ESMTP/TLS/AES128-SHA; 31 Jan 2014 18:44:56 +0530 Received: from NDA-HCLC-CSHT06.HCLC.CORP.HCL.IN (10.33.64.81) by NDA-CORP-HT03.CORP.HCL.IN (10.248.64.35) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 31 Jan 2014 18:44:50 +0530 Received: from NDA-HCLC-MBS03.HCLC.CORP.HCL.IN ([10.33.64.34]) by NDA-HCLC-CSHT06.HCLC.CORP.HCL.IN ([::1]) with mapi id 14.03.0123.003; Fri, 31 Jan 2014 18:44:49 +0530 From: Stuti Awasthi To: "user@hadoop.apache.org" Subject: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1 Thread-Topic: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1 Thread-Index: Ac8eg8Iux+OMPjmwTJi+9t5CoiSB5w== Date: Fri, 31 Jan 2014 13:14:48 +0000 Message-ID: <2270F3695BF2C04DB9FE975FB546FF3C21531E6D@NDA-HCLC-MBS03.hclc.corp.hcl.in> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.33.64.124] Content-Type: multipart/alternative; boundary="_000_2270F3695BF2C04DB9FE975FB546FF3C21531E6DNDAHCLCMBS03hcl_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_2270F3695BF2C04DB9FE975FB546FF3C21531E6DNDAHCLCMBS03hcl_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi All, I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP reque= st at port 50070 on dfs is not working properly. I have an Hadoop cluster which is operating from several days. Recently we = are also not able to see dfshealth.jsp page from webconsole. Problems : 1. http://:50070/dfshealth.jsp shows following error HTTP ERROR: 404 Problem accessing /. Reason: NOT_FOUND 2. SNN is not able to roll edits : ERROR in SecondaryNameNode Log java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=3D1 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpUR= LConnection.java:1401) at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileCli= ent(TransferFsImage.java:160) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(Se= condaryNameNode.java:347) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(Se= condaryNameNode.java:336) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInf= ormation.java:1093) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.download= CheckpointFiles(SecondaryNameNode.java:336) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckp= oint(SecondaryNameNode.java:411) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(S= econdaryNameNode.java:312) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(Seco= ndaryNameNode.java:275) ERROR in Namenode Log 2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNames= ystem: Roll Edit Log from 10.139.9.231 2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNames= ystem: Cannot roll edit log, edits.new files already exists in all healthy = directories: /usr/lib/hadoop/storage/dfs/nn/current/edits.new Namenode logs which suggest that webserver is started on 50070 successfully: 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port return= ed by webServer.getConnectors()[0].getLocalPort() before open() is -1. Open= ing the listener on 50070 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.ge= tLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() ret= urned 50070 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound= to port 50070 2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNod= e: Web-server up at: HOSTNAME:50070 Hdfs-site.xml dfs.replication 2 dfs.name.dir /usr/lib/hadoop/storage/dfs/nn dfs.data.dir /usr/lib/hadoop/storage/dfs/dn dfs.permissions false dfs.webhdfs.enabled true dfs.http.address HOSTNAME:50070 dfs.secondary.http.address HOSTNAME:50090 fs.checkpoint.dir /usr/lib/hadoop/storage/dfs/snn /etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host fi= le but the issue was not resolved) 127.0.0.1 localhost IP1 Hostname1 # Namenode- vm01 - itself IP2 Hostname2 # DataNode- vm02 ........ # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters Note : All Hadoop daemons are executing fine and the jobs are running prope= rly. How to resolve this issue, I have tried many options provided on different = forums but still facing the same issue. I belive that this can cause a major problem later as my edits are not gett= ing rolled into fsimage.. This can cause me a data loss in case of failure. Please suggest Thanks Stuti ::DISCLAIMER:: ---------------------------------------------------------------------------= ------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and inte= nded for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as informa= tion could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in trans= mission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability = on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the = author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, disse= mination, copying, disclosure, modification, distribution and / or publication of this message without the prior written= consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please= delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses = and other defects. ---------------------------------------------------------------------------= ------------------------------------------------------------------------- --_000_2270F3695BF2C04DB9FE975FB546FF3C21531E6DNDAHCLCMBS03hcl_ Content-Type: text/html; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable

Hi All,

 

I am suddenly started facing issue on Hadoop Cluster= . Seems like HTTP request at port 50070 on dfs is not working properly.

I have an Hadoop cluster which is operating from sev= eral days. Recently we are also not able to see dfshealth.jsp page from web= console.

 

Problems :

1. http://<Hostname>:50070/dfshealth.jsp shows following error<= o:p>

 

HTTP ERROR: 404

Problem accessing /. Reason:

NOT_FOUND

 

2. SNN is not able to roll edits :   =       

ERROR in SecondaryNameNode Log

java.io.FileNotFoundException= : http://HOSTNAME:50070/getimage?getimage=3D1

     = ;  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpU= RLConnection.java:1401)

     = ;  at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileCl= ient(TransferFsImage.java:160)

     = ;  at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(S= econdaryNameNode.java:347)

     = ;  at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(S= econdaryNameNode.java:336)

     = ;  at java.security.AccessController.doPrivileged(Native Method)<= /o:p>

     = ;  at javax.security.auth.Subject.doAs(Subject.java:416)

     = ;  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupIn= formation.java:1093)

     = ;  at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloa= dCheckpointFiles(SecondaryNameNode.java:336)

     = ;  at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheck= point(SecondaryNameNode.java:411)

     = ;  at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(= SecondaryNameNode.java:312)

     = ;  at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(Sec= ondaryNameNode.java:275)

 

ERROR in Namenode Log

2014-01-31 18:15:12,046 INFO = org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.= 139.9.231

2014-01-31 18:15:12,046 WARN = org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, = edits.new files already exists in all healthy directories:

  /usr/lib/hadoop/storag= e/dfs/nn/current/edits.new

 

 

 

Namenode logs which suggest that webserver is starte= d on 50070 successfully:

2014-01-31 14:42:35,208 INFO = org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors= ()[0].getLocalPort() before open() is -1. Opening the listener on 50070

2014-01-31 14:42:35,209 INFO = org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 w= ebServer.getConnectors()[0].getLocalPort() returned 50070

2014-01-31 14:42:35,209 INFO = org.apache.hadoop.http.HttpServer: Jetty bound to port 50070

2014-01-31 14:42:35,378 INFO = org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME= :50070

 

 

Hdfs-site.xml

<configuration>

    <proper= ty>

     = ;   <name>dfs.replication</name>

     = ;   <value>2</value>

    </prope= rty>

 

    <proper= ty>

     = ;   <name>dfs.name.dir</name>

     = ;   <value>/usr/lib/hadoop/storage/dfs/nn</value>

    </prope= rty>

 

    <proper= ty>

     = ;   <name>dfs.data.dir</name>

     = ;   <value>/usr/lib/hadoop/storage/dfs/dn</value>

    </prope= rty>

 

    <proper= ty>

     = ;   <name>dfs.permissions</name>

     = ;   <value>false</value>

    </prope= rty>

<property>

  <name>dfs.webhdf= s.enabled</name>

  <value>true</= value>

</property><= /span>

 

<property>

  <name>dfs.http.a= ddress</name>

  <value>HOSTNAME:= 50070</value>

</property><= /span>

 

<property>

  <name>dfs.second= ary.http.address</name>

  <value>HOSTNAME:= 50090</value>

</property><= /span>

 

<property>

  <name>fs.checkpo= int.dir</name>

  <value>/usr/lib/= hadoop/storage/dfs/snn</value>

</property><= /span>

 

</configuration>

 

 

/etc/hosts (Note I have also tried by comment= ing 127.0.0.1 entry in host file but the issue was not resolved)=

 

127.0.0.1   &n= bsp;   localhost

 

IP1    Hostnam= e1         # Namenode- vm01 - itself

IP2    Hostnam= e2         # DataNode- vm02

……..

 

# The following lines are des= irable for IPv6 capable hosts

::1     i= p6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix<= /o:p>

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

 

 

Note : All Hadoop daemons are= executing fine and the jobs are running properly.

 

How to resolve this issue, I = have tried many options provided on different forums but still facing the s= ame issue.

I belive that this can cause = a major problem later as my edits are not getting rolled into fsimage.. Thi= s can cause me a data loss in case of failure.

 

Please suggest

 

Thanks

Stuti

 

 

 



::DISCLAIMER= ::
---------------------------------------------------------------------= ---------------------------------------------------------------------------= ----

The contents of thi= s e-mail and any attachment(s) are confidential and intended for the named = recipient(s) only.
E-mail transmission is not guaranteed to be secure or= error-free as information could be intercepted, corrupted,
lost, destr= oyed, arrive late or incomplete, or may contain viruses in transmission. Th= e e mail and its contents
(with or without referred errors) shall there= fore not attach any liability on the originator or HCL or its affiliates. <= br>Views or opinions, if any, presented in this email are solely those of t= he author and may not necessarily reflect the
views or opinions of HCL = or its affiliates. Any form of reproduction, dissemination, copying, disclo= sure, modification,
distribution and / or publication of this message w= ithout the prior written consent of authorized representative of
HCL is= strictly prohibited. If you have received this email in error please delet= e it and notify the sender immediately.
Before opening any email and/or= attachments, please check them for viruses and other defects.

-------------------= ---------------------------------------------------------------------------= ------------------------------------------------------

--_000_2270F3695BF2C04DB9FE975FB546FF3C21531E6DNDAHCLCMBS03hcl_--