hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adarsh Sharma <adarsh.sha...@orkash.com>
Subject Re: HTTP Error
Date Fri, 08 Jul 2011 04:40:51 GMT
Thanks , Still don't understand the issue.

My name node has repeatedly show these logs :

2011-07-08 09:36:31,365 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
ugi=hadoop,hadoop    ip=/MAster-IP   cmd=listStatus    
src=/home/hadoop/system    dst=null    perm=null
2011-07-08 09:36:31,367 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 2 on 9000, call delete(/home/hadoop/system, true) from 
Master-IP:53593: error: 
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete 
/home/hadoop/system. Name node is in safe mode.
The ratio of reported blocks 0.8293 has not reached the threshold 
0.9990. Safe mode will be turned off automatically.
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete 
/home/hadoop/system. Name node is in safe mode.
The ratio of reported blocks 0.8293 has not reached the threshold 
0.9990. Safe mode will be turned off automatically.
    at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
    at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
    at 
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)


And one of my data node shows the below logs :

2011-07-08 09:49:56,967 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: 
DNA_REGISTER
2011-07-08 09:49:59,962 WARN 
org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting 
down: org.apache.hadoop.ipc.RemoteException: 
org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 
192.168.0.209:50010 is attempting to report storage ID 
DS-218695497-SLave_IP-50010-1303978807280. Node SLave_IP:50010 is 
expected to serve this storage.
        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:3920)
        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:2891)
        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:715)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

        at org.apache.hadoop.ipc.Client.call(Client.java:740)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at $Proxy4.blockReport(Unknown Source)
        at 
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:756)
        at 
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1186)
        at java.lang.Thread.run(Thread.java:619)

2011-07-08 09:50:00,072 INFO org.apache.hadoop.ipc.Server: Stopping 
server on 50020
2011-07-08 09:50:00,072 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 1 on 50020: exiting
2011-07-08 09:50:00,074 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 2 on 50020: exiting
2011-07-08 09:50:00,074 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 0 on 50020: exiting
2011-07-08 09:50:00,076 INFO org.apache.hadoop.ipc.Server: Stopping IPC 
Server listener on 50020
2011-07-08 09:50:00,077 INFO org.apache.hadoop.ipc.Server: Stopping IPC 
Server Responder
2011-07-08 09:50:00,077 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup 
to exit, active threads is 1
2011-07-08 09:50:00,078 WARN 
org.apache.hadoop.hdfs.server.datanode.DataNode: 
DatanodeRegistration(SLave_IP:50010, 
storageID=DS-218695497-192.168.0.209-50010-1303978807280, 
infoPort=50075, ipcPort=50020):DataXceiveServer: 
java.nio.channels.AsynchronousCloseException
        at 
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
        at 
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
        at 
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:130)
        at java.lang.Thread.run(Thread.java:619)

2011-07-08 09:50:00,394 INFO 
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Exiting 
DataBlockScanner thread.
2011-07-08 09:50:01,079 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup 
to exit, active threads is 0
2011-07-08 09:50:01,183 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: 
DatanodeRegistration(192.168.0.209:50010, 
storageID=DS-218695497-192.168.0.209-50010-1303978807280, 
infoPort=50075, ipcPort=50020):Finishing DataNode in: 
FSDataset{dirpath='/hdd1-1/data/current'}
2011-07-08 09:50:01,183 INFO org.apache.hadoop.ipc.Server: Stopping 
server on 50020
2011-07-08 09:50:01,183 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup 
to exit, active threads is 0
2011-07-08 09:50:01,185 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ws14-suru-lin/

Also my dfsdmin report shows :

bash-3.2$ bin/hadoop dfsadmin -report
Safe mode is ON
Configured Capacity: 59069984768 (55.01 GB)
Present Capacity: 46471880704 (43.28 GB)
DFS Remaining: 45169745920 (42.07 GB)
DFS Used: 1302134784 (1.21 GB)
DFS Used%: 2.8%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: IP:50010
Decommission Status : Normal
Configured Capacity: 59069984768 (55.01 GB)
DFS Used: 1302134784 (1.21 GB)
Non DFS Used: 12598104064 (11.73 GB)
DFS Remaining: 45169745920(42.07 GB)
DFS Used%: 2.2%
DFS Remaining%: 76.47%
Last contact: Fri Jul 08 10:03:40 IST 2011

But I have 2 datanodes.Safe mode is on from the last 1 hour. I know the 
command to leave it manually.
I think the problem arises due to non start up of one of my datanodes. 
How could i solve this problem .

Also for

  HTTP ERROR: 404

/dfshealth.jsp

RequestURI=/dfshealth.jsp

/Powered by Jetty:// <http://jetty.mortbay.org/> error,

I manually check through below command at all nodes 

On Master :

ash-3.2$ /usr/java/jdk1.6.0_18/bin/jps 
7548 SecondaryNameNode
7395 NameNode
7628 JobTracker
7713 Jps

And also on slaves :

[root@ws33-shiv-lin ~]# /usr/java/jdk1.6.0_20/bin/jps 
5696 DataNode
5941 Jps
5818 TaskTracker




Thanks



Jeff.Schmitz@shell.com wrote:
> Adarsh,
>
> You could also run from command line
>
> [root@xxxxxxx bin]# ./hadoop dfsadmin -report
> Configured Capacity: 1151948095488 (1.05 TB)
> Present Capacity: 1059350446080 (986.6 GB)
> DFS Remaining: 1056175992832 (983.64 GB)
> DFS Used: 3174453248 (2.96 GB)
> DFS Used%: 0.3%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 5 (5 total, 0 dead)
>
>
>
>
> -----Original Message-----
> From: dhruv21@gmail.com [mailto:dhruv21@gmail.com] On Behalf Of Dhruv
> Kumar
> Sent: Thursday, July 07, 2011 10:01 AM
> To: common-user@hadoop.apache.org
> Subject: Re: HTTP Error
>
> 1) Check with jps to see if all services are functioning.
>
> 2) Have you tried appending dfshealth.jsp at the end of the URL as the
> 404
> says?
>
> Try using this:
> http://localhost:50070/dfshealth.jsp
>
>
>
> On Thu, Jul 7, 2011 at 7:13 AM, Adarsh Sharma
> <adarsh.sharma@orkash.com>wrote:
>
>   
>> Dear all,
>>
>> Today I am stucked with the strange problem in the running hadoop
>>     
> cluster.
>   
>> After starting hadoop by bin/start-all.sh, all nodes are started. But
>>     
> when
>   
>> I check through web UI ( MAster-Ip:50070), It shows :
>>
>>
>>   HTTP ERROR: 404
>>
>> /dfshealth.jsp
>>
>> RequestURI=/dfshealth.jsp
>>
>> /Powered by Jetty:// <http://jetty.mortbay.org/>
>> /
>>
>> /I check by command line that hadoop cannot able to get out of safe
>>     
> mode.
>   
>> /
>>
>> /I know , manually command to leave safe mode
>> /
>>
>> /bin/hadoop dfsadmin -safemode leave
>> /
>>
>> /But How can I make hadoop  run properly and what are the reasons of
>>     
> this
>   
>> error
>> /
>>
>> /
>> Thanks
>> /
>>
>>
>>
>>     
>
>   


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message