hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-751) Namenode constantly using up 100% CPU
Date Thu, 30 Nov 2006 02:59:22 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-751?page=comments#action_12454521 ] 
            
Koji Noguchi commented on HADOOP-751:
-------------------------------------

The error messages I initially copy&pasted are probably not directly related to this problem.
Once I increased the HADOOP_HEAPSIZE, this high CPU load problem went away.

Raghu pointed out 
"If increase of 500mb fixed the issue, wouldn't the main suspect be Java  GC?"

"attaching gdb to the pid shows jvm is handling "failed allocation" most of the time. I don't
know if it is possible to check java stack for a given pid. I tried attaching multiple times
and stack looks pretty much the same each time: "

#0  0xf771bea8 in instanceKlass::oop_adjust_pointers ()
   from libjvm.so
#1  0xf7937999 in PSMarkSweepDecorator::adjust_pointers ()
   from libjvm.so
#2  0xf79383af in PSOldGen::adjust_pointers () from libjvm.so
#3  0xf79370d6 in PSMarkSweep::mark_sweep_phase3 ()
   from libjvm.so
#4  0xf7936184 in PSMarkSweep::invoke_no_policy ()
   from libjvm.so
#5  0xf7940e7d in PSScavenge::invoke () from libjvm.so
#6  0xf790b40f in ParallelScavengeHeap::failed_mem_allocate ()
   from libjvm.so
#7  0xf79dfa6f in VM_ParallelGCFailedAllocation::doit ()
   from libjvm.so
#8  0xf79df466 in VM_Operation::evaluate () from libjvm.so
#9  0xf79dea77 in VMThread::evaluate_operation ()
   from libjvm.so
#10 0xf79dece5 in VMThread::loop () from libjvm.so
#11 0xf79de850 in VMThread::run () fromlibjvm.so
#12 0xf78fe338 in _start () from libjvm.so
#13 0xf7fd4341 in start_thread () from /lib/tls/libpthread.so.0
#14 0xf7f686fe in clone () from /lib/tls/libc.so.6 





> Namenode constantly using up 100% CPU
> -------------------------------------
>
>                 Key: HADOOP-751
>                 URL: http://issues.apache.org/jira/browse/HADOOP-751
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.8.0
>            Reporter: Koji Noguchi
>
> Trying to figure out the exact cause, but CPU load of the namenode is constantly 100%
without any file exchanges.
> .log file of the namenode showing the following exception about every 20 seconds.
> 2006-11-27 16:02:07,199 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020
call error: java.io.IOException: java.util.NoSuchElementException
> java.io.IOException: java.util.NoSuchElementException
>   at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1029)
>   at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:176)
>   at org.apache.hadoop.dfs.DatanodeDescriptor.getBlocks(DatanodeDescriptor.java:96)
>   at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1446)
>   at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:506)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:387)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:518)
> On some of the datanodes, .log file is showing
> 2006-11-27 16:03:57,542 WARN org.apache.hadoop.dfs.DataNode: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: java.util.NoSuchElementException
>   at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1029)
>   at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:176)
>   at org.apache.hadoop.dfs.DatanodeDescriptor.getBlocks(DatanodeDescriptor.java:96)
>   at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1446)
>   at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:506)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:387)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:518)
>   at org.apache.hadoop.ipc.Client$Connection.run(Client.java:248)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message