From hdfs-issues-return-235862-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Thu Sep 27 20:35:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0EAFE180652 for ; Thu, 27 Sep 2018 20:35:05 +0200 (CEST) Received: (qmail 12306 invoked by uid 500); 27 Sep 2018 18:35:04 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 11974 invoked by uid 99); 27 Sep 2018 18:35:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Sep 2018 18:35:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 8A06FC02C1 for ; Thu, 27 Sep 2018 18:35:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id agAkQMGm9C_e for ; Thu, 27 Sep 2018 18:35:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id D218B5F4DA for ; Thu, 27 Sep 2018 18:35:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 44F63E2633 for ; Thu, 27 Sep 2018 18:35:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9F90F23FB0 for ; Thu, 27 Sep 2018 18:35:00 +0000 (UTC) Date: Thu, 27 Sep 2018 18:35:00 +0000 (UTC) From: =?utf-8?Q?=C3=8D=C3=B1igo_Goiri_=28JIRA=29?= To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-13945) TestDataNodeVolumeFailure is Flaky MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-13945?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1663= 0875#comment-16630875 ]=20 =C3=8D=C3=B1igo Goiri commented on HDFS-13945: ------------------------------------ Thanks [~ayushtkn] for opening this. Can you give an overview of what the fix is? * Why is {{getInstanceStorageDir()}} not working and you have to use the fu= ll method? * Why you create the file later now? > TestDataNodeVolumeFailure is Flaky > ---------------------------------- > > Key: HDFS-13945 > URL: https://issues.apache.org/jira/browse/HDFS-13945 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Ayush Saxena > Assignee: Ayush Saxena > Priority: Major > Attachments: HDFS-13945-01.patch > > > The test is failing in trunk since long. > Reference - > [https://builds.apache.org/job/PreCommit-HDFS-Build/25140/testReport/juni= t/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnde= rReplicationAfterVolFailure/] > =C2=A0 > [https://builds.apache.org/job/PreCommit-HDFS-Build/25135/testReport/juni= t/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnde= rReplicationAfterVolFailure/] > =C2=A0 > [https://builds.apache.org/job/PreCommit-HDFS-Build/25133/testReport/juni= t/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnde= rReplicationAfterVolFailure/] > =C2=A0 > [https://builds.apache.org/job/PreCommit-HDFS-Build/25104/testReport/juni= t/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnde= rReplicationAfterVolFailure/] > =C2=A0 > =C2=A0 > Stack Trace - > =C2=A0 > Timed out waiting for condition. Thread diagnostics: Timestamp: 2018-09-2= 6 03:32:07,162 "IPC Server handler 2 on 33471" daemon prio=3D5 tid=3D2931 t= imed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(= Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupp= ort.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Cond= itionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.c= oncurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apa= che.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:288) at org.apac= he.hadoop.ipc.Server$Handler.run(Server.java:2668) "IPC Server handler 3 on= 34285" daemon prio=3D5 tid=3D2646 timed_waiting java.lang.Thread.State: TI= MED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.= locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.l= ocks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSy= nchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(Link= edBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(Ca= llQueueManager.java:288) at org.apache.hadoop.ipc.Server$Handler.run(Server= .java:2668) "org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d2ee4cd" daemo= n prio=3D5 tid=3D2633 timed_waiting java.lang.Thread.State: TIMED_WAITING a= t java.lang.Thread.sleep(Native Method) at org.apache.hadoop.util.JvmPauseM= onitor$Monitor.run(JvmPauseMonitor.java:192) at java.lang.Thread.run(Thread= .java:748) "IPC Server Responder" daemon prio=3D5 tid=3D2766 runnable java.= lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Nativ= e Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) = at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.= nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.Sel= ectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.ipc.Server$Resp= onder.doRunLoop(Server.java:1334) at org.apache.hadoop.ipc.Server$Responder= .run(Server.java:1317) "org.eclipse.jetty.server.session.HashSessionManager= @1287fc65Timer" daemon prio=3D5 tid=3D2492 timed_waiting java.lang.Thread.S= tate: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.con= current.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.conc= urrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abstract= QueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolEx= ecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java= .util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Schedule= dThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.ge= tTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecu= tor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadP= oolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run= (Thread.java:748) "qtp548667392-2533" daemon prio=3D5 tid=3D2533 timed_wait= ing java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Me= thod) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:= 215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.ut= il.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jett= y.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at or= g.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.ja= va:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThread= Pool.java:626) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.= hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@63818b03" daemon = prio=3D5 tid=3D2521 timed_waiting java.lang.Thread.State: TIMED_WAITING at = java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.name= node.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4045) at ja= va.lang.Thread.run(Thread.java:748) "BP-1973654218-172.17.0.2-1537975830395= heartbeating to localhost/127.0.0.1:43522" daemon prio=3D5 tid=3D2640 time= d_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Na= tive Method) at org.apache.hadoop.hdfs.server.datanode.IncrementalBlockRepo= rtManager.waitTillNextIBR(IncrementalBlockReportManager.java:158) at org.ap= ache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor= .java:717) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPS= erviceActor.java:844) at java.lang.Thread.run(Thread.java:748) "AsyncAppend= er-Dispatcher-Thread-38" daemon prio=3D5 tid=3D70 in Object.wait() java.lan= g.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native= Method) at java.lang.Object.wait(Object.java:502) at org.apache.log4j.Asyn= cAppender$Dispatcher.run(AsyncAppender.java:548) at java.lang.Thread.run(Th= read.java:748) "IPC Server listener on 43522" daemon prio=3D5 tid=3D2499 ru= nnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epo= llWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrappe= r.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java= :93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at su= n.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorIm= pl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener.r= un(Server.java:1159) "IPC Server handler 8 on 34285" daemon prio=3D5 tid=3D= 2651 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe= .park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Lo= ckSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronize= r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.= util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at o= rg.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:288) at or= g.apache.hadoop.ipc.Server$Handler.run(Server.java:2668) "nioEventLoopGroup= -30-1" prio=3D10 tid=3D2824 runnable java.lang.Thread.State: RUNNABLE at su= n.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArra= yWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.d= oSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSele= ct(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:97) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(Select= edSelectionKeySetSelector.java:62) at io.netty.channel.nio.NioEventLoop.sel= ect(NioEventLoop.java:754) at io.netty.channel.nio.NioEventLoop.run(NioEven= tLoop.java:410) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run= (SingleThreadEventExecutor.java:131) at io.netty.util.concurrent.DefaultThr= eadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at j= ava.lang.Thread.run(Thread.java:748) "IPC Server handler 7 on 43522" daemon= prio=3D5 tid=3D2515 timed_waiting java.lang.Thread.State: TIMED_WAITING at= sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupp= ort.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQ= ueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.jav= a:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueu= e.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager= .java:288) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2668) "q= tp1255640117-2483" daemon prio=3D5 tid=3D2483 runnable java.lang.Thread.Sta= te: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at su= n.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.E= PollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.Selector= Impl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.selec= t(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(Manag= edSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProduc= er.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strat= egy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:= 249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produc= eConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.s= trategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at or= g.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) a= t org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.jav= a:589) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.hdfs.ser= ver.datanode.DataXceiverServer@2b2f4b0a" daemon prio=3D5 tid=3D2815 runnabl= e java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.ac= cept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSoc= ketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(Serve= rSocketChannelImpl.java:250) at sun.nio.ch.ServerSocketAdaptor.accept(Serve= rSocketAdaptor.java:100) at org.apache.hadoop.hdfs.net.TcpPeerServer.accept= (TcpPeerServer.java:85) at org.apache.hadoop.hdfs.server.datanode.DataXceiv= erServer.run(DataXceiverServer.java:145) at java.lang.Thread.run(Thread.jav= a:748) "IPC Server handler 8 on 37367" daemon prio=3D5 tid=3D2779 timed_wai= ting > =C2=A0 > Timed out waiting for condition. Thread diagnostics: Timestamp: 2018-09-2= 6 03:32:07,162 "IPC Server handler 2 on 33471" daemon prio=3D5 tid=3D2931 t= imed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(= Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupp= ort.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Cond= itionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.c= oncurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apa= che.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:288) at org.apac= he.hadoop.ipc.Server$Handler.run(Server.java:2668) "IPC Server handler 3 on= 34285" daemon prio=3D5 tid=3D2646 timed_waiting java.lang.Thread.State: TI= MED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.= locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.l= ocks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSy= nchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(Link= edBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(Ca= llQueueManager.java:288) at org.apache.hadoop.ipc.Server$Handler.run(Server= .java:2668) "org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d2ee4cd" daemo= n prio=3D5 tid=3D2633 timed_waiting java.lang.Thread.State: TIMED_WAITING a= t java.lang.Thread.sleep(Native Method) at org.apache.hadoop.util.JvmPauseM= onitor$Monitor.run(JvmPauseMonitor.java:192) at java.lang.Thread.run(Thread= .java:748) "IPC Server Responder" daemon prio=3D5 tid=3D2766 runnable java.= lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Nativ= e Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) = at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.= nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.Sel= ectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.ipc.Server$Resp= onder.doRunLoop(Server.java:1334) at org.apache.hadoop.ipc.Server$Responder= .run(Server.java:1317) "org.eclipse.jetty.server.session.HashSessionManager= @1287fc65Timer" daemon prio=3D5 tid=3D2492 timed_waiting java.lang.Thread.S= tate: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.con= current.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.conc= urrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abstract= QueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolEx= ecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java= .util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Schedule= dThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.ge= tTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecu= tor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadP= oolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run= (Thread.java:748) "qtp548667392-2533" daemon prio=3D5 tid=3D2533 timed_wait= ing java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Me= thod) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:= 215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.ut= il.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jett= y.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at or= g.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.ja= va:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThread= Pool.java:626) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.= hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@63818b03" daemon = prio=3D5 tid=3D2521 timed_waiting java.lang.Thread.State: TIMED_WAITING at = java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.name= node.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4045) at ja= va.lang.Thread.run(Thread.java:748) "BP-1973654218-172.17.0.2-1537975830395= heartbeating to localhost/127.0.0.1:43522" daemon prio=3D5 tid=3D2640 time= d_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Na= tive Method) at org.apache.hadoop.hdfs.server.datanode.IncrementalBlockRepo= rtManager.waitTillNextIBR(IncrementalBlockReportManager.java:158) at org.ap= ache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor= .java:717) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPS= erviceActor.java:844) at java.lang.Thread.run(Thread.java:748) "AsyncAppend= er-Dispatcher-Thread-38" daemon prio=3D5 tid=3D70 in Object.wait() java.lan= g.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native= Method) at java.lang.Object.wait(Object.java:502) at org.apache.log4j.Asyn= cAppender$Dispatcher.run(AsyncAppender.java:548) at java.lang.Thread.run(Th= read.java:748) "IPC Server listener on 43522" daemon prio=3D5 tid=3D2499 ru= nnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epo= llWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrappe= r.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java= :93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at su= n.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorIm= pl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener.r= un(Server.java:1159) "IPC Server handler 8 on 34285" daemon prio=3D5 tid=3D= 2651 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe= .park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Lo= ckSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronize= r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.= util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at o= rg.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:288) at or= g.apache.hadoop.ipc.Server$Handler.run(Server.java:2668) "nioEventLoopGroup= -30-1" prio=3D10 tid=3D2824 runnable java.lang.Thread.State: RUNNABLE at su= n.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArra= yWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.d= oSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSele= ct(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:97) at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(Select= edSelectionKeySetSelector.java:62) at io.netty.channel.nio.NioEventLoop.sel= ect(NioEventLoop.java:754) at io.netty.channel.nio.NioEventLoop.run(NioEven= tLoop.java:410) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run= (SingleThreadEventExecutor.java:131) at io.netty.util.concurrent.DefaultThr= eadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at j= ava.lang.Thread.run(Thread.java:748) "IPC Server handler 7 on 43522" daemon= prio=3D5 tid=3D2515 timed_waiting java.lang.Thread.State: TIMED_WAITING at= sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupp= ort.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQ= ueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.jav= a:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueu= e.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager= .java:288) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2668) "q= tp1255640117-2483" daemon prio=3D5 tid=3D2483 runnable java.lang.Thread.Sta= te: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at su= n.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.E= PollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.Selector= Impl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.selec= t(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(Manag= edSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProduc= er.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strat= egy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:= 249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produc= eConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.s= trategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at or= g.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) a= t org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.jav= a:589) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.hdfs.ser= ver.datanode.DataXceiverServer@2b2f4b0a" daemon prio=3D5 tid=3D2815 runnabl= e java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.ac= cept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSoc= ketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(Serve= rSocketChannelImpl.java:250) at sun.nio.ch.ServerSocketAdaptor.accept(Serve= rSocketAdaptor.java:100) at org.apache.hadoop.hdfs.net.TcpPeerServer.accept= (TcpPeerServer.java:85) at org.apache.hadoop.hdfs.server.datanode.DataXceiv= erServer.run(DataXceiverServer.java:145) at java.lang.Thread.run(Thread.jav= a:748) "IPC Server handler 8 on 37367" daemon prio=3D5 tid=3D2779 timed_wai= ting -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org