hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <jason.had...@gmail.com>
Subject Re: Many child processes dont exit
Date Mon, 22 Feb 2010 16:09:12 GMT
Someone is using a threadpool that does not have daemon priority
threads, and that is not shutdown before the main method returns.
The non daemon threads prevent the jvm from exiting.
We had this problem for a while and modified the Child.main to exit,
rather than trying to work out and fix the third party library that
ran the thread pool. This thecnique does of  prevent jvm reuse.

On Sat, Feb 20, 2010 at 6:49 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> Do you have System.exit() as the last line in your main() ?
>    Job job = createSubmittableJob(conf, otherArgs);
>    System.exit(job.waitForCompletion(true)? 0 : 1);
>
>
> On Sat, Feb 20, 2010 at 12:32 AM, Zheng Lv <lvzheng19800619@gmail.com>wrote:
>
>> Hello Ted,
>>  Yes. Every hour a job will be created and started, and when it finished,
>> it will maintain. The logs looks like normal, do you know what can lead to
>> this happen?Thank you.
>>    LvZheng
>> 2010/2/20 Ted Yu <yuzhihong@gmail.com>
>>
>> > Did the number of child processes increase over time ?
>> >
>> > On Friday, February 19, 2010, Zheng Lv <lvzheng19800619@gmail.com>
>> wrote:
>> > > Hello Edson,
>> > >   Thank you for your reply. I don't want to kill them, I want to know
>> why
>> > > these child processes don't exit, and to know how to make them exit
>> > > successfully when they finish. Any ideas? Thank you.
>> > >     LvZheng
>> > >
>> > > 2010/2/18 Edson Ramiro <erlfilho@gmail.com>
>> > >
>> > >> Do you want to kill them ?
>> > >>
>> > >> if yes, you can use
>> > >>
>> > >>  ./bin/slaves.sh pkill java
>> > >>
>> > >> but it will kill the datanode and tasktracker processes
>> > >> in all slaves and you'll need to start these processes again.
>> > >>
>> > >> Edson Ramiro
>> > >>
>> > >>
>> > >> On 14 February 2010 22:09, Zheng Lv <lvzheng19800619@gmail.com>
>> wrote:
>> > >>
>> > >> > any idea?
>> > >> >
>> > >> > 2010/2/11 Zheng Lv <lvzheng19800619@gmail.com>
>> > >> >
>> > >> > > Hello Everyone,
>> > >> > >   We often find many child processes in datanodes, which
have
>> > already
>> > >> > > finished for long time. And following are the jstack log:
>> > >> > >   Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01
>> mixed
>> > >> > mode):
>> > >> > > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422
waiting
>> on
>> > >> > > condition [0x0000000000000000]
>> > >> > >    java.lang.Thread.State: RUNNABLE
>> > >> > > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826
>> runnable
>> > >> > > [0x000000004100a000]
>> > >> > >    java.lang.Thread.State: RUNNABLE
>> > >> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native
Method)
>> > >> > >         at
>> > >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> > >> > >         at
>> > >> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> > >> > >         at
>> > >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> > >> > >         - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
>> > >> > >         - locked <0x00002aaab9b5f710> (a
>> > >> > > java.util.Collections$UnmodifiableSet)
>> > >> > >         - locked <0x00002aaab9b5f680> (a
>> > sun.nio.ch.EPollSelectorImpl)
>> > >> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > >> > >         at java.lang.Thread.run(Thread.java:619)
>> > >> > > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825
>> waiting
>> > on
>> > >> > > condition [0x0000000041604000]
>> > >> > >    java.lang.Thread.State: WAITING (parking)
>> > >> > >         at sun.misc.Unsafe.park(Native Method)
>> > >> > >         - parking to wait for  <0x00002aaab9b61620>
(a
>> > >> > >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > >> > >         at
>> > >> > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> > >> > >         at
>> > >> > >
>> > >> >
>> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa
>> >
>>
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Mime
View raw message