commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Gamache <cgama...@gmail.com>
Subject Re: [Re] Proper use of Executors
Date Wed, 21 Jun 2017 15:48:04 GMT
Ok so, it's hard to tell if this made much of a difference. There might be
a few less messages? Hard to be sure.

I'm still getting those messages, but I'm now suspecting Selenium is
generating the majority of these messages. It is behaving badly when it
comes to killing its spawned phantomjs processes, thereby leaving the
executors open. Seems like a recurring theme, eh Siegfried? ;)

Selenium is co-opting the Apache Commons Exec. It looks like it is using
ExecuteWatchdog.INFINITE_TIMEOUT, so no help there. I see they are making a
pretty good effort to kill the processes. I won't rule-out that I'm doing
it wrong, but I'm pretty sure

} finally {
  driver.quit();
}

is all I'm responsible for if I want Selenium to close phantomjs processes.
I do have some room to upgrade Selenium and PhantomJS which might help. I
didn't see any meaningful changes in Selenium that would lead me to believe
that an upgrade would fix the issue. But, it's a good practice and any
patches will have to be made at the head of the project anyway.

Thanks for all the help!

CG


On Mon, Jun 19, 2017 at 4:11 PM, Siegfried Goeschl <
siegfried.goeschl@it20one.com> wrote:

> Staying tuned :-)
>
> > On 5 Jun 2017, at 13:59, Chris Gamache <cgamache@gmail.com> wrote:
> >
> > Hi Siegfried,
> >
> > I have implemented the executor watchdogs and shutdown hooks. I will
> know if our efforts have been fruitful when we make our first restart which
> I'm sure will be soon. I promise I'll write back and let you know how it
> goes.
> >
> > Thank you vary much for checking in with me!
> >
> >> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <
> siegfried.goeschl@it20one.com> wrote:
> >>
> >> Hi Chris,
> >>
> >> any new findings from your side?
> >>
> >> Thanks in advance,
> >>
> >> Siegfried Goeschl
> >>
> >>> On 28 May 2017, at 21:41, Siegfried Goeschl <
> siegfried.goeschl@it20one.com> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> there are couple of things to consider
> >>>
> >>> * You are using a PumpStreamHander but the STDERR is not consumed.
> Each process has an internal buffer (size depends on the OS) and when the
> buffer is full any write to STDERR is blocked
> >>> * That could happen if the process being executed actually writes some
> error messages :-)
> >>> * Are you 100% sure that the processes will terminate? See
> ExecuteWatchdog
> >>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
> >>>
> >>> Thanks in advance,
> >>>
> >>> Siegfried Goeschl
> >>>
> >>>
> >>>> On 27 May 2017, at 14:27, Chris Gamache <cgamache@gmail.com> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
> >>>>
> >>>> I'm having an issue where my Tomcat server is bleeding out hundreds
of
> >>>> threads and all of the memory in the form of Executors that I'm
> running but
> >>>> don't seem to be closing down ... When the server finally grinds to
a
> halt
> >>>> I have to restart. When I do it looks like this at shutdown time:
> >>>>
> >>>> <snip>
> >>>>
> >>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> [Exec
> >>>> Default Executor] but has failed to stop it. This is very likely to
> create
> >>>> a memory leak. Stack trace of thread:
> >>>> java.lang.Object.wait(Native Method)
> >>>> java.lang.Object.wait(Object.java:502)
> >>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> >>>> org.apache.commons.exec.DefaultExecutor.executeInternal(
> DefaultExecutor.java:364)
> >>>> org.apache.commons.exec.DefaultExecutor.access$200(
> DefaultExecutor.java:48)
> >>>> org.apache.commons.exec.DefaultExecutor$1.run(
> DefaultExecutor.java:200)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> [Exec
> >>>> Stream Pumper] but has failed to stop it. This is very likely to
> create a
> >>>> memory leak. Stack trace of thread:
> >>>> java.io.FileInputStream.readBytes(Native Method)
> >>>> java.io.FileInputStream.read(FileInputStream.java:255)
> >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> >>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
> >>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> </snip>
> >>>>
> >>>> And my thread dump is a mile long.
> >>>>
> >>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's
> the
> >>>> relevant code. It is called from a regular method in a regular class,
> >>>> nothing fancy:
> >>>>
> >>>> CommandLine cmdLine = CommandLine.parse(command.toString());
> >>>> DefaultExecutor executor = new DefaultExecutor();
> >>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> >>>> executor.setStreamHandler(esh);
> >>>> executor.execute(cmdLine);
> >>>>
> >>>> `is` and `os` are passed in on the constructor. Their opens and
> closes are
> >>>> managed well and cleaned up on the outside of this class...
> >>>> Are there further steps I'm missing to ensure the threads I'm
> creating are
> >>>> getting shut down properly and the resources they are using are being
> >>>> returned?
> >>>>
> >>>> Any help is much appreciated.
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message