commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Gamache <cgama...@gmail.com>
Subject Re: Help needed: commons-exec CLOSE_WAIT problem
Date Sat, 06 Feb 2016 02:53:23 GMT
I saw that! I couldn't have done it without you. Many thanks!

CG

Sent from my iPhone

> On Feb 5, 2016, at 12:24 PM, Siegfried Goeschl <siegfried.goeschl@gmail.com> wrote:
> 
> Hi Chris,
> 
> according to https://github.com/SeleniumHQ/selenium/pull/1545 the PR is merged and closed
:-)
> 
> Cheers,
> 
> Siegfried Goeschl
> 
>> On 27.01.2016, at 14:52, Siegfried Göschl <siegfried.goeschl@it20one.com>
wrote:
>> 
>> Hi Chris,
>> 
>> glad to help - please note that I took the freedom to post my findings at https://github.com/SeleniumHQ/selenium/issues/1080
- could you do the same :-)
>> 
>> Cheers,
>> 
>> Siegfried Goeschl
>> 
>> 
>> ----- Ursprüngliche Mail -----
>> Von: "Chris Gamache" <cgamache@gmail.com>
>> An: "Commons Users List" <user@commons.apache.org>
>> Gesendet: Mittwoch, 27. Januar 2016 14:43:32
>> Betreff: Re: Help needed: commons-exec CLOSE_WAIT problem
>> 
>> I really appreciate the thorough inspection you have given. With your help
>> I think I've targeted the real problem...
>> 
>> GhostDriver can't exist outside of phantomjs. GhostDriver is an
>> implementation of WebDriver Wire Protocol which runs on phantomjs's
>> JavaScript engine. It can't run without phantomjs.
>> 
>> We can rest easy that commons-exec is not holding open the file handle.
>> That's good. I think the PhantomJSDriver is firing a copy of phantomjs up
>> using commons-exec, but it's not communicating with the service through
>> commons-exec. Upon closer inspection selenium-java is communicating through
>> tcp/ip to a WebDriver server phantomjs is popping up on random available
>> ports. This happens to be an instance of some class that implements
>> org.apache.http.client.HttpClient, by default it looks to
>> be org.apache.http.impl.client.CloseableHttpClient ... I think that when
>> commons-exec terminates phantomjs, thereby terminating the GhostDriver
>> server, the HttpClient is still attached to that port.
>> 
>> Here's my take on the real problem:
>> 
>> I think the selenium folks didn't consider port closing
>> (org.apache.http.client.HttpClient doesn't have a close method to
>> implement) a necessary step and they figured that if whatever service the
>> HttpClient was attached to shut itself down that the port would
>> automatically close itself down also. The standard use for selenium usually
>> doesn't require it to run for days and days on end. I can't fault them for
>> missing it.
>> 
>> That being said, they made this difficult to override in consumer classes
>> by keeping a lot of the guts of the connections hidden behind private
>> classes and private member data.
>> 
>> So, I'll need to dig into selenium-java, change a few interfaces to make
>> sure that whatever HttpClient is used will have a close() hook, and then
>> make sure that whatever is driving the HttpClient has access to the close
>> hook on shutdown, and fires it. When the HttpClient is closed, the file
>> descriptor should disappear.
>> 
>> Thank you again for your help. It was indispensable!
>> 
>> CG
>> 
>> 
>> On Tue, Jan 26, 2016 at 4:07 PM, Siegfried Göschl <
>> siegfried.goeschl@it20one.com> wrote:
>> 
>>> Hi Chris,
>>> 
>>> I played with your GitHub repo [
>>> https://github.com/cgamache/openfile](https://github.com/cgamache/openfile)
>>> 
>>> ### 1. First Take
>>> 
>>> Your test program actually starts a JVM and the Selenium library start
>>> multiple "phantomjs" executable to run the test as shown below
>>> 
>>> ```
>>> application> ps -ef | grep phantom
>>> 501  1021   460   0  9:26PM ttys000    0:00.71 java -jar openfile.jar
>>> /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs
>>> 501  1073  1062   0  9:26PM ttys000    0:00.08
>>> /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs --webdriver=36418
>>> --webdriver-logfile=/Users/sgoeschl/work/github/cgamache/openfile/target/phantomjsdriver.log
>>> ```
>>> 
>>> So commons-exec is used under the hood to run the "phantomjs" binary and
>>> make sure nothing hangs (output & error thread pumping, watchdog, ..)
>>> 
>>> When the "WebDriver" has finished its work there
>>> 
>>> * is no ***phantomjs*** process running
>>> * is only your program running as shown below
>>> 
>>> ```
>>> application> ps -ef | grep phantom
>>> 501  1021   460   0  9:23PM ttys000    0:00.72 java -jar openfile.jar
>>> /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs
>>> 501  1034   469   0  9:23PM ttys001    0:00.00 grep phantom
>>> ```
>>> 
>>> And as described we have a couple of ports in CLOSE_WAIT stemming from
>>> your program
>>> 
>>> ```
>>> application> lsof -p 1021 | grep CLOSE_WAIT
>>> java    1021 sgoeschl   16u    IPv6 0x78418468470e25c7       0t0     TCP
>>> localhost:50581->localhost:40560 (CLOSE_WAIT)
>>> java    1021 sgoeschl   18u    IPv6 0x784184685efd95a7       0t0     TCP
>>> localhost:50599->localhost:45319 (CLOSE_WAIT)
>>> java    1021 sgoeschl   19u    IPv6 0x784184685efdc0a7       0t0     TCP
>>> localhost:50620->localhost:17950 (CLOSE_WAIT)
>>> java    1021 sgoeschl   20u    IPv6 0x78418468470e3087       0t0     TCP
>>> localhost:50641->localhost:41916 (CLOSE_WAIT)
>>> java    1021 sgoeschl   21u    IPv6 0x78418468470e4b67       0t0     TCP
>>> localhost:50661->localhost:25113 (CLOSE_WAIT)
>>> ```
>>> 
>>> As far as common-exec is concerned everything is fine - the
>>> ***phantomjs*** processes where nicely executed and no process is left
>>> 
>>> So you have 5 ports in limbo which exactly corresponds to the number of
>>> created ***phantom.js*** instances
>>> 
>>> ```
>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs
>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs
>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs
>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs
>>> INFO: executable: /usr/local/Cellar/phantomjs/2.1.0/bin/phantomjs
>>> ```
>>> 
>>> This nicely corresponds to your test code
>>> 
>>> ```
>>> public static final int REPS = 5;
>>> 
>>> for (; i < REPS; i++) {
>>>  WebDriver driver = new PhantomJSDriver(caps);
>>>  driver.get("http://www.apache.org");
>>>  driver.quit();
>>> }
>>> ```
>>> 
>>> In other words each ***phantomjs*** process leaves exactly on port behind
>>> :-)
>>> 
>>> 
>>> ### 2. Second Take
>>> 
>>> So where is this port coming from? The next session - starting the test
>>> programm leaves the 5 ports behind
>>> 
>>> ```
>>> application> lsof -p 1329 | grep CLOSE_WAIT
>>> java    1329 sgoeschl   16u    IPv6 0x784184686037bb07       0t0     TCP
>>> localhost:51358->localhost:30177 (CLOSE_WAIT)
>>> java    1329 sgoeschl   18u    IPv6 0x784184685f5c3b47       0t0     TCP
>>> localhost:51379->localhost:cadkey-tablet (CLOSE_WAIT)
>>> java    1329 sgoeschl   19u    IPv6 0x784184686037f0c7       0t0     TCP
>>> localhost:51400->localhost:43648 (CLOSE_WAIT)
>>> java    1329 sgoeschl   20u    IPv6 0x784184686037c5c7       0t0     TCP
>>> localhost:51418->localhost:34505 (CLOSE_WAIT)
>>> java    1329 sgoeschl   21u    IPv6 0x78418468470e1047       0t0     TCP
>>> localhost:51438->localhost:30905 (CLOSE_WAIT)
>>> ```
>>> 
>>> All of ports are actually found in phantomjsdriver.log (cadkey-tablet is
>>> actually port 1400)
>>> 
>>> ```
>>> [INFO  - 2016-01-26T20:51:19.697Z] GhostDriver - Main - running on port
>>> 30177
>>> [INFO  - 2016-01-26T20:51:21.685Z] GhostDriver - Main - running on port
>>> 1400
>>> [INFO  - 2016-01-26T20:51:23.427Z] GhostDriver - Main - running on port
>>> 43648
>>> [INFO  - 2016-01-26T20:51:25.156Z] GhostDriver - Main - running on port
>>> 34505
>>> [INFO  - 2016-01-26T20:51:26.887Z] GhostDriver - Main - running on port
>>> 30905
>>> ```
>>> 
>>> So the ports are actually used/opened for ***GhostDriver*** and left there
>>> even when ***driver.quit()*** was called
>>> 
>>> 
>>> ### 3. Conclusion
>>> 
>>> * ASFAIK it is not commons-exec issue since all ***phantomjs*** were
>>> cleanly terminated
>>> * Each ***PhantomJSDriver*** invocation leaves a port in ***CLOSE_WAIT***
>>> behind
>>> * Those ports are linked to ***GhostDriver - Main - running on port
>>> 30177***
>>> 
>>> 
>>> Cheers,
>>> 
>>> Siegfried Goeschl
>>> 
>>> 
>>> 
>>> ----- Ursprüngliche Mail -----
>>> Von: "Chris Gamache" <cgamache@gmail.com>
>>> An: "Commons Users List" <user@commons.apache.org>
>>> Gesendet: Dienstag, 26. Januar 2016 04:02:41
>>> Betreff: Re: Help needed: commons-exec CLOSE_WAIT problem
>>> 
>>> Most defnintely. This was tested on OSX, but should work properly on Linux
>>> also. You'll need to install phantomjs.
>>> 
>>> You can clone this repo (https://github.com/cgamache/openfile) and build
>>> with Maven. The com.codeborne:phantomjsdriver artifact has all of selenium
>>> packaged inside.
>>> 
>>> To run,
>>> 
>>> $ java -jar openfile.jar /path/to/phantomjs
>>> 
>>> To examine,
>>> 
>>> $ lsof -p 38270 | grep CLOSE_WAIT | wc -l
>>> 
>>> Jan 25, 2016 9:46:17 PM
>>> org.openqa.selenium.phantomjs.PhantomJSDriverService <init>
>>> 
>>> ....... super chatty........
>>> 
>>> [INFO  - 2016-01-26T02:46:41.169Z] ShutdownReqHand - _handle - About to
>>> shutdown
>>> 
>>> 
>>> Sleeping for 30 seconds so you can examine file handles... you should see 5
>>> 
>>> Then you can use the lsof command at the top of the output, complete with
>>> process ID, to check the CLOSE_WAIT file handles.
>>> 
>>> Thanks for taking a look!
>>> 
>>> CG
>>> 
>>> 
>>> 
>>> On Mon, Jan 25, 2016 at 5:28 PM, Siegfried Göschl <
>>> siegfried.goeschl@it20one.com> wrote:
>>> 
>>>> Hi Chris,
>>>> 
>>>> there could be couple of reasons for this behaviour - is there a minimal
>>>> setup to reproduce the problem?
>>>> 
>>>> Thanks in advance
>>>> 
>>>> Siegfried Goeschl
>>>> 
>>>> ----- Ursprüngliche Mail -----
>>>> Von: "Chris Gamache" <cgamache@gmail.com>
>>>> An: user@commons.apache.org
>>>> Gesendet: Montag, 25. Januar 2016 22:34:34
>>>> Betreff: Help needed: commons-exec CLOSE_WAIT problem
>>>> 
>>>> Hi commons-exec folks,
>>>> 
>>>> Hoping you can help me figure this out. Selenium Java uses commons-exec
>>> 1.3
>>>> under the hood to communicate with phantomjs. When it fires up it opens
>>> up
>>>> a pipe that you can see with lsof:
>>>> 
>>>> # lsof -p 19947
>>>> 
>>>> ...
>>>> java    19947 user   64     PIPE 0xd2cd00ccca85f9d     16384
>>>> ->0xd2cd00ca9fbbf9d
>>>> java    19947 user   66     PIPE 0xd2cd00ca31d445d     16384
>>>> ->0xd2cd00ca31d4c9d
>>>> 
>>>> Then we get this in lsof as selenium is driving phantomjs:
>>>> 
>>>> java    19947 user   62u    IPv6 0xd2cd00cc904879d       0t0      TCP
>>>> localhost:49757->localhost:23795 (ESTABLISHED)
>>>> 
>>>> Then after selenium closes and terminates the executor-- properly, as I
>>>> observed by stepping through the code as it executes, but maybe someone
>>>> knows otherwise-- we can see in lsof:
>>>> 
>>>> java    19947 user  62u    IPv6 0xd2cd00cc904879d       0t0      TCP
>>>> localhost:49757->localhost:23795 (CLOSE_WAIT)
>>>> 
>>>> And each successive web driver instance that gets created cause those to
>>>> build up and build up and build up until you run out of file handles.
>>>> 
>>>> ... So are there any considerations for using commons-exec that the
>>>> selenium folks might not be addressing during process destruction which
>>>> might manifest themselves in these file handles just hanging out, taking
>>> up
>>>> space? This can't be the correct/uncorrectable behavior.
>>>> 
>>>> I happen to be using Selenium Java 2.49.1 which is the latest version as
>>> of
>>>> this moment, and Java 8. It seems like this has been broken for quite
>>> some
>>>> time though -- https://github.com/SeleniumHQ/selenium/issues/1080
>>>> 
>>>> Please advise! Thanks!
>>>> 
>>>> CG
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>>> For additional commands, e-mail: user-help@commons.apache.org
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>> For additional commands, e-mail: user-help@commons.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


Mime
View raw message