brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BROOKLYN-467) HttpFeed sometimes slow to detect https connection failure: tweak timeouts?
Date Wed, 05 Apr 2017 15:47:41 GMT
Aled Sage created BROOKLYN-467:
----------------------------------

             Summary: HttpFeed sometimes slow to detect https connection failure: tweak timeouts?
                 Key: BROOKLYN-467
                 URL: https://issues.apache.org/jira/browse/BROOKLYN-467
             Project: Brooklyn
          Issue Type: Improvement
    Affects Versions: 0.10.0
            Reporter: Aled Sage
            Priority: Minor


While testing an entity's {{HttpFeed}}, to confirm that it marked the entity as on-fire when
unreachable, it sometimes took a surprising long time to respond to the https failure.

Most times it took less than a couple of seconds, but sometimes it took approx 4 minutes.

To simulate failure of the entity, I executed (on the entity's VM):
{noformat}
sudo /sbin/iptables -I INPUT -p TCP --dport 8443 -j REJECT
{noformat}

Looking at the output of jstack in the Brooklyn instance, I saw the thread below stuck at
this point for several minutes:
{noformat}
"brooklyn-execmanager-acAIl5eD-842" #4280 daemon prio=5 os_prio=31 tid=0x00007ff8d89dc000
nid=0x16637 runnable [0x0000700009a86000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        - locked <0x00000007be3657c0> (a java.lang.Object)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        - locked <0x00000007be365880> (a java.lang.Object)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:553)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:412)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.brooklyn.util.http.HttpTool.execAndConsume(HttpTool.java:578)
        at org.apache.brooklyn.util.http.HttpTool.httpGet(HttpTool.java:527)
        at org.apache.brooklyn.util.http.executor.apacheclient.HttpExecutorImpl.execute(HttpExecutorImpl.java:78)
        at org.apache.brooklyn.feed.http.HttpFeed$2.call(HttpFeed.java:385)
        at org.apache.brooklyn.feed.http.HttpFeed$2.call(HttpFeed.java:373)
        at org.apache.brooklyn.core.feed.Poller$PollJob$1.run(Poller.java:75)
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:157)
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:150)
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:363)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:451)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:529)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

This suggests to me that it's caused by a very long tcp timeout. I'm guessing that when I
configured iptables, it was in the middle of negotiating the https connection. It sits there
waiting for an appropriate response, before timing out.

Perhaps we should configure more aggressive timeouts?




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message