hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Cabatoff (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HTTPCORE-483) High CPU usage after enabling chunked https responses
Date Wed, 30 Aug 2017 18:26:00 GMT

    [ https://issues.apache.org/jira/browse/HTTPCORE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147753#comment-16147753
] 

Nick Cabatoff commented on HTTPCORE-483:
----------------------------------------

Just thought you'd like to know that it's all working smoothly now that we've moved off the
deprecated APIs.  Thanks again for your help Oleg.

> High CPU usage after enabling chunked https responses
> -----------------------------------------------------
>
>                 Key: HTTPCORE-483
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-483
>             Project: HttpComponents HttpCore
>          Issue Type: Bug
>          Components: HttpCore NIO
>    Affects Versions: 4.4.6
>         Environment: Centos 7
>            Reporter: Nick Cabatoff
>         Attachments: fg1-initial-unchunked.svg, fg2-initial-chunked.svg, fg3-patched-unchunked.svg,
fg4-patched-chunked.svg
>
>
> I wanted persistent connections with keep-alives, and I discovered that the reason I
wasn't getting them was because my responses didn't have a content-length or transfer-encoding:
chunked.  So I did body.setChunked( true ), passed that to response.setEntity(), and it worked.
 But I noticed that my app was pegged at 100% CPU in top.  I found that the app was fine when
it started, but after servicing a single HTTPS request with chunked responses, it got stuck
in this 100% cpu state.  It turns out that the reactor thread was busy looping:  despite there
being no connections open after servicing that one request, in AbstractIOReactor.execute()
the selector.select() call continued to return non-zero, and processEvent() always found a
key that was writable.  Stepping through the code I noticed that SSLIOSession.updateEventMask()
wasn't closing the session:
> {noformat}
>         // Graceful session termination
>         if (this.status == CLOSING && this.sslEngine.isOutboundDone()
>                 && (this.endOfStream || this.sslEngine.isInboundDone())) {
>             this.status = CLOSED;
>         }
> {noformat}
> The status was CLOSING and endOfStream was true, but sslEngine.isOutboundDone() returned
false.  
> I looked at httpcore 5 and found that a new check had been added just above:
> {noformat}
> diff --git a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> --- a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> +++ b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> @@ -371,6 +371,11 @@
>  
>      private void updateEventMask() {
>          // Graceful session termination
> +        if (this.status == CLOSING && !this.outEncrypted.hasData()) {
> +            this.sslEngine.closeOutbound();
> +        }
> +
> +        // Graceful session termination
>          if (this.status == CLOSING && this.sslEngine.isOutboundDone()
>                  && (this.endOfStream || this.sslEngine.isInboundDone())) {
>              this.status = CLOSED;
> {noformat}
> I applied this patch to my httpcore 4.4.6 and the problem went away.  Now after servicing
a request with chunked responses the session gets closed properly and the reactor is properly
idle. 
> There's still another problem outstanding though.  Request processing now consumes way
more CPU than it used to.  Doing one request per second without chunked responses, my app
consumes ~20% of one core; with chunked responses it's closer to 100%.  
> I'm attaching flame graphs (see [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html]):
(1) my starting point without chunked respones; (2) with chunked respones; (3) after applying
the above patch, without chunked responses; (4) after applying patch, with chunked responses.
 Note the similarity between (2) and (4): in both cases, much of the Java code (green) is
mostly consumed with epoll calls.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message