Mailing-List: contact dev-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpComponents Project" <dev@hc.apache.org>
Date: Wed, 23 Aug 2017 22:03:00 +0000 (UTC)
From: "Oleg Kalnichevski (JIRA)" <jira@apache.org>
To: dev@hc.apache.org
Message-ID: <JIRA.13097035.1503498312000.106078.1503525780202@Atlassian.JIRA>
In-Reply-To: <JIRA.13097035.1503498312000@Atlassian.JIRA>
References: <JIRA.13097035.1503498312000@Atlassian.JIRA> <JIRA.13097035.1503498312139@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (HTTPCORE-483) High CPU usage after
 enabling chunked https responses
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 23 Aug 2017 22:03:07 -0000


    [ https://issues.apache.org/jira/browse/HTTPCORE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139216#comment-16139216 ] 

Oleg Kalnichevski edited comment on HTTPCORE-483 at 8/23/17 10:02 PM:
----------------------------------------------------------------------

I am not sure what exactly this proves or disproves. With 11 byte payloads chunk-coding effectively doubles the size of encoded message content. 

You are very welcome to suggest ways of reducing epolling or optimizing {{ChunkDecoder}} though.

PS: Not to mention that chunk-coding may cause messages to get split into multiple IP frames, which would explain extra epolling.

Oleg


was (Author: olegk):
I am not sure what exactly this proves or disproves. With 11 byte payloads chunk-coding effectively doubles the size of encoded message content. 

You are very welcome to suggest ways of reduce epolling or optimizing {{ChunkDecoder}} though.

Oleg

> High CPU usage after enabling chunked https responses
> -----------------------------------------------------
>
>                 Key: HTTPCORE-483
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-483
>             Project: HttpComponents HttpCore
>          Issue Type: Bug
>          Components: HttpCore NIO
>    Affects Versions: 4.4.6
>         Environment: Centos 7
>            Reporter: Nick Cabatoff
>         Attachments: fg1-initial-unchunked.svg, fg2-initial-chunked.svg, fg3-patched-unchunked.svg, fg4-patched-chunked.svg
>
>
> I wanted persistent connections with keep-alives, and I discovered that the reason I wasn't getting them was because my responses didn't have a content-length or transfer-encoding: chunked.  So I did body.setChunked( true ), passed that to response.setEntity(), and it worked.  But I noticed that my app was pegged at 100% CPU in top.  I found that the app was fine when it started, but after servicing a single HTTPS request with chunked responses, it got stuck in this 100% cpu state.  It turns out that the reactor thread was busy looping:  despite there being no connections open after servicing that one request, in AbstractIOReactor.execute() the selector.select() call continued to return non-zero, and processEvent() always found a key that was writable.  Stepping through the code I noticed that SSLIOSession.updateEventMask() wasn't closing the session:
> {noformat}
>         // Graceful session termination
>         if (this.status == CLOSING && this.sslEngine.isOutboundDone()
>                 && (this.endOfStream || this.sslEngine.isInboundDone())) {
>             this.status = CLOSED;
>         }
> {noformat}
> The status was CLOSING and endOfStream was true, but sslEngine.isOutboundDone() returned false.  
> I looked at httpcore 5 and found that a new check had been added just above:
> {noformat}
> diff --git a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> --- a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> +++ b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> @@ -371,6 +371,11 @@
>  
>      private void updateEventMask() {
>          // Graceful session termination
> +        if (this.status == CLOSING && !this.outEncrypted.hasData()) {
> +            this.sslEngine.closeOutbound();
> +        }
> +
> +        // Graceful session termination
>          if (this.status == CLOSING && this.sslEngine.isOutboundDone()
>                  && (this.endOfStream || this.sslEngine.isInboundDone())) {
>              this.status = CLOSED;
> {noformat}
> I applied this patch to my httpcore 4.4.6 and the problem went away.  Now after servicing a request with chunked responses the session gets closed properly and the reactor is properly idle. 
> There's still another problem outstanding though.  Request processing now consumes way more CPU than it used to.  Doing one request per second without chunked responses, my app consumes ~20% of one core; with chunked responses it's closer to 100%.  
> I'm attaching flame graphs (see [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html]): (1) my starting point without chunked respones; (2) with chunked respones; (3) after applying the above patch, without chunked responses; (4) after applying patch, with chunked responses.  Note the similarity between (2) and (4): in both cases, much of the Java code (green) is mostly consumed with epoll calls.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org