From dev-return-23680-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Wed Oct 31 19:28:12 2012 Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4AEBADDF3 for ; Wed, 31 Oct 2012 19:28:12 +0000 (UTC) Received: (qmail 3941 invoked by uid 500); 31 Oct 2012 19:28:11 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 3888 invoked by uid 500); 31 Oct 2012 19:28:11 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 3827 invoked by uid 99); 31 Oct 2012 19:28:11 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Oct 2012 19:28:11 +0000 Date: Wed, 31 Oct 2012 19:28:11 +0000 (UTC) From: "Adam Kocoloski (JIRA)" To: dev@couchdb.apache.org Message-ID: <1994100751.52552.1351711691564.JavaMail.jiratomcat@arcas> In-Reply-To: <1056915614.51010.1351700713425.JavaMail.jiratomcat@arcas> Subject: [jira] [Commented] (COUCHDB-1583) improve the flow control in ibrowse streams MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/COUCHDB-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488134#comment-13488134 ] Adam Kocoloski commented on COUCHDB-1583: ----------------------------------------- Thanks for opening this ticket Bob. Sorry for not firing off a topic branch sooner. When we started using the new replicator internally at Cloudant we noticed that long-running replications would ultimately consume a ton of memory on the node mediating the replication and crash. I tracked it down to the mailbox for the process consuming the _changes feed over HTTP, and from there to a misuse of ibrowse's stream_next API. Admittedly the API is super-confusing; it should have a big read warning box telling you that you may receive multiple messages for each invocation of the function. The patch is working well for us, and given that the commit landed directly on master I'm going to go ahead and resolve this one. > improve the flow control in ibrowse streams > ------------------------------------------- > > Key: COUCHDB-1583 > URL: https://issues.apache.org/jira/browse/COUCHDB-1583 > Project: CouchDB > Issue Type: Improvement > Components: Replication > Reporter: Bob Dionne > Assignee: Adam Kocoloski > Fix For: 1.3 > > > The flow control in ibrowse's async response streams is tricky. We call > stream_next to pull more data off the socket, but it seems that ibrowse > will sometimes split that data into multiple messages. If we call > stream_next for each message we process we end up with an overflowing > mailbox. > This commit[1] changes the consumer so that it clears out the mailbox before > calling stream_next. > [1] 8ccf696f833a0f0a453d -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira