From dev-return-4985-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Mon Jul 06 18:02:54 2009 Return-Path: Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: (qmail 63788 invoked from network); 6 Jul 2009 18:02:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 6 Jul 2009 18:02:54 -0000 Received: (qmail 48842 invoked by uid 500); 6 Jul 2009 18:03:04 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 48790 invoked by uid 500); 6 Jul 2009 18:03:04 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 48780 invoked by uid 99); 6 Jul 2009 18:03:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 18:03:04 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of adam.kocoloski@gmail.com designates 209.85.198.239 as permitted sender) Received: from [209.85.198.239] (HELO rv-out-0506.google.com) (209.85.198.239) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 18:02:54 +0000 Received: by rv-out-0506.google.com with SMTP id g37so1154893rvb.35 for ; Mon, 06 Jul 2009 11:02:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=j42I38kvxsBZ3jlS2NT6wMCOBucWhtRmPtbpx7rvPaE=; b=W1uyICbIMnbh8Wh6zIuovpCWVFUWVtgpdIRAfFpmp1zQ79JBrOs/aNadk/PZg+X9P7 siPMkeWBrcex878TmDzZKJepqna+ZjCtuoOxh6Djjd4EdA2XQ56X3H/pC0LznYlcLdKf PhBU1JlNUeaIwos0mI2+Hz8QmAN8LgJfr9pcQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=WC1syBxK19u894usP/Pe06egyr0TLk76wwCNX/9k/q/sNRyqHlRBOEdlQInnON1ZoE qq4+l0Xc1yD0Ph6QcjhWEiQvfDf0Mraf/jm55pcYppgjo6L62d8NDDJoQd5ZtbnbolGV ikH5kSBSeo6KrqPNRpBHZPwEJjGhsmzqH47Kk= Received: by 10.140.255.19 with SMTP id c19mr1573209rvi.10.1246903349349; Mon, 06 Jul 2009 11:02:29 -0700 (PDT) Received: from COMPTON-SEVEN-THIRTY-SEVEN.MIT.EDU (COMPTON-SEVEN-THIRTY-SEVEN.MIT.EDU [18.109.7.226]) by mx.google.com with ESMTPS id g22sm3186732rvb.15.2009.07.06.11.02.26 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 06 Jul 2009 11:02:28 -0700 (PDT) Sender: Adam Kocoloski Message-Id: From: Adam Kocoloski To: dev@couchdb.apache.org In-Reply-To: <08DABC07-CD7E-4A4B-A297-906B94A52F4C@erlang-consulting.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Subject: Re: Timetable for 0.10.0 Date: Mon, 6 Jul 2009 14:02:25 -0400 References: <3418A46D-FE98-4782-9D5A-D45DE1FD899D@apache.org> <20090703232147.GC7898@tumbolia.org> <51301D85-CDAD-4341-88EC-10D95616EE60@apache.org> <08DABC07-CD7E-4A4B-A297-906B94A52F4C@erlang-consulting.com> X-Mailer: Apple Mail (2.935.3) X-Virus-Checked: Checked by ClamAV on apache.org Hi Tamas, in this case I'm talking about letting the user decide when to switch the socket back into {active,once} mode to receive the next message. We like this feature because it allows us to incrementally process the result of a call to _changes without using any memory to actually store the response (which may be 100s of MBs). I poked around lhttpc when ETC released it, but you're right that it's still missing a number of features that we use in the replicator. Off the top of my head I can name * chunked downloads * chunked uploads * streaming response bodies Regarding binaries, our solution so far has been to make requests that we can reasonably expect to have large response bodies (e.g. attachment downloads) on dedicated connections outside the ibrowse connection pool and garbage_collect() those processes as necessary. Memory management is certainly a concern for us, and I'm curious to see how the new version of ibrowse behaves now that it's using binaries internally as well. Thanks for bringing lhttpc to our attention, but at the moment it seems that only ibrowse offers the features that we would like in our replicator HTTP client. Cheers, Adam On Jul 6, 2009, at 6:00 AM, Tamas Nagy wrote: > Hi Adam, > > What kind of control of the socket behaviour? lhttpc might be a good > candidate as well as it is steadily building up its feature set with > things which are necessary for couchdb. (like chunked). > Arguably ibrowse is a much mature client supporting a lot of > different options (and lhttpc might not have all the required > features yet), but with the recent introduction of using binaries > combined with the long lived processes inside ibrowse can result in > nasty memory blowups as binaries are reference counted in the VM > hence the GC might not be able to get rid of the huge binaries fast > enough during data transfer. > > Regards, > Tamas > > Tamas Nagy > Erlang Training & Consulting > http://www.erlang-consulting.com > > On 4 Jul 2009, at 01:02, Adam Kocoloski wrote: > >> On Jul 3, 2009, at 7:28 PM, Chris Anderson wrote: >> >>> Especially if we can get the replicator based on _changes, and >>> then truly deprecate the update_notification process >> >> Chandru Mullaparthi gave us a nice assist on that front today with >> an update to ibrowse that lets us control the socket behavior. As >> far as I know ibrowse is the only Erlang HTTP client that does this >> correctly. One month will be more than enough time to build a >> replicator based on _changes now that this piece of the puzzle is >> resolved. >> >> Adam >> >