From user-return-10142-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Mon Apr 19 08:06:24 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 4065 invoked from network); 19 Apr 2010 08:06:23 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 19 Apr 2010 08:06:23 -0000 Received: (qmail 82776 invoked by uid 500); 19 Apr 2010 08:06:22 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 82595 invoked by uid 500); 19 Apr 2010 08:06:20 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 82587 invoked by uid 99); 19 Apr 2010 08:06:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 08:06:19 +0000 X-ASF-Spam-Status: No, hits=1.7 required=10.0 tests=AWL,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of fredrik.widlund@qbrick.com designates 62.13.40.40 as permitted sender) Received: from [62.13.40.40] (HELO mail.qbrick.com) (62.13.40.40) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 08:06:14 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.qbrick.com (Postfix) with ESMTP id 6DFB8519E2 for ; Mon, 19 Apr 2010 10:05:52 +0200 (CEST) Received: from mail.qbrick.com ([127.0.0.1]) by localhost (mail0.p0.w0.local [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 48855-01-42 for ; Mon, 19 Apr 2010 10:05:51 +0200 (CEST) Received: from exchange0.ad.qbrick.com (exchange0.ad.qbrick.com [10.16.0.4]) by mail.qbrick.com (Postfix) with ESMTP id 77B66519CC for ; Mon, 19 Apr 2010 10:05:51 +0200 (CEST) Received: from exchange0.ad.qbrick.com ([10.16.0.4]) by exchange0.ad.qbrick.com ([10.16.0.4]) with mapi; Mon, 19 Apr 2010 10:07:20 +0200 From: Fredrik Widlund To: "user@couchdb.apache.org" Content-Class: urn:content-classes:message Date: Mon, 19 Apr 2010 10:07:19 +0200 Subject: RE: CouchDB and Hadoop_ Thread-Topic: CouchDB and Hadoop_ Thread-Index: AcrdmCweElV7b4BMSrOzXc2QuBzflwB/TMpg Message-ID: References: <4BC7AC4B.1050401@gmail.com> <5CFF84BC-DE85-446A-8CF0-9E9DD1B4FAF6@googlemail.com> In-Reply-To: Accept-Language: sv-SE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: sv-SE, en-US Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Hi, The case I've tested so far is using couch in the following setup (which is= a small part of what would be a production level setup for us) - two bidirectionally synced nodes - <50 writes/s to node A, each updating a unique doc - <50 writes/s to node B, each updating a unique doc - <50 reads/s from each node - regular compacting the database containing the docs The two nodes run on quad (e5520) cpu with 16G ram. CPU ramp down and up to= 400% (i.e. full load on all cores) every few seconds. Couch 0.11.0 crashes= regularly, which has been reported and is being worked on from what I unde= rstand. Also, the replications tasks breaks and has to be restarted very of= ten, probably due to the problem above. Now, I've received a temporary patch as a possible work-around for the cras= hes and I haven't tested this case with the work-around yet, but I would as= sume this hopefully sorts out the crashes, but not the cpu load. Kind regards, Fredrik Widlund -----Original Message----- From: Randall Leeds [mailto:randall.leeds@gmail.com] Sent: den 16 april 2010 21:06 To: user@couchdb.apache.org Subject: Re: CouchDB and Hadoop_ Hey Fredrik, I'm one of the couchdb-lounge developers. I'd like to understand better what your performance concerns are. Why are you concerned about replicating a large number of changes? A distributed file system would be doing the same thing but at a lower level. If such a system were to work you'd be saving only HTTP and JSON overhead vs replication. If the replicator is too slow, that is something that can possibly be improved. If you're concerned about the runtime impact of replication this is tunable via the [replicator] configuration section. couchdb-lounge already uses nginx for distributing simple GET and PUT operations to documents and a python-twisted daemon to handle views. The twisted daemon has configurable caching (with the one caveat that the cache is currently unbounded, so the daemon needs to be restarted periodically.... I should really fix this :-P). It should be possible to chain any standard nginx caching modules in front of the lounge proxy module. If you have other concerns or would like to investigate more, ping me on irc (tilgovi) or join us over on http://groups.google.com/group/couchdb-lounge -Randall On Fri, Apr 16, 2010 at 09:54, Fredrik Widlund wrote: > > > Thanks, I will! We will actually use nginx for "dumb" caching, but add an= api layer in between the cache and the couch. Also we actually need to mir= ror data to provide HA, and the performance issues we're having are more ab= out constantly replicating a large number of changes than accelerating the = reads. I'm not sure if couchdb-lounge would address this. > > We did stumble upon a bug that's being addressed and we we're also provid= ed with a temporary work-around and it could be due to that, but with a qui= te modest load we periodically kept hitting the roof of a e5520 quad-core s= o I'm a bit worried about the performance aspect. > > Kind regards, > Fredrik Widlund > > -----Ursprungligt meddelande----- > Fr=E5n: David Coallier [mailto:david.coallier@gmail.com] > Skickat: den 16 april 2010 18:06 > Till: user@couchdb.apache.org > =C4mne: Re: CouchDB and Hadoop_ > > On 16 April 2010 16:22, Fredrik Widlund wrot= e: >> >> >> Well, we're building a solution on Couch and replication on a relatively= large scale and saying "it just works" doesn't really describe it for us. = I really like the Couch design but it's a bit of a challenge making it work= , for us. I can describe the case if you like. >> >> Also we already have a decentralized distributed file system layer (whic= h often is a natural part of a cloud solution I suppose) so if we could run= it on top of that it would lessen the complexity of the overall solution. >> >> In any case it was a quick comment to the Hadoop question, and maybe it = just wouldn't work that way. You could in general discuss atomic operations= /locking and performance implications by moving synchronization to a lower = abstraction layer I guess. >> > > > You should look into couchdb-lounge . It should resolve most of your > "sharding" replication issues :) > > -- > David Coallier > >