Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.161.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=c92H6XzfUAMgOHDq6TBuO4Yjhf9kCLZtnpOAxmbzt0OcWIavRq8uAnukx86SnDj61M
         DyiPDzDalvxFemKWKCjGX8S4Xc4DhwS4zlpcZJ8nQwS8cLyWkhCKZ0ww5jOh7LPREoUc
         Hgvzy98Ktln821z9rztRPoezIsyXrEP98oDoU=
MIME-Version: 1.0
In-Reply-To: <AANLkTim5OrCzgMZu4Qe7wqOxirJN9kubs1VFyFw3tD1t@mail.gmail.com>
References: <AANLkTim5OrCzgMZu4Qe7wqOxirJN9kubs1VFyFw3tD1t@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Mon, 20 Sep 2010 16:51:38 -0400
Message-ID: <AANLkTin3sOEG4Om58eHVTbC5+p8bBGQh6XdwAeZcpn=q@mail.gmail.com>
Subject: Re: distributed map-reduce views
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1

> How would doing something like this with CouchDB and Lounge compare
> with using Hadoop and HBase?

Remember that CouchDB and Hadoop serve different purposes. CouchDB is
a data store, where as Hadoop is a data processing platform. While
they both have "MapReduce" functionality they aren't quite the same
thing.

In CouchDB, when we use Map/Reduce, we create a single persistent
index of data using map and reduce operators. These indexes can then
be queried using single key or range lookups. Because of the
properties of Map/Reduce we're capable of updating these indexes
incrementally.

Hadoop on the other hand is meant to handle arbitrary pipelines of
data processing. Ie, users can configure Hadoop to run multiple stages
of Map/Reduce in order to produce some desired output. The
intermediate stages are not intended to be persistent and query-able.
I'm not familiar enough to know how people use HBase in conjunction
with Hadoop other than I believe its generally a data source. I don't
know if it stores intermediate results or not. As far as I know,
Hadoop doesn't provide incremental indexing.

As Randal points out, there are various differences in implementation,
but its also important to understand the data store vs. data
processing differences of the two systems.

HTH,
Paul Davis