Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of wilson.jim.r@gmail.com
 designates 74.125.82.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Rvw9l3MeZgigJ1BjjufpQbGePybYz07r7bDHbduSCANO3ytzQBsHaRRJFGaBAYEhfA
         71aNtLr7UQ6DG0j3Opw+AlvaW+WzPwUJLzvNR/4LEOGVI5TocucBgyBm/e13r19CgSnE
         T1ZVYeoTUadnT6Uzur7izJvCSR3g2hDGWngPI=
MIME-Version: 1.0
In-Reply-To: <g2mc32d31441004160112u8babd6ccwb8407e83d825595a@mail.gmail.com>
References: <4BC7AC4B.1050401@gmail.com>
	 <g2mc32d31441004160112u8babd6ccwb8407e83d825595a@mail.gmail.com>
Date: Mon, 19 Apr 2010 12:46:41 -0400
Message-ID: <i2gac08e8d1004190946n981d2406j3d4c09f0e5eaf736@mail.gmail.com>
Subject: Re: CouchDB and Hadoop
From: "Jim R. Wilson" <wilson.jim.r@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Steve,

Clarification points:  Hadoop is not a filesystem, it's an
implementation of MapReduce.  For sharing files around the cluster,
Hadoop uses HDFS (the Hadoop Distributed File System) by default, and
can also use other filesystems (I believe it supports Amazon S3
storage if the cluster is in EC2).

So, the questions become:

* Could the data file for a CouchDB database be stored in HDFS?
* Could the MapReduce tasks executed by CouchDB be offloadad to Hadoop?

I think the answers to both are "probably".  How much work it would
take to implement such a system is an open question.  I suspect
storing the data file on HDFS would be easier than offloading the
mapreduce tasks.

As far as handling the Java to Erlang/JavaScript mismatch, I think
that particular piece can be addressed by using Hadoop Streaming[1].
I have done a fair amount of work using Python to work on JSON objects
over Hadoop Streaming - Erlang/JavaScript should be no different.

The real question in my mind is, "why do any of this?".  Both Hadoop
and CouchDB are fine systems with particular goals in mind.  I'm not
convinced there's significant value in Frankensteining them together.

Just my $0.02

[1] http://hadoop.apache.org/common/docs/r0.15.2/streaming.html

-- Jim R. Wilson (jimbojw)


On Fri, Apr 16, 2010 at 4:12 AM, Suhail Ahmed <suhailski@gmail.com> wrote:
> Sure It can be done but for me the whole Java to Erlang layer would be a
> mess since they are so different. The better way to go about doing this
> would to be implement a distributed file system like Hadoop underneath Co=
uch
> for same effect.
>
> On Fri, Apr 16, 2010 at 1:16 AM, Steve-Mustafa Ismail Mustafa <
> m.i.mustafa@gmail.com> wrote:
>
>> I swear, I spent over an hour going through the mailing list trying to f=
ind
>> an answer.
>>
>> I know that CouchDB is a document oriented DB and I know that Hadoop is =
a
>> File System and that both implement Map/Reduce. =A0But is it possible to=
 have
>> them stacked with Hadoop being the FS in use and CouchDB being the DB? T=
his
>> way, wouldn't you get the distributed/clustered FS abilities of Hadoop i=
n
>> addition to the powerful retrieval abilities of CouchDB?
>>
>> If its not possible, and I suspect that it is so, _why_? Don't they oper=
ate
>> on two seperate levels? Wouldn't CouchDB sort of replace HBase?
>>
>> Thanks in advance for any and all replies
>>
>