Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: couchdb-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of damienkatz@gmail.com
 designates 66.249.82.236 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer;
        b=x0QD5R2adDNlVXPiOiW4Gi5gwmayJro7IgkPhtRdyFVPasaWuw0gP3W3eJfhUovLOd/9cmbAAB1M6un/hQ4nQ87K7/PWThgSroXZ7broOsab8WcbE0lh45EvoO/EuOewM7ImfhSuE2HJHiFwZvYcJAtyrqHilDnIMxMibb3BCJo=
Message-Id: <0080D71A-A0DC-42CB-A950-44E8308DDFAB@gmail.com>
From: Damien Katz <damienkatz@gmail.com>
To: couchdb-user@incubator.apache.org
In-Reply-To: <e282921e0803161644k340c335j4f61b5b1b42f3d4f@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v919.2)
Subject: Re: PUT / POST tradeoffs
Date: Sun, 16 Mar 2008 20:42:01 -0400
References: <e282921e0803161644k340c335j4f61b5b1b42f3d4f@mail.gmail.com>


On Mar 16, 2008, at 7:44 PM, Chris Anderson wrote:

> Couchers,
>
> I've been diving into CouchDB lately, and seeing how it's a great fit
> for my application. I've run into some questions about how to record
> information in an efficient way. Here's a description of what I'm
> trying to do, and a couple of methods I've come up with. Hopefully
> someone on the list can give me some insight into how determine what
> the pros and cons of each approach are.
>
> Let's say I'm crawling the web, looking for embeds of YouTube videos
> on blogs and such. When I come across one, I'll be recording:
>
> the YouTube video URL.
> the URL the video was embedded on.
> a snippet of the context in which it was found.
>
> In the end I'd like to give people the ability to see where their
> videos are being embedded. E.g. start from a video and find the embeds
> of it from across the web.
>
> I'll be recrawling some blogs quite frequently, so I have this idea
> about how to avoid duplicate content:
>
> I calculate an MD5 hash from the information I want to store in a
> deterministic way, so processing the same page twice creates identical
> computed hash values. I use the hash values as document_ids, and PUT
> the data to CouchDB, with no _rev attribute. CouchDB will rejects the
> PUTs of duplicate data with a conflict. In my application I just
> ignore the conflict, as all it means is that I've already put that
> data there (maybe in an earlier crawl).
>
> The alternative approach is to forgo the MD5 hash calculation, and
> POST the parsed data into CouchDB, creating a new record with an
> arbitrary id. I imagine that I would end up with a lot of identical
> data in this case, and it would become the job of the
> Map/Combine/Reduce process to filter duplicates while creating the
> lookup indexes.
>
> I suppose my question boils down to this: Are there unforeseen costs
> to building a high percentage of failing PUTs into my application
> design? It seems like the most elegant way to ensure unique data. But
> perhaps I am putting too high a premium on unique data - I suppose in
> the end it depend on the cost to compute a conflict, vs the ongoing
> cost of calculating reductions across redundant data sets.

I don't see any problems with this approach. For you purposes, using  
MD5 hashes of data should work just fine.

>
>
> Thanks for any insights!
> Chris
>
> -- 
> Chris Anderson
> http://jchris.mfdz.com