couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From J Chris Anderson <jch...@gmail.com>
Subject Re: Idea for View & Bulk Insert combination for ad-hoc database changes
Date Tue, 20 Apr 2010 16:03:14 GMT

On Apr 19, 2010, at 6:29 PM, Michael Genereux wrote:

> The more I think about what I was suggesting, the less I think it
> needs to be transactional.  If I really cared, for example in the
> update version, I would include a "pricing version" field that the
> document's price has been increased to the new price and watch that
> pricing version if I ran the update again.
> 
> I was really thinking of an easy way to create a bulk insert from a
> view.  The "CouchDB way" is "it's not transactional across documents."
> This is fine with me and an understanding of this should be a
> requirement of switching to CouchDB.  If someone doesn't get this, my
> suggested feature is not the biggest of their problems if switching
> from transactional SQL.  My workaround for this is currently looking
> like a Ruby library that calls a view with all the logic love in
> Javascript, tweaks the JSON into a bulk insert, and shoves the thing
> back in.  (Batch processingTM coming soon.)  This is just an
> inefficient way to do it.
> 
> As far as compaction, I don't want to sit around waiting for a
> compaction on a massive database that is only modifying 5% of its
> records.
> 

If you go back an take a look at Mikeal's node.js _changes handler, you'll see that it exactly
what you want (minus the shipping as part of CouchDB part).

The only thing that could be better is script that queries a view and works to make it empty
(and that is only more efficient if your view already exists - otherwise you have to scan
the entire DB to build the view, and if it's a one-time thing, you'd be better of using a
filtered _changes query and avoid building a throwaway index.

Chris

> Loving the feedback BTW!  Thank you everyone!
> 
> On Mon, Apr 19, 2010 at 12:49 PM, Randall Leeds <randall.leeds@gmail.com> wrote:
>> Ad-hoc db-wide transformations suddenly strike me as a generic form of the
>> compactor. The compactor is an identity update function. It achieves a full
>> scan of the db by putting results in a new .couch file. It runs until it has
>> caught up to the tail of the current file (using a changes listener) and
>> then swaps the files. I believe a good patch here is one that creates a code
>> path the compactor can share.
>> 
>> The side bonus is that this implementation is transactional.
>> 
>> On Apr 19, 2010 2:51 PM, "Martin Murphy" <martin.murphy@whiteboard-it.com>
>> wrote:
>> 
>> I agree.  Some format of bulk changes needs to be integrated into futon
>> before the masses will adopt.
>> 
>> On Mon, Apr 19, 2010 at 1:21 PM, Michael Genereux <mgenereu@gmail.com>wrote:
>> 
>>> Thanks!  Okay, I need to reread "_changes" and the "all_or_nothing" on
>>> the bulk_api.  Also "Document Update Handlers" was way too complicated
>>> and wanted a record id for what I was looking to do.  I have a CouchDB
>>> database holding a log and I want to chuck old data easily.  Losing
>>> quick manipulation when switching to CouchDB seems like a barrier of
>>> entry.  I don't know of any situation where I haven't used SQL
>>> (database), sed (ascii document), or other ad hoc concepts to
>>> manipulate records/files/documents.
>>> 
>>> Thanks for the feedback and it was expedient so thanks for that too.
>> 
>>> 
>>> On Mon, Apr 19, 2010 at 9:45 AM, J Chris Anderson <jchris@apache.org>
>>> wrote:
>>>> 
>>>> On Apr 18...
>> --
>> Martin Murphy
>> Whiteboard-it
>> (205) 910-0720
>> 


Mime
View raw message