Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of a.zatvornitskiy@gmail.com
 designates 209.85.215.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHdjipKHENV3eOU0q5bFD5_Pzgsk8dsoZbvpOqg+7XA++jSjpg@mail.gmail.com>
References: 
 <CAKGaSrRxQcLT7BM567bPbPiOu_mS_YjgV-+6MBZ9YA5Qsp9_aQ@mail.gmail.com>
	<CAHdjipKHENV3eOU0q5bFD5_Pzgsk8dsoZbvpOqg+7XA++jSjpg@mail.gmail.com>
Date: Thu, 14 Feb 2013 12:05:34 +0200
Message-ID: 
 <CAKGaSrR9FVpsWE3FV8hj1kRvu6W31cQV5RW_nMSPBrJo27JiiA@mail.gmail.com>
Subject: Re: I've just released a first preview of couch_normalizer
From: =?UTF-8?B?0JDQu9C10LrRgSBaYXR2b3JuaXRza2l5?= <a.zatvornitskiy@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=e89a8f23485553e83404d5ac663b

--e89a8f23485553e83404d5ac663b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi! I pleased that you find it useful.

In short:

1. Take a short look on {num_workers, 5} configuration options. It means
that for target db a couch_normalizer will use 5 workers, each worker works
under OTP. A particular worker spawn its own child process for applying
particular scenarios for current document. You should see log messages in
case one of them ends or fails (for many reasons such a conflict, scenario
logic, etc). Check this code http://goo.gl/TasYp for more details.

2. I think that some kind of final notification should be. Feel free to
submit an issue. Is it what you want? http://goo.gl/UThkT

3. Yeah, Alex I know. That why we used the Elixir and its amazing
functional meta-programming features.


On Wed, Feb 13, 2013 at 5:47 PM, Alexander Shorin <kxepal@gmail.com> wrote:

> That is awesome, Alex!
>
> This is the thing I always dream when our document schemes receives
> new  update and there is need to apply they for large amount of
> databases. Trick with local database mirror and replications works,
> but it's not very fast.
>
> Few questions:
>
> 1. What happens it normalization fails due to some reasons? For
> example, due to conflict update. Have I start whole normalization
> again in this case or it will just write few notes about conflict in
> logs and will try apply script to document one more time?
>
> 2. Is it possible to receive final result of the normalization job and
> error description if it fails? Continuous querying _active_tasks looks
> not optimal and it's possible to miss the moment when normalization
> fails.
>
> 3. Only Erlang and Elixir migration scripts are possible, right? Or
> it's possible to use any scripts that supports CouchDB stdio
> communication protocol? I suppose many of us who faced same problem
> already have Python/Ruby/whatever-lang scripts that successfully
> handles scheme normalization and well tested. It would be helpful to
> not force them made whole job again.
>
> --
> ,,,^..^,,,
>
>
> On Wed, Feb 13, 2013 at 7:04 PM, =D0=90=D0=BB=D0=B5=D0=BA=D1=81 Zatvornit=
skiy
> <a.zatvornitskiy@gmail.com> wrote:
> > Hi everybody!
> >
> > A couch_normalizer v0.6 is out!
> >
> > The couch_normalizer designed as a standard Apache CouchDB httpd handle=
r
> > and uses a Rails db migration approach. Written both in Erlang and
> Elixir.
> > Works well on production and has a great IO performance.
> >
> > For example:
> >
> > % Starts a normalization process.
> >
> > % curl -v -XPOST -H"Content-Type: application/json"
> > http://127.0.0.1:5984/db/_normalize
> >
> > % =3D> {"ok":"Normalization process has been started (<0.174.0>)."}
> >
> >
> > % Gets a normalization process execution status.
> >
> > % curl -v http://127.0.0.1:5984/_active_tasks
> >
> > % =3D>
> >
> [{"pid":"<0.174.0>","continue":false,"db":"db","docs_conflicted":0,"docs_=
deleted":0,"docs_normalized":0,"docs_read":3000,"finished_on":1358513508,"n=
um_workers":5,"started_on":1358513508,"type":"normalization","updated_on":1=
358513508}]
> >
> > As a result, it allows to deploy migration scripts (aka scenarios) and
> > change big amount of documents as fast as possible (without HTTP overhe=
ad
> > and some kind of 'delayed jobs') via internal CouchDB functions, such a=
s
> > couch_db:open_doc/2, couch_db:update_doc/3 and so on.
> >
> > Check more: https://github.com/datahogs/couch_normalizer
> >
> > If you want to contribute, feel free to open an Github issue or submit =
a
> > pull request or ping me offline)
> >
> > It still under scoping and development.
>

--e89a8f23485553e83404d5ac663b--