couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tibor Gemes <tib...@gmail.com>
Subject Re: tinkering with limits while replicating
Date Tue, 05 Feb 2013 19:06:51 GMT
A simle example which filters by type:

fun({Doc}, {Req}) ->
        DocType = couch_util:get_value(<<"type">>, Doc),
        case DocType of
                undefined -> false;
                <<"mytype">> ->       true;
                _ -> false
        end
end.


This is another example. A bit more complicated. This receives a values
query parameter (comma separated). The filter lets through any design docs,
any _deleted docs, and the mytype doc only if its value propery is
contained in the query parameter:

fun({Doc}, {Req}) ->
        DocId = couch_util:get_value(<<"_id">>, Doc),
        DocType = couch_util:get_value(<<"type">>, Doc),
        DocValue = couch_util:get_value(<<"value">>, Doc),
        DocDeleted = couch_util:get_value(<<"_deleted">>, Doc),

        {Query} = couch_util:get_value(<<"query">>,Req, {[]}),
        ValuesParam = couch_util:get_value(<<"values">>,Query),
        Values = binary:split(ValuesParam, <<",">>, [global]),
        case {DocId, DocDeleted}  of
        {<<"_design/", _/binary>>, _} -> true;
        {_, true} -> true;
        _ ->
                case DocType of
                        undefined -> false;
                        <<"mytype">> -> lists:any(fun(E) -> E =:= DocValue
end, Values);
                        _ -> false
                end
        end
end.

Those might be suboptimal but works perfectly for me.


Üdv:

Gémes Tibor
CodWell Kft
+3630-3430856


On 5 February 2013 19:43, Dave Cottlehuber <dch@jsonified.com> wrote:

> Hi Tibor,
>
> Are you able to share your code? There's not much erlang examples around.
>
> A+
> Dave
>
> On 5 February 2013 19:39, Tibor Gemes <tibber@gmail.com> wrote:
> > I've rewritten my filters into erlang and the continuous filtered
> > replications work faster with a few magnitude. T
> > 2013.02.05. 19:23, "Nathan Vander Wilt" <nate-lists@calftrail.com> ezt
> írta:
> >
> >> Hi Stephen,
> >>
> >> I've been doing some tests related to replication lately too
> >> (continuous+filtered in my case). I suspect the reason Futon hangs is
> >> because your whole VM is running out of RAM due to your very high
> >> os_process_limit. I went in to a bit more detail in
> >>
> http://mail-archives.apache.org/mod_mbox/couchdb-dev/201302.mbox/%3c70278F4A-FD08-4818-89B7-EA1B0AF846F5@gmail.com%3ebutthis
setting basically determines the size of the couchjs worker pool —
> >> you'd probably rather have a bit of contention for the pool at a
> reasonable
> >> size (maybe ~100 per GB free, tops?) than start paging.
> >>
> >> hth,
> >> -natevw
> >>
> >>
> >>
> >> On Feb 4, 2013, at 5:15 PM, Stephen Bartell wrote:
> >>
> >> > Hi all,
> >> >
> >> > I'm hitting some limits while replicating , I'm hoping someone could
> >> advise.
> >> > Im running this in a VM on my macbook with the following allocated
> >> resources:
> >> > ubuntu 11.04
> >> > 4 cores @ 2.3ghz
> >> > 8 gb mem
> >> >
> >> > I'm doing a one-to-many replication.
> >> > 1) I create one db named test.
> >> > 2) Then create [test_0 .. test_99] databases.
> >> > 3) I then set up replications from test -> [test_0 .. test_99].  100
> >> replications total.
> >> > 4) I finally go to test and create a doc, hit save.
> >> >
> >> > When I hit save, futon becomes completely unresponsive for around
> 10sec.
> >>  It eventually returns to normal behavior.
> >> >
> >> > Tailing the couchdb log I find waves of the following errors:
> >> > [Tue, 05 Feb 2013 00:46:26 GMT] [info] [<0.6936.1>] Retrying POST
> >> request to http://admin:*****@localhost:5984/test_25/_revs_diff in 1.0
> >> seconds due to error {code,503}
> >> >
> >> > I see that the replicator is finding the server to be unresponsive.
>  The
> >> waves of these messages show that replicator retries in 0.25 sec, then
> 0.5
> >> sec, then 1sec, then 2sec.  This is expected.  Everything settles done
> >> after about 4 retries.
> >> >
> >> > So my first thought is resource limits.  I threw the book at it and
> set :
> >> > 1) max_dbs_open: 500
> >> > 2) os_process_limit: 5000
> >> > 3) http_connections: 20000
> >> > 4) ulimit -Sn 4096 (the hard limit is 4096)
> >> >
> >> > I really don't know whats reasonable for these values relative to how
> >> many replications I am setting up.  So these values, save max_dbs_open,
> >>  are all stabs in the dark.
> >> >
> >> > No change in performance.
> >> >
> >> > So, I'm at a loss now.  what can I do to get all this to work? Or what
> >> am I doing wrong?  And note that this is only a test.  I aim to
> quadruple
> >>  the amount of replications and have lots and lots of insertions on the
> so
> >> called "test" database.  Actually, there will be several of these
> >> one-to-many databases.
> >> >
> >> > I've heard people get systems up to thousands of dbs and replicators
> >> running just fine.  So I hope Im just not offering to right sacrifices
> up
> >> to couchdb yet.
> >> >
> >> > Thanks for any insight,
> >> >
> >> > sb
> >> >
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message