couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Smith <jeremyasm...@yahoo.com>
Subject Re: Erlang reduce functions
Date Mon, 28 May 2012 13:37:38 GMT
Thank you for you explanation, looking forward to moving a number of our views to Erlang

Jeremy


________________________________
 From: david martin <david.martin@lymegreen.co.uk>
To: user@couchdb.apache.org 
Sent: Monday, 28 May 2012, 13:50
Subject: Re: Erlang reduce functions
 
On 28/05/12 10:54, Jeremy Smith wrote:
> Thank you, that works perfectly.
>
> I guess the only way to understand how this works is for me to understand more about
Erlang, if you have any pointers/tips for learning Erlang I would be very grateful (I come
from a Perl, C, Java background)

In my humble opinion, your map function shows you have a very good understanding of Erlang
for having only started yesterday.

Your problem was not fully understanding the Couchdb reduce function.
From what I have gathered, it has two possible phases of operation, "normal" reduce and "re-reduce"
reduce.
Whether it is in normal reduce or re-reduce reduce is indicated by the third parameter being
true or false.

I have been experimenting in the area of Erlang view functions and am pleased to say that,
so far,
all the Erlang function modules that Couchdb uses are available in the Erlang View function.
The function can be debugged using the built in Log("string") Function combined with formatting
with  statements like

  Log(io_lib:format("X = ~p Val = ~p", [ Key,Value]))

Compile and run time errors are (very) fully  reported in the couch.log

I am using an even  balance of six cores without any specific attempts to use explicit parallel
maps.

The use of parallel maps would however enable heavy processing  of things such as serious
crypto functions
i.e. sha512  or complex Perl Compatible Regular Expressions (PCRE)
from the re module to be run in parallel over different Key/Values for each Document in the
View.

Try that in Javascript!

Very long serial computations can cause the gen-server handling the view to
time out to stop runaway processes in the View code.

http://blog.vmoroz.com/2011/01/erlang-pmap.html


The problem of not seeming to allow recursive calls in the view function has to be
addressed by the use of the Y-combinator

http://blog.vmoroz.com/2010/10/recursive-anonymous-function-in-erlang.html

thanks to Victor Moroz for erudite explanations, well worth taking the time to understand
(if that is possible!).

I believe that the true versatility and power of Couchdb can only be realised
using Erlang Views and there is very little in the way of examples on the web.

If there are any real specific interest cases I might even post some examples somewhere.

> Many thanks
> Jeremy
>
>
> ________________________________
>   From: david martin<david.martin@lymegreen.co.uk>
> To: user@couchdb.apache.org
> Sent: Monday, 28 May 2012, 9:40
> Subject: Re: Erlang reduce functions
>
> try this
> fun(Keys,Values,ReReduce)->
>       case ReReduce of
>           true->lists:sum(Values);
>           false->length(Values)
>       end
> end.
>
>
> On 27/05/12 16:33, Jeremy Smith wrote:
>> Hi,
>>
>> I am new to erlang, just started today, because we were advised to move some of our
views to native erlang views, I am trying to write a reduce function:
>>
>> The Values its using seem to be in the form:
>> [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1]]
>>
>>
>> so I have tried to do the following to get a count of all the 1's, but it only gives
me the count of the outermost lists (in the above case 3, instead of 60(ish))
>>
>> My Reduce funtion:
>> fun(Keys, Values, ReReduce) ->   lists:flatlength(Values) end.
>>
>>
>> My Map function looks like this:
>> fun({Doc}) ->
>>        Emitter = fun(Doc) ->
>>            Date           = proplists:get_value(<<"publish_date">>, 
 Doc),
>>
>>            Year = string:substr(erlang:bitstring_to_list(Date), 1, 4),
>>            Emit(erlang:list_to_bitstring(Year), 1)
>>        end,
>>
>>        HasRequiredFields = fun(Doc) ->
>>            case {proplists:is_defined(<<"a_number">>, Doc), proplists:is_defined(<<"publish">>,
Doc), proplists:is_defined(<<"publish_date">>, Doc)} of
>>                {true, true, true} ->
>>                    Emitter(Doc);
>>
>>                _->
>>                    false
>>            end
>>        end,
>>
>>        HasRequiredFields(Doc)
>> end.
>>
>> Cheers
>> Jeremy
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message