couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Logging into a database
Date Sun, 18 Oct 2009 15:37:43 GMT
On Oct 18, 2009, at 11:31 AM, Benoit Chesneau wrote:

> On Sun, Oct 18, 2009 at 5:07 PM, Adam Kocoloski  
> <kocolosk@apache.org> wrote:
>> On Oct 17, 2009, at 9:15 PM, Jan Lehnardt wrote:
>>
>>> Hey all,
>>>
>>> I had an itch to scratch and now there is an experimental branch  
>>> that
>>> enables logging to a database:
>>>
>>>  http://github.com/janl/couchdb/commits/log-to-db
>>>
>>> Check out the test case on how it is supposed to work:
>>>
>>>
>>> http://github.com/janl/couchdb/blob/5cbfbeaddea2152e35b88cae515152f78d6b7544/share/www/script/test/log_database.js
>>>
>>> You just create a `database = dbname` entry in your configuration  
>>> under
>>> the `[log]` section and CouchDB will start to write log entries to  
>>> the
>>> specified database.
>>>
>>> Depending on the log level this means that each single request to  
>>> CouchDB
>>> triggers multiple writes to the log database. The logger, for now,  
>>> defaults
>>> to delayed_commits. A future improvement, could be using the  
>>> `batch=ok`
>>> module.
>>>
>>> I just wanted to throw this out here inspired by Chris' "releasing  
>>> things
>>> before I go to sleep for the night".
>>>
>>> Let me know what you think.
>>>
>>> Cheers
>>> Jan
>>> --
>>
>> Hi Jan, cool!  I do have a word of caution, though ...
>>
>> I tried this a long time ago but never released it because it  
>> caused the
>> logging queue to back up pretty badly under a moderate read load.   
>> Logging
>> events are asynchronous (as they should be), so if the logging  
>> event handler
>> is slower than the system generating the events its entirely  
>> possible for
>> the message queue in the gen_event manager to back up.   
>> Unfortunately,
>> Erlang has this ugly positive feedback loop where a large message  
>> queue will
>> dramatically slow down that process.  Eventually (assuming a  
>> constant read
>> load), the gen_event manager process used all available memory and  
>> things
>> went kaboom.
>
> maybe using gen_event2 from rabbitmq project would solve that since  
> it buffer ?
> http://hg.rabbitmq.com/rabbitmq-server/file/4bb959b80bac/src/gen_server2.erl
>
> - benoit

Is there a gen_event2?  That link goes to gen_server2.  In principle a  
gen_event2 would help, if you knew that the load on the server would  
eventually drop to the point that the logger could catch up.  I'd be  
fairly nervous about betting on that in production, though.  Best,

Adam


Mime
View raw message