couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe David Manana <fdman...@apache.org>
Subject Re: View group compactions shutting down in trunk
Date Wed, 14 Sep 2011 15:04:23 GMT
Mike, this is the simplest approach I was talking about earlier, in
case you want to try it:

http://friendpaste.com/1s0In2cLBhWcdsAWEqAw1E

regards,

On Wed, Sep 14, 2011 at 6:54 AM, Filipe David Manana
<fdmanana@apache.org> wrote:
> Makes sense Mike. I think the simplest is to make the view compactor keep
> the database open.
> I've been thinking in the meanwhile for other approaches however.
> I'll get back to this soon.
>
> Thanks for reporting this and testing
>
> On Wednesday, September 14, 2011, Mike Leddy <mike@loop.com.br> wrote:
>>
>> Thanks for the patch Filipe, but I am pretty sure that the compaction
>> daemon is not involved in this.
>>
>> I removed all item in the 'compactions' config and restarted couchdb.
>> As ?CONFIG_ETS is empty that compact_loop does nothing and
>> maybe_compact_db is never called.
>>
>> Next I started a view compaction in Futon whilst doing the inserts
>> that cause the constant lru recovery. As expected the result was the
>> the same:
>>
>> "Shutting down view group server, monitored db is closing" in the
>> log at the same time the view group compaction was shut down.
>>
>> I had never seen this before as I had no reason to compact these
>> large views whilst having so many concurrent accesses on the
>> smaller databases.
>>
>> Regards,
>>
>> Mike
>>
>> On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
>>> Mike,
>>>
>>> Can you try the following patch?
>>>
>>> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>>>
>>> Basically it keeps the respective database open until view compaction
>>> finishes.
>>>
>>> Thanks for sharing this
>>>
>>> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mike@loop.com.br> wrote:
>>> > Hello,
>>> >
>>> > I've been experimenting with trunk and the new compaction daemon,
>>> > and discovered that some of my larger views were never being fully
>>> > compacted.
>>> >
>>> > Basically I am encountering a large number of these situations:
>>> >
>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view
>>> > group server, monitored db is closing.
>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon
-
>>> > an error ocurred while compacting the view group `base` from database
>>> > `XXXX-1301529600-1303948799`: shutdown
>>> >
>>> > I suspect that since I have a large number of other smaller databases
>>> > (approx 2500) that are constantly being updated there is a constant
>>> > stream of databases being closed as a result of least recently used
>>> > recovery - I currently have max_open_dbs at 500.
>>> >
>>> > This does not affect database compaction as a database is never
>>> > considered idle when compacting.
>>> >
>>> > Consequently the larger views are shutdown before completion. Also,
>>> > when a new attempt to re-compact is processed the partially complete
>>> > *.compact.view file is truncated. As a result these large views
>>> > never get fully compacted.
>>> >
>>> > If the view group compaction continued where it left off I guess
>>> > the impact would be minimal.
>>> >
>>> > Has anyone else seen this ?
>>> >
>>> > Regards,
>>> >
>>> > Mike
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>
>>
>>
>
> --
> Filipe David Manana,
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."
>
>



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Mime
View raw message