couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe David Manana <fdman...@apache.org>
Subject Re: View group compactions shutting down in trunk
Date Thu, 15 Sep 2011 06:19:02 GMT
Mike, I was able to create a test case to reproduce your issue and
validate the last patch. I created an issue in Jira for it:

https://issues.apache.org/jira/browse/COUCHDB-1283

regards,

On Wed, Sep 14, 2011 at 8:04 AM, Filipe David Manana
<fdmanana@apache.org> wrote:
> Mike, this is the simplest approach I was talking about earlier, in
> case you want to try it:
>
> http://friendpaste.com/1s0In2cLBhWcdsAWEqAw1E
>
> regards,
>
> On Wed, Sep 14, 2011 at 6:54 AM, Filipe David Manana
> <fdmanana@apache.org> wrote:
>> Makes sense Mike. I think the simplest is to make the view compactor keep
>> the database open.
>> I've been thinking in the meanwhile for other approaches however.
>> I'll get back to this soon.
>>
>> Thanks for reporting this and testing
>>
>> On Wednesday, September 14, 2011, Mike Leddy <mike@loop.com.br> wrote:
>>>
>>> Thanks for the patch Filipe, but I am pretty sure that the compaction
>>> daemon is not involved in this.
>>>
>>> I removed all item in the 'compactions' config and restarted couchdb.
>>> As ?CONFIG_ETS is empty that compact_loop does nothing and
>>> maybe_compact_db is never called.
>>>
>>> Next I started a view compaction in Futon whilst doing the inserts
>>> that cause the constant lru recovery. As expected the result was the
>>> the same:
>>>
>>> "Shutting down view group server, monitored db is closing" in the
>>> log at the same time the view group compaction was shut down.
>>>
>>> I had never seen this before as I had no reason to compact these
>>> large views whilst having so many concurrent accesses on the
>>> smaller databases.
>>>
>>> Regards,
>>>
>>> Mike
>>>
>>> On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
>>>> Mike,
>>>>
>>>> Can you try the following patch?
>>>>
>>>> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>>>>
>>>> Basically it keeps the respective database open until view compaction
>>>> finishes.
>>>>
>>>> Thanks for sharing this
>>>>
>>>> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mike@loop.com.br> wrote:
>>>> > Hello,
>>>> >
>>>> > I've been experimenting with trunk and the new compaction daemon,
>>>> > and discovered that some of my larger views were never being fully
>>>> > compacted.
>>>> >
>>>> > Basically I am encountering a large number of these situations:
>>>> >
>>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down
view
>>>> > group server, monitored db is closing.
>>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction
daemon -
>>>> > an error ocurred while compacting the view group `base` from database
>>>> > `XXXX-1301529600-1303948799`: shutdown
>>>> >
>>>> > I suspect that since I have a large number of other smaller databases
>>>> > (approx 2500) that are constantly being updated there is a constant
>>>> > stream of databases being closed as a result of least recently used
>>>> > recovery - I currently have max_open_dbs at 500.
>>>> >
>>>> > This does not affect database compaction as a database is never
>>>> > considered idle when compacting.
>>>> >
>>>> > Consequently the larger views are shutdown before completion. Also,
>>>> > when a new attempt to re-compact is processed the partially complete
>>>> > *.compact.view file is truncated. As a result these large views
>>>> > never get fully compacted.
>>>> >
>>>> > If the view group compaction continued where it left off I guess
>>>> > the impact would be minimal.
>>>> >
>>>> > Has anyone else seen this ?
>>>> >
>>>> > Regards,
>>>> >
>>>> > Mike
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>> --
>> Filipe David Manana,
>>
>> "Reasonable men adapt themselves to the world.
>>  Unreasonable men adapt the world to themselves.
>>  That's why all progress depends on unreasonable men."
>>
>>
>
>
>
> --
> Filipe David Manana,
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."
>



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Mime
View raw message