incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe David Manana <fdman...@apache.org>
Subject Re: View group compactions shutting down in trunk
Date Wed, 14 Sep 2011 13:54:39 GMT
Makes sense Mike. I think the simplest is to make the view compactor keep
the database open.
I've been thinking in the meanwhile for other approaches however.
I'll get back to this soon.

Thanks for reporting this and testing

On Wednesday, September 14, 2011, Mike Leddy <mike@loop.com.br> wrote:
>
> Thanks for the patch Filipe, but I am pretty sure that the compaction
> daemon is not involved in this.
>
> I removed all item in the 'compactions' config and restarted couchdb.
> As ?CONFIG_ETS is empty that compact_loop does nothing and
> maybe_compact_db is never called.
>
> Next I started a view compaction in Futon whilst doing the inserts
> that cause the constant lru recovery. As expected the result was the
> the same:
>
> "Shutting down view group server, monitored db is closing" in the
> log at the same time the view group compaction was shut down.
>
> I had never seen this before as I had no reason to compact these
> large views whilst having so many concurrent accesses on the
> smaller databases.
>
> Regards,
>
> Mike
>
> On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
>> Mike,
>>
>> Can you try the following patch?
>>
>> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>>
>> Basically it keeps the respective database open until view compaction
finishes.
>>
>> Thanks for sharing this
>>
>> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mike@loop.com.br> wrote:
>> > Hello,
>> >
>> > I've been experimenting with trunk and the new compaction daemon,
>> > and discovered that some of my larger views were never being fully
>> > compacted.
>> >
>> > Basically I am encountering a large number of these situations:
>> >
>> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view
group server, monitored db is closing.
>> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon
-
an error ocurred while compacting the view group `base` from database
`XXXX-1301529600-1303948799`: shutdown
>> >
>> > I suspect that since I have a large number of other smaller databases
>> > (approx 2500) that are constantly being updated there is a constant
>> > stream of databases being closed as a result of least recently used
>> > recovery - I currently have max_open_dbs at 500.
>> >
>> > This does not affect database compaction as a database is never
>> > considered idle when compacting.
>> >
>> > Consequently the larger views are shutdown before completion. Also,
>> > when a new attempt to re-compact is processed the partially complete
>> > *.compact.view file is truncated. As a result these large views
>> > never get fully compacted.
>> >
>> > If the view group compaction continued where it left off I guess
>> > the impact would be minimal.
>> >
>> > Has anyone else seen this ?
>> >
>> > Regards,
>> >
>> > Mike
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>
>
>

-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message