bloodhound-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jure Zitnik <>
Subject Re: [RFC] File descriptor overflow running the test suite WAS: [BEP-0003] #355 - Summary of the work done towards multi-product support (2013-02-16)
Date Wed, 20 Feb 2013 10:43:57 GMT

On 2/20/13 9:58 AM, Olemis Lang wrote:
> Nonetheless the issue is definitely related to this . I've been
> tracking down this with PyDev and the source of all evil lays inside
> `MultiProductSystem.upgrade_environment` method and is quite bizarre .
> Immediately after executing the statement
> {{{
> #!py
>                      db("CREATE TEMPORARY TABLE %s_temp AS SELECT %s FROM %s" %
>                          (table, cols, table))
> }}}

Thanks for pointing that out, that is in fact the actual source of the 
problem. According to [1] and [2], sqlite creates temporary database on 
the filesystem and keeps it there until the database connection is 
closed. That happens despite the fact that the main database that we're 
using is in-memory.

r1448062 fixes the issue by not creating temporary tables but rather 
tables in the main database when migrating the tables.

> ... for the first time in `db_installed_version < 3` branch two new
> file descriptors are open and not closed even after
> {{{
> #!py
>                      db("DROP TABLE %s_temp" % table)
> }}}
> It strikes to me that this statement is inside a loop but new fds
> allocated just once inside the loop , but a new pair once again after
> a new method call . That's the sustained +2 fd rate I mentioned before
> .

The statement is in the loop as the loop is executed for all tables that 
need to be migrated ;)

> @jure : even after fixing this I recommend to remove MP upgrade inside
> EnvironmentStub.__init__

The leaking fd issue is fixed. Multi product upgrade inside 
EnvironmentStub.__init__  was added to make sure that the database is 
always upgraded to multi-product schema as the SQL translator is 
installed in any case. Not upgrading the schema would cause translated 
SQLs to hit invalid schema.


[1] sqlite CREATE TABLE -
[2] sqlite inmemory database -

View raw message