bloodhound-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olemis Lang <>
Subject Re: [RFC] File descriptor overflow running the test suite WAS: [BEP-0003] #355 - Summary of the work done towards multi-product support (2013-02-16)
Date Wed, 20 Feb 2013 19:05:29 GMT
On 2/20/13, Jure Zitnik <> wrote:
> Hi,


> On 2/20/13 9:58 AM, Olemis Lang wrote:
>> Nonetheless the issue is definitely related to this . I've been
>> tracking down this with PyDev and the source of all evil lays inside
>> `MultiProductSystem.upgrade_environment` method and is quite bizarre .
> Thanks for pointing that out, that is in fact the actual source of the
> problem. According to [1] and [2], sqlite creates temporary database on
> the filesystem and keeps it there until the database connection is
> closed. That happens despite the fact that the main database that we're
> using is in-memory.
> r1448062 fixes the issue by not creating temporary tables but rather
> tables in the main database when migrating the tables.

/me trying it ... will update spreadsheet afterwards as if this was
yesterdays result

>> It strikes to me that this statement is inside a loop but new fds
>> allocated just once inside the loop , but a new pair once again after
>> a new method call . That's the sustained +2 fd rate I mentioned before
>> .
> The statement is in the loop as the loop is executed for all tables that
> need to be migrated ;)

yes , that's pretty obvious ;)
I was just trying to make my point in that this is related to a new
method call and apparently another SQLite connection created .

>> @jure : even after fixing this I recommend to remove MP upgrade inside
>> EnvironmentStub.__init__
> The leaking fd issue is fixed.

I toast to that !

> Multi product upgrade inside
> EnvironmentStub.__init__  was added to make sure that the database is
> always upgraded to multi-product schema as the SQL translator is
> installed in any case. Not upgrading the schema would cause translated
> SQLs to hit invalid schema.

I've been running MP test suite by removing that upgrade in __init__
and everything seems to be fine provided that upgrade will be
performed in TC setUp method . Keeping it there might cause some
trouble , especially considering the env override mechanism I've been
using to convert test cases (i.e. env property with nop setter ;) .
Notice that some env stub objects are deliberately ignored and dropped
as a consequence .



View raw message