incubator-bloodhound-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olemis Lang <ole...@gmail.com>
Subject Re: [RFC] File descriptor overflow running the test suite WAS: [BEP-0003] #355 - Summary of the work done towards multi-product support (2013-02-16)
Date Wed, 20 Feb 2013 08:58:11 GMT
On 2/20/13, Matevž Bradač <matevzb@gmail.com> wrote:
> I thought I saw the "etilqs" somewhere else, so after a bit of googling it
> rang a bell.
> It's "sqlite" in reverse, and the files created seem to originate from
> calling sqlite3_open()

hahaha ... I spent a bloody whole day to discover
xD

> and providing NULL/empty string as the filename. From
> http://www.sqlite.org/c3ref/open.html:
> =====
> If the filename is an empty string, then a private, temporary on-disk
> database will be created.
> This private database will be automatically deleted as soon as the database
> connection is closed.
> =====
>
[...]
> So the temp files are still identified, but
> ** anybody smart enough to figure out
[...]

<joke>
If these so called smart guys were in a Google+ circle ... I'm
officially out ... :P
</joke>

Enough joking ... it's almost 4:00am and I still need to sleep ... well ...

[...]
> */
> #ifndef SQLITE_TEMP_FILE_PREFIX
> # define SQLITE_TEMP_FILE_PREFIX "etilqs_"
> #endif
> […]
>
> I'm not sure how trac creates the db for tests, but it seems not all db's
> are of in-memory type.
>

The DB is in-memory . That's a fact and may be checked in a similar
way to the one mentioned in my first message . You'll notice that
initially there's no extra file descriptor allocated .

Nonetheless the issue is definitely related to this . I've been
tracking down this with PyDev and the source of all evil lays inside
`MultiProductSystem.upgrade_environment` method and is quite bizarre .
Immediately after executing the statement

{{{
#!py

                    db("CREATE TEMPORARY TABLE %s_temp AS SELECT %s FROM %s" %
                        (table, cols, table))

}}}

... for the first time in `db_installed_version < 3` branch two new
file descriptors are open and not closed even after

{{{
#!py

                    db("DROP TABLE %s_temp" % table)

}}}

It strikes to me that this statement is inside a loop but new fds
allocated just once inside the loop , but a new pair once again after
a new method call . That's the sustained +2 fd rate I mentioned before
.

@matevz : you r on the right track , it's about temporary tables ...
thanks for the pointer ;)

@jure : even after fixing this I recommend to remove MP upgrade inside
EnvironmentStub.__init__

PS: c u tomorrow ... if I ever wake up ... :)

-- 
Regards,

Olemis.

Mime
View raw message