clerezza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Seaborne <a...@apache.org>
Subject Re: Out of memory issue on tdb storage provider tests
Date Fri, 06 Dec 2013 09:28:15 GMT
On 05/12/13 22:56, Enrico Daga wrote:
> Hi all,
> I am having difficulties compiling clerezza, precisely
>
> [INFO] Building Clerezza - SCB Jena TDB Storage Provider 0.7-SNAPSHOT
>
> ...
>    TdbTcProviderTest>TcProviderTest.testGraphDeletion:433 ? OutOfMemory Java
> heap...
>    TdbTcProviderTest>TcProviderTest.testGetGraph:99 ? OutOfMemory Java heap
> space
>    TdbTcProviderTest>TcProviderTest.testGetTriplesGraph:459 ? OutOfMemory
> Java he...
>    TdbTcProviderTest>TcProviderTest.testGraphIsNotMutable:339 ? OutOfMemory
> Java ...
>    TdbTcProviderTest.testCreateGraph ? OutOfMemory Java heap space
>    TdbTcProviderTest.testGetTriplesMGraph ? OutOfMemory Java heap space
>    TdbTcProviderTest.testGetTriples ? OutOfMemory Java heap space
>    TdbTcProviderTest.testDeleteEntity ? OutOfMemory Java heap space
>    TdbTcProviderTest.testCreateMGraphNoDuplicateNames ? OutOfMemory Java
> heap spa...
>    TdbTcProviderTest.testCreateMGraphExtended ? OutOfMemory Java heap space
>    TdbTcProviderTest.testGetMGraph ? OutOfMemory Java heap space
>    TdbTcProviderTest.testCreateGraphNoDuplicateNames ? OutOfMemory Java heap
> spac...
>    TdbTcProviderTest.testCreateGraphWithInitialCollection ? OutOfMemory Java
> heap...
>    TdbTcProviderTest.testCreateGraphExtended ? OutOfMemory Java heap space
>
> I tried increasing the heap until 3g (I have 4 on this machine).

Is your machine a 32bit machine with 32 bit java?  I'm unclear here 
because you say heap is 3G and I thought the 32 bit limit was 1.5G-ish.

Does change in heap size move the point of failure?

(TDB can't use memory mapped files on a 32 bit system so it uses 
old-style file IO and caches in heap; downside, uses heap.)

Otherwise - internally there is a cache of open databases because 
opening one, and having it filesystem-cache-cold is prohibitively 
expensive.

If the tests are creating different DB each time, this might be a 
partial cause of heap usage.

There is StoreConnection.reset() and related operations to carefully 
manipulate this cache.  TDB uses these for @BeforeClass/@AfterClass 
clean tests.

For testing, in-memory TDB databases are very useful.  The default is a 
fresh, uncached daatset on each call of TDBFactory.createDataset().

Your disk will rattle less. The tests will run faster.  They have been 
reported as faster than normal Jena in-memory datasets but I don't see 
why.  They work by having a RAM disk like are that bytes are written 
into and copied out of so it is good simulation of a disk - no 
accidental sharing of data.

	Andy

>
> Any hint?
>
> Thank you!
> Enrico
>


Mime
View raw message