Mailing-List: contact dev-help@clerezza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@clerezza.apache.org
Received-SPF: pass (nike.apache.org: domain of andy.seaborne.apache@gmail.com
 designates 74.125.82.169 as permitted sender)
Message-ID: <52A198AF.2020100@apache.org>
Date: Fri, 06 Dec 2013 09:28:15 +0000
From: Andy Seaborne <andy@apache.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.1.1
MIME-Version: 1.0
To: dev@clerezza.apache.org
Subject: Re: Out of memory issue on tdb storage provider tests
References: 
 <CAGTWk79fjtOR+iuMGZvHqZ1GLQggDZ07qwZhMjLJhYxNZvPqiw@mail.gmail.com>
In-Reply-To: 
 <CAGTWk79fjtOR+iuMGZvHqZ1GLQggDZ07qwZhMjLJhYxNZvPqiw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 05/12/13 22:56, Enrico Daga wrote:
> Hi all,
> I am having difficulties compiling clerezza, precisely
>
> [INFO] Building Clerezza - SCB Jena TDB Storage Provider 0.7-SNAPSHOT
>
> ...
>    TdbTcProviderTest>TcProviderTest.testGraphDeletion:433 ? OutOfMemory Java
> heap...
>    TdbTcProviderTest>TcProviderTest.testGetGraph:99 ? OutOfMemory Java heap
> space
>    TdbTcProviderTest>TcProviderTest.testGetTriplesGraph:459 ? OutOfMemory
> Java he...
>    TdbTcProviderTest>TcProviderTest.testGraphIsNotMutable:339 ? OutOfMemory
> Java ...
>    TdbTcProviderTest.testCreateGraph ? OutOfMemory Java heap space
>    TdbTcProviderTest.testGetTriplesMGraph ? OutOfMemory Java heap space
>    TdbTcProviderTest.testGetTriples ? OutOfMemory Java heap space
>    TdbTcProviderTest.testDeleteEntity ? OutOfMemory Java heap space
>    TdbTcProviderTest.testCreateMGraphNoDuplicateNames ? OutOfMemory Java
> heap spa...
>    TdbTcProviderTest.testCreateMGraphExtended ? OutOfMemory Java heap space
>    TdbTcProviderTest.testGetMGraph ? OutOfMemory Java heap space
>    TdbTcProviderTest.testCreateGraphNoDuplicateNames ? OutOfMemory Java heap
> spac...
>    TdbTcProviderTest.testCreateGraphWithInitialCollection ? OutOfMemory Java
> heap...
>    TdbTcProviderTest.testCreateGraphExtended ? OutOfMemory Java heap space
>
> I tried increasing the heap until 3g (I have 4 on this machine).

Is your machine a 32bit machine with 32 bit java?  I'm unclear here 
because you say heap is 3G and I thought the 32 bit limit was 1.5G-ish.

Does change in heap size move the point of failure?

(TDB can't use memory mapped files on a 32 bit system so it uses 
old-style file IO and caches in heap; downside, uses heap.)

Otherwise - internally there is a cache of open databases because 
opening one, and having it filesystem-cache-cold is prohibitively 
expensive.

If the tests are creating different DB each time, this might be a 
partial cause of heap usage.

There is StoreConnection.reset() and related operations to carefully 
manipulate this cache.  TDB uses these for @BeforeClass/@AfterClass 
clean tests.

For testing, in-memory TDB databases are very useful.  The default is a 
fresh, uncached daatset on each call of TDBFactory.createDataset().

Your disk will rattle less. The tests will run faster.  They have been 
reported as faster than normal Jena in-memory datasets but I don't see 
why.  They work by having a RAM disk like are that bytes are written 
into and copied out of so it is good simulation of a disk - no 
accidental sharing of data.

	Andy

>
> Any hint?
>
> Thank you!
> Enrico
>