clerezza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Seaborne <a...@apache.org>
Subject Re: Out of memory issue on tdb storage provider tests
Date Sat, 07 Dec 2013 16:05:38 GMT
> I can't see how the tdb
> caching itself could led to a OOM on all tests after a second or the like:

Enrico - sorry, nor can I.

On 64 bit, as you have, the node table cache is still in-heap even if 
indexes are not, but if changing the heap size does not make a 
difference, then it does not look like accumulated TDB junk.

If all the tests fail and fail at the same time, in the same place, 
regardless of heap, it does suggest that the test itself is causing 
massive allocation somehow.

As it seems to fail quickly, can you point jvisualvm/YourKit/... at a 
test run?  (If you find it is TDB contributing, then do let me know.)

	Andy


On 06/12/13 21:45, Enrico Daga wrote:
> Hi Andy, all,
>
>
> On 6 December 2013 09:28, Andy Seaborne <andy@apache.org> wrote:
>
>> On 05/12/13 22:56, Enrico Daga wrote:
>>
>>> Hi all,
>>> I am having difficulties compiling clerezza, precisely
>>>
>>> [INFO] Building Clerezza - SCB Jena TDB Storage Provider 0.7-SNAPSHOT
>>>
>>> ...
>>>     TdbTcProviderTest>TcProviderTest.testGraphDeletion:433 ? OutOfMemory
>>> Java
>>> heap...
>>>     TdbTcProviderTest>TcProviderTest.testGetGraph:99 ? OutOfMemory Java
>>> heap
>>> space
>>>     TdbTcProviderTest>TcProviderTest.testGetTriplesGraph:459 ? OutOfMemory
>>> Java he...
>>>     TdbTcProviderTest>TcProviderTest.testGraphIsNotMutable:339 ?
>>> OutOfMemory
>>> Java ...
>>>     TdbTcProviderTest.testCreateGraph ? OutOfMemory Java heap space
>>>     TdbTcProviderTest.testGetTriplesMGraph ? OutOfMemory Java heap space
>>>     TdbTcProviderTest.testGetTriples ? OutOfMemory Java heap space
>>>     TdbTcProviderTest.testDeleteEntity ? OutOfMemory Java heap space
>>>     TdbTcProviderTest.testCreateMGraphNoDuplicateNames ? OutOfMemory Java
>>> heap spa...
>>>     TdbTcProviderTest.testCreateMGraphExtended ? OutOfMemory Java heap
>>> space
>>>     TdbTcProviderTest.testGetMGraph ? OutOfMemory Java heap space
>>>     TdbTcProviderTest.testCreateGraphNoDuplicateNames ? OutOfMemory Java
>>> heap
>>> spac...
>>>     TdbTcProviderTest.testCreateGraphWithInitialCollection ? OutOfMemory
>>> Java
>>> heap...
>>>     TdbTcProviderTest.testCreateGraphExtended ? OutOfMemory Java heap
>>> space
>>>
>>> I tried increasing the heap until 3g (I have 4 on this machine).
>>>
>>
>> Is your machine a 32bit machine with 32 bit java?  I'm unclear here
>> because you say heap is 3G and I thought the 32 bit limit was 1.5G-ish.
>>
> $ mvn --version && java -version
> Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
> 13:51:28+0000)
> Maven home: /opt/local/share/java/maven3
> Java version: 1.6.0_65, vendor: Apple Inc.
> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
> Default locale: en_US, platform encoding: MacRoman
> OS name: "mac os x", version: "10.9", arch: "x86_64", family: "mac"
> java version "1.6.0_65"
> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>
>
>>
>> Does change in heap size move the point of failure?
>>
>
> I checked with 1g and 3g and no, changing the heap size does not change
> anything, all tests fail (after the same time each)
>
>
>> (TDB can't use memory mapped files on a 32 bit system so it uses old-style
>> file IO and caches in heap; downside, uses heap.)
>>
>> Otherwise - internally there is a cache of open databases because opening
>> one, and having it filesystem-cache-cold is prohibitively expensive.
>>
>> If the tests are creating different DB each time, this might be a partial
>> cause of heap usage.
>>
> This is the case, at a first look [1].
>
>
>>
>> There is StoreConnection.reset() and related operations to carefully
>> manipulate this cache.  TDB uses these for @BeforeClass/@AfterClass clean
>> tests
>
>
>> For testing, in-memory TDB databases are very useful.  The default is a
>> fresh, uncached daatset on each call of TDBFactory.createDataset().
>>
>> Your disk will rattle less. The tests will run faster.  They have been
>> reported as faster than normal Jena in-memory datasets but I don't see why.
>>   They work by having a RAM disk like are that bytes are written into and
>> copied out of so it is good simulation of a disk - no accidental sharing of
>> data.
>
> I understand, having tests with an in memory TDB would be better. But there
> should be something wrong with the tests, because I can't see how the tdb
> caching itself could led to a OOM on all tests after a second or the like:
>
> testCreateGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 1.125 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testGetTriplesMGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 0.948 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testGetTriples(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 1.108 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testDeleteEntity(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 0.95 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testCreateMGraphNoDuplicateNames(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 1.101 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testCreateMGraphExtended(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 0.927 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testGetMGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 1.088 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testCreateGraphNoDuplicateNames(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 0.988 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testCreateGraphWithInitialCollection(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 1.256 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> testCreateGraphExtended(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>   Time elapsed: 0.935 sec  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
>
> Bests,
>
> Enrico
>
>
>>
>>
>>          Andy
>>
>>
>>
>>> Any hint?
>>>
>>> Thank you!
>>> Enrico
>>>
>>>
>>
>
> [1]
> https://git-wip-us.apache.org/repos/svn?p=clerezza.git;a=blob;f=parent/rdf.jena.tdb.storage/src/test/java/org/apache/clerezza/rdf/jena/tdb/storage/TdbTcProviderTest.java;h=3ab68476ec397f63715155c51e9c08c9e773681a;hb=HEAD
>


Mime
View raw message