clerezza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Daga <enricod...@gmail.com>
Subject Re: Out of memory issue on tdb storage provider tests
Date Fri, 06 Dec 2013 21:45:38 GMT
Hi Andy, all,


On 6 December 2013 09:28, Andy Seaborne <andy@apache.org> wrote:

> On 05/12/13 22:56, Enrico Daga wrote:
>
>> Hi all,
>> I am having difficulties compiling clerezza, precisely
>>
>> [INFO] Building Clerezza - SCB Jena TDB Storage Provider 0.7-SNAPSHOT
>>
>> ...
>>    TdbTcProviderTest>TcProviderTest.testGraphDeletion:433 ? OutOfMemory
>> Java
>> heap...
>>    TdbTcProviderTest>TcProviderTest.testGetGraph:99 ? OutOfMemory Java
>> heap
>> space
>>    TdbTcProviderTest>TcProviderTest.testGetTriplesGraph:459 ? OutOfMemory
>> Java he...
>>    TdbTcProviderTest>TcProviderTest.testGraphIsNotMutable:339 ?
>> OutOfMemory
>> Java ...
>>    TdbTcProviderTest.testCreateGraph ? OutOfMemory Java heap space
>>    TdbTcProviderTest.testGetTriplesMGraph ? OutOfMemory Java heap space
>>    TdbTcProviderTest.testGetTriples ? OutOfMemory Java heap space
>>    TdbTcProviderTest.testDeleteEntity ? OutOfMemory Java heap space
>>    TdbTcProviderTest.testCreateMGraphNoDuplicateNames ? OutOfMemory Java
>> heap spa...
>>    TdbTcProviderTest.testCreateMGraphExtended ? OutOfMemory Java heap
>> space
>>    TdbTcProviderTest.testGetMGraph ? OutOfMemory Java heap space
>>    TdbTcProviderTest.testCreateGraphNoDuplicateNames ? OutOfMemory Java
>> heap
>> spac...
>>    TdbTcProviderTest.testCreateGraphWithInitialCollection ? OutOfMemory
>> Java
>> heap...
>>    TdbTcProviderTest.testCreateGraphExtended ? OutOfMemory Java heap
>> space
>>
>> I tried increasing the heap until 3g (I have 4 on this machine).
>>
>
> Is your machine a 32bit machine with 32 bit java?  I'm unclear here
> because you say heap is 3G and I thought the 32 bit limit was 1.5G-ish.
>
$ mvn --version && java -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
13:51:28+0000)
Maven home: /opt/local/share/java/maven3
Java version: 1.6.0_65, vendor: Apple Inc.
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: "mac os x", version: "10.9", arch: "x86_64", family: "mac"
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)


>
> Does change in heap size move the point of failure?
>

I checked with 1g and 3g and no, changing the heap size does not change
anything, all tests fail (after the same time each)


> (TDB can't use memory mapped files on a 32 bit system so it uses old-style
> file IO and caches in heap; downside, uses heap.)
>
> Otherwise - internally there is a cache of open databases because opening
> one, and having it filesystem-cache-cold is prohibitively expensive.
>
> If the tests are creating different DB each time, this might be a partial
> cause of heap usage.
>
This is the case, at a first look [1].


>
> There is StoreConnection.reset() and related operations to carefully
> manipulate this cache.  TDB uses these for @BeforeClass/@AfterClass clean
> tests


> For testing, in-memory TDB databases are very useful.  The default is a
> fresh, uncached daatset on each call of TDBFactory.createDataset().
>
> Your disk will rattle less. The tests will run faster.  They have been
> reported as faster than normal Jena in-memory datasets but I don't see why.
>  They work by having a RAM disk like are that bytes are written into and
> copied out of so it is good simulation of a disk - no accidental sharing of
> data.

I understand, having tests with an in memory TDB would be better. But there
should be something wrong with the tests, because I can't see how the tdb
caching itself could led to a OOM on all tests after a second or the like:

testCreateGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 1.125 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testGetTriplesMGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 0.948 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testGetTriples(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 1.108 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testDeleteEntity(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 0.95 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateMGraphNoDuplicateNames(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 1.101 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateMGraphExtended(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 0.927 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testGetMGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 1.088 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateGraphNoDuplicateNames(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 0.988 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateGraphWithInitialCollection(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 1.256 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateGraphExtended(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
 Time elapsed: 0.935 sec  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space

Bests,

Enrico


>
>
>         Andy
>
>
>
>> Any hint?
>>
>> Thank you!
>> Enrico
>>
>>
>

[1]
https://git-wip-us.apache.org/repos/svn?p=clerezza.git;a=blob;f=parent/rdf.jena.tdb.storage/src/test/java/org/apache/clerezza/rdf/jena/tdb/storage/TdbTcProviderTest.java;h=3ab68476ec397f63715155c51e9c08c9e773681a;hb=HEAD

-- 
Enrico Daga

--
http://www.enridaga.net
skype: enri-pan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message