Mailing-List: contact dev-help@clerezza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@clerezza.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: <52A34752.6030308@apache.org>
References: 
 <CAGTWk79fjtOR+iuMGZvHqZ1GLQggDZ07qwZhMjLJhYxNZvPqiw@mail.gmail.com>
	<52A198AF.2020100@apache.org>
	<CAGTWk7-3HVUQ-yAb3OOWVV-96Cz7rG9PLdWZ=jnqfF-9Sgau5g@mail.gmail.com>
	<52A34752.6030308@apache.org>
Date: Tue, 10 Dec 2013 17:28:21 +0100
Message-ID: 
 <CALvhUEXjpjN3b2WzO=bqc6YXUx-tZOvVgXey8PQgUBQYMFsmGw@mail.gmail.com>
Subject: Re: Out of memory issue on tdb storage provider tests
From: =?ISO-8859-1?Q?Reto_Bachmann=2DGm=FCr?= <reto@wymiwyg.com>
To: dev@clerezza.apache.org
Content-Type: multipart/alternative; boundary=001a11c1ba20cda58204ed3099d9

--001a11c1ba20cda58204ed3099d9
Content-Type: text/plain; charset=ISO-8859-1

Hi

I can't reproduce the problem, with

MAVEN_OPTS="-Xmx1g -XX:MaxPermSize=128m" mvn clean install

things compile fine here.

Cheers,
Reto

$ mvn --version && java -version
Apache Maven 3.0.3 (r1075438; 2011-02-28 18:31:09+0100)
Maven home: /home/reto/lib/apache-maven-3.0.3
Java version: 1.7.0_25, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.11.0-14-generic", arch: "amd64", family:
"unix"
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4ubuntu3)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)


On Sat, Dec 7, 2013 at 5:05 PM, Andy Seaborne <andy@apache.org> wrote:

> I can't see how the tdb
>> caching itself could led to a OOM on all tests after a second or the like:
>>
>
> Enrico - sorry, nor can I.
>
> On 64 bit, as you have, the node table cache is still in-heap even if
> indexes are not, but if changing the heap size does not make a difference,
> then it does not look like accumulated TDB junk.
>
> If all the tests fail and fail at the same time, in the same place,
> regardless of heap, it does suggest that the test itself is causing massive
> allocation somehow.
>
> As it seems to fail quickly, can you point jvisualvm/YourKit/... at a test
> run?  (If you find it is TDB contributing, then do let me know.)
>
>         Andy
>
>
>
> On 06/12/13 21:45, Enrico Daga wrote:
>
>> Hi Andy, all,
>>
>>
>> On 6 December 2013 09:28, Andy Seaborne <andy@apache.org> wrote:
>>
>>  On 05/12/13 22:56, Enrico Daga wrote:
>>>
>>>  Hi all,
>>>> I am having difficulties compiling clerezza, precisely
>>>>
>>>> [INFO] Building Clerezza - SCB Jena TDB Storage Provider 0.7-SNAPSHOT
>>>>
>>>> ...
>>>>     TdbTcProviderTest>TcProviderTest.testGraphDeletion:433 ?
>>>> OutOfMemory
>>>> Java
>>>> heap...
>>>>     TdbTcProviderTest>TcProviderTest.testGetGraph:99 ? OutOfMemory Java
>>>> heap
>>>> space
>>>>     TdbTcProviderTest>TcProviderTest.testGetTriplesGraph:459 ?
>>>> OutOfMemory
>>>> Java he...
>>>>     TdbTcProviderTest>TcProviderTest.testGraphIsNotMutable:339 ?
>>>> OutOfMemory
>>>> Java ...
>>>>     TdbTcProviderTest.testCreateGraph ? OutOfMemory Java heap space
>>>>     TdbTcProviderTest.testGetTriplesMGraph ? OutOfMemory Java heap
>>>> space
>>>>     TdbTcProviderTest.testGetTriples ? OutOfMemory Java heap space
>>>>     TdbTcProviderTest.testDeleteEntity ? OutOfMemory Java heap space
>>>>     TdbTcProviderTest.testCreateMGraphNoDuplicateNames ? OutOfMemory
>>>> Java
>>>> heap spa...
>>>>     TdbTcProviderTest.testCreateMGraphExtended ? OutOfMemory Java heap
>>>> space
>>>>     TdbTcProviderTest.testGetMGraph ? OutOfMemory Java heap space
>>>>     TdbTcProviderTest.testCreateGraphNoDuplicateNames ? OutOfMemory
>>>> Java
>>>> heap
>>>> spac...
>>>>     TdbTcProviderTest.testCreateGraphWithInitialCollection ?
>>>> OutOfMemory
>>>> Java
>>>> heap...
>>>>     TdbTcProviderTest.testCreateGraphExtended ? OutOfMemory Java heap
>>>> space
>>>>
>>>> I tried increasing the heap until 3g (I have 4 on this machine).
>>>>
>>>>
>>> Is your machine a 32bit machine with 32 bit java?  I'm unclear here
>>> because you say heap is 3G and I thought the 32 bit limit was 1.5G-ish.
>>>
>>>  $ mvn --version && java -version
>> Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
>> 13:51:28+0000)
>> Maven home: /opt/local/share/java/maven3
>> Java version: 1.6.0_65, vendor: Apple Inc.
>> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/
>> Contents/Home
>> Default locale: en_US, platform encoding: MacRoman
>> OS name: "mac os x", version: "10.9", arch: "x86_64", family: "mac"
>> java version "1.6.0_65"
>> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>>
>>
>>
>>> Does change in heap size move the point of failure?
>>>
>>>
>> I checked with 1g and 3g and no, changing the heap size does not change
>> anything, all tests fail (after the same time each)
>>
>>
>>  (TDB can't use memory mapped files on a 32 bit system so it uses
>>> old-style
>>> file IO and caches in heap; downside, uses heap.)
>>>
>>> Otherwise - internally there is a cache of open databases because opening
>>> one, and having it filesystem-cache-cold is prohibitively expensive.
>>>
>>> If the tests are creating different DB each time, this might be a partial
>>> cause of heap usage.
>>>
>>>  This is the case, at a first look [1].
>>
>>
>>
>>> There is StoreConnection.reset() and related operations to carefully
>>> manipulate this cache.  TDB uses these for @BeforeClass/@AfterClass clean
>>> tests
>>>
>>
>>
>>  For testing, in-memory TDB databases are very useful.  The default is a
>>> fresh, uncached daatset on each call of TDBFactory.createDataset().
>>>
>>> Your disk will rattle less. The tests will run faster.  They have been
>>> reported as faster than normal Jena in-memory datasets but I don't see
>>> why.
>>>   They work by having a RAM disk like are that bytes are written into and
>>> copied out of so it is good simulation of a disk - no accidental sharing
>>> of
>>> data.
>>>
>>
>> I understand, having tests with an in memory TDB would be better. But
>> there
>> should be something wrong with the tests, because I can't see how the tdb
>> caching itself could led to a OOM on all tests after a second or the like:
>>
>> testCreateGraph(org.apache.clerezza.rdf.jena.tdb.storage.
>> TdbTcProviderTest)
>>   Time elapsed: 1.125 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testGetTriplesMGraph(org.apache.clerezza.rdf.jena.tdb.
>> storage.TdbTcProviderTest)
>>   Time elapsed: 0.948 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testGetTriples(org.apache.clerezza.rdf.jena.tdb.storage.
>> TdbTcProviderTest)
>>   Time elapsed: 1.108 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testDeleteEntity(org.apache.clerezza.rdf.jena.tdb.storage.
>> TdbTcProviderTest)
>>   Time elapsed: 0.95 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testCreateMGraphNoDuplicateNames(org.apache.clerezza.rdf.
>> jena.tdb.storage.TdbTcProviderTest)
>>   Time elapsed: 1.101 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testCreateMGraphExtended(org.apache.clerezza.rdf.jena.tdb.
>> storage.TdbTcProviderTest)
>>   Time elapsed: 0.927 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testGetMGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
>>   Time elapsed: 1.088 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testCreateGraphNoDuplicateNames(org.apache.clerezza.rdf.jena.tdb.storage.
>> TdbTcProviderTest)
>>   Time elapsed: 0.988 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testCreateGraphWithInitialCollection(org.apache.clerezza.
>> rdf.jena.tdb.storage.TdbTcProviderTest)
>>   Time elapsed: 1.256 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>> testCreateGraphExtended(org.apache.clerezza.rdf.jena.tdb.
>> storage.TdbTcProviderTest)
>>   Time elapsed: 0.935 sec  <<< ERROR!
>> java.lang.OutOfMemoryError: Java heap space
>>
>> Bests,
>>
>> Enrico
>>
>>
>>
>>>
>>>          Andy
>>>
>>>
>>>
>>>  Any hint?
>>>>
>>>> Thank you!
>>>> Enrico
>>>>
>>>>
>>>>
>>>
>> [1]
>> https://git-wip-us.apache.org/repos/svn?p=clerezza.git;a=
>> blob;f=parent/rdf.jena.tdb.storage/src/test/java/org/
>> apache/clerezza/rdf/jena/tdb/storage/TdbTcProviderTest.java;h=
>> 3ab68476ec397f63715155c51e9c08c9e773681a;hb=HEAD
>>
>>
>

--001a11c1ba20cda58204ed3099d9--