incubator-clerezza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reto Bachmann-Gmuer <reto.bachm...@trialox.org>
Subject Re: leak but where after parsing rdf files?
Date Sat, 22 Jan 2011 11:38:40 GMT
To me this looks like a leak, it successfully parses the same bund of files
a few time and then it has an OutOfMemoryError:

$ MAVEN_OPTS="-Xmx215m -Xms128m"  mvn  exec:java -o -e
+ Error stacktraces are turned on.
[INFO]
NOTE: Maven is executing in offline mode. Any artifacts not already in your
local
repository will be inaccessible.

[INFO] Scanning for projects...
[INFO]
------------------------------------------------------------------------
[INFO] Building Unnamed -
org.apache.clerezza:turtlememory:bundle:0.10-incubating-SNAPSHOT
[INFO]    task-segment: [exec:java]
[INFO]
------------------------------------------------------------------------
[INFO] Preparing exec:java
[INFO] No goals needed for project - skipping
[INFO] [exec:java {execution: default-cli}]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/reto/.m2/repository/org/slf4j/slf4j-log4j12/1.5.8/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/reto/.m2/repository/org/slf4j/slf4j-simple/1.5.8/slf4j-simple-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test6.ttl
Sat Jan 22 12:30:44 CET 2011 69394744
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test9.ttl
Sat Jan 22 12:30:45 CET 2011 35188872
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test8.ttl
Sat Jan 22 12:30:47 CET 2011 47366808
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test7.ttl
Sat Jan 22 12:30:48 CET 2011 47203240
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test6.ttl
Sat Jan 22 12:30:49 CET 2011 42741384
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test5.ttl
Sat Jan 22 12:30:51 CET 2011 42775024
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test4.ttl
Sat Jan 22 12:30:52 CET 2011 53483456
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test3.ttl
Sat Jan 22 12:30:53 CET 2011 53054312
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test2.ttl
Sat Jan 22 12:30:55 CET 2011 48124560
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test1.ttl
Sat Jan 22 12:30:57 CET 2011 38761672
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test9.ttl
Sat Jan 22 12:31:00 CET 2011 35990832
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test8.ttl
Sat Jan 22 12:31:02 CET 2011 47626544
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test7.ttl
Sat Jan 22 12:31:04 CET 2011 42677864
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test6.ttl
Sat Jan 22 12:31:06 CET 2011 40464624
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test5.ttl
Sat Jan 22 12:31:07 CET 2011 48232728
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test4.ttl
Sat Jan 22 12:31:08 CET 2011 76654184
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test3.ttl
Sat Jan 22 12:31:09 CET 2011 40184992
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test2.ttl
Sat Jan 22 12:31:11 CET 2011 25407456
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test1.ttl
Sat Jan 22 12:31:13 CET 2011 39777768
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test9.ttl
Sat Jan 22 12:31:15 CET 2011 22957128
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test8.ttl
Sat Jan 22 12:31:17 CET 2011 54125432
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test7.ttl
Sat Jan 22 12:31:18 CET 2011 55672280
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test6.ttl
Sat Jan 22 12:31:20 CET 2011 44336424
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test5.ttl
Sat Jan 22 12:31:21 CET 2011 72590304
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test4.ttl
Sat Jan 22 12:31:23 CET 2011 70973640
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test3.ttl
Sat Jan 22 12:31:24 CET 2011 54795824
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test2.ttl
Sat Jan 22 12:31:26 CET 2011 37645640
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test1.ttl
Sat Jan 22 12:31:29 CET 2011 39334808
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test9.ttl
Sat Jan 22 12:31:31 CET 2011 36357600
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test8.ttl
Sat Jan 22 12:31:34 CET 2011 49300320
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test7.ttl
Sat Jan 22 12:31:35 CET 2011 49912136
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test6.ttl
Sat Jan 22 12:31:37 CET 2011 41279608
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test5.ttl
Sat Jan 22 12:31:39 CET 2011 66985536
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test4.ttl
Sat Jan 22 12:31:40 CET 2011 61069440
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test3.ttl
Sat Jan 22 12:31:41 CET 2011 32386376
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test2.ttl
Sat Jan 22 12:31:43 CET 2011 44755344
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test1.ttl
Sat Jan 22 12:31:45 CET 2011 37499360
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test9.ttl
Sat Jan 22 12:31:48 CET 2011 33931904
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test8.ttl
Sat Jan 22 12:31:50 CET 2011 31057256
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test7.ttl
Sat Jan 22 12:31:51 CET 2011 51560088
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test6.ttl
Sat Jan 22 12:31:53 CET 2011 41348008
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test5.ttl
Sat Jan 22 12:31:54 CET 2011 67824056
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test4.ttl
Sat Jan 22 12:31:56 CET 2011 69011608
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test3.ttl
Sat Jan 22 12:31:57 CET 2011 30993920
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test2.ttl
Sat Jan 22 12:31:59 CET 2011 34794960
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test1.ttl
Sat Jan 22 12:32:01 CET 2011 37394448
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test9.ttl
Sat Jan 22 12:32:03 CET 2011 35800560
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test8.ttl
Sat Jan 22 12:32:05 CET 2011 35942216
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test7.ttl
Sat Jan 22 12:32:06 CET 2011 49681512
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test6.ttl
Sat Jan 22 12:32:08 CET 2011 41445104
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test5.ttl
Sat Jan 22 12:32:10 CET 2011 44588072
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test4.ttl
Sat Jan 22 12:32:11 CET 2011 39404008
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test3.ttl
Sat Jan 22 12:32:12 CET 2011 46719432
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test2.ttl
Sat Jan 22 12:32:14 CET 2011 42149400
parsing:
file:/home/reto/projects/apache/clerezza/issues/CLEREZZA-384/turtlememory/target/classes/test/turtle/test1.ttl
[INFO]
------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO]
------------------------------------------------------------------------
[INFO] An exception occured while executing the Java class. null

Java heap space
[INFO]
------------------------------------------------------------------------
[INFO] Trace
org.apache.maven.lifecycle.LifecycleExecutionException: An exception occured
while executing the Java class. null
    at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:719)
    at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:569)
    at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:539)
    at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
    at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)
    at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
    at
org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
    at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
    at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception
occured while executing the Java class. null
    at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:346)
    at
org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
    at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
    ... 17 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:291)
    at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.lang.String.toCharArray(String.java:2737)
    at org.apache.xerces.impl.dv.util.Base64.decode(Unknown Source)
    at org.apache.xerces.impl.dv.xs.Base64BinaryDV.getActualValue(Unknown
Source)
    at org.apache.xerces.impl.dv.xs.XSSimpleTypeDecl.getActualValue(Unknown
Source)
    at org.apache.xerces.impl.dv.xs.XSSimpleTypeDecl.validate(Unknown
Source)
    at com.hp.hpl.jena.datatypes.xsd.XSDDatatype.parse(XSDDatatype.java:249)
    at com.hp.hpl.jena.datatypes.BaseDatatype.isValid(BaseDatatype.java:100)
    at
org.openjena.riot.checker.CheckerLiterals.validateByDatatypeJena(CheckerLiterals.java:112)
    at
org.openjena.riot.checker.CheckerLiterals.validateByDatatype(CheckerLiterals.java:107)
    at
org.openjena.riot.checker.CheckerLiterals.checkLiteral(CheckerLiterals.java:80)
    at
org.openjena.riot.checker.CheckerLiterals.checkLiteral(CheckerLiterals.java:58)
    at
org.openjena.riot.system.ParserProfileChecker.createTypedLiteral(ParserProfileChecker.java:113)
    at
org.openjena.riot.system.ParserProfileBase.create(ParserProfileBase.java:157)
    at
org.openjena.riot.lang.LangTurtleBase.tokenAsNode(LangTurtleBase.java:480)
    at org.openjena.riot.lang.LangTurtleBase.node(LangTurtleBase.java:330)
    at
org.openjena.riot.lang.LangTurtleBase.triplesNode(LangTurtleBase.java:358)
    at
org.openjena.riot.lang.LangTurtleBase.objectList(LangTurtleBase.java:340)
    at
org.openjena.riot.lang.LangTurtleBase.predicateObjectItem(LangTurtleBase.java:273)
    at
org.openjena.riot.lang.LangTurtleBase.predicateObjectList(LangTurtleBase.java:265)
    at
org.openjena.riot.lang.LangTurtleBase.triples(LangTurtleBase.java:245)
    at
org.openjena.riot.lang.LangTurtleBase.triplesSameSubject(LangTurtleBase.java:206)
    at
org.openjena.riot.lang.LangTurtle.oneTopLevelElement(LangTurtle.java:34)
    at
org.openjena.riot.lang.LangTurtleBase.runParser(LangTurtleBase.java:132)
    at org.openjena.riot.lang.LangBase.parse(LangBase.java:75)
    at
org.openjena.riot.system.JenaReaderTurtle2.readWorker(JenaReaderTurtle2.java:34)
    at
org.openjena.riot.system.JenaReaderRIOT.readImpl(JenaReaderRIOT.java:124)
    at org.openjena.riot.system.JenaReaderRIOT.read(JenaReaderRIOT.java:40)
    at org.openjena.riot.system.JenaReaderRIOT.read(JenaReaderRIOT.java:55)
    at com.hp.hpl.jena.rdf.model.impl.ModelCom.read(ModelCom.java:202)
    at test.turtle.TestWithFiles.parse(TestWithFiles.java:37)
    at test.turtle.TestWithFiles.main(TestWithFiles.java:48)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 1 minute 37 seconds
[INFO] Finished at: Sat Jan 22 12:32:16 CET 2011
[INFO] Final Memory: 132M/190M
[INFO]
------------------------------------------------------------------------




On Sat, Jan 22, 2011 at 12:29 PM, Reto Bachmann-Gmuer <
reto.bachmann@trialox.org> wrote:

> In did its much faster!
>
> Still there is the issue that might be caused by this fixed-size
> hard-referenced caches. If I start with -Xmx190 it parser test6.ttl
> successfull the first time and gets a memory exception the second time.
>
> Reto
>
>
> On Fri, Jan 21, 2011 at 12:37 PM, Andy Seaborne <
> andy.seaborne@epimorphics.com> wrote:
>
>>
>>
>> On 20/01/11 19:43, Reto Bachmann-Gmuer wrote:
>>
>>> HI Andy
>>>
>>> I've committed an application that uses directly jena without clerezza
>>> stuff
>>>
>>
>>
>>  in the middle that demonstarts the problem.
>>>
>>> Starting it with
>>>
>>> MAVEN_OPTS="-Xmx256m -Xms128m"  mvn clean install exec:java -o -e
>>>
>>> it will fail at one of the files, howver if I change the order in which
>>> the
>>> files are to be parsed and put the file it was failing at at the
>>> begginning
>>> it suceeds parsing this file and will fail at another one.
>>>
>>> the app is here:
>>>
>>> http://svn.apache.org/viewvc/incubator/clerezza/issues/CLEREZZA-384/turtlememory
>>>
>>
>> Not entirely without clerezza stuff - the POM does not work standalone.
>>
>> After some POM hacking, I got it working.  I take it the test is
>> "TestWithFiles".
>>
>> It's not using RIOT because that's not in the Jena download yet.
>>
>> Add
>>
>>    <dependency>
>>      <groupId>com.hp.hpl.jena</groupId>
>>      <artifactId>arq</artifactId>
>>      <version>2.8.7</version>
>>    </dependency>
>>
>> and either:
>>
>>        com.hp.hpl.jena.query.ARQ.init() ;
>>
>> or
>>
>>        org.openjena.riot.SysRIOT.wireIntoJena() ;
>>
>> With this the test passes (and much faster as well).
>>
>>
>> The test is not just parsing.  It's storing the results in a model so the
>> space needed included complete storage of the model.
>>
>> Only a small increase in -Xmx (e.g. 350m) and the test passes.
>>
>> The test fails in the first pass over the files if it's going to fail. I
>> suspect that one or more internal systems have fixed size caches.  Jena
>> does.  JavaCC has expanding buffering (and you have some very large
>> literals).
>>
>> Jena's caches are bounded by number of slots so churning based on large
>> literals will need to settle down before any conclusions of a memory leak
>> can be made.  Hence failing on the first pass is not suggestive of a memory
>> leak.  This is backed up by the fact file order matters.
>>
>> JavaCC used by the old parser uses expanding buffers and your long
>> literals will force those larger and hence the runtime working space is
>> higher on a single file parse.  RIOT uses a fixed size buffer and builds the
>> large literals directly into the string to be used as the RDF node.
>>
>> As increasing the heap means that the test runs and the test fails in the
>> first pass over the files if it is going to fail, I conclude it's various
>> caches filling up and just not fitting. I guess it passes at 256m with RIOT
>> by chance.  Slightly less overhead meaning that caches just happen to fit.
>>
>> There is a streaming interface to RIOT in org.openjena.riot.RiotReader.
>>
>>        Andy
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message