incubator-jena-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Helsen (JIRA)" <>
Subject [jira] [Commented] (JENA-91) extremely large buffer is being created in ObjectFileStorage
Date Mon, 08 Aug 2011 23:30:28 GMT


Simon Helsen commented on JENA-91:

There is some additional observation. When I change the line ByteBuffer bb = ByteBuffer.allocate(len)
;  into ByteBuffer bb = ByteBuffer.allocateDirect(len) ;  I notice something quite remarkable:
I can see short, but massive memory spikes. E.g. for the most part the memory occupied by
my server takes about 500Mb, but it sometimes spikes to 1.8 or 2.3 in case even 3.8Gb, but
only for a very short moment and then goes back its "normal" consumption

This change does permit me to keep running my tests, but obviously this is not a desirable
change since allocateDirect relies on memory-mapped I/O. Strangely, I did see the same problems
when running TDB as a whole in memory mapped I/O.

I don't understand enough about how this bit of code is supposed to work to debug this in
other further meaningful way. Perhaps I need to understand first why the allocations are so
large. Also, do you run tests on windows systems once in a while?

> extremely large buffer is being created in ObjectFileStorage
> ------------------------------------------------------------
>                 Key: JENA-91
>                 URL:
>             Project: Jena
>          Issue Type: Bug
>          Components: TDB
>         Environment: Windows (and I presume any little endian system)
>            Reporter: Simon Helsen
>            Priority: Critical
> I tried to debug the OME and check why a bytebuffer is causing my native memory to explode
in almost no time. It all seems to happen in this bit of code in com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage
(lines 243 onwards)
>   // No - it's in the underlying file storage.
>         lengthBuffer.clear() ;
>         int x =, loc) ;
>         if ( x != 4 )
>             throw new FileException(""+loc+")["+filesize+"]["+file.size()+"]:
Failed to read the length : got "+x+" bytes") ;
>         int len = lengthBuffer.getInt(0) ;
>         ByteBuffer bb = ByteBuffer.allocate(len) ;
> My debugger shows that x==4. It also shows the lengthBuffer has the following content:
[111, 110, 61, 95]. This amounts to the value of len=1869495647, which is rather a lot :-)
Obviously, the next statement (ByteBuffer.allocate) causes the OME.

This message is automatically generated by JIRA.
For more information on JIRA, see:


View raw message