community-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Vesse <rve...@dotnetrdf.org>
Subject Re: Disk space requirement for building on Windows
Date Tue, 10 Mar 2015 11:07:04 GMT
Using an alternative approach would not make any difference

It is a fundamental bug in Windows memory mapped files that means that a
JVM can never guarantee to completely release memory mapped files while
the JVM is alive.

Andy has posted this many times on threads about TDB on Windows in the
past.  No workaround we could attempt could ever solve the issue on
Windows so there is really no point in expending effort changing something
low level that otherwise works fine across multiple platforms.

Rob

On 10/03/2015 10:25, "Stian Soiland-Reyes" <stain@apache.org> wrote:

> Thanks, Rob!
>
>I tried looking yesterday at ways to reduce the disk space
>requirements when building on Windows - including truncating the files
>after closing. This seems to require deep changes into TDBs
>ChannelManager which keeps the corresponding FileChannels - perhaps a
>new method for that purpose?
>
>https://github.com/apache/jena/blob/master/jena-tdb/src/main/java/com/hp/h
>pl/jena/tdb/base/file/ChannelManager.java
>
>
>It seems on Windows with Oracle/OpenJDK you can call System.gc() to
>(hopefully) release the ByteBuffers that lock the memory regions (and
>then making the files deletable) - but this adds a significant
>overhead. The dispose methods on ByteBufferImpls are not easily
>accessible - you would need some introspection hackery to get hold of
>that cleaner() and that would of course only work on Oracle/OpenJDK.
>as fc.map() still does the same thing.
>
>Close your eyes -  GPL3!
>https://github.com/stain/jdk8u/blob/master/src/share/classes/java/nio/Dire
>ct-X-Buffer.java.template#L72
>http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b
>132/java/nio/DirectByteBuffer.java/#72
>
>
>I tried using the FileChannels from JDK7 NIO2 (e.g.
>FileChannel.open(Path)) instead of through RandomAccessFile - but it
>did not make any difference
>
>
>Perhaps System.gc() is not worth it in general (* on Windows) when
>closing a dataset - I tried to modify the ChannelManager to always do
>this on release, it meant each test in jena-jdbc-tdb took 1.5s instead
>of 0.2s, but it did allow me to delete the used folders from target/
>while the JVM/test was running.
>
>For the tests we could do something like for every 10 tests do
>System.gc() and wipe the old data.
>
>Perhaps Fuseki 2 could do System.gc() on [Remove] && SystemTDB.isWindows.
>
>
>
>On 10 March 2015 at 10:00, ASF GitHub Bot (JIRA) <jira@apache.org> wrote:
>>
>>     [ 
>>https://issues.apache.org/jira/browse/JENA-897?page=com.atlassian.jira.pl
>>ugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354617#com
>>ment-14354617 ]
>>
>> ASF GitHub Bot commented on JENA-897:
>> -------------------------------------
>>
>> Github user asfgit closed the pull request at:
>>
>>     https://github.com/apache/jena/pull/41
>>
>>
>>> jena-jdbc-tdb tests use %TEMP% instead of target/
>>> -------------------------------------------------
>>>
>>>                 Key: JENA-897
>>>                 URL: https://issues.apache.org/jira/browse/JENA-897
>>>             Project: Apache Jena
>>>          Issue Type: Bug
>>>          Components: JDBC
>>>    Affects Versions: Jena 2.12.1, Jena 2.13.0
>>>         Environment: Windowx 8.0 x64, C: with 34 GB free
>>>            Reporter: Stian Soiland-Reyes
>>>            Priority: Critical
>>>             Fix For: Jena 2.13.1
>>>
>>>
>>> .. and thus mvn clean install on Windows will easily consume 37 GB on
>>>C: and run out of disk space - even if Jena is built on a larger
>>>partition.
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>
>
>
>-- 
>Stian Soiland-Reyes
>Apache Taverna (incubating), Apache Commons RDF (incubating)
>http://orcid.org/0000-0001-9842-9718





Mime
View raw message