community-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stian Soiland-Reyes <st...@apache.org>
Subject Re: Disk space requirement for building on Windows
Date Tue, 10 Mar 2015 11:14:34 GMT
The System.gc() is not guaranteed to work - but it did work for me.
That is what I meant by a 'best effort' within the test. The test does
not require all those old test folders to hang around.

Could we do a workaround for the test to set say a tinier block-size
instead of 8 MB?


As I have been unable to build all of Jena on Windows - what is the
actual disk space requirement? It should at least be documented in the
README.

On 10 March 2015 at 11:07, Rob Vesse <rvesse@dotnetrdf.org> wrote:
> Using an alternative approach would not make any difference
>
> It is a fundamental bug in Windows memory mapped files that means that a
> JVM can never guarantee to completely release memory mapped files while
> the JVM is alive.
>
> Andy has posted this many times on threads about TDB on Windows in the
> past.  No workaround we could attempt could ever solve the issue on
> Windows so there is really no point in expending effort changing something
> low level that otherwise works fine across multiple platforms.
>
> Rob
>
> On 10/03/2015 10:25, "Stian Soiland-Reyes" <stain@apache.org> wrote:
>
>> Thanks, Rob!
>>
>>I tried looking yesterday at ways to reduce the disk space
>>requirements when building on Windows - including truncating the files
>>after closing. This seems to require deep changes into TDBs
>>ChannelManager which keeps the corresponding FileChannels - perhaps a
>>new method for that purpose?
>>
>>https://github.com/apache/jena/blob/master/jena-tdb/src/main/java/com/hp/h
>>pl/jena/tdb/base/file/ChannelManager.java
>>
>>
>>It seems on Windows with Oracle/OpenJDK you can call System.gc() to
>>(hopefully) release the ByteBuffers that lock the memory regions (and
>>then making the files deletable) - but this adds a significant
>>overhead. The dispose methods on ByteBufferImpls are not easily
>>accessible - you would need some introspection hackery to get hold of
>>that cleaner() and that would of course only work on Oracle/OpenJDK.
>>as fc.map() still does the same thing.
>>
>>Close your eyes -  GPL3!
>>https://github.com/stain/jdk8u/blob/master/src/share/classes/java/nio/Dire
>>ct-X-Buffer.java.template#L72
>>http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b
>>132/java/nio/DirectByteBuffer.java/#72
>>
>>
>>I tried using the FileChannels from JDK7 NIO2 (e.g.
>>FileChannel.open(Path)) instead of through RandomAccessFile - but it
>>did not make any difference
>>
>>
>>Perhaps System.gc() is not worth it in general (* on Windows) when
>>closing a dataset - I tried to modify the ChannelManager to always do
>>this on release, it meant each test in jena-jdbc-tdb took 1.5s instead
>>of 0.2s, but it did allow me to delete the used folders from target/
>>while the JVM/test was running.
>>
>>For the tests we could do something like for every 10 tests do
>>System.gc() and wipe the old data.
>>
>>Perhaps Fuseki 2 could do System.gc() on [Remove] && SystemTDB.isWindows.
>>
>>
>>
>>On 10 March 2015 at 10:00, ASF GitHub Bot (JIRA) <jira@apache.org> wrote:
>>>
>>>     [
>>>https://issues.apache.org/jira/browse/JENA-897?page=com.atlassian.jira.pl
>>>ugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354617#com
>>>ment-14354617 ]
>>>
>>> ASF GitHub Bot commented on JENA-897:
>>> -------------------------------------
>>>
>>> Github user asfgit closed the pull request at:
>>>
>>>     https://github.com/apache/jena/pull/41
>>>
>>>
>>>> jena-jdbc-tdb tests use %TEMP% instead of target/
>>>> -------------------------------------------------
>>>>
>>>>                 Key: JENA-897
>>>>                 URL: https://issues.apache.org/jira/browse/JENA-897
>>>>             Project: Apache Jena
>>>>          Issue Type: Bug
>>>>          Components: JDBC
>>>>    Affects Versions: Jena 2.12.1, Jena 2.13.0
>>>>         Environment: Windowx 8.0 x64, C: with 34 GB free
>>>>            Reporter: Stian Soiland-Reyes
>>>>            Priority: Critical
>>>>             Fix For: Jena 2.13.1
>>>>
>>>>
>>>> .. and thus mvn clean install on Windows will easily consume 37 GB on
>>>>C: and run out of disk space - even if Jena is built on a larger
>>>>partition.
>>>
>>>
>>>
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v6.3.4#6332)
>>
>>
>>
>>--
>>Stian Soiland-Reyes
>>Apache Taverna (incubating), Apache Commons RDF (incubating)
>>http://orcid.org/0000-0001-9842-9718
>
>
>
>



-- 
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons RDF (incubating)
http://orcid.org/0000-0001-9842-9718

Mime
View raw message