ant-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Loren Kratzke (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
Date Tue, 07 Apr 2015 16:46:12 GMT

    [ https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483468#comment-14483468
] 

Loren Kratzke edited comment on IVY-1197 at 4/7/15 4:45 PM:
------------------------------------------------------------

I would be happy to provide you with a project that will reproduce the issue. I can and will
do that. 

Generally speaking from a high level, the utility classes are calling convenience methods
and writing to streams that ultimately buffer the data being written. There is buffering,
then more buffering, and even more buffering until you have multiple copies of the entire
content of the stream stored in over sized buffers (because they double in size when they
fill up). Oddly, the twist is that the JVM hits a limit no matter how much RAM you allocate.
Once the buffers total more than about ~1GB (which is what happens with a 100-200MB upload)
the JVM refuses to allocate more buffer space (even if you jack up the RAM to 20GB, no cigar).
Honestly, there is no benefit in buffering any of this data to begin with, it is just a side
effect of using high level copy methods. There is no memory ballooning at all when the content
is written directly to the network.

I will provide a test project and note the break points where you can debug and watch the
process walk all the way down the isle to an OOME. I will have this for you asap.


was (Author: qphase):
I would be happy to provide you with a project that will reproduce the issue. I can and will
do that. 

Generally speaking from a high level, the utility classes are calling convenience methods
and writing to streams that ultimately buffer the data being written. There is buffering,
then more buffering, and even more buffering until you have multiple copies of the entire
content of the stream stored in over sized buffers (because they double in size when they
fill up). Oddly, the twist is that the JVM hits a limit no matter how much RAM you allocate.
Once the buffers total more than about ~1GB (which is what happens with a 100-200MB upload)
the JVM refuses to allocate more buffer space (even is you jack up the RAM to 20GB, no cigar).
Honestly, there is no benefit in buffering any of this data to begin with, it is just a side
effect of using high level copy methods. There is no memory ballooning at all when the content
is written directly to the network.

I will provide a test project and note the break points where you can debug and watch the
process walk all the way down the isle to an OOME. I will have this for you asap.

> OutOfMemoryError during ivy:publish
> -----------------------------------
>
>                 Key: IVY-1197
>                 URL: https://issues.apache.org/jira/browse/IVY-1197
>             Project: Ivy
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 2.0
>            Reporter: Michael Rumpf
>         Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt, org.apache.ivy.util.url.HttpClientHandler.patch
>
>
> When publishing a large file, an OutOfMemoryError occurs.
> {code}
> [ivy:publish] 	published ppg to XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> BUILD FAILED
> /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
> /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
> 	at java.util.Arrays.copyOf(Arrays.java:2786)
> 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> 	at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
> 	at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
> 	at org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
> 	at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
> 	at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
> 	at org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
> 	at org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
> 	at org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
> 	at org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
> 	at org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
> 	at org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
> 	at org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
> 	at org.apache.ivy.Ivy.publish(Ivy.java:600)
> 	at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
> 	at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
> 	at org.apache.tools.ant.Task.perform(Task.java:348)
> 	at org.apache.tools.ant.Target.execute(Target.java:390)
> 	at org.apache.tools.ant.Target.performTasks(Target.java:411)
> 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
> 	at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
> 	at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
> 	at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
> 	at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Total time: 14 minutes 24 seconds
> Finished: FAILURE
> {code}
> The size of the file that is being uploaded is: 687712714, so around 650-700MB.
> The publish task is part of a Hudson Ant build where the artefacts are published to an
Artifactory repository at the end.
> I have given the Job 1300MB for the max heap size.
> It seems as if the whole file is loaded into memory for the upload.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message