tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Jaquemet <olivier.jaque...@jalios.com>
Subject Re: OutOfMemory on large file download with AJP and cachingAllowed=false
Date Tue, 23 Apr 2019 15:29:44 GMT
On 23/04/2019 16:12, Christopher Schultz wrote:
> Olivier,
Hi Christopher,
Thanks for you answer.
> On 4/23/19 05:58, Olivier Jaquemet wrote:
>> Hi all,
>>
>> We were able to reproduce a OutOfMemory error when using AJP and
>> the Resources cachingAllowed=false directive. It looks like a bug
>> of AJP connector(s), as it does not occurs with other HTTP
>> connectors.
>>
>> Could you confirm the behavior described below is indeed bug ? (if
>> you do, I'll create the issue on bugzilla)
>>
>> To reproduce :
>>
>> * Use latest tomcat 8.5 version (tested with Tomcat 8.5.40) * Add
>> an AJP connector to server.xml <Connector port="8009"
>> protocol="AJP/1.3" redirectPort="8443" />
> nb: no compression
>
> nb: NIO connector is in use ; no APR (see stack trace for thread name)
>
>> * Add the following directive to context.xml : <Resources
>> cachingAllowed="false" />
> Okay. Why context.xml, by the way?
I don't even know (yet...) why this setting was added in the first place 
in the environment where it was present... !
so why this file... I don't know either :)
>> * Create a large file in the samples webapp, for example : cd
>> webapps/examples dd if=/dev/zero of=large.txt bs=1k count=2000000
> ~2GiB static file
>
>> * Start Tomcat with a 1024 mb heap size (JAVA_OPTS="-Xms1024m
>> -Xmx1024m" * Configure Apache HTTPD to use mod_proxy_ajp, or mod_jk
>> (both will have the same issue) [1] * Start Apache HTTPD * Download
>> file through default HTTP connector
>> http://localhost:8080/examples/large.txt --> OK * Download file
>> through Apache/AJP http://localhost/examples/large.txt --> BUG :
>> OutOfMemory error occurs Exception in thread
>> "ajp-nio-8009-exec-10" java.lang.OutOfMemoryError: Java heap space
>> at
>>
>> org.apache.catalina.webresources.FileResource.getContent(FileResource.
> java:207)
>>   at
>>
>> org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServl
> et.java:992)
>>   at
>>
>> org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:
> 438)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
>> ...
> That's ... interesting. What if you request the file more than once
> via the default connector? I would have expected these code paths to
> be relatively the same.
There are no problems at all if I request the file many times in // 
using the default HTTP connector.
> Can you generate a heap dump when the OOME occurs? Use the following
> JVM startup options:
>
> - -XX:+HeapDumpOnOutOfMemoryError
> - -XX:HeapDumpPath=/path/to/heap-dump-file
Sure. Here are heap dumps and log file from two different run :
https://www.dropbox.com/sh/q6iqpe42fxvsvin/AAARR4uOnn-qJ4PCg8e_Ll8La?dl=0
> If you generate a heap dump, are you able to navigate it? I've heard
> that Eclipse MAT is good. I've mostly used YourKit and it's quite
> good, able to pick-out the "big things" and help locate what kind of
> object(s) is/are taking up all the space.

I've used MAT some times before, and indeed it's a really nice tool !
However, if you are familiar with YourKit, you might find the culprit 
faster than me...
Can you have a look a the heap dump I shared above ?

I did have a look at it with MAT... however I could not find any 
definitive conclusion ...
There is a BufferedWriter of an AccessLogValve which reference a large 
char array with \0 (like the content of the large generated file...).
There are many (leaking?) 
java.util.zip.ZipFile$ZipFileInflaterInputStream (however you said there 
were no compression, though here they are)
...

> Reading the code for FileResource.getContent, it's clear that the
> entire file is being loaded into memory, which obviously isn't going
> to work, here. I'm wondering why that's happening since streaming is
> the correct behavior when caching=false. Also strange is that
> DefaultServlet will attempt to call FileResource.getContent() -- which
> returns a byte[] -- and, if that returns null, it will call
> FileResource.getInputStream which ... calls this.getContent. So this
> looks like a special-case for FileResource just trying to implement
> that interface in the simplest way possible.
>
> FileResource seems to implement in-memory caching whether it's enabled
> or not.
>
> I can't understand why this doesn't fail for the other kind of
> connector. Everything else is the same? You have two separate
> connectors in one instance, or are you changing the connector between
> tests?

Everything is exactly the same as I have only one instance with two 
separate connectors (AJP+HTTP).

One last (confusing) information in form of an exception, that I could 
*not* reproduce our test environments, but that I saw on the server 
where the symptom first occurred.
I don't think this is related to the OOM, but it might be another 
symptom of the same resource configuration.

java.lang.ArrayIndexOutOfBoundsException: Unable to return 
[/path/to/file/being/downloaded.ext] as a byte array since the resource 
is [2,637,615,704] bytes in size which is larger than the maximum size 
of a byte array
     at 
org.apache.catalina.webresources.FileResource.getContent(FileResource.java:196)
     at 
org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:1000)
     at 
org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:438)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
     at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
     at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
     at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
[...]
     at 
org.apache.catalina.servlets.DefaultServlet.service(DefaultServlet.java:418)
     at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
     at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
     at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
     at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
     at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
     at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
     at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
     at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
     at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
     at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:486)
     at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
     at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
     at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
     at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
     at java.lang.Thread.run(Thread.java:748)

Olivier

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message