tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: OutOfMemory on large file download with AJP and cachingAllowed=false
Date Tue, 23 Apr 2019 14:12:43 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Olivier,

On 4/23/19 05:58, Olivier Jaquemet wrote:
> Hi all,
> 
> We were able to reproduce a OutOfMemory error when using AJP and
> the Resources cachingAllowed=false directive. It looks like a bug
> of AJP connector(s), as it does not occurs with other HTTP
> connectors.
> 
> Could you confirm the behavior described below is indeed bug ? (if
> you do, I'll create the issue on bugzilla)
> 
> To reproduce :
> 
> * Use latest tomcat 8.5 version (tested with Tomcat 8.5.40) * Add
> an AJP connector to server.xml <Connector port="8009"
> protocol="AJP/1.3" redirectPort="8443" />

nb: no compression

nb: NIO connector is in use ; no APR (see stack trace for thread name)

> * Add the following directive to context.xml : <Resources
> cachingAllowed="false" />

Okay. Why context.xml, by the way?

> * Create a large file in the samples webapp, for example : cd
> webapps/examples dd if=/dev/zero of=large.txt bs=1k count=2000000

~2GiB static file

> * Start Tomcat with a 1024 mb heap size (JAVA_OPTS="-Xms1024m
> -Xmx1024m" * Configure Apache HTTPD to use mod_proxy_ajp, or mod_jk
> (both will have the same issue) [1] * Start Apache HTTPD * Download
> file through default HTTP connector 
> http://localhost:8080/examples/large.txt --> OK * Download file
> through Apache/AJP http://localhost/examples/large.txt --> BUG :
> OutOfMemory error occurs Exception in thread
> "ajp-nio-8009-exec-10" java.lang.OutOfMemoryError: Java heap space 
> at
> 
> org.apache.catalina.webresources.FileResource.getContent(FileResource.
java:207)
>
>  at
> 
> org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServl
et.java:992)
>
>  at
> 
> org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:
438)
>
> 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
> ...

That's ... interesting. What if you request the file more than once
via the default connector? I would have expected these code paths to
be relatively the same.

Can you generate a heap dump when the OOME occurs? Use the following
JVM startup options:

- -XX:+HeapDumpOnOutOfMemoryError
- -XX:HeapDumpPath=/path/to/heap-dump-file

If you generate a heap dump, are you able to navigate it? I've heard
that Eclipse MAT is good. I've mostly used YourKit and it's quite
good, able to pick-out the "big things" and help locate what kind of
object(s) is/are taking up all the space.

Reading the code for FileResource.getContent, it's clear that the
entire file is being loaded into memory, which obviously isn't going
to work, here. I'm wondering why that's happening since streaming is
the correct behavior when caching=false. Also strange is that
DefaultServlet will attempt to call FileResource.getContent() -- which
returns a byte[] -- and, if that returns null, it will call
FileResource.getInputStream which ... calls this.getContent. So this
looks like a special-case for FileResource just trying to implement
that interface in the simplest way possible.

FileResource seems to implement in-memory caching whether it's enabled
or not.

I can't understand why this doesn't fail for the other kind of
connector. Everything else is the same? You have two separate
connectors in one instance, or are you changing the connector between
tests?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAly/HVsACgkQHPApP6U8
pFgkLg//SGrPQ86VAyamI2nK0Ai67C1Y33+pBqEqlPP4UzPUu8cjnQd8cigvGqrv
HvgJJJrsDEA5XRpS7I2Xw/rUMHHyXjalarCHr0dHQdYbL108HLDlXSzhOBD/viA+
PZIeMjWAlcpZfRk1Eg9pZtlAbzANi8X6EdbKutpkMo+6clUmAaAMLI4C3e/OInKA
IqfUqqEseeS6Bdi51GZoNfm8yspWz5hEcBSoWI5H5kTQCbLZ4Wpf4D2RXFI5Cj1s
rVaGG9ne47QQKa1dS5iQGBVFE8MwLWPIW7rkyIrRbVaNWd54mX18cL/hMmQuiOfd
9D3KrUm6j8TeP48OcD5LFW5uz8k92cOZ114cVlql5ndifBp1XdC2TXddGjiC4Uf8
8DhG94wGIiv6U51v1XEtF9u7/b4d8UMsEsqpzZ/EcN85Bs5ZHrzGzDrhgq/SrsC3
tutVODV78FWM5daxvilFhzXAtbaF+wqFzakGIeHQ8QefQFR9rEPvMKxrrMKJgvM2
2DLNNvKUmBZICYDEahzHTbcdaELVRwbQwO5cElk1rcBp66EhG4DY0wgvimeQbnjA
iPjMG2VCSIXsVghV0qgLTPk8B8S00i/e18TYl46ciJpTvLE5tleDS/kRb75oj3Hm
aWOZ+qAAYpAiP1vsUFyHQfWYBIqfK7h7fRjruWDfBAtabbPRRyg=
=VhGf
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message