tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Fisk <af...@limepeer.com>
Subject Re: Any clue on this, please? Uploading large files - out of memory
Date Wed, 03 Dec 2003 19:31:43 GMT
I unfortunately don't have the time to step through each and every 
thread where these errors are occuring, although I wish I did.  The 
question is, has someone done this?  It's about the most tedious coding 
process I know of, so it just wouldn't surprise me if no one's actually 
done it.  Do you know what threads these errors occur in?  If so, do you 
know when they occur?  Can you reproduce them?

Clearly there are legitimate OOMEs, there just much rarer than the 
illegitimate ones.  It's the legitimacy or illegitimacy that's tough to 
determine.  I'm sure this process has happened, but then again, maybe not.

-Adam


Shapira, Yoav wrote:

> Howdy,
> I would throw out one more piece of advice: consider jakarta commons
> fileupload component.  It is well-designed with respect to handling
> large files.
> 
> As to the "OutOfMemoryErrors are, in fact, always bugs" claim -- well,
> that's the most amusing thing I've heard today ;)  If they are bugs,
> find the buggy code.
> 
> Yoav Shapira
> Millennium ChemInformatics
> 
> 
> 
>>-----Original Message-----
>>From: Fabrizio Nesti [mailto:nesti@medialab.sissa.it]
>>Sent: Wednesday, December 03, 2003 2:01 PM
>>To: Tomcat Developers List
>>Subject: Re: Any clue on this, please? Uploading large files - out of
>>memory
>>
>>Indeed I am not 100% sure of the real cause of the OOME below.
>>However, as far as my request is concerned, it seems that the upload
>>component that we use (Echopoint's one, quite cool) _could_ (and
> 
> should)
> 
>>have used an InputStream.
>>
>>So this is enough for me to go bother them and no longer the tomcat-dev
> 
> :).
> 
>>I was thinking that tomcat was the critical point, so I wrote here just
> 
> to
> 
>>be sure.
>>
>>In case you need further testing please let me know, for the rest it's
> 
> up
> 
>>to
>>you tomcat developers... and... thanks for your good work.
>>
>>cheers,
>>Fabrizio
>>
>>
>>On Wed, 3 Dec 2003, Adam Fisk wrote:
>>
>>
>>>I've heard mention on this list many times of these OutOfMemoryErrors
>>>not being bugs.  I work on a Java app that experiences very high
> 
> network
> 
>>>traffic load, however, and it's been my experience that
>>>OutOfMemoryErrors are, in fact, always bugs regardless of how
> 
> tempting
> 
>>>it is to chalk it up to something else.
>>>
>>>What makes everyone so certain it's not a bug or multiple bugs?
> 
> Since
> 
>>>you don't get stack traces and at best can pin down the thread name
> 
> for
> 
>>>OutOfMemoryErrors, they take a lot of time to track down.  In my
>>>experience, though, they tend to result from an unaccounted for
>>>degenerate request coming that causes the code to, well, consume all
> 
> the
> 
>>>available memory (i.e., a bug).
>>>
>>>-Adam
>>>
>>>
>>>Shapira, Yoav wrote:
>>>
>>>
>>>>Howdy,
>>>>This belongs on the user, not dev, list.  It's most likely a simple
>>>>issue (not a bug) with you not allocating enough memory to the JVM.
>>>>Please pursue this issue on the user list.
>>>>
>>>>Yoav Shapira
>>>>Millennium ChemInformatics
>>>>
>>>>
>>>>
>>>>
>>>>>-----Original Message-----
>>>>>From: Fabrizio Nesti [mailto:nesti@medialab.sissa.it]
>>>>>Sent: Wednesday, December 03, 2003 1:35 PM
>>>>>To: tomcat-dev@jakarta.apache.org
>>>>>Subject: Any clue on this, please? Uploading large files - out of
>>>>
>>>>memory
>>>>
>>>>
>>>>>Hi,
>>>>>any comment on this "out of memory" with large file upload?
>>>>>
>>>>>This error seems recurring to a bunch of users, but I'm wondering
> 
> if
> 
>>>>it's a
>>>>
>>>>
>>>>>problem in the tomcat implementation, or if there are other layers
> 
> to
> 
>>>>>blame.
>>>>>
>>>>>Thanks for any light on this darkness, since I've a tight schedule
> 
> on
> 
>>>>>this issue... unfortunately..
>>>>>cheers,
>>>>>fabrizio
>>>>>
>>>>>
>>>>>---------- Forwarded message ----------
>>>>>Date: Mon, 1 Dec 2003 14:36:57 +0100 (MET)
>>>>>From: Fabrizio Nesti <nesti@medialab.sissa.it>
>>>>>To: tomcat-dev@jakarta.apache.org
>>>>>Subject: Uploading large files - out of memory exception
>>>>>
>>>>>Dear tomcat developers,
>>>>>
>>>>>I've noticed a problem while uploading files with tomcat 4.1.x.
>>>>>
>>>>>- When uploading large files (say larger than 10MB) tomcat throws
> 
> an
> 
>>>>out
>>>>
>>>>
>>>>>of memory exception.
>>>>>
>>>>>- The problem can be avoided raising the memory allocation (as
> 
> found in
> 
>>>>>other similar messages, -Xms128m -Xmx512m) but this is not a
>>>>
>>>>solution,
>>>>
>>>>
>>>>>since it depends on the memory and just makes the limit higher.
>>>>>
>>>>>I do not know how the actual download works (we are using the
>>>>>EchoPoint fileupload component, for that matters) but it seems that
>>>>>the whole file is slurped in memory _before_ passing the request to
> 
> a
> 
>>>>>user servlet. Indeed it seems that our download component is never
>>>>>invoked (see the error below).
>>>>>
>>>>>Is there a configuration option to avoid this behavior, or there's
>>>>>nothing to do but limit the filesize?
>>>>>
>>>>>Thanks in advance for any clue and
>>>>>cheers,
>>>>>Fabrizio
>>>>>
>>>>>
>>>>>PS: The error:
>>>>>
>>>>>...
>>>>>description The server encountered an internal error (Internal
> 
> Server
> 
>>>>>Error)
>>>>>that prevented it from fulfilling this request.
>>>>>
>>>>>exception
>>>>>
>>>>>javax.servlet.ServletException: Servlet execution threw an
> 
> exception
> 
>>>>>      at
>>>
>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> 
> c
> 
>>>>atio
>>>>
>>>>
>>>>>nFilterChain.java:269)
>>>>>      at
>>>
>>>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
> 
> l
> 
>>>>terC
>>>>
>>>>
>>>>>hain.java:193)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> 
> l
> 
>>>>ve.j
>>>>
>>>>
>>>>>ava:256)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext
> 
> .
> 
>>>>invo
>>>>
>>>>
>>>>>keNext(StandardPipeline.java:643)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
> 
> :
> 
>>>>480)
>>>>
>>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> 
> l
> 
>>>>ve.j
>>>>
>>>>
>>>>>ava:191)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext
> 
> .
> 
>>>>invo
>>>>
>>>>
>>>>>keNext(StandardPipeline.java:643)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
> 
> :
> 
>>>>480)
>>>>
>>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2
> 
> 4
> 
>>>>17)
>>>>
>>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> 
> v
> 
>>>>a:18
>>>>
>>>>
>>>>>0)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext
> 
> .
> 
>>>>invo
>>>>
>>>>
>>>>>keNext(StandardPipeline.java:643)
>>>>>      at
>>>
>>>org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcher
> 
> V
> 
>>>>alve
>>>>
>>>>
>>>>>.java:171)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext
> 
> .
> 
>>>>invo
>>>>
>>>>
>>>>>keNext(StandardPipeline.java:641)
>>>>>      at
>>>
>>>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
> 
> v
> 
>>>>a:17
>>>>
>>>>
>>>>>2)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext
> 
> .
> 
>>>>invo
>>>>
>>>>
>>>>>keNext(StandardPipeline.java:641)
>>>>>      at
>>>
>>>org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:5
> 
> 7
> 
>>>>7)
>>>>
>>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext
> 
> .
> 
>>>>invo
>>>>
>>>>
>>>>>keNext(StandardPipeline.java:641)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
> 
> :
> 
>>>>480)
>>>>
>>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
> 
> e
> 
>>>>.jav
>>>>
>>>>
>>>>>a:174)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext
> 
> .
> 
>>>>invo
>>>>
>>>>
>>>>>keNext(StandardPipeline.java:643)
>>>>>      at
>>>
>>>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
> 
> :
> 
>>>>480)
>>>>
>>>>
>>>>>      at
>>>
>>>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
>>>
>>>>>      at
>>>
>>>org.apache.catalina.connector.http.HttpProcessor.process(HttpProcessor
> 
> .
> 
>>>>java
>>>>
>>>>
>>>>>:1040)
>>>>>      at
>>>
>>>org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.jav
> 
> a
> 
>>>>:115
>>>>
>>>>
>>>>>1)
>>>>>      at java.lang.Thread.run(Thread.java:536)
>>>>>
>>>>>root cause
>>>>>
>>>>>java.lang.OutOfMemoryError
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>---------------------------------------------------------------------
>>>
>>>>>To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
>>>>>For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>This e-mail, including any attachments, is a confidential business
>>
>>communication, and may contain information that is confidential,
>>proprietary and/or privileged.  This e-mail is intended only for the
>>individual(s) to whom it is addressed, and may not be saved, copied,
>>printed, disclosed or used by anyone else.  If you are not the(an)
> 
> intended
> 
>>recipient, please immediately delete this e-mail from your computer
> 
> system
> 
>>and notify the sender.  Thank you.
>>
>>>>
>>>>
> ---------------------------------------------------------------------
> 
>>>>To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
>>>>For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
>>>>
>>>>
>>>>.
>>>>
>>>
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
>>>
>>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
> 
> 
> 
> 
> 
> This e-mail, including any attachments, is a confidential business communication, and
may contain information that is confidential, proprietary and/or privileged.  This e-mail
is intended only for the individual(s) to whom it is addressed, and may not be saved, copied,
printed, disclosed or used by anyone else.  If you are not the(an) intended recipient, please
immediately delete this e-mail from your computer system and notify the sender.  Thank you.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
> 
> 
> .
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org


Mime
View raw message