tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Reese <are...@yahoo-inc.com>
Subject RE: Cores with FlushableGzipOutputStream
Date Thu, 16 Feb 2012 20:45:03 GMT


> -----Original Message-----
> From: Allen Reese [mailto:areese@yahoo-inc.com]
> Sent: Thursday, February 09, 2012 1:28 PM
> To: Tomcat Users List
> Cc: Lars Anderson
> Subject: RE: Cores with FlushableGzipOutputStream
> 
> 
> 
> > -----Original Message-----
> > From: Konstantin Kolinko [mailto:knst.kolinko@gmail.com]
> > Sent: Thursday, February 09, 2012 1:00 PM
> > To: Tomcat Users List
> > Subject: Re: Cores with FlushableGzipOutputStream
> >
> > 2012/2/10 Allen Reese <areese@yahoo-inc.com>:
> > > Try again now that I'm subscribed.
> > >
> > >
> > >> -----Original Message-----
> > >> From: Allen Reese
> > >> Sent: Thursday, February 09, 2012 12:03 PM
> > >> To: 'users@tomcat.apache.org'
> > >> Cc: Lars Anderson
> > >> Subject: Cores with FlushableGzipOutputStream
> > >>
> > >> We've just upgraded from tomcat 6.0.33 to 6.0.35 and started
> having
> > >> the JVM core on our production boxes.
> > >>
> > >> I'm trying to determine what the next course of action should be
> > here.
> > >> I have an Oracle Support contract, but they don't seem to see this
> > as
> > >> a JVM issue, and blame it on a native lib.
> > >>
> > >>
> > >> Thanks.
> > >>
> > >> Allen Reese
> > >> Core Platforms
> > >> Yahoo!, Inc.
> > >>
> > >> Running on linux x86-64, jdk 6u27, 6u29, 6u30, 7u2
> > >>
> > >> We run several tests and the output is:
> > >>
> > >> Jdk   | Version       | flags
> > >>       |
> > >> 6u30  | 6.0.33        | compression enabled
> > >>       | works
> > >> 6u30  | 6.0.35        | compression enabled
> > >>       | cores
> > >> 6u30  | 6.0.35        | compression disabled
> > >>       | works
> > >> 6u30  | 6.0.35        | Remove changes to
> FlushableGzipOutputStream
> > >> [1]
> > >>       | works
> > >> 6u30  | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
> > >>       | cores
> > >>
> > >> 7u2   | 6.0.35        | compression enabled
> > >>       | cores
> > >> 7u2   | 6.0.35        | compression disabled
> > >>       |
> > >> 7u2   | 6.0.35        | Remove changes to
> FlushableGzipOutputStream
> > >> [1]
> > >>       |
> > >> 7u2   | 6.0.35    | -Dsun.zip.disableMemoryMapping=true
> > >>       | cores
> > >>
> > >> https://issues.apache.org/bugzilla/show_bug.cgi?id=52121
> > >>
> > >> I filed an SR with Oracle, as this looks like a JVM bug and got
> the
> > >> following response:
> > >>
> > >> Generic Note
> > >> ------------------------
> > >> Hi Allen,
> > >>
> > >> Thank you for sending the hotspot error logs (hs_err_pid<pid>).
> > >> Each one of them has verbiage that indicates the problem is not
> > >> with
> > Java,
> > >> but with native code:
> > >>
> > >> # Problematic frame:
> > >> # C [libzip.so+0x77e3] char+0xa3
> > >> #
> > >> # If you would like to submit a bug report, please visit:
> > >> # http://java.sun.com/webapps/bugreport/crash.jsp
> > >> # The crash happened outside the Java Virtual Machine in native
> > code.
> > >> # See problematic frame for where to report the bug.
> > >> #
> > >>
> > >> The case description also noted:
> > >>
> > >> Rolling back this patch to tomcat increases stability:
> > >> http://svn.apache.org/viewvc?view=revision&revision=1197382
> > >>
> > >> Again, this points to software other than Java. The Java defect
> > >> mentioned, 4813885, was fixed in June of 2009.
> > >> =====================
> > >>
> > >> Allen Reese
> > >> Core Platforms
> > >> Yahoo!, Inc.
> > >>
> > >> [1]: Patch to remove changes to FlushableGZIPOutputStream from
> > 6.0.35.
> > >>
> > >>(...)
> > >>
> >
> > Hi!
> >
> > If you revert r1197382 you will be facing BZ 52121 [1].
> >
> >
> > Overall, BZ 52121 itself was about a bug in Deflater implementation
> in
> > JRE. The difference is that I tried to make the new implementation as
> > safe as possible,  making sure that Deflater.setLevel() calls do not
> > occur in a row, but there is some data between them.
> >
> > Either way Deflater API does not have such limitation on the use of
> > setLevel(), so the root cause of BZ 52121 is a JRE bug.
> >
> >
> > I think that the current code is safe and you hitting a different
> > issue. Your "better stability" is likely a luck caused by a different
> > state of Deflater thanks to different implementation at Tomcat side.
> >
> > Maybe you have some web resource that consistently produces a
> response
> > that triggers this issue.
> >
> > From experience of BZ 52121 we were lucky that someone was able to
> > provide sample data that reproduced the issue.  It looks that
> > originally it was some product listing generated by JSP page. Maybe
> > you can find something similar.
> >
> > That data is now part of Tomcat testsuite (see
> > TestFlushableGZIPOutputStream class).  Note that the reproducer is
> > rather simple and does not use Tomcat, but just the streams I/O.
> >
> >
> > Anyway, a JRE should not fail fatally with a core dump regardless of
> > what public API the program uses.
> Thanks.  This is what I was after, I'm going to put this thread in my
> SR with Oracle along with Mark's reply.
> 
> I'll see if we can get the data that triggers it, it's a high volume
> system and the cores come after about 3 to 4 hours.

I'm looking at this again, and I'm trying do determine how to enhance the test case [2] to
reproduce what I'm seeing. 

My question is, I have all of the data in pcap format, so I can extract the chunks, what is
the best way to feed this through.
Do I decompress it all and write it all at once through the stream?
Or is this going to matter based on what order the chunks got written in.

--Allen


[2]
http://svn.apache.org/viewvc/tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java?view=markup


> > [1] https://issues.apache.org/bugzilla/show_bug.cgi?id=52121
> >
> > Best regards,
> > Konstantin Kolinko
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message