Return-Path: X-Original-To: apmail-tomcat-dev-archive@www.apache.org Delivered-To: apmail-tomcat-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EB90C9609 for ; Fri, 4 Nov 2011 19:57:18 +0000 (UTC) Received: (qmail 18376 invoked by uid 500); 4 Nov 2011 19:57:18 -0000 Delivered-To: apmail-tomcat-dev-archive@tomcat.apache.org Received: (qmail 18323 invoked by uid 500); 4 Nov 2011 19:57:18 -0000 Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Developers List" Delivered-To: mailing list dev@tomcat.apache.org Received: (qmail 18314 invoked by uid 99); 4 Nov 2011 19:57:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Nov 2011 19:57:18 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Nov 2011 19:57:13 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 375D823888FD for ; Fri, 4 Nov 2011 19:56:52 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1197731 - in /tomcat/tc7.0.x/trunk: ./ java/org/apache/coyote/http11/filters/ test/org/apache/coyote/http11/filters/ webapps/docs/ Date: Fri, 04 Nov 2011 19:56:51 -0000 To: dev@tomcat.apache.org From: markt@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111104195652.375D823888FD@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: markt Date: Fri Nov 4 19:56:51 2011 New Revision: 1197731 URL: http://svn.apache.org/viewvc?rev=1197731&view=rev Log: Fix https://issues.apache.org/bugzilla/show_bug.cgi?id=52121 Based on a test case by David Marcks. Improvements to test case and actual fix by kkolinko. Added: tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java - copied, changed from r1197387, tomcat/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/bug52121-part1 (props changed) - copied unchanged from r1197343, tomcat/trunk/test/org/apache/coyote/http11/filters/bug52121-part1 tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/bug52121-part2 (props changed) - copied unchanged from r1197343, tomcat/trunk/test/org/apache/coyote/http11/filters/bug52121-part2 Modified: tomcat/tc7.0.x/trunk/ (props changed) tomcat/tc7.0.x/trunk/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.java tomcat/tc7.0.x/trunk/webapps/docs/changelog.xml Propchange: tomcat/tc7.0.x/trunk/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Nov 4 19:56:51 2011 @@ -1 +1 @@ -/tomcat/trunk:1156115,1156171,1156276,1156304,1156519,1156530,1156602,1157015,1157018,1157151,1157198,1157204,1157810,1157832,1157834,1157847,1157908,1157939,1158155,1158160,1158176,1158195,1158198-1158199,1158227,1158331,1158334-1158335,1158426,1160347,1160592,1160611,1160619,1160626,1160639,1160652,1160720-1160721,1160772,1160774,1160776,1161303,1161310,1161322,1161339,1161486,1161540,1161549,1161584,1162082,1162149,1162169,1162721,1162769,1162836,1162932,1163630,1164419,1164438,1164469,1164480,1164567,1165234,1165247-1165248,1165253,1165273,1165282,1165309,1165331,1165338,1165347,1165360-1165361,1165367-1165368,1165602,1165608,1165677,1165693,1165721,1165723,1165728,1165730,1165738,1165746,1165765,1165777,1165918,1165921,1166077,1166150-1166151,1166290,1166366,1166620,1166686,1166693,1166752,1166757,1167368,1167394,1169447,1170647,1171692,1172233-1172234,1172236,1172269,1172278,1172282,1172556,1172610,1172664,1172689,1172711,1173020-1173021,1173082,1173088,1173090,1173096 ,1173241,1173256,1173288,1173333,1173342,1173461,1173614,1173630,1173659,1173722,1174061,1174239,1174322,1174325,1174329-1174330,1174337-1174339,1174343,1174353,1174799,1174882,1174884,1174983,1175155,1175158,1175167,1175182,1175190,1175201,1175272,1175275,1175283,1175582,1175589-1175590,1175594,1175602,1175613,1175633,1175690,1175713,1175798,1175889,1175896,1175907,1176584,1176590,1176799,1177050,1177060,1177125,1177152,1177160,1177245,1177850,1177862,1177978,1178209,1178228,1178233,1178449,1178542,1178681,1178684,1178721,1179268,1179274,1180261,1180865,1180891,1180894,1180907,1181028,1181123,1181125,1181136,1181291,1181743,1182796,1183078,1183105,1183142,1183328,1183339-1183340,1183492-1183494,1183605,1184917,1184919,1185018,1185020,1185200,1185588,1185626,1185756,1185758,1186011,1186042-1186045,1186104,1186123,1186137,1186153,1186254,1186257,1186377-1186379,1186479-1186480,1186712,1186743,1186750,1186763,1186890-1186892,1186894,1186949,1187018,1187027-1187028,1187381,1187 755,1187775,1187827,1188301,1188303-1188305,1188399,1188822,1188930-1188931,1189116,1189129,1189183,1189240,1189256,1189386,1189413-1189414,1189477,1189685,1189805,1189857,1189864,1189882,1190034,1190185,1190279,1190339,1190371,1190388-1190389,1190474,1190481,1194915,1195222-1195223,1195531,1195899,1195905,1195943,1195949,1195953,1195955,1195965,1195968,1196175,1196212,1196223,1196304-1196305,1196735,1196825,1196827,1197299-1197300,1197339-1197340 +/tomcat/trunk:1156115,1156171,1156276,1156304,1156519,1156530,1156602,1157015,1157018,1157151,1157198,1157204,1157810,1157832,1157834,1157847,1157908,1157939,1158155,1158160,1158176,1158195,1158198-1158199,1158227,1158331,1158334-1158335,1158426,1160347,1160592,1160611,1160619,1160626,1160639,1160652,1160720-1160721,1160772,1160774,1160776,1161303,1161310,1161322,1161339,1161486,1161540,1161549,1161584,1162082,1162149,1162169,1162721,1162769,1162836,1162932,1163630,1164419,1164438,1164469,1164480,1164567,1165234,1165247-1165248,1165253,1165273,1165282,1165309,1165331,1165338,1165347,1165360-1165361,1165367-1165368,1165602,1165608,1165677,1165693,1165721,1165723,1165728,1165730,1165738,1165746,1165765,1165777,1165918,1165921,1166077,1166150-1166151,1166290,1166366,1166620,1166686,1166693,1166752,1166757,1167368,1167394,1169447,1170647,1171692,1172233-1172234,1172236,1172269,1172278,1172282,1172556,1172610,1172664,1172689,1172711,1173020-1173021,1173082,1173088,1173090,1173096 ,1173241,1173256,1173288,1173333,1173342,1173461,1173614,1173630,1173659,1173722,1174061,1174239,1174322,1174325,1174329-1174330,1174337-1174339,1174343,1174353,1174799,1174882,1174884,1174983,1175155,1175158,1175167,1175182,1175190,1175201,1175272,1175275,1175283,1175582,1175589-1175590,1175594,1175602,1175613,1175633,1175690,1175713,1175798,1175889,1175896,1175907,1176584,1176590,1176799,1177050,1177060,1177125,1177152,1177160,1177245,1177850,1177862,1177978,1178209,1178228,1178233,1178449,1178542,1178681,1178684,1178721,1179268,1179274,1180261,1180865,1180891,1180894,1180907,1181028,1181123,1181125,1181136,1181291,1181743,1182796,1183078,1183105,1183142,1183328,1183339-1183340,1183492-1183494,1183605,1184917,1184919,1185018,1185020,1185200,1185588,1185626,1185756,1185758,1186011,1186042-1186045,1186104,1186123,1186137,1186153,1186254,1186257,1186377-1186379,1186479-1186480,1186712,1186743,1186750,1186763,1186890-1186892,1186894,1186949,1187018,1187027-1187028,1187381,1187 755,1187775,1187827,1188301,1188303-1188305,1188399,1188822,1188930-1188931,1189116,1189129,1189183,1189240,1189256,1189386,1189413-1189414,1189477,1189685,1189805,1189857,1189864,1189882,1190034,1190185,1190279,1190339,1190371,1190388-1190389,1190474,1190481,1194915,1195222-1195223,1195531,1195899,1195905,1195943,1195949,1195953,1195955,1195965,1195968,1196175,1196212,1196223,1196304-1196305,1196735,1196825,1196827,1197299-1197300,1197339-1197340,1197343,1197382,1197386-1197387,1197480,1197578 Modified: tomcat/tc7.0.x/trunk/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.java URL: http://svn.apache.org/viewvc/tomcat/tc7.0.x/trunk/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.java?rev=1197731&r1=1197730&r2=1197731&view=diff ============================================================================== --- tomcat/tc7.0.x/trunk/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.java (original) +++ tomcat/tc7.0.x/trunk/java/org/apache/coyote/http11/filters/FlushableGZIPOutputStream.java Fri Nov 4 19:56:51 2011 @@ -35,58 +35,74 @@ public class FlushableGZIPOutputStream e super(os); } - private static final byte[] EMPTYBYTEARRAY = new byte[0]; - private boolean hasData = false; - /** - * Here we make sure we have received data, so that the header has been for - * sure written to the output stream already. + * It is used to reserve one byte of real data so that it can be used when + * flushing the stream. */ + private byte[] lastByte = new byte[1]; + private boolean hasLastByte = false; + @Override - public synchronized void write(byte[] bytes, int i, int i1) - throws IOException { - super.write(bytes, i, i1); - hasData = true; + public synchronized void write(byte[] bytes) throws IOException { + write(bytes, 0, bytes.length); } @Override - public synchronized void write(int i) throws IOException { - super.write(i); - hasData = true; + public synchronized void write(byte[] bytes, int offset, int length) + throws IOException { + if (length > 0) { + flushLastByte(); + if (length > 1) { + super.write(bytes, offset, length - 1); + } + rememberLastByte(bytes[offset + length - 1]); + } } @Override - public synchronized void write(byte[] bytes) throws IOException { - super.write(bytes); - hasData = true; + public synchronized void write(int i) throws IOException { + flushLastByte(); + rememberLastByte((byte) (i & 0xFF)); } @Override - public synchronized void flush() throws IOException { - if (!hasData) { - return; // do not allow the gzip header to be flushed on its own - } - - // trick the deflater to flush - /** - * Now this is tricky: We force the Deflater to flush its data by - * switching compression level. As yet, a perplexingly simple workaround - * for - * http://developer.java.sun.com/developer/bugParade/bugs/4255743.html - */ - if (!def.finished()) { - def.setInput(EMPTYBYTEARRAY, 0, 0); - - def.setLevel(Deflater.NO_COMPRESSION); - deflate(); + public void close() throws IOException { + flushLastByte(); + super.close(); + } - def.setLevel(Deflater.DEFAULT_COMPRESSION); - deflate(); + private void rememberLastByte(byte b) { + lastByte[0] = b; + hasLastByte = true; + } - out.flush(); + private void flushLastByte() throws IOException { + if (hasLastByte) { + super.write(lastByte, 0, 1); + hasLastByte = false; } + } - hasData = false; // no more data to flush + @Override + public synchronized void flush() throws IOException { + if (hasLastByte) { + // - do not allow the gzip header to be flushed on its own + // - do not do anything if there is no data to send + + // trick the deflater to flush + /** + * Now this is tricky: We force the Deflater to flush its data by + * switching compression level. As yet, a perplexingly simple workaround + * for + * http://developer.java.sun.com/developer/bugParade/bugs/4255743.html + */ + if (!def.finished()) { + def.setLevel(Deflater.NO_COMPRESSION); + flushLastByte(); + def.setLevel(Deflater.DEFAULT_COMPRESSION); + } + } + out.flush(); } /* Copied: tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java (from r1197387, tomcat/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java) URL: http://svn.apache.org/viewvc/tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java?p2=tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java&p1=tomcat/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java&r1=1197387&r2=1197731&rev=1197731&view=diff ============================================================================== --- tomcat/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java (original) +++ tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/TestFlushableGZIPOutputStream.java Fri Nov 4 19:56:51 2011 @@ -20,10 +20,16 @@ import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.File; import java.io.FileInputStream; -import java.io.InputStream; +import java.io.IOException; import java.io.OutputStream; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; import java.util.zip.GZIPInputStream; +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; + import org.junit.Test; import org.apache.catalina.util.IOTools; @@ -38,27 +44,25 @@ public class TestFlushableGZIPOutputStre public void testBug52121() throws Exception { ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream(); - ByteArrayOutputStream sink = new ByteArrayOutputStream(); - OutputStream output = new FlushableGZIPOutputStream(byteOutStream); File sourcesDir = new File("test/org/apache/coyote/http11/filters/"); - InputStream input; + List parts = new ArrayList(); + byte[] part; - input = new FileInputStream(new File(sourcesDir, "bug52121-part1")); - try { - IOTools.flow(input, output); - } finally { - input.close(); - } + part = loadFile(new File(sourcesDir, "bug52121-part1")); + parts.add(part); + flowBytes(part, output); output.flush(); - input = new FileInputStream(new File(sourcesDir, "bug52121-part2")); - try { - IOTools.flow(input, output); - } finally { - input.close(); - } + part = loadFile(new File(sourcesDir, "bug52121-part2")); + parts.add(part); + flowBytes(part, output); + output.flush(); + + part = "data2".getBytes("ASCII"); + parts.add(part); + output.write(part); output.flush(); output.close(); @@ -67,6 +71,49 @@ public class TestFlushableGZIPOutputStre new ByteArrayInputStream(byteOutStream.toByteArray()); GZIPInputStream inflaterStream = new GZIPInputStream(byteInStream); - IOTools.flow(inflaterStream, sink); + ByteArrayOutputStream sink = new ByteArrayOutputStream(); + try { + IOTools.flow(inflaterStream, sink); + } finally { + sink.close(); + } + + byte[] decompressedBytes = sink.toByteArray(); + int originalLength = 0; + for (byte[] bytes : parts) { + assertArrayEquals(bytes, Arrays.copyOfRange(decompressedBytes, + originalLength, originalLength + bytes.length)); + originalLength += bytes.length; + } + assertEquals(originalLength, decompressedBytes.length); + } + + /** + * Loads file into memory. + */ + private byte[] loadFile(File file) throws IOException { + ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream(); + FileInputStream input = new FileInputStream(file); + try { + IOTools.flow(input, byteOutStream); + } finally { + input.close(); + } + return byteOutStream.toByteArray(); + } + + /** + * Writes data to the stream and returns the size of the file. + */ + private void flowBytes(byte[] bytes, OutputStream output) + throws IOException { + // Could use output.write(), but IOTools writes in small portions, and + // that is more natural + ByteArrayInputStream byteInStream = new ByteArrayInputStream(bytes); + try { + IOTools.flow(byteInStream, output); + } finally { + byteInStream.close(); + } } } Propchange: tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/bug52121-part1 ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Propchange: tomcat/tc7.0.x/trunk/test/org/apache/coyote/http11/filters/bug52121-part2 ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Modified: tomcat/tc7.0.x/trunk/webapps/docs/changelog.xml URL: http://svn.apache.org/viewvc/tomcat/tc7.0.x/trunk/webapps/docs/changelog.xml?rev=1197731&r1=1197730&r2=1197731&view=diff ============================================================================== --- tomcat/tc7.0.x/trunk/webapps/docs/changelog.xml (original) +++ tomcat/tc7.0.x/trunk/webapps/docs/changelog.xml Fri Nov 4 19:56:51 2011 @@ -201,6 +201,11 @@ In HTTP connectors: self-guard against using a non-recycled input buffer. Requests will be rejected with response status 400. (kkolinko) + + 52121: Fix possible output corruption when compression is + enabled for a connector and the response is flushed. Includes a test + caes provided by David Marcks. (kkolinko/markt) + --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org For additional commands, e-mail: dev-help@tomcat.apache.org