From users-return-264669-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Thu May 3 16:47:25 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D6B0C180625 for ; Thu, 3 May 2018 16:47:24 +0200 (CEST) Received: (qmail 50923 invoked by uid 500); 3 May 2018 14:47:22 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 50912 invoked by uid 99); 3 May 2018 14:47:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 May 2018 14:47:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 7AD30180787 for ; Thu, 3 May 2018 14:47:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.085 X-Spam-Level: *** X-Spam-Status: No, score=3.085 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FORGED_HOTMAIL_RCVD2=1.187, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=hotmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Q6IRLO6hZ4-R for ; Thu, 3 May 2018 14:47:19 +0000 (UTC) Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-oln040092068098.outbound.protection.outlook.com [40.92.68.98]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 3D4645FB3B for ; Thu, 3 May 2018 14:47:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=XXl5fEnS5zzXUwvcogQokuUilFvi7hiP/dAqPHL8dVE=; b=Mk0dYESgZmL0eJt9EsesMl3cZVFC0Wzfrt1VrHA8CLsf+klVBgi3yFuaaS7a9jpIvZYS5SydSPSudnvrCAcJE+vRg+63RNeAWbKRNty1HD1DZdfGBxDDb/dupiIov3lU6WeBgG31MNxT8Da0NNXgro3+J+alGsdTWYJ471Ikr6s4DlYeh5DllRp81etOkyc00V8dRu6PdfHRgUYVl30z/5FNYYtfIRlaHdGW54AzIrtUgys1+vviols0PH2PRSaizhfzOCBzF69epWzl6XkQkEb2TgIonHEu2HhiU3AdIRbQsamWgBfw3efsTpuyyqs6iIZt5/hPd808OBFEHEyHGQ== Received: from HE1EUR02FT006.eop-EUR02.prod.protection.outlook.com (10.152.10.55) by HE1EUR02HT140.eop-EUR02.prod.protection.outlook.com (10.152.11.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.715.13; Thu, 3 May 2018 14:47:12 +0000 Received: from DB6P189MB0565.EURP189.PROD.OUTLOOK.COM (10.152.10.58) by HE1EUR02FT006.mail.protection.outlook.com (10.152.10.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.715.13 via Frontend Transport; Thu, 3 May 2018 14:47:12 +0000 Received: from DB6P189MB0565.EURP189.PROD.OUTLOOK.COM ([fe80::489e:ae3b:5cce:adf2]) by DB6P189MB0565.EURP189.PROD.OUTLOOK.COM ([fe80::489e:ae3b:5cce:adf2%13]) with mapi id 15.20.0715.024; Thu, 3 May 2018 14:47:11 +0000 From: R Mundell To: "users@tomcat.apache.org" Subject: =?Windows-1252?Q?Intermittently_a_Tomcat=92s_7's_response_loses_all_(non-?= =?Windows-1252?Q?Coyote-added)_headers?= Thread-Topic: =?Windows-1252?Q?Intermittently_a_Tomcat=92s_7's_response_loses_all_(non-?= =?Windows-1252?Q?Coyote-added)_headers?= Thread-Index: AdPi7ZIXHsANP8ILRJSRrPigbnvXgw== Date: Thu, 3 May 2018 14:47:11 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:00AE5D8B507E928FA2884BC81D673622DBC0A9C8115DAD934BD784E7C4246CB0;UpperCasedChecksum:D55D659126E09CA457CA58485A29A5DEB6642A82F817A4246B04092608B9684A;SizeAsReceived:6966;Count:43 x-tmn: [ZApF/YHsfpTjWwzkryBi89l4sl59jH7L] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;HE1EUR02HT140;7:usQkiNlKneiPc5abdsHHx60SUi0zqwjRy0/naaxv33h0Rcr17Mij0+69xEBosXdYj1R+Hmf3q6F+YfZbNeyjJjBjLxHzU1t1glDni6QWTkWRCcvuyOxE4qMRQTTvhqFTwEiP1wx2OWLUlPt/qgM3mIb6CKAl+Zo62+uc98IXlU3Vvn7Oh+E/mOkphnqRznzaBhAYPDx9KtAGTPn2V83/rvDFWl83RVy6cdL22wOXzjLkXvl+9n70OBaVNOdrb/6t x-incomingheadercount: 43 x-eopattributedmessage: 0 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(201702061078)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1603101448)(1601125374)(1701031045);SRVR:HE1EUR02HT140; x-ms-traffictypediagnostic: HE1EUR02HT140: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(444000031);SRVR:HE1EUR02HT140;BCL:0;PCL:0;RULEID:;SRVR:HE1EUR02HT140; x-forefront-prvs: 066153096A x-forefront-antispam-report: SFV:NSPM;SFS:(7070007)(98901004);DIR:OUT;SFP:1901;SCL:1;SRVR:HE1EUR02HT140;H:DB6P189MB0565.EURP189.PROD.OUTLOOK.COM;FPR:;SPF:None;LANG:; x-microsoft-antispam-message-info: 19jvltKE+U7sQX7gWF0zxdfp0tP3woLj9ZoBdh2VV+Bv5zVzJGio3uuCdAZ9G5KpUBTYZ0/p/UebCSTclh5YtZoe/wnjhCzJcDh1DYJNMqONs/mpluX6d4aIXEMg7KFgtEa/0Oxu18Uj6+UskQjpEAqlwgPvckXz2aYW5TfhZ3QlgcbmJKhW7aHrx2p3nUmU Content-Type: multipart/alternative; boundary="_000_DB6P189MB056511D5F53969A27C8B6A6A95870DB6P189MB0565EURP_" MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 247709d1-4727-45bb-993a-08d5b104c0f7 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: dd759f05-a917-4aa0-a2f5-4cc35c50e0c8 X-MS-Exchange-CrossTenant-Network-Message-Id: 247709d1-4727-45bb-993a-08d5b104c0f7 X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: dd759f05-a917-4aa0-a2f5-4cc35c50e0c8 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 May 2018 14:47:11.7918 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1EUR02HT140 --_000_DB6P189MB056511D5F53969A27C8B6A6A95870DB6P189MB0565EURP_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Attention Tomcat developers who know how the Coyote bit of Tomcat works=85 = I=92ve got a tricky one for you! :-) We=92re struggling with a puzzling problem where intermittently, calls to a= servlet are delivered back to the client with all of the headers missing e= xcept the ones that Coyote adds. In roughly 9,999 out of 10,000 attempts the response returns correctly (whi= ch includes a number of headers which are added by filters, a body response= which is gzipped by the servlet and a =93Content-Encoding=94 header which = indicates it=92s gzipped, and the correct =93Content-Length=94 header). But, intermittently, approximately 1 time out of 10,000 requests, the respo= nse back to the client contains only 3 headers: =93Date=94, =93Server=94, a= nd a =93Transfer-Encoding=94 header set to =93chunked=94 (even though the r= esponse body is not chunked). The result of this is that the gzipped body i= s presented back to the client with the =93Content-Encoding=94 header missi= ng so it=92s rendered unreadable by the browser, and all of the other heade= rs are missing. (We=92ve determined this by being able to capture a couple = of instances of this issue in Fiddler). Looking at the Coyote source code it would appear that if the =93Content-Le= ngth=94 header isn=92t presented to Coyote from the upstream handling of th= e filters and servlet then it=92ll default to a chunked encoding, which exp= lains why the response is getting chunked! So, basically - the filters run without error, the servlet runs without err= or, but the when the response is received back into Coyote=92s AbstractHttp= 11Processor.process() method, the headers have disappeared! Annoyingly we can=92t find a recreatable test case, but anecdotally it=92s = when the server is heavily loaded, but there are still plenty of pooled wor= kers available, and plenty of memory. This is a very high volume server (pr= obably millions of requests a day) so adding verbose logging within Coyote = isn=92t a realistic option. I don=92t believe it=92s actually possible for the filters or the servlet t= o remove headers even if they want to (they don=92t appear to have any acce= ss to the MimeHeaders of the Coyote =93Response=94 object, and the HttpServ= letResponse object doesn=92t have any methods which allow header removal). The only clue we have is occasionally we see =93INFO: Encountered a non-rec= ycled response and recycled it forcedly.=94 in the logs, but these errors a= re infrequent compared to the number of instances of the problem, and the t= imestamps of these errors don=92t correspond to when users have reported th= e error. My best guess would be somehow the Coyote =93Request=94 object is accidenta= lly accessed by two threads concurrently to service two requests, and one r= equest is calling the Request.recycle() (which is calling headers.recycle()= which removes all of the headers) but I=92m struggling to see how that cou= ld happen. Note we are not using async servlets. I appreciate 7.0.72 isn=92t the very latest and greatest version but I can= =92t find anything in the change log for later 7.x, 8.x, 8.5.x or 9.x relea= ses which indicate this is a known issue or resolved. Any ideas about how to further diagnose this would be very gratefully recei= ved. Otherwise our next step is to blindly update to later versions and jus= t hope it goes away. Thanks, Richard --_000_DB6P189MB056511D5F53969A27C8B6A6A95870DB6P189MB0565EURP_--