From users-return-265005-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Tue Jul 3 10:31:50 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id AA95E180632 for ; Tue, 3 Jul 2018 10:31:49 +0200 (CEST) Received: (qmail 30288 invoked by uid 500); 3 Jul 2018 08:31:48 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 30277 invoked by uid 99); 3 Jul 2018 08:31:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jul 2018 08:31:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 841D1D173D for ; Tue, 3 Jul 2018 08:31:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.971 X-Spam-Level: * X-Spam-Status: No, score=1.971 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=pojo-pl.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id fZ1zfQytR7KF for ; Tue, 3 Jul 2018 08:31:45 +0000 (UTC) Received: from mail-yb0-f195.google.com (mail-yb0-f195.google.com [209.85.213.195]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 0ED195F36B for ; Tue, 3 Jul 2018 08:31:45 +0000 (UTC) Received: by mail-yb0-f195.google.com with SMTP id k127-v6so407690ybk.6 for ; Tue, 03 Jul 2018 01:31:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pojo-pl.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=SfZOH4tEKRiGtSNl6m6ftwauoaLHXKf4rVp7J03zpD8=; b=Kzz/h4UBzt0Q0BH4QiTRUyq9CnvHqWinERao9rWCcYpU22qbbnWlKh1y/hrAU1hpYt UkDwrZbXXP+OhL6ltzVsaD7k/pmxkPMnXczrE6oDE8B9eCROpWXJfOf7Y5xVSS0ruM/b GVqQb6D3lJpgtm5j3qKVCLTbUFY/TIK8XNLQ+WkPg0fFwdhvA5LphOVzV7M45v+pGYu2 hJkeJOyMPCAg79zPuj55zs3MAfa9aoZZsS1jjksOfh1auuaSnQFj5GacxYn0SjQIvYHM ThfsYgCDgiI1BDBKm6hWOmRtpkt6nminMbCWVzV1ohcBKc/ei0s1UPxNOAd6lqtH4Td4 js/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=SfZOH4tEKRiGtSNl6m6ftwauoaLHXKf4rVp7J03zpD8=; b=ADd+IzBrOrRf62gnUU7I2IHL8nFAetuxg1BzoMlDWuhpCc79eQW5ffGN90cgMLxTv2 tyP4j5hdhSoXQ8T/X28MDmTH0HL3rb0G/knzaefRq9J/TTf3pQVVvgr9lnqEhYi7sQZA 3VMWMXueX866dIOSWniF7sa4f8woiWn5IEn6cWsV025wnTOU56ACGB7OJ1AYmhZvYVyq +tjPZAqARScvq4hf62pQ9dnRPa/6jR0TzWokhvAuwKpdCnBvXNwq0K3AB1OCXWzcr8iO UCs+ltFykUm2dloVyUEsxhFCpfrZur0UNsWZjSU4ZJZW6JF5yijQdDG5llAe27SawZrj uH7A== X-Gm-Message-State: APt69E1ZCVUZVTjgJcKSEuzTw1+RWY6ltd8arWW5oUTWUVKQYJxZIwOp Jc6SVJe86B1J3kfFXxIBmYougDkeDz9n68uMC0Tfww== X-Google-Smtp-Source: AAOMgpfCKR08n6LNRNmYnzgGOxW1N3cUvn7rINPbBzmRo33+5EP985R8MquNPO7y6xYkN2ygE5kspmLXcm6BI1KL9fg= X-Received: by 2002:a25:99c6:: with SMTP id q6-v6mr3990504ybo.137.1530606703797; Tue, 03 Jul 2018 01:31:43 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a81:8787:0:0:0:0:0 with HTTP; Tue, 3 Jul 2018 01:31:43 -0700 (PDT) X-Originating-IP: [89.74.215.144] In-Reply-To: References: From: =?UTF-8?Q?Piotr_Jo=C5=84ski?= Date: Tue, 3 Jul 2018 10:31:43 +0200 Message-ID: Subject: Re: Bug in org.apache.tomcat.util.http.fileupload.MultipartStream.ItemInputStream#read(byte[], int, int) To: Tomcat Users List Content-Type: multipart/alternative; boundary="000000000000defabd0570142468" --000000000000defabd0570142468 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable BTW instead of ******************* you have to use some real values. of course I have just obfuscated values and left headers because they can matter. On 3 July 2018 at 10:30, Piotr Jo=C5=84ski wrote: > Hi, thanks for reply! > Here is sample snippet that i use to reproduce it: > ------------------------------------------------------------ > ---------------------------------------------------------- > #!/usr/bin/env bash > echo; > echo; > echo; > echo; > echo; > echo; > curl 'http://localhost:8083/************' \ > -H 'Origin: ***************' \ > -H 'X-Client-Version: 6.5.2-rc0' \ > -H 'Authorization: ********************' \ > -H 'Content-Type: multipart/form-data;boundary=3D"=3D=3D=3D=3D=3D=3D= boundary=3D=3D=3D=3D=3D=3D"' > \ > -H 'Accept: application/json, text/plain, */*' \ > -H 'Referer: *****************' \ > -H 'X-Client-ID: *************' \ > -H 'X-Request-Id: 16613F22FBF46FA9BA441125219C2B13' \ > -H 'X-B3-TraceId: 16613f22fbf46fb9ba441125219c3b03' \ > -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 > (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' \ > -H 'DNT: 1' \ > --data-binary $'--=3D=3D=3D=3D=3D=3Dboundary=3D=3D=3D=3D=3D=3D\nCont= ent-disposition: > form-data; name=3D"some-name"; filename=3D"some-file"\r\nContent-Type: > application/http\r\nContent-ID: req0\r\n\r\nPUT /api/***** > HTTP/1.1\nContent-Type: application/json\n\n{******** > json-here********}\r\n\r\n--=3D=3D=3D=3D=3D=3Dboundary=3D=3D=3D=3D=3D=3D-= -' \ > --compressed > echo; > echo; > echo; > echo; > echo; > echo; > ------------------------------------------------------------ > ---------------------------------------------------------- > > 1. Originally I have sent multipart/mixed request, but tomcat does not > support it at all! Tomcat supports only form-data, however in RFC, the ve= ry > first example is about mixed: https://www.w3.org/Protocols/rfc1341/7_2_ > Multipart.html > 2. I had to add Content-disposition: form-data; name=3D"some-name"; > filename=3D"some-file" because without that it was failing due to missing > field name and filename. > > I hope you manage to reproduce it. The simplest way is to use srping boot > app with tomcat, which is chosen by default and send that request. Good > luck! > > Thanks ! > > On 3 July 2018 at 10:24, R=C3=A9my Maucherat wrote: > >> On Mon, Jul 2, 2018 at 5:14 PM Piotr Jo=C5=84ski wrot= e: >> >> > Hi, of course I use it together with multipart request. >> > I have spring boot 2 + zuul on tomcat 8.5.31. And I cannot proxy traff= ic >> > with multipart request due to that error. >> > I know that available return the right number of bytes but later you >> have >> > method makeAvailable() which tries to read more than allowed! Some >> greedy >> > developer wrote that :) >> > Please check unit tests which I added. The should explain you >> everything. >> > >> >> Hum, ok, but there's no multipart boundary in your test. Do you have an >> example of multipart content that fails to be processed correctly ? >> makeAvailable will never try to read beyond the boundary position. >> Personally, I don't see it as a problem if there are exceptions trying t= o >> process non multipart content, but this sort of cleaner error handling i= s >> often added. >> >> >> > >> > Also here is example issue: >> > >> > https://stackoverflow.com/questions/3263809/apache-commons- >> file-upload-stream-ended-unexpectedly >> > I saw a lot of them -- all unresolved. >> > >> >> Maybe, but this one is about Tomcat 6, quite a while ago. >> >> Fileupload is a separate component. Of course, we do fix and update it a= s >> needed. >> >> R=C3=A9my >> >> >> > >> > On 2 July 2018 at 16:58, R=C3=A9my Maucherat wrote: >> > >> > > On Mon, Jul 2, 2018 at 4:35 PM Piotr Jo=C5=84ski = wrote: >> > > >> > > > Java: openjdk version "1.8.0_163" >> > > > OpenJDK Runtime Environment (Zulu 8.28.0.1-linux64) (build >> > 1.8.0_163-b01) >> > > > OpenJDK 64-Bit Server VM (Zulu 8.28.0.1-linux64) (build 25.163-b01= , >> > mixed >> > > > mode) >> > > > >> > > > OS: Ubuntu 18.04 Linux local 4.15.0-23-generic #25-Ubuntu SMP Wed >> May >> > 23 >> > > > 18:02:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >> > > > >> > > > >> > > > The problem is that >> > > > >> > > > org.apache.tomcat.util.http.fileupload.MultipartStream. >> > > ItemInputStream#read(byte[], >> > > > int, int) method does not update pos field after reading from >> buffer / >> > > > stream. >> > > > >> > > >> > > pos is set to the next separator which (IMO) should work fine since >> > > available returns the right amount of bytes which are allowed to be >> read. >> > > Doesn't this work for you ? This is not a generic utility class, it'= s >> > > supposed to be used with multipart content, is it at least what you >> are >> > > doing ? >> > > >> > > R=C3=A9my >> > > >> > > >> > > > >> > > > Unfortunately I cannot provide full example as this is private >> project. >> > > > >> > > > Here are sample unit tests. First reproduces the error and second >> use >> > > > reflection to set proper field value to simulate proper behaviour: >> > > > >> > > > package org.apache.tomcat.util.http.fileupload; >> > > > >> > > > import org.junit.jupiter.api.Test; >> > > > >> > > > import java.io.ByteArrayInputStream; >> > > > import java.io.IOException; >> > > > import java.lang.reflect.Field; >> > > > >> > > > import static org.assertj.core.api.Assertions.assertThat; >> > > > >> > > > class ItemInputStreamTest { >> > > > >> > > > @Test >> > > > void Should_Read_Bytes_But_Throws_Exception() throws >> IOException { >> > > > // given >> > > > byte[] bytes =3D new byte[]{1, 2, 3}; >> > > > final ByteArrayInputStream inputStream =3D new >> > > > ByteArrayInputStream(bytes); >> > > > final MultipartStream.ProgressNotifier progressNotifier = =3D >> new >> > > > MultipartStream.ProgressNotifier(null, 1111); >> > > > final MultipartStream multipartStream =3D new >> > > > MultipartStream(inputStream, >> > > > >> > > bytes, >> > > > >> > > > progressNotifier); >> > > > MultipartStream.ItemInputStream itemInputStream =3D >> > > > multipartStream.new ItemInputStream(); >> > > > >> > > > // when >> > > > byte[] buffer =3D new byte[8196]; >> > > > int result =3D itemInputStream.read(buffer, 0, 8196); >> > > > >> > > > // then >> > > > assertThat(result).isEqualTo(3); >> > > > } >> > > > >> > > > @Test >> > > > void Should_Read_Bytes_Fixed() throws IOException, >> > > > NoSuchFieldException, IllegalAccessException { >> > > > // given >> > > > byte[] bytes =3D new byte[]{1, 2, 3}; >> > > > final ByteArrayInputStream inputStream =3D new >> > > > ByteArrayInputStream(bytes); >> > > > final MultipartStream.ProgressNotifier progressNotifier = =3D >> new >> > > > MultipartStream.ProgressNotifier(null, 1111); >> > > > final MultipartStream multipartStream =3D new >> > > > MultipartStream(inputStream, >> > > > >> > > bytes, >> > > > >> > > > progressNotifier); >> > > > MultipartStream.ItemInputStream itemInputStream =3D >> > > > multipartStream.new ItemInputStream(); >> > > > >> > > > Field pos =3D itemInputStream.getClass() >> > > > .getDeclaredField("pos"); >> > > > pos.setAccessible(true); >> > > > pos.set(itemInputStream, 3); >> > > > >> > > > // when >> > > > byte[] buffer =3D new byte[8196]; >> > > > int result =3D itemInputStream.read(buffer, 0, 8196); >> > > > >> > > > // then >> > > > assertThat(result).isEqualTo(3); >> > > > } >> > > > } >> > > > >> > > >> > >> > > --000000000000defabd0570142468--