Mailing-List: contact users-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@httpd.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Luke Meyer <lmeyer@vmware.com>
To: "users@httpd.apache.org" <users@httpd.apache.org>
Date: Fri, 20 Jan 2012 12:10:28 -0800
Thread-Topic: script to reproduce gzip+chunked bug
Thread-Index: AczXr425h3f0HhH0TFOVw7AdHsJk3Q==
Message-ID: 
 <03E840D17E263A48A5766AD576E0423A03DA36CE10@exch-mbx-111.vmware.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: multipart/mixed;
	boundary="_002_03E840D17E263A48A5766AD576E0423A03DA36CE10exchmbx111vmw_"
MIME-Version: 1.0
Subject: [users@httpd] script to reproduce gzip+chunked bug

--_002_03E840D17E263A48A5766AD576E0423A03DA36CE10exchmbx111vmw_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I've been tracking down an elusive problem that affects POST requests made =
to httpd with the body gzipped and chunked, deflated with mod_deflate and p=
assed on to a backend. I found that most of the time it worked; but occasio=
nally (sending exactly the same thing multiple times) the contents wouldn't=
 arrive at the backend intact. I think it has to do with the chunking, whic=
h varies depending on networking.

A bit of an edge case, obviously, but I think I have the tool to reproduce =
it. I've created the attached script, which will gzip a test file and send =
it with any specific chunk sizes I want. Can I get some feedback on whether=
 my implementation is correct? I ask because I seem to have uncovered a com=
pletely different bug along the way: if the last chunk is 8 bytes or smalle=
r, httpd returns 400 (doesn't understand the request). Wireshark seems to d=
e-chunk and un-zip the packets it captures just fine, and as long as the ch=
unk size isn't too small, so does httpd. So I think it's right, but I want =
to know if I'm doing something stupid before I file it as a bug.

Script operation is pretty simple; once you have perl and IO::Socket::INET =
and Compress::Zlib in place, just run it against a server like so:

./chunk-zip-test.pl -server=3Dlocalhost -port=3D80 -url=3D/ -file=3D/usr/sh=
are/dict/words -size=3D40
(or just let the script use the defaults you see at the top)

I've only tested it on Linux so far. Could you have a look and let me know?

--_002_03E840D17E263A48A5766AD576E0423A03DA36CE10exchmbx111vmw_
Content-Type: application/octet-stream; name="chunk-zip-test.pl"
Content-Description: chunk-zip-test.pl
Content-Disposition: attachment; filename="chunk-zip-test.pl"; size=3372;
	creation-date="Fri, 20 Jan 2012 11:57:19 GMT";
	modification-date="Fri, 20 Jan 2012 09:33:09 GMT"
Content-Transfer-Encoding: base64

IyEvdXNyL2Jpbi9wZXJsIC13cwojIEV4YW1wbGUgdXNhZ2U6CiMgLi9jaHVuay16aXAtdGVzdC5w
bCAtcG9ydD04MCAtc2l6ZT00MAoKdXNlIHN0cmljdDsKdXNlIHZhcnMgcXcoICRzZXJ2ZXIgJHBv
cnQgJHVybCAkZmlsZSAkYWxnbyAkc2l6ZSAkbXJlcSApOwp1c2UgSU86OlNvY2tldDo6SU5FVDsK
dXNlIENvbXByZXNzOjpabGliOwoKIyB0aGVzZSBwYXJhbXMgY2FuIGJlIHNwZWNpZmllZCBhdCBj
b21tYW5kIGxpbmU7IGRlZmF1bHRzIGFyZSBoZXJlCgkkc2VydmVyIHx8PSAibG9jYWxob3N0IjsK
CSRwb3J0IHx8PSAiODAiOwoJJHVybCAgfHw9ICIvY2h1bmt0ZXN0L2luZGV4LmpzcCI7CgkkZmls
ZSB8fD0gImNodW5rLXRlc3QtZmlsZS50eHQiOwoJJGFsZ28gfHw9ICJkZWZhdWx0IjsgIyBhbGdv
cml0aG0gZm9yIHBpY2tpbmcgY2h1bmsgc2l6ZXMKCSRzaXplIHx8PSAxMDAwOyAjICJzaXplIiBw
YXJhbWV0ZXIgZm9yIGNodW5rIHNpemUgYWxnb3JpdGhtCgkkbXJlcSB8fD0gMjA7ICNtYXggcmVx
dWVzdHMgdG8gdHJ5CgojIGNob29zZSBhbiBhbGdvcml0aG0gZm9yIHNlbmRpbmcgY2h1bmsgc2l6
ZXMKbXkgJGNodW5rX3N0YXRlID0gMDsKbXkgJGNodW5rX3NpemVfYWxnbyA9IHsKCWRlZmF1bHQg
PT4gc3ViIHsgJHNpemUgfSwKCWFsdGR1YiA9PiBzdWIgeyAoJGNodW5rX3N0YXRlKysgJSAyID09
IDApID8gJHNpemUgOiAkc2l6ZSAqIDIgfSwKCXJhbmRvbSA9PiBzdWIgeyAxICsgaW50IHJhbmQg
JHNpemUgfSwKfS0+eyRhbGdvfSBvciBkaWUgIm5vIHN1Y2ggYWxnbyAkYWxnb1xuIjsKCgpteSAk
Q1JMRiA9ICJceDBkXHgwYSI7ICMgZXhwbGljaXQgbGluZSBlbmRpbmcgdXNlZCB3LyBIVFRQCiMg
ZGVmaW5lIHRoZSBIVFRQIHJlcXVlc3QgaGVhZGVyCm15ICRoZWFkZXIgPSBqb2luICRDUkxGLCAo
c3BsaXQgL1xuLywgPDwiSEVBREVSIik7ClBPU1QgJHVybCBIVFRQLzEuMQpIb3N0OiAkc2VydmVy
OiRwb3J0CkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbgpDb25uZWN0aW9uOiBjbG9zZQpDb250ZW50
LUVuY29kaW5nOiBnemlwClRyYW5zZmVyLUVuY29kaW5nOiBjaHVua2VkCkhFQURFUgpjaG9tcCAk
aGVhZGVyOwokaGVhZGVyIC49ICRDUkxGIHggMjsKCiMgcmVhZCBpbiB0aGUgdGVzdCBmaWxlCm9w
ZW4gbXkgJGZoLCAnPCcsICRmaWxlIG9yIGRpZSAiY291bGRuJ3Qgb3BlbiAnJGZpbGUnOiAkIVxu
IjsKYmlubW9kZSAkZmggb3IgZGllICJjb3VsZG4ndCBiaW5tb2RlIGZpbGU6ICQhXG4iOwokZmgt
PnJlYWQobXkgJGNvbnRlbnQsIDEwXzAwMF8wMDApIG9yIGRpZSAiY291bGRuJ3QgcmVhZCAkZmls
ZTogJCFcbiI7Cm15ICRmaWxlX3NpemUgPSBsZW5ndGggJGNvbnRlbnQ7CgojIGRlZmxhdGUgdGhl
IHRlc3QgZmlsZSBmb3Igc3RyZWFtaW5nCm15KCRkZWZsYXRvciwgJHN0YXR1cykgPSBkZWZsYXRl
SW5pdCgtV2luZG93Qml0cyA9PiAzMSk7CiRzdGF0dXMgPT0gWl9PSyBvciBkaWUgImZhaWxlZCB0
byBjcmVhdGUgZGVmbGF0b3I6ICRzdGF0dXMiOwoobXkgJHppcHBlZCwgJHN0YXR1cykgPSAkZGVm
bGF0b3ItPmRlZmxhdGUoJGNvbnRlbnQpOwokc3RhdHVzID09IFpfT0sgb3IgZGllICJjb3VsZG4n
dCBjb21wcmVzczogJCEiOwokemlwcGVkIC49ICRkZWZsYXRvci0+Zmx1c2g7CgpwcmludCAidW56
aXBwZWQ6ICRmaWxlX3NpemVcbiI7CnByaW50ICJ6aXBwZWQ6ICIuIGxlbmd0aCgkemlwcGVkKS4i
XG4iOwoKbXkgJHRyaWVzID0gMDsgIyBudW1iZXIgb2YgdHJpZXMgdG8gcHJvZHVjZSB0aGUgYnVn
ClRSWTogd2hpbGUoJHRyaWVzKysgPCAkbXJlcSkgeyAjIHRyeSB1bnRpbCB3ZSBmYWlsIG9yIGFy
ZSBraWxsZWQKCgkjIGNyZWF0ZSBhIGNvbm5lY3Rpb24gZm9yIHNlbmRpbmcgdGhlIEhUVFAgcmVx
dWVzdAoJbXkgJHNvY2tldCA9IG5ldyBJTzo6U29ja2V0OjpJTkVUKCAKCQkJUHJvdG8gICAgPT4g
J3RjcCcsCgkJCVBlZXJBZGRyID0+ICRzZXJ2ZXIsCgkJCVBlZXJQb3J0ID0+ICRwb3J0LAoJCQlU
aW1lb3V0ICA9PiAxMCwKCQkpIG9yIGRpZSAiY291bGRuJ3Qgb3BlbiBzb2NrZXQ6ICQhXG4iOwoJ
JHNvY2tldC0+YXV0b2ZsdXNoKDEpOwoJJHNvY2tldC0+c2VuZCgkaGVhZGVyKSBvciBkaWUgImVy
cm9yIHNlbmRpbmcgaGVhZGVyIHRvIHNvY2tldDogJCEiOwoKCSMgd3JpdGUgdGhlIGd6aXBwZWQg
Y29udGVudCBpbiBjaHVua3MKCW15ICRzZW50ID0gMDsKCW15ICRsYXN0X2NodW5rX3NpemUgPSAt
MTsKCVJFQUQ6IHdoaWxlKDEpIHsKCQlteSAkY2h1bmtfc2l6ZSA9ICRjaHVua19zaXplX2FsZ28t
PigpOwoKCQkjIGdldCBzb21lIHN0dWZmICh1cCB0byAkY2h1bmtfc2l6ZSkgZnJvbSB0aGUgemlw
cGVkIGNvbnRlbnQuLi4KCQlteSAkZGF0YSA9IHN1YnN0cigkemlwcGVkLCAkc2VudCwgJGNodW5r
X3NpemUpOwoJCSRjaHVua19zaXplID0gbGVuZ3RoICRkYXRhOwoKCQkjIGFuZCBzZW5kIHRoZSBj
aHVuayB0byB0aGUgc2VydmVyCgkJJHNvY2tldC0+c2VuZChzcHJpbnRmICcleCVzJywgJGNodW5r
X3NpemUsICRDUkxGIC4gJGRhdGEgLiAkQ1JMRikKCQkJb3IgZGllICJlcnJvciBzZW5kaW5nIGhl
YWRlciB0byBzb2NrZXQ6ICQhIjsKCQkkc2VudCArPSAkY2h1bmtfc2l6ZTsKCgkJIyBpZiB3ZSBh
cmUgYXQgdGhlIGVuZCwgZmluaXNoIHRoZSB0cmFuc21pc3Npb24uCgkJbGFzdCBSRUFEIGlmICRj
aHVua19zaXplID09IDA7CgkJJGxhc3RfY2h1bmtfc2l6ZSA9ICRjaHVua19zaXplOwoJfQoKCSMg
Y2hlY2sgdGhlIHJlc3BvbnNlIGZvciB0aGUgbWF0Y2hpbmcgWC1TaXplIGhlYWRlciB3aGljaCB0
aGUgSlNQIHNob3VsZCBzZXQKCWRlZmluZWQgJHNvY2tldC0+cmVjdihteSAkcmVzcG9uc2UsIDEw
XzAwMCwgdW5kZWYpIG9yIGRpZSAiZXJyb3Igd2hpbGUgcmVjZWl2aW5nIHJlc3BvbnNlOiAkISI7
Cgkkc29ja2V0LT5jbG9zZTsKCXVubGVzcyggJHJlc3BvbnNlID1+IC9eIFgtU2l6ZTogXHMgKFxk
KykveHNtaSApIHsKCQlwcmludCAiVW5leHBlY3RlZCByZXNwb25zZSBmb3IgcmVxdWVzdCAkdHJp
ZXM6XG4kcmVzcG9uc2UiOwoJCXByaW50ICJMYXN0IGNodW5rIHNpemUgd2FzICRsYXN0X2NodW5r
X3NpemUuXG4iOwoJCWxhc3QgVFJZOwoJfQoJaWYoJDEgIT0gJGZpbGVfc2l6ZSkgewoJCXByaW50
ICJGb3IgcmVxdWVzdCAkdHJpZXM7IHNpemVzIERJRCBOT1QgbWF0Y2guXG4iOwoJCXByaW50ICJG
aWxlIHNpemUgaXMgJGZpbGVfc2l6ZTsgcmVxdWVzdCBzaXplIHdhcyAkMS5cbiI7CgkJcHJpbnQg
Ikxhc3QgY2h1bmsgc2l6ZSB3YXMgJGxhc3RfY2h1bmtfc2l6ZS5cbiI7CgkJbGFzdCBUUlk7Cgl9
CglwcmludCAiQ29tcGxldGVkIHJlcXVlc3QgJHRyaWVzOyBzaXplcyBtYXRjaGVkLlxuIjsKfQoK
ZXhpdCAwOwoK


--_002_03E840D17E263A48A5766AD576E0423A03DA36CE10exchmbx111vmw_
Content-Type: text/plain; charset=us-ascii


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
--_002_03E840D17E263A48A5766AD576E0423A03DA36CE10exchmbx111vmw_--