perl-modperl mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Moseley <>
Subject Re: mod_perl and Transfer-Encoding: chunked
Date Wed, 03 Jul 2013 18:44:31 GMT
Hi Jim,

This is the Transfer-Encoding: chunked I was writing about:

On Wed, Jul 3, 2013 at 11:34 AM, Jim Schueler <>wrote:

> I played around with chunking recently in the context of media streaming:
> The client is only requesting a "chunk" of data.  "Chunking" is how media
> players perform a "seek".  It was originally implemented for FTP transfers:
>  E.g, to transfer a large file in (say 10K) chunks.  In the case that you
> describe below, if no Content-Length is specified, that indicates "send the
> remainder".
> From what I know, a "chunk" request header is used this way to specify the
> server response.  It does not reflect anything about the data included in
> the body of the request.  So first, I would ask if you're confused about
> this request information.
> Hypothetically, some browsers might try to upload large files in small
> chunks and the "chunk" header might reflect a push transfer.  I don't know
> if "chunk" is ever used for this purpose.  But it would require the
> following characteristics:
>   1.  The browser would need to originally inquire if the server is
>       capable of this type of request.
>   2.  Each chunk of data will arrive in a separate and independent HTTP
>       request.  Not necessarily in the order they were sent.
>   3.  Two or more requests may be handled by separate processes
>       simultaneously that can't be written into a single destination.
>   4.  Somehow the server needs to request a resend if a chunk is missing.
>       Solving this problem requires an imaginitive use of HTTP.
> Sounds messy.  But might be appropriate for 100M+ sized uploads.  This
> *may* reflect your situation.  Can you please confirm?
> For a single process, the incoming content-length is unnecessary. Buffered
> I/O automatically knows when transmission is complete.  The read() argument
> is the buffer size, not the content length.  Whether you spool the buffer
> to disk or simply enlarge the buffer should be determined by your hardware
> capabilities.  This is standard IO behavior that has nothing to do with
> HTTP chunk.  Without a "Content-Length" header, after looping your read()
> operation, determine the length of the aggregate data and pass that to
> Catalyst.
> But if you're confident that the complete request spans several smaller
> (chunked) HTTP requests, you'll need to address all the problems I've
> described above, plus the problem of re-assembling the whole thing for
> Catalyst.  I don't know anything about Plack, maybe it can perform all this
> required magic.
> Otherwise, if the whole purpose of the Plack temporary file is to pass a
> file handle, you can pass a buffer as a file handle.  Used to be
> IO::String, but now that functionality is built into the core.
> By your last paragraph, I'm really lost.  Since you're already passing the
> request as a file handle, I'm guessing that Catalyst creates the
> tempororary file for the *response* body.  Can you please clarify?  Also,
> what do you mean by "de-chunking"?  Is that the same think as re-assembling?
> Wish I could give a better answer.  Let me know if this helps.
> -Jim
> On Tue, 2 Jul 2013, Bill Moseley wrote:
>  For requests that are chunked (Transfer-Encoding: chunked and no
>> Content-Length header) calling $r->read returns unchunked data from the
>> socket.
>> That's indeed handy.  Is that mod_perl doing that un-chunking or is it
>> Apache?
>> But, it leads to some questions.
>> First, if $r->read reads unchunked data then why is there a
>> Transfer-Encoding header saying that the content is chunked?   Shouldn't
>> that header be removed?   How does one know if the content is chunked or
>> not, otherwise?
>> Second, if there's no Content-Length header then how does one know how
>> much
>> data to read using $r->read?
>> One answer is until $r->read returns zero bytes, of course.  But, is
>> that guaranteed to always be the case, even for, say, pipelined requests?
>> My guess is yes because whatever is de-chunking the request knows to stop
>> after reading the last chunk, trailer and empty line.   Can
>> anyone elaborate
>> on how Apache/mod_perl is doing this?
>> Perhaps I'm approaching this incorrectly, but this is all a bit untidy.
>> I'm using Catalyst and Catalyst needs a Content-Length.  So, I have a
>> Plack
>> Middleware component that creates a temporary file writing the buffer from
>> $r->read( my $buffer, 64 * 1024 ) until that returns zero bytes.  I pass
>> this file handle onto Catalyst.
>> Then, for some content-types, Catalyst (via HTTP::Body) writes the body to
>> another temp file.    I don't know how Apache/mod_perl does its
>> de-chunking,
>> but I can call $r->read with a huge buffer length and Apache returns that.
>>  So, maybe Apache is buffering to disk, too.
>> In other words, for each tiny chunked JSON POST or PUT I'm creating two
>> (or
>> three?) temp files which doesn't seem ideal.
>> --
>> Bill Moseley

Bill Moseley

View raw message