httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dean gaudet <>
Subject Re: BUFF, IOL, Chunking, and Unicode in 2.0 (long)
Date Tue, 02 May 2000 16:09:28 GMT
On Tue, 2 May 2000, Martin Kraemer wrote:

> * iol's sit below BUFF. Therefore, they don't have enough information
>   to know which part of the written byte stream is net client data,
>   and which part is protocol information (chunks, MIME headers for
>   multipart/*).

there's not much stopping you from writing an iol which takes a BUFF * in
its initialiser, and then bcreating a second BUFF, and bpushing your iol.

	/* this is in r->pool rather than r->connection->pool because
	 * we expect to create & destroy this inside request boundaries
	 * and if we stuck it in r->connection->pool the storage wouldn't
	 * be reclaimed earlier enough on pipelined connections.
	 * also, no need for buffering in new_buff because the translation
	 * layer can easily assume lower level BUFF is doing the buffering.
	new_buff = ap_bcreate(r->pool, B_WR);
		ap_utf8_to_ebcdic(r->pool, r->connection->client));
	r->connection->client = new_buff;

main problem is that the new_buff only works for writing, and you
potentially need a separate conversion layer for reading from the

shouldn't be too hard to split up r->connection->client into a read and
write half.

think of iol as the equivalent of the low level read/write, and BUFF
as the equivalent of FILE *.  there's a reason for both layers in
the interface.

> * iol's don't allow simplification of today's chunking code. It is
>   spread thruout buff.c and there's a very hairy balance between
>   efficiency and code correctness. Re-adding (EBCDIC/UTF) conversion,
>   possibly with sup[port for multi byte character sets (MBCS), would
>   make a code nightmare out of it. (buff.c in 1.3 was "almost" a
>   nightmare because we had onlu single byte translations.

as i've said before, i welcome anyone to do it otherwise without adding
network packets, without adding unnecessary byte copies, and without
making it even more complex.  until you've tried it, it's pretty easy
to just say "this is a mess".  once you've tried it i suspect you'll
discover why it is a mess.

that said, i'm still trying to prove to myself that the zero-copy
crud necessary to clean this up can be done in a less complex manner.

> * Putting conversion to a hierarchy level any higher than buff.c is no
>   solution either: for chunks, as well as for multipart headers and
>   buffering boundaries, we need character set translation. Pulling it
>   to a higher level means that a lot of redundant information has to
>   be passed down and up.

huh?  HTTP is in ASCII -- you don't need any conversion -- if a chunking
BUFF below a converting BUFF/iol is writing those things in ascii
it works.  no?  at least that's my understanding of the code in 1.3.

you wouldn't do the extra BUFF layer above until after you've written
the headers into the plain-text BUFF.

i would expect you'd:

	write headers through plain text BUFF
	push conversion BUFF
	run method
	pop conversion BUFF
	pump multipart header
	push conversion BUFF
	pop conversion BUFF

> In my understanding, we need a layered buff.c (which I number from 0
> upwards):

you've already got it :)

>     | Caller: using ap_bputs() | or ap_bgets/apbwrite etc.
>     +--------------------------+
>     | Layer 3: Buffered I/O    | gets/puts/getchar functionality
>     +--------------------------+
>     | Layer 2: Code Conversion | (optional conversions)
>     +--------------------------+
>     | Layer 1: Chunking Layer  | Adding chunks on writes
>     +--------------------------+
>     | Layer 0: Binary Output   | bwrite/bwritev, error handling
>     +--------------------------+
>     | iol_* functionality      | basic i/o
>     +--------------------------+
>     | apr_* functionality      |

there are two cases you need to consider:

chunking and a partial write occurs -- you need to keep track of how much
of the chunk header/trailer was written so that on the next loop around
(which happens in the application at the top) you continue where you
left off.

and more importantly at the moment, and easier to grasp -- consider what
happens when you've got a pipelined connection.  a dozen requests come
in from the client, and apache-1.3 will send back the minimal number
of packets.  2.0-current still needs fixing in this area (specifically
saferead needs to be implemented).

for example, suppose the client sends one packet:

	GET /images/a.gif HTTP/1.1
	Host: foo

	GET /images/b.gif HTTP/1.1
	Host: foo

suppose that a.gif and b.gif are small 200 byte files.

apache-1.3 sends back one response packet:

	HTTP/1.1 OK

	a.gif body
	HTTP/1.1 OK

	b.gif body

consider what happens with your proposal.  in between each of those
requests you remove the buffering -- which means you have to flush a
packet boundary.  so your proposal generates two network packets.

like i've said before on this topic -- if all unixes had TCP_CORK,
it'd be a breeze.  but only linux has TCP_CORK.

you pretty much require a layer of buffering right above the iol which
talks to the network.

and once you put that layer of buffering there, you might as well merge
chunking into it, because chunking needs buffering as well (specifically
for the async i/o case).

and then you either have to double-buffer, or you can only stack
non-buffered layers above it.  fortunately, character-set conversion
should be doable without any buffering.

*or* you implement a zero-copy library, and hope it all works out in
the end.


View raw message