hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Johnson <e...@tibco.com>
Subject Re: streaming request body
Date Tue, 24 Feb 2004 15:14:57 GMT
John Keyes wrote:

>> For (a), Oleg's response is correct. You might easily be confused, in 
>> the sense that HttpClient's API inverts the control. It is not that 
>> you write to an "OutputStream" to send your data, it is that you 
>> provide HttpClient with an "InputStream", and it reads that stream 
>> and sends the data. HttpClient is designed to accomodate your 
>> concern, and if your configuration is correct (as per the examples), 
>> it will not buffer the entire contents of your InputStream, but 
>> rather read it and send it in small chunks. As another post points 
>> you, you may still have to buffer what you're sending to *disk*, but 
>> not to memory.
>
>
> So you think buffering all requests to disk to support streaming is an 
> acceptable solution? If I am dealing with XXX,000 of requests that 
> sure as hell would suck with all the disk I/O going on. Does this not 
> suggest that there is a problem with the architecture?

Many on the mailing list are aware of architectural limitations in the 
2.0 design of HttpClient. This was a conscious compromise that we made 
many months ago to live within certain constraints, with the key 
trade-off being a "final" version of the 2.0 implementation sooner. This 
is apparently good choice for you too, in that you've started using it 
actively!

This very issue you raise is on the list of possible tasks to address 
for the 3.0 release, as per someone else's post (see:
http://nagoya.apache.org/eyebrowse/ReadMsg?listName=commons-httpclient-dev@jakarta.apache.org&msgNo=6015
). See bug http://nagoya.apache.org/bugzilla/show_bug.cgi?id=26070 
referred to in that post.

You should read the discussion there, as it also describes an 
implementation approach to get around the specific limitation. If you'd 
like to second the request to get the change in for 3.0, or provide 
additional work-arounds, or add to the discussion, you might look there. 
A patch to address the issue would be welcome, I'm sure.

>
>> As for (b), this is again under your control via 
>> HttpMethod.getResponseBodyAsStream(). As with (a), you can also 
>> invoke HttpClient such that it does cache the entire contents 
>> (HttpMethod.getResponseBodyAsString() ).
>>
>> In both cases, it is possible to get the behavior that you desire.
>
>
> Not it is not. Again think of XXX,000 of requests. 

I have thought of many requests. I still maintain it is possible. Your 
argument may be that it requires more coding on your part for it to work 
well, or that it requires massive disk caching, which could dramatically 
affect performance. I don't disagree.

>
>
>> Connection pooling is only part of the concern. HttpClient supports 
>> HTTP 1.1 persistent connections. It doesn't expose the underlying 
>> socket's InputStream and OutputStream. If it did, it cound not ensure 
>> that persistent connections work properly.
>
>
> I still don't see the problem. The OutputStream and InputStream can be 
> wrapped so there is no loss of control. Why do you think control would 
> be lost?

We're saying the same thing here. I'm saying they're not exposed, and 
you're saying they could be wrapped, thus hiding them. Since they are 
already hidden, your issue would seem to be a problem with *how* they 
are exposed (or not). Again, comments and feedback or a patch for bug 
26070 would be welcome.

-Eric.



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org


Mime
View raw message