hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kalnichevski, Oleg" <oleg.kalnichev...@bearingpoint.com>
Subject RE: streaming request body
Date Tue, 24 Feb 2004 16:34:38 GMT

> come along) and work from there - we need to process around 70,000 
> requests a minute.

But not concurrently, right? So, the memory overhead is (No of concurrent connections) * (buffer
size). Even if you had 1,000 concurrent SOAP requests, with 2K buffer you would still end
up with 2,048 * 1,000 bytes. Allow me to speculate that even a fairly modern PDA would be
able to afford that amount of memory overhead. Am I missing something too?

Oleg

-----Original Message-----
From: John Keyes [mailto:john@integralsource.com]
Sent: Tuesday, February 24, 2004 17:29
To: Commons HttpClient Project
Subject: Re: streaming request body


On 24 Feb 2004, at 16:22, Kalnichevski, Oleg wrote:

>
>> But a *segment* will be held in memory prior to writing to the output
>> stream though.  For XXX,000 requests I think this is an unreasonable
>> memory overhead.
>
> John,
> Just to make sure I understand you correctly, you are saying that your 
> application will be processing XXX,000 requests *concurrently*? What 
> kind of application is it, if I may ask you?

It's a SOAP processor - we just want to stop using the J2SDK connection 
class, hide the connection class behind an API (in case other impls 
come along) and work from there - we need to process around 70,000 
requests a minute.

-John K

>
> Oleg
>
>
> -----Original Message-----
> From: John Keyes [mailto:john@integralsource.com]
> Sent: Tuesday, February 24, 2004 16:59
> To: Commons HttpClient Project
> Subject: Re: streaming request body
>
>
> On 24 Feb 2004, at 14:36, Stefan Dingfelder wrote:
>
>> John Keyes schrieb:
>>
>>>> For (a), Oleg's response is correct. You might easily be confused,
>>>> in  the sense that HttpClient's API inverts the control. It is not
>>>> that  you write to an "OutputStream" to send your data, it is that
>>>> you  provide HttpClient with an "InputStream", and it reads that
>>>> stream and  sends the data. HttpClient is designed to accomodate
>>>> your concern, and  if your configuration is correct (as per the
>>>> examples), it will not  buffer the entire contents of your
>>>> InputStream, but rather read it and  send it in small chunks. As
>>>> another post points you, you may still  have to buffer what you're
>>>> sending to *disk*, but not to memory.
>>>
>>>
>>> So you think buffering all requests to disk to support streaming is
>>> an  acceptable solution?  If I am dealing with XXX,000 of requests
>>> that  sure as hell would suck with all the disk I/O going on.  Does
>>> this not  suggest that there is a problem with the architecture?
>>>
>> I am missing something here from both views. Maybe I am wrong but as I
>> understand it, I can provide any InputStream. And that must not be a
>> file on disk (which I dislike also - except for large files or live
>> streams that cannot be put to memory in total) but can be any object
>> in memory. So in case of sending it there should be no problem..
>> Correct?
>
> Correct.
>
> But a *segment* will be held in memory prior to writing to the output
> stream though.  For XXX,000 requests I think this is an unreasonable
> memory overhead.
>
> I am looking at avoiding using Sun's connection class as it buffers all
> of the content prior to writing to the wire.
>
> -John K
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: 
> commons-httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> commons-httpclient-dev-help@jakarta.apache.org
>
>
> ***********************************************************************
> ****************************
> The information in this email is confidential and may be legally 
> privileged.  Access to this email by anyone other than the intended 
> addressee is unauthorized.  If you are not the intended recipient of 
> this message, any review, disclosure, copying, distribution, 
> retention, or any action taken or omitted to be taken in reliance on 
> it is prohibited and may be unlawful.  If you are not the intended 
> recipient, please reply to or forward a copy of this message to the 
> sender and delete the message, any attachments, and any copies thereof 
> from your system.
> ***********************************************************************
> ****************************
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: 
> commons-httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> commons-httpclient-dev-help@jakarta.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org


***************************************************************************************************
The information in this email is confidential and may be legally privileged.  Access to this
email by anyone other than the intended addressee is unauthorized.  If you are not the intended
recipient of this message, any review, disclosure, copying, distribution, retention, or any
action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. 
If you are not the intended recipient, please reply to or forward a copy of this message to
the sender and delete the message, any attachments, and any copies thereof from your system.
***************************************************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org


Mime
View raw message