Mailing-List: contact modules-dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: modules-dev@httpd.apache.org
Received-SPF: pass (herse.apache.org: domain of mcqueenorama@gmail.com
 designates 66.249.92.171 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=d6nodw1UWBO51e7gFI5Sn4JBsVLVAf31WUsrsa93H+4EfU0uu0H6ZqXLg3qz2LuZuiq0nRDASNWhNzIWhnh+/Ewi8WfPjxcmPoC9NwsBwDLfjj8A1c9jjgZ3W3guLID9QP+Wa1uQC5rZvAlUuZ0R4pxTazzDTTavEEc5zcvfuGI=
Message-ID: <5b3fa8f0611130618g33de133dv40c5f49cec8c2f30@mail.gmail.com>
Date: Mon, 13 Nov 2006 06:18:37 -0800
From: "Brian McQueen" <mcqueenorama@gmail.com>
To: modules-dev@httpd.apache.org
Subject: Re: Huge memory consumption while storing the entire response
Cc: "Sai Jai Ganesh Gurubaran" <sgurubar@cordys.com>
In-Reply-To: <318C84E390AAFD4E9BDADA9E3639067405424E@ex-ind-u0.vanenburg.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <318C84E390AAFD4E9BDADA9E3639067405424E@ex-ind-u0.vanenburg.com>

Sorry, but I just noticed that the messages to which I referred you
actually went to the apreq mailing list:

<apreq-dev@httpd.apache.org>

If you can't find them I'll send them to you directly.

On 11/13/06, Sai Jai Ganesh Gurubaran <sgurubar@cordys.com> wrote:
> Hi All,
>         We were developing a content manager module on Apache 2.0.
> For this we have the module hooked to the out filter of Apache (and
> there are no other filters).
>
> The base code was taken from Apache 2.0 book (conversion of text to
> upper case).
>
> We notice that the memory consumption is very high per httpd and
> eventually Apache has to be restarted within 25 mins.
>
> On analysis, we find that the main problem is in the way we store the
> complete response for further analysis.
>
> Here is the main code that we use:
>
>
> [CODE]
>
>
> //----------------------------------------------------------------------
> -------
> /*
>  *      Checks if the input bucket carries text/html chunks of
> information.
>  *  For other than text/html chunks, this filter hands over the same
>  *      to the next filter in chain.
>  *      For text/html, this breaks the pipe line, accumulates the full
> response
>  *
>  */
> static apr_status_t testOutFilter_run(ap_filter_t *f,
>                                         apr_bucket_brigade *pbbIn)
> {
>     request_rec* r = f->r;
>     conn_rec* c = r->connection;
>     apr_bucket* pbktIn;
>     apr_bucket_brigade* pbbOut;
>     int flag_EOS = FALSE;
>
>     // To take into account all the MIME Types related to text
>     if ( (r->content_type != NULL) &&
> (strstr(r->content_type,"text")==NULL))
>     {
>                 /*
>                  * For MIME types other than text type send bucket
> brigade as received
>                  */
>
>                 return ap_pass_brigade(f->next,pbbIn);
>     }
>     else
>     {
>                 if (f->ctx)
>                 {
>                         pbbOut = (apr_bucket_brigade *) f->ctx;
>                 }
>                 else
>                 {
>                         pbbOut=apr_brigade_create(r->pool,
> c->bucket_alloc);
>                         f->ctx = pbbOut;
>                         ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r,
>                                         "First BB of :%s",r->uri);
>                 }
>
>                 APR_BRIGADE_FOREACH(pbktIn,pbbIn)
>                 {
>                         apr_bucket *pbktOut;
>
>                         if(APR_BUCKET_IS_EOS(pbktIn))
>                         {
>                                 apr_bucket
> *pbktEOS=apr_bucket_eos_create(c->bucket_alloc);
>                                 APR_BRIGADE_INSERT_TAIL(pbbOut,pbktEOS);
>                                 flag_EOS = TRUE;
>                                 continue;
>                         }
>                         //Copy the in-buckets to the out-buckets
>                         apr_bucket_copy(pbktIn,&pbktOut);
>                         APR_BRIGADE_INSERT_TAIL(pbbOut,pbktOut);
>                 }
>
>                 /* To get the entire content to a pointer to char */
>                 if(flag_EOS == TRUE)
>                 {
>                         // Custom Code goes here
>                         // ...........
>                         // ...........
>                         // ...........
>
>                         return ap_pass_brigade(f->next,pbbOut);
>
>                 }
>                 else
>                 {
>                         return OK;
>                 }
>         }
> }
>
>
> Here is some statistics we gathered:
> 1. Module with the custom code - Apache mem growth is rapid and 25 mins
> we need to restart
> 2. Module without the custom code - Apache mem growth is still rapid and
> 25-30 mins max before we restart
> 3. Apache without the response gathering part - stable for long hours
> (per httpd mem consumption is 44 MB)
>
> Can anyone tell us the best way to gather the entire response without
> consuming all the memory?
>
>
> Any help will be greatly appreciated.
> Ganesh
>
>
> ***********************************************************************
> The information in this message is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this
> message by anyone else is unauthorized. If you are not the
> intended recipient, any disclosure, copying, or distribution of the
> message, or any action or omission taken by you in reliance on
> it is prohibited and may be unlawful. Please immediately contact
> the sender if you have received this message in error. This email
> does not constitute any commitment from Cordys Holding BV or
> any of its subsidiaries except when expressly agreed in a written
> agreement between the intended recipient and
> Cordys Holding BV or its subsidiaries.
> ***********************************************************************
>
>
>