quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Graham Dumpleton <grah...@dscpl.com.au>
Subject Re: Mod_python: Memory leak - bug or common feature?
Date Wed, 17 Jan 2007 10:07:43 GMT

On 17/01/2007, at 8:54 PM, export@hope.cz wrote:

>
> Thank you for your reply.
> Now I use
> Apache 2.2.4
> Mod_python 3.3.0.b
> Python 2.3.5
> The results are more or less the same when I used
> Apache 2.0
> and mod_python 3.1.3-1
> Python 2.3.4
>
> Memory increases when I read  (filter.read()) or write( filter.write 
> (s).
> In otherwords, memory is not released after  I wrote data to a file 
> ( before next reading).
>
> Is  my presumption correct that the memory should be released
> during each
> reading/writing in the filter?
> Or do I make a mistake somewhere?

You are giving conflicting information. Previously you said you were  
running
out of resources which suggested runaway memory use. Ie., process size
just kept growing and growing until machine memory exhausted. Is that  
what
is happening or were you more echoing your concerns rather than what was
happening.

What you will probably see after having restarted Apache and then  
doing an
initial upload is that memory use by Apache will grow as it reads in  
more and
more of the file. Once the complete request has been serviced, Apache  
will
continue to use at least as much memory as it allocated by that  
point. You
will not see the Apache process size shrink as once the Apache process
has allocated memory, it notionally does not give it back to the  
operating system
until the process has quit. This doesn't mean it is lost though as it  
is put on
a free list within the Apache process itself and when a subsequent  
upload of
same file for example occurs, it will just use that free memory.  
Thus, process
size will grow on the first request but if you keep doing the same  
request, the
process size should stay reasonably stable.

Note that how much the Apache process grows to initially can depend a  
bit
on how a handler deals with a file upload. If the response handler  
does a
req.read() and then processes it all in one go, it will have to hold  
the whole
file in memory and so if it is a big file it will use a lot. If  
instead the response
handler read in the file in small blocks at a time using req.read 
(blocksize)
and process each block and then discarded it, the memory use by Apache
will be less as it doesn't have to hold the whole file in memory but  
more or
less just what it is processing.

It is hard to tell if you do have a problem with a memory leak or  
whether you
just understand how memory allocation works for processes.

What are you using to monitor memory use? Does memory use keep growing
continually after every request or does it plateau after the initial  
increases
with the firs requests?

Graham

> Thank you for help
>
> Regards,
> Lad.
>
>
> > On 16/01/2007, at 6:54 PM, export@hope.cz wrote:
> >
> > > I have this, very simple input filter
> > >
> > > def inputfilter(filter):
> > >     if filter.req.method != 'POST':
> > >         filter.pass_on()
> > >         return
> > >     filter.req.log_error('first read')
> > >     s = filter.read()
> > >     while s:
> > >         filter.req.log_error('writing (%s)' % len(s))
> > >         filter.write(s)
> > >         #filter.flush()
> > >         s = filter.read()
> > >     if s is None:
> > >         filter.req.log_error('closing')
> > >         filter.close()
> > >
> > > I would like to use an input filter  for uploading large files.
> > > I would expect that after each writing/reading, filter releases
> > > memory.But instead memory
> > > increases during reading/writing  and no memory is released  
> after  each
> > > reading/writing. Is it a common feature that the filter during
> > > reading/writing does  NOT release memory and memory increases  
> until the
> > > whole file is uploaded? Or is  my presumption correct that the  
> memory
> > > should be released  during each reading/writing in the filter?  
> Thank you
> > > for help and reply.
> >
> > Since I don't recollect you saying what version of mod_python you  
> are
> > using, I would suggest you upgrade to mod_python 3.3.0b if you  
> aren't
> > using it already. Once you have confirmed that you still have  
> the  problem
> > with that version let us know. Also state which versions of  
> Apache and
> > Python you are using at the same time.
> >
> > For 3.3.0b see:
> >
> >    http://httpd.apache.org/modules/python-download.cgi
> >    http://nicolas.lehuen.com/download/mod_python/
> >
> > The latter has Windows binaries for various Apache/Python combos.
> >
> > Graham


Mime
View raw message