perl-modperl mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clinton Gormley <cl...@traveljury.com >
Subject Re: Reducing memory usage using fewer cgi programs
Date Sat, 25 Oct 2008 11:34:05 GMT
On Sat, 2008-10-25 at 05:59 +0200, Michael Lackhoff wrote:
> On 24.10.2008 15:03 Michael Peters wrote:
> 
> > This is only true if those structures were created during run time and go out of
scope at run time. 
> > If they are generated at compile time or attached to global variables or package
level variables, 
> > they will not be re-used by Perl.

> Wait a minute, I would like to do exactly that: use a config module in
> startup.pl that loads some massive config hashes in the hope that the
> memory they use will be shared:

I think there was some confusion here about what was being said.  

Michael Peters' comment about memory reuse was saying that:
- if at runtime, you load a large memory structure
- then let those variables go out of scope
- then that memory will be come available for reuse by Perl

but 
- if you load that same large memory structure at server startup
- and the variables don't go out of scope, 
- then that memory stays used

> 
> package MyConfig;
> our $aHugeConfigHash = load_data_from_config_file();
> 
> (well sort of, it is actually wrapped in an accessor but that gets its
> data from the package variable)
> 
> Are you saying, I cannot share the memory this way? And if so, is there
> an alternative?

I do exactly this - a lot of my website is config driven. At startup, I
load a large data structure (7,000 lines of YAML ~~ 700kB in memory),
that I can access from any module.  

This data structure is considered by my app to be read-only, but not
enforced.

        Plug: You may want to look at Config::Merge (my module for
        loading a tree of YAML / JSON / XML / Perl / INI /
        Config::General files into a single config hash:
        
        http://search.cpan.org/~drtech/Config-Merge-1.00/Merge.pm

Similarly, I preload almost all the modules that I will use at compile
time. I exclude certain seldom used large modules, such as PDF::API2,
which is require'd when needed.

This makes forking a new child very fast, thanks to copy-on-write in
linux.

Unfortunately, in Perl, there is no way to make sure that
data-that-will-never-change and compiled code is stored on a separate
page from data-that-will-change. So, it is likely that, while your
config data starts out completely shared, over time, it will become less
so.

As Stas Bekman said in his Improving mod_perl Sites' Performance
article:

   http://www.perl.com/pub/a/2002/07/30/mod_perl.html

don't aim to make MaxRequestsPerChild = 10000, as the amount of memory
each child consumes will just increase over time, while creating a new
child is cheap, thanks to your preloading.

You may want to look at Linux::Smaps

   http://search.cpan.org/author/OPI/Linux-Smaps-0.06/lib/Linux/Smaps.pm

and at smem.pl (a script which uses Linux::Smaps to print out memory
usage of a process)

   http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html

to get some idea of just how shared the memory in your httpd processes
are.


Some examples of smem.pl output from my live site: (compare shared vs
private clean + private dirty)

PARENT PROCESS:
---------------
    VMSIZE:     135036 kb
    RSS:         45784 kb total
                 36616 kb shared
                  1012 kb private clean
                  8156 kb private dirty

OLDEST CHILD:
-------------
    VMSIZE:     143780 kb
    RSS:         53140 kb total
                 33912 kb shared
                    12 kb private clean
                 19216 kb private dirty

YOUNGEST CHILD:
---------------
    VMSIZE:     138052 kb
    RSS:         47172 kb total
                 36272 kb shared
                     0 kb private clean
                 10900 kb private dirty



hth

Clint



Mime
View raw message