perl-modperl mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clinton Gormley < >
Subject Re: Reducing memory usage using fewer cgi programs
Date Sat, 25 Oct 2008 11:34:05 GMT
On Sat, 2008-10-25 at 05:59 +0200, Michael Lackhoff wrote:
> On 24.10.2008 15:03 Michael Peters wrote:
> > This is only true if those structures were created during run time and go out of
scope at run time. 
> > If they are generated at compile time or attached to global variables or package
level variables, 
> > they will not be re-used by Perl.

> Wait a minute, I would like to do exactly that: use a config module in
> that loads some massive config hashes in the hope that the
> memory they use will be shared:

I think there was some confusion here about what was being said.  

Michael Peters' comment about memory reuse was saying that:
- if at runtime, you load a large memory structure
- then let those variables go out of scope
- then that memory will be come available for reuse by Perl

- if you load that same large memory structure at server startup
- and the variables don't go out of scope, 
- then that memory stays used

> package MyConfig;
> our $aHugeConfigHash = load_data_from_config_file();
> (well sort of, it is actually wrapped in an accessor but that gets its
> data from the package variable)
> Are you saying, I cannot share the memory this way? And if so, is there
> an alternative?

I do exactly this - a lot of my website is config driven. At startup, I
load a large data structure (7,000 lines of YAML ~~ 700kB in memory),
that I can access from any module.  

This data structure is considered by my app to be read-only, but not

        Plug: You may want to look at Config::Merge (my module for
        loading a tree of YAML / JSON / XML / Perl / INI /
        Config::General files into a single config hash:

Similarly, I preload almost all the modules that I will use at compile
time. I exclude certain seldom used large modules, such as PDF::API2,
which is require'd when needed.

This makes forking a new child very fast, thanks to copy-on-write in

Unfortunately, in Perl, there is no way to make sure that
data-that-will-never-change and compiled code is stored on a separate
page from data-that-will-change. So, it is likely that, while your
config data starts out completely shared, over time, it will become less

As Stas Bekman said in his Improving mod_perl Sites' Performance

don't aim to make MaxRequestsPerChild = 10000, as the amount of memory
each child consumes will just increase over time, while creating a new
child is cheap, thanks to your preloading.

You may want to look at Linux::Smaps

and at (a script which uses Linux::Smaps to print out memory
usage of a process)

to get some idea of just how shared the memory in your httpd processes

Some examples of output from my live site: (compare shared vs
private clean + private dirty)

    VMSIZE:     135036 kb
    RSS:         45784 kb total
                 36616 kb shared
                  1012 kb private clean
                  8156 kb private dirty

    VMSIZE:     143780 kb
    RSS:         53140 kb total
                 33912 kb shared
                    12 kb private clean
                 19216 kb private dirty

    VMSIZE:     138052 kb
    RSS:         47172 kb total
                 36272 kb shared
                     0 kb private clean
                 10900 kb private dirty



View raw message