perl-modperl mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stas Bekman <>
Subject Re: ithreads with modperl
Date Fri, 09 Jan 2004 21:26:39 GMT
Elizabeth Mattijsen wrote:
> At 15:17 -0500 1/9/04, Perrin Harkins wrote:
>> On Fri, 2004-01-09 at 14:52, Stas Bekman wrote:
>>>  We really need more real world benchmarks to make a good judgement. 
>>> It's
>>>  probably quite certain that the performance is going to be worse if 
>>> you spawn
>>>  threads, but don't deploy the benefits available exclusively to threads
>>  > (shared opcode tree, shared vars, etc).
>> That reminds me, does anyone know what happened with the shared opcode
>> tree?  Does it not work, or is it just dwarfed by the size of the
>> non-shared stuff?  The size problems these guys are having seem to point
>> to little or no sharing happening between threads.
> I'm sure you know my PerlMonks article "Things yuu need to know before 
> programming Perl ithreads" ( 
> ).
> I recently ran a little test that showed (at least to Devel::Size) that 
> you have _at least_ about 250Kbyte of "data" that needs to be copied 
> between threads if you _only_ do:
>   use threads;
>   use threads::shared;
> And I'm not sure whether this number isn't too low, because I don't know 
> for sure whether the CV's in the stash haven't been counted correctly.  
> If they were not, then you would come at about 400Kbyte of "data" for a 
> _bare_ thread.
> Loading a few modules, each with their initializations, add up _very_ 
> quickly to several Mbytes of "data" that needs to be cloned _every_ time 
> you start a thread.  And these are _not_ simple copies: all of the 
> stashes need to be walked to make sure that all the [SAHC]V's are 
> properly copied to the  thread's copy.  So it's taking a _lot_ of CPU as 
> well...
> So yes, in general I think you can say that the data copied for each 
> thread, quickly dwarves whatever optrees are shared.

How is this different from fork? When you fork, OS shares all memory pages 
between the parent and the child. As variables are modified, memory pages 
become dirty and unshared. With forking mutable (vars) and non-mutable 
(OpCodes) share the same memory pages, so ones a mutable variable changes, the 
opcodes allocated from the same memory page, get unshared too. So you get more 
and more memory unshared as you go. in the long run (unless you use size 
limiting tools) all the memory gets unshared.

With ithreads, opcode tree is always shared and mutable data is copied at the 
very beginning. So your memory consumption should be exactly the same after 
the first request and after 1000's request (assuming that you don't allocate 
any memory at run time). Here you get more memory consumed at the beginning of 
the spawned thread, but it stays the same.

So let's say you have 8MB Opcode Tree and 4MB of mutable data. The process 
totalling at 12MB. Using fork you will start off with all 12MB shared and get 
memory unshared as you go. With threads, you will start off with 4MB upfront 
memory consumption and it'll stay the same. Now if in your fork setup you 
braket at 8MB with a size limiting tool to restart, you will get the same 4MB 
overhead per process. Besides equal memory usage you get better run-time 
performance with threads, because it doesn't need to copy dirty pages as with 
forks (everything was done at the perl_clone, which can be arranged long 
before the request is served) (and you get a slowdown at the same time because 
of context management).

So, as you can see it's quite possible that threads will perform better than 
forks and consume equal or less amount of memory if the opcode tree is bigger 
than the mutable data.

Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker     mod_perl Guide --->

Reporting bugs:
Mail list info:

View raw message