httpd-modules-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ingo Walz <ingo.w...@googlemail.com>
Subject Re: apr_hash_t and global scope
Date Thu, 12 Dec 2013 15:34:32 GMT
Am 12.12.2013 11:27, schrieb Nick Kew:
> On 12 Dec 2013, at 02:41, Ingo Walz wrote:
>
>> So socache with shared memory (shmcb) is the way you would suggest?
> How important is the sharing in your case?  If it's just about performance,
> you may be better-off accepting that each process will repeat the work
> of populating the hash, if that's a smaller overhead than IPC.
>
> If sharing is necessary, then socache will indeed serve, though you can
> leave the choice of socache engine to your config.
Well, depends. Of course, I want to get the best out of my 
implementation, so best-case scenario might be:
- no global (no idea if I ever can avoid that though, I guess not if shm 
will be used) or thread mutex
- no external dependencies (such as memcache e.g.)

On the other side, it should have a clear "scaling pattern". As of now, 
it only depends on the number of possible HTML rewrites (due to the 
number of lookups in apr_hash_t) and the number of assets delivered by 
factor one. Changing that on my current environment might blow up factor 
1 by factor 3 from a memory perspective. And thus changes on different 
servers with a different number of workers - which is a kind of 
unacceptable for me.
Before I do that, I'd introduce a smaller overhead on the way how it's 
stored but still have a clear scaling pattern.

But as I said, I'm still trying to evaluate what exactly is the best 
case scenario in details and which options do I have.

Memcache would be a great opportunity either, because it will give me a 
"Cluster capable" implementation for free, if I share the cache across 
multiple servers - but that's a use-case for the future and not of 
interest right now.

If I use server->process->pool, does every worker internally has it's 
own process->pool scope? Or is it 1 pool used as a parent process? 
Checking my process list gave me 3 workers running as my apache user and 
1 as root - so I guess there is a parent process, used to bind the 
socket and "dispatch" the work to the workers - but I've no idea if 
something is shared between those processes at all (beside the 
dispatched request itself).

>
> I was also a little uneasy about your original description: the use of a
> forever-pool becomes a memory leak if it is populated with per-request data.
> Modules that do such things usually create their own pool (as a child of
> the process pool) which they can then maintain privately.
>
Really a memory leak and not a potential double free or even a segfault? 
Because the request pool is cleaned up before the server pool and the 
reference, pointing to the memory segment remains in the process pool - 
but it might be freed (or even re-used by another process). I made an 
apr_pstrndup from the request data, assigned to server->process->pool to 
avoid this issue. I've also tried an own pool, but server->process->pool 
does 100% fit my use-case (I thought).

I'll try to work with socache inside the implementation to see if it 
covers all my needs in detail.
But I still try to understand why it happens. Will see if I can figure 
out what's going on currently (gdb is not easy to use in this particular 
use-case ... ), try socache and get back to you, once I have the first 
results.


Thanks!

Ingo


Mime
View raw message