httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yehuda Katz <yeh...@ymkatz.net>
Subject Re: New RewriteMap Help/Suggestions
Date Fri, 26 Apr 2013 05:16:40 GMT
On Thu, Apr 25, 2013 at 10:35 AM, Jim Riggs <apache-lists@riggs.me> wrote:

> So, I have created a crude, working proof-of-concept of this. It basically
> copies all of the functionality of the txt maps, including the cache, but
> in the lookup_map_regexpfile() function, it compiles the regexp for each
> line, attempts a match, and returns the backref-substituted replacement.
> (This pair gets cached.) This works beautifully as is, but it is horribly
> inefficient to have to compile the REs every time we come in with a new
> key/URL. So, I was thinking of precompiling all of them and see three
> options:
>
> 1. Precompile and store all of the REs at config load time.
>
1a. Precompile and store all of the REs at config load time or when the map
file is updated.

> 2. Compile and store all of the REs the first time we hit
> lookup_map_regexpfile() or when the map file is updated.
> 3. Compile and store each RE as we read through the map file in
> lookup_map_regexpfile() until a match is found and bail (full list will be
> built over time).
>
> #1 is nice, because all of the work is done up front and will be fast from
> then on. The problem, though, is that I would like this map to
> reload/refresh if the map file gets changed like the other types do. #2 and
> #3 solve this. With #2 I worry about performance of compiling everything if
> the map file gets updated and we get a thundering herd. With #3 there is
> some coordination to manage with respect to which lines have been compiled
> and which ones haven't.
>
I think #3 is not a great idea for the same reason you mentioned.

I have actually seen the problem that you mention in #2 in a live
environment with a (poorly-designed) custom module. Each request tries to
clear the cached results and build them again, very quickly overloading the
server.

You could potentially use something like ap_hook_monitor to watch the file
for changes, paired with 1a (not sure how much load that might add). In my
regular apache module reference (Nick Kew's Apache Modules Book which I
keep on my office bookshelf) it is mentioned quickly (pages 67, 268, 337).

- Y

Mime
View raw message