Mailing-List: contact modperl-help@perl.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (nike.apache.org: domain of pharkins@gmail.com designates
 209.85.212.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <54D0734F.1060308@streppone.it>
References: 
 <CANJD0wKmBWkTh2ccoeS+vhoBWDcGwKRVLan+QVkPkhwXboBCcw@mail.gmail.com>
	<54D0734F.1060308@streppone.it>
Date: Tue, 3 Feb 2015 11:38:23 -0500
Message-ID: 
 <CAC0_be0Xa2k1QPv9TqSVDAVkJ8uQFv1rzKnpG9gLzqK2vxPSdA@mail.gmail.com>
Subject: Re: mod_perl for multi-process file processing?
From: Perrin Harkins <pharkins@gmail.com>
To: Cosimo Streppone <cosimo@streppone.it>
Cc: Alan Raetz <alanraetz@gmail.com>, mod_perl list <modperl@perl.apache.org>
Content-Type: multipart/alternative; boundary=f46d043c7cac098539050e31b364

--f46d043c7cac098539050e31b364
Content-Type: text/plain; charset=UTF-8

Cache::FastMmap is a great module for sharing read/write data, but it can't
compete with the speed of loading it all into memory before forking as Alan
said he plans to do.

- Perrin

On Tue, Feb 3, 2015 at 2:05 AM, Cosimo Streppone <cosimo@streppone.it>
wrote:

> Alan Raetz wrote:
>
>  So I have a perl application that upon startup loads about ten perl
>> hashes (some of them complex) from files. This takes up a few GB of
>> memory and about 5 minutes. It then iterates through some cases and
>> reads from (never writes) these perl hashes. To process all our cases,
>> it takes about 3 hours (millions of cases). We would like to speed up
>> this process. I am thinking this is an ideal application of mod_perl
>> because it would allow multiple processes but share memory.
>>
>
> Sure you could use modperl for this.
> I would also consider at least these alternatives:
>
> - use Cache::FastMmap, https://metacpan.org/pod/Cache::FastMmap
>   Load up your data with a loader script, and forget about it.
>   Cache::FastMmap also works with modperl.
>
> - use a network server, like memcached or redis to store your
>   read-only data, and use a lightweight network protocol (on localhost)
>   to get the data.
>
> In both cases, reading from multiple processes will not be a problem.
> The cheapest solution for the consumer part (the "cases" above)
> would be to use a command like "parallel" to fire up as many copies
> of your consumer script as you can afford.
>
> Hope this helps,
>
> --
> Cosimo
>
>

--f46d043c7cac098539050e31b364
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Cache::FastMmap is a great module for sharing read/write d=
ata, but it can&#39;t compete with the speed of loading it all into memory =
before forking as Alan said he plans to do.<div><br></div><div>- Perrin</di=
v></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, F=
eb 3, 2015 at 2:05 AM, Cosimo Streppone <span dir=3D"ltr">&lt;<a href=3D"ma=
ilto:cosimo@streppone.it" target=3D"_blank">cosimo@streppone.it</a>&gt;</sp=
an> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left:1px #ccc solid;padding-left:1ex"><span class=3D"">Alan Raetz wr=
ote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
So I have a perl application that upon startup loads about ten perl<br>
hashes (some of them complex) from files. This takes up a few GB of<br>
memory and about 5 minutes. It then iterates through some cases and<br>
reads from (never writes) these perl hashes. To process all our cases,<br>
it takes about 3 hours (millions of cases). We would like to speed up<br>
this process. I am thinking this is an ideal application of mod_perl<br>
because it would allow multiple processes but share memory.<br>
</blockquote>
<br></span>
Sure you could use modperl for this.<br>
I would also consider at least these alternatives:<br>
<br>
- use Cache::FastMmap, <a href=3D"https://metacpan.org/pod/Cache::FastMmap"=
 target=3D"_blank">https://metacpan.org/pod/<u></u>Cache::FastMmap</a><br>
=C2=A0 Load up your data with a loader script, and forget about it.<br>
=C2=A0 Cache::FastMmap also works with modperl.<br>
<br>
- use a network server, like memcached or redis to store your<br>
=C2=A0 read-only data, and use a lightweight network protocol (on localhost=
)<br>
=C2=A0 to get the data.<br>
<br>
In both cases, reading from multiple processes will not be a problem.<br>
The cheapest solution for the consumer part (the &quot;cases&quot; above)<b=
r>
would be to use a command like &quot;parallel&quot; to fire up as many copi=
es<br>
of your consumer script as you can afford.<br>
<br>
Hope this helps,<span class=3D"HOEnZb"><font color=3D"#888888"><br>
<br>
-- <br>
Cosimo<br>
<br>
</font></span></blockquote></div><br></div>

--f46d043c7cac098539050e31b364--