httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tony Miller" <jusanotheran...@gmail.com>
Subject Re: [users@httpd] Apache Multi Process Limit
Date Sun, 10 Jun 2007 21:11:30 GMT
You understood my question absolutely fine, Vincent. The script parses out
the data once per request to my server. As for why I was using Apache to
parent these scripts, I have four words: I was being stupid! I am from a web
development and design background so I am too used to checking any script
via a browser rather than a command line.

Anyway, your suggestion did the trick. I ran the script via command line and
now i can run as many instances as i want. Muchos gracias!

Regards,
Tony

On 6/9/07, Vincent Bray <noodlet@gmail.com> wrote:
>
> On 09/06/07, Angelo Bourik <jusanotherangel@gmail.com> wrote:
> > Hello all,
>
> Hi,
>
> If I understood your question correctly, you've written a spider
> script that runs via apache. That script connects to a remote site,
> fetches a page by id and parses out the data you need, once per
> request to your own server (or does it run all 100,000 requests per
> single request to your server?).
>
> My first question is, why are you running your script under apache,
> rather than as a command line process? I don't see any advantage to
> having apache parent these scripts for you.
>
> As regards having more than one script running at the same time,
> that's perhaps a matter of cpu usage, though it does sound odd that
> your server is not at least starting to run the others. Perhaps your
> script opens and locks your database in such a way that the others are
> blocked?
>
> I'd recommend running your spider on the command line, outside of the
> server. If you really want to speed it up, split the spider in to a
> fetching thread group that is network bound, and a parsing thread
> group that munges the queued responses. Or, just write your script in
> a faster language :-)
>
> --
> noodl
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Mime
View raw message