httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Gabrielsen" <apa...@arton.no>
Subject Re: [users@httpd] Robots.txt and virtualhost
Date Sat, 05 Jun 2004 21:58:23 GMT
Ah, I see! Thanks for clearing that up.

Thomas

----- Original Message ----- 
From: "Jon Berg" <jon.berg@turtlemeat.com>
To: <users@httpd.apache.org>
Sent: Saturday, June 05, 2004 8:05 PM
Subject: Re: [users@httpd] Robots.txt and virtualhost


: hi,
:
: There is no way of doing a query to apache and finding all virtual hosts
: on an ip. There is a way to do this with dns, dnslocator.com.
: Or I guess if Google saves every ip that they get when doing
: a dns lookup they could merge this together and find it. But I don't
: see any point of doing it.
:
: So a new virtual host would be like a new "host" on the Internet for
: a searchbot.
:
:
:
: Jon Berg.
: ----
: http://tubegator.com
:
: Thomas Gabrielsen wrote:
:
: > Wow! Nine exclamation marks! I guess you have a great knowledge on this
: > subject then ;)
: > So I dare to ask you a cuple of other questions:
: > - When a robots crawling the net, is it scanning internet for which ip's
: > hosting webservers, and then again finds what domains are hosted there?
If
: > thats the case, how do they do that? Does webservers (in this case:
Apache)
: > give out that information about which domain is hosted when querying the
IP
: > (mainhost).
: > If this is the case, the newer virtualhost should have the same amount
of
: > visits as the mainhost by now so this can't be the case.....!?
: >
: > Can you (or anybody) tell me more about how robots work, or where I can
read
: > more about it? Specially the google bot.
: >
: > Thanks,
: > Thomas
: >
: > ----- Original Message ----- 
: > From: "Zoe Ballz" <zoe@zoeballz.tv>
: > To: <users@httpd.apache.org>
: > Sent: Saturday, June 05, 2004 1:31 AM
: > Subject: Re: [users@httpd] Robots.txt and virtualhost
: >
: >
: > Hiya
: >
: > You are right - thats exactly the reason !!!!!!!!!
: >
: > Zoe
: >
: > http://zoeballz.net - Website Hosting for the Transgendered Community
: >   ----- Original Message ----- 
: >   From: Thomas Gabrielsen
: >   To: users@httpd.apache.org
: >   Sent: Friday, June 04, 2004 7:44 PM
: >   Subject: [users@httpd] Robots.txt and virtualhost
: >
: >
: >   Hi
: >
: >   I have a question about Robots.txt files and virtualhost's. When I
have
: > one
: >   main directory and one (named) virtualhost. Both are sharing one ip.
: > Should
: >   I add then something special in the robots.txt file that are placed in
the
: >   main directory pointing the robots to the virtualhost directory? The
: > reason
: >   I ask is that I get MANY requests for robots.txt in the main directory
: >   compared to the virtualhost. Now, the mainhost has been up for a long
time
: >   compared to the virtualhost (which has been up for a week) so that
could
: > be
: >   the reason. I just wanted to ask you fine people to be sure :)
: >
: >   Thanks,
: >   Thomas Gabrielsen
: >
: >
: >
: >
: >   ---------------------------------------------------------------------
: >   The official User-To-User support forum of the Apache HTTP Server
Project.
: >   See <URL:http://httpd.apache.org/userslist.html> for more info.
: >   To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
: >      "   from the digest: users-digest-unsubscribe@httpd.apache.org
: >   For additional commands, e-mail: users-help@httpd.apache.org
: >
: >
: >
: > ---------------------------------------------------------------------
: > The official User-To-User support forum of the Apache HTTP Server
Project.
: > See <URL:http://httpd.apache.org/userslist.html> for more info.
: > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
: >    "   from the digest: users-digest-unsubscribe@httpd.apache.org
: > For additional commands, e-mail: users-help@httpd.apache.org
: >
: >
: >
:
:
: ---------------------------------------------------------------------
: The official User-To-User support forum of the Apache HTTP Server Project.
: See <URL:http://httpd.apache.org/userslist.html> for more info.
: To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
:    "   from the digest: users-digest-unsubscribe@httpd.apache.org
: For additional commands, e-mail: users-help@httpd.apache.org
:
:



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message