httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Gabrielsen" <apa...@arton.no>
Subject Re: [users@httpd] Robots.txt and virtualhost
Date Sat, 05 Jun 2004 17:54:16 GMT
Wow! Nine exclamation marks! I guess you have a great knowledge on this
subject then ;)
So I dare to ask you a cuple of other questions:
- When a robots crawling the net, is it scanning internet for which ip's
hosting webservers, and then again finds what domains are hosted there? If
thats the case, how do they do that? Does webservers (in this case: Apache)
give out that information about which domain is hosted when querying the IP
(mainhost).
If this is the case, the newer virtualhost should have the same amount of
visits as the mainhost by now so this can't be the case.....!?

Can you (or anybody) tell me more about how robots work, or where I can read
more about it? Specially the google bot.

Thanks,
Thomas

----- Original Message ----- 
From: "Zoe Ballz" <zoe@zoeballz.tv>
To: <users@httpd.apache.org>
Sent: Saturday, June 05, 2004 1:31 AM
Subject: Re: [users@httpd] Robots.txt and virtualhost


Hiya

You are right - thats exactly the reason !!!!!!!!!

Zoe

http://zoeballz.net - Website Hosting for the Transgendered Community
  ----- Original Message ----- 
  From: Thomas Gabrielsen
  To: users@httpd.apache.org
  Sent: Friday, June 04, 2004 7:44 PM
  Subject: [users@httpd] Robots.txt and virtualhost


  Hi

  I have a question about Robots.txt files and virtualhost's. When I have
one
  main directory and one (named) virtualhost. Both are sharing one ip.
Should
  I add then something special in the robots.txt file that are placed in the
  main directory pointing the robots to the virtualhost directory? The
reason
  I ask is that I get MANY requests for robots.txt in the main directory
  compared to the virtualhost. Now, the mainhost has been up for a long time
  compared to the virtualhost (which has been up for a week) so that could
be
  the reason. I just wanted to ask you fine people to be sure :)

  Thanks,
  Thomas Gabrielsen




  ---------------------------------------------------------------------
  The official User-To-User support forum of the Apache HTTP Server Project.
  See <URL:http://httpd.apache.org/userslist.html> for more info.
  To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
     "   from the digest: users-digest-unsubscribe@httpd.apache.org
  For additional commands, e-mail: users-help@httpd.apache.org



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message