httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Berg <jon.b...@turtlemeat.com>
Subject Re: [users@httpd] Robots.txt and virtualhost
Date Sat, 05 Jun 2004 18:05:33 GMT
hi,

There is no way of doing a query to apache and finding all virtual hosts 
on an ip. There is a way to do this with dns, dnslocator.com.
Or I guess if Google saves every ip that they get when doing
a dns lookup they could merge this together and find it. But I don't
see any point of doing it.

So a new virtual host would be like a new "host" on the Internet for
a searchbot.



Jon Berg.
----
http://tubegator.com

Thomas Gabrielsen wrote:

> Wow! Nine exclamation marks! I guess you have a great knowledge on this
> subject then ;)
> So I dare to ask you a cuple of other questions:
> - When a robots crawling the net, is it scanning internet for which ip's
> hosting webservers, and then again finds what domains are hosted there? If
> thats the case, how do they do that? Does webservers (in this case: Apache)
> give out that information about which domain is hosted when querying the IP
> (mainhost).
> If this is the case, the newer virtualhost should have the same amount of
> visits as the mainhost by now so this can't be the case.....!?
> 
> Can you (or anybody) tell me more about how robots work, or where I can read
> more about it? Specially the google bot.
> 
> Thanks,
> Thomas
> 
> ----- Original Message ----- 
> From: "Zoe Ballz" <zoe@zoeballz.tv>
> To: <users@httpd.apache.org>
> Sent: Saturday, June 05, 2004 1:31 AM
> Subject: Re: [users@httpd] Robots.txt and virtualhost
> 
> 
> Hiya
> 
> You are right - thats exactly the reason !!!!!!!!!
> 
> Zoe
> 
> http://zoeballz.net - Website Hosting for the Transgendered Community
>   ----- Original Message ----- 
>   From: Thomas Gabrielsen
>   To: users@httpd.apache.org
>   Sent: Friday, June 04, 2004 7:44 PM
>   Subject: [users@httpd] Robots.txt and virtualhost
> 
> 
>   Hi
> 
>   I have a question about Robots.txt files and virtualhost's. When I have
> one
>   main directory and one (named) virtualhost. Both are sharing one ip.
> Should
>   I add then something special in the robots.txt file that are placed in the
>   main directory pointing the robots to the virtualhost directory? The
> reason
>   I ask is that I get MANY requests for robots.txt in the main directory
>   compared to the virtualhost. Now, the mainhost has been up for a long time
>   compared to the virtualhost (which has been up for a week) so that could
> be
>   the reason. I just wanted to ask you fine people to be sure :)
> 
>   Thanks,
>   Thomas Gabrielsen
> 
> 
> 
> 
>   ---------------------------------------------------------------------
>   The official User-To-User support forum of the Apache HTTP Server Project.
>   See <URL:http://httpd.apache.org/userslist.html> for more info.
>   To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>      "   from the digest: users-digest-unsubscribe@httpd.apache.org
>   For additional commands, e-mail: users-help@httpd.apache.org
> 
> 
> 
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
> 
> 
> 


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message