httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "RuneImp" <r...@imptech.net>
Subject Re: robots.txt
Date Fri, 18 Jan 2002 00:53:31 GMT
It will block all robots from the site. If you only
wish to block all robots from certain directories
then you would do:

User-agent: *
Disallow: /badsite1/
Disallow: /badsite2/

If you only want certain robots to not see certain
directories then:

User-agent: Scooter
Disallow: /badsite1/
Disallow: /badsite2/


-=- RuneImp
ImpTech - Web Design, Hosting & Computer Tech
http://imptech.net
rune@imptech.net


----- Original Message ----- 
From: "Raquel Rice" <raquel@thericehouse.net>
To: <users@httpd.apache.org>
Sent: Thursday, January 17, 2002 4:39 PM
Subject: Re: robots.txt


On Thu, 17 Jan 2002 15:52:38 -0800
RuneImp "RuneImp" <rune@imptech.net> wrote:

> Put this in your file
> 
> User-agent: *
> Disallow: /
> 
> 
> -=- RuneImp
> ImpTech - Web Design, Hosting & Computer Tech
> http://imptech.net
> rune@imptech.net
> 

Thanks, but won't that block everything?

BTW ... here's a good site for info:
http://www.robotstxt.org/wc/robots.html

-- 
Raquel
============================================================
Men never do evil so completely and cheerfully as when they do it
with religious conviction.
  --Blaise Pascal


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message