httpd-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From br...@hyperreal.org
Subject cvs commit: apachen/htdocs/manual/misc howto.html
Date Mon, 26 Jan 1998 07:12:59 GMT
brian       98/01/25 23:12:59

  Modified:    htdocs/manual/misc howto.html
  Log:
  tsk tsk, randy.  Can't find this on covalent.net either.
  
  Revision  Changes    Path
  1.4       +8 -1      apachen/htdocs/manual/misc/howto.html
  
  Index: howto.html
  ===================================================================
  RCS file: /export/home/cvs/apachen/htdocs/manual/misc/howto.html,v
  retrieving revision 1.3
  retrieving revision 1.4
  diff -u -r1.3 -r1.4
  --- howto.html	1997/06/04 11:42:57	1.3
  +++ howto.html	1998/01/26 07:12:58	1.4
  @@ -130,7 +130,14 @@
   <P><code>robots.txt</code> provides a means to request that robots limit
their
   activities at the site, or more often than not, to leave the site alone.</P>
   
  -<P>When the first robots were developed, they had a bad reputation for sending hundreds/thousands
of requests to each site, often resulting in the site being overloaded. Things have improved
dramatically since then, thanks to <A HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html">
Guidelines for Robot Writers</A>, but even so, some robots may <A HREF="http://www.zyzzyva.com/robots/alert/">exhibit
unfriendly behavior</A> which the webmaster isn't willing to tolerate, and will want
to stop.</P>
  +<P>When the first robots were developed, they had a bad reputation for
  +sending hundreds/thousands of requests to each site, often resulting
  +in the site being overloaded. Things have improved dramatically since
  +then, thanks to <A
  +HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html">
  +Guidelines for Robot Writers</A>, but even so, some robots may exhibit
  +unfriendly behavior which the webmaster isn't willing to tolerate, and
  +will want to stop.</P>
   
   <P>Another reason some webmasters want to block access to robots, is to
   stop them indexing dynamic information. Many search engines will use the
  
  
  

Mime
View raw message