httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dean Gaudet <dgau...@arctic.org>
Subject Re: Browser capabilities?
Date Wed, 01 Oct 1997 18:41:01 GMT
MSIE 4 and navi^H^H^H^Hcommunicator 4 both support crawling and both
support robots.txt.

Dean

On Wed, 1 Oct 1997, Lars Eilebrecht wrote:

> According to Rob Hartill:
> 
> [...]
> >  Ooo, I already play big brother on User-Agents with 'crawl' in the name.
> 
> BTW, Lynx has also a -crawl/-traversal mode, but User-Agent isn't changed.
> I wonder if any 'crawling' browser supports robots.txt (Lynx does not)?
> 
> [...]
> > > Doesn't look like Netscape has wisened up yet.
> >  
> >  I pleaded with them a while back to make the crawl mode show itself.
> >  Oh well, it's their own foot they're shooting.
> 
> Netscape? Crawl-mode? *scratch*   What version?
> 
> 
> ciao...
> -- 
> Lars Eilebrecht                  - LSD melts in your mind, not in your hands.
> sfx@unix-ag.org
> http://www.si.unix-ag.org/~sfx/
> 


Mime
View raw message