httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Eilebrecht <Lars.Eilebre...@unix-ag.org>
Subject Re: Browser capabilities?
Date Wed, 01 Oct 1997 10:16:19 GMT
According to Rob Hartill:

[...]
>  Ooo, I already play big brother on User-Agents with 'crawl' in the name.

BTW, Lynx has also a -crawl/-traversal mode, but User-Agent isn't changed.
I wonder if any 'crawling' browser supports robots.txt (Lynx does not)?

[...]
> > Doesn't look like Netscape has wisened up yet.
>  
>  I pleaded with them a while back to make the crawl mode show itself.
>  Oh well, it's their own foot they're shooting.

Netscape? Crawl-mode? *scratch*   What version?


ciao...
-- 
Lars Eilebrecht                  - LSD melts in your mind, not in your hands.
sfx@unix-ag.org
http://www.si.unix-ag.org/~sfx/

Mime
View raw message