httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Eilebrecht <>
Subject Re: Browser capabilities?
Date Wed, 01 Oct 1997 10:16:19 GMT
According to Rob Hartill:

>  Ooo, I already play big brother on User-Agents with 'crawl' in the name.

BTW, Lynx has also a -crawl/-traversal mode, but User-Agent isn't changed.
I wonder if any 'crawling' browser supports robots.txt (Lynx does not)?

> > Doesn't look like Netscape has wisened up yet.
>  I pleaded with them a while back to make the crawl mode show itself.
>  Oh well, it's their own foot they're shooting.

Netscape? Crawl-mode? *scratch*   What version?

Lars Eilebrecht                  - LSD melts in your mind, not in your hands.

View raw message