manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Schuch <markus_sch...@web.de>
Subject [Webcrawler Connector] Feature for ignoring meta/rel robots tags/attributes
Date Sat, 25 Feb 2017 22:02:20 GMT
Hi,

what do you think about adding the possibility to ignore meta/rel robots
tags/attributes?

I know, such a thing is an unpolite behavior for a webcrawler, but we
already have the feature to ignore the robots.txt and for me it was
unexpected, when i configured the crawler to ignore robots.txt but it
still respected the meta/rel robots tags/attibutes.

I will open a ticket, if there are no objections.

Thanks in advance.
Markus

Mime
View raw message