nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Laurenz <mlaur...@slmedien.de>
Subject Nutch ignores robots.txt
Date Wed, 02 Nov 2011 13:13:40 GMT
Hi there,
Because a Nutch client seems to cause errors on our web server, we changed robots.txt for
www.kinoundco.de<http://www.kinoundco.de> to disallow Nutch. Unfortunately we still
get requests:

2011-11-01 05:50:35 W3SVC4 MB 62.128.28.16 GET /Rango/+(Math.random()*100000)+ - 80 - 188.40.65.130
HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 373 263 0
2011-11-01 05:52:23 W3SVC4 MB 62.128.28.16 GET /default.aspx aspxerrorpath=/Atemlos-Gefaehrliche-Wahrheit/+(Math.random()*100000)+
80 - 188.40.65.130 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 200 0 32449 315 15
2011-11-01 05:59:15 W3SVC4 MB 62.128.28.16 GET /Kleine-wahre-Luegen/+(Math.random()*100000)+
- 80 - 188.40.65.130 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 401 277 15
2011-11-01 05:59:31 W3SVC4 MB 62.128.28.16 GET /Nachtasyl/+(Math.random()*100000)+ - 80 -
188.40.65.130 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 381 267 0
2011-11-01 06:35:30 W3SVC4 MB 62.128.28.16 GET /Auf-der-anderen-Seite-der-Leinwand-100-Jahre-Moviemento/+(Math.random()*100000)+
- 80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 473 313 15
2011-11-01 06:35:33 W3SVC4 MB 62.128.28.16 GET /Zoowaerter/+(Math.random()*100000)+ - 80 -
78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 383 268 31
2011-11-01 06:35:40 W3SVC4 MB 62.128.28.16 GET /default.aspx aspxerrorpath=/Sascha/+(Math.random()*100000)+
80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 200 0 32449 292 62
2011-11-01 06:36:35 W3SVC4 MB 62.128.28.16 GET /Beschissenheit-der-Dinge/+(Math.random()*100000)+
- 80 - 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 411 282 31
2011-11-01 06:38:14 W3SVC4 MB 62.128.28.16 GET /Auf-der-Suche/+(Math.random()*100000)+ - 80
- 78.46.90.27 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 389 271 15
2011-11-01 06:39:55 W3SVC4 MB 62.128.28.16 GET /Fall/+(Math.random()*100000)+ - 80 - 78.46.90.27
HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 371 262 15
2011-11-01 07:51:10 W3SVC4 MB 62.128.28.16 GET /Midnight-in-Paris/+(Math.random()*100000)+
- 80 - 176.9.26.236 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 397 275 0
2011-11-01 07:51:40 W3SVC4 MB 62.128.28.16 GET /Betty-Anne-Waters/+(Math.random()*100000)+
- 80 - 176.9.26.236 HTTP/1.0 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2
- - www.kinoundco.de 302 0 397 275 15

Sincerely,
Max
Maximilian Laurenz
S&L Medien Gruppe GmbH
Aidenbachstraße 54
81379 München
Tel. +49 89 790862-49
Fax +49 89 790862-55
mlaurenz@slmedien.de<mailto:mlaurenz@slmedien.de>
http://www.slmedien.de
S&L Medien Gruppe GmbH | Geschäftsführung: Maria-Theresia von Seidlein, Torsten Weihrich,
Olaf Wiehler | Sitz der Gesellschaft: München | Amtsgericht München | HRB 99977


Mime
View raw message