nutch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kevin chen <kevinc...@bdsing.com>
Subject Re: How can nutch crawl the content of a dynamic url with a query string?
Date Sun, 27 Sep 2009 01:36:22 GMT
By default, nutch skips URLs containing certain characters. To change
it, open regex-urlfilter.txt, comment out the following line.

# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]


On Sun, 2009-09-27 at 03:55 +0800, Shawn Young wrote:
> Hi all,
> 
> 	I have a question, if a web page's url likes
> http://www.test.com/test.php?gid=1111111 ,how can nutch crawl its content?
> I've had a try, but it seems that nutch ignores the query string
> 'gid=1111111' of the url.
> 
> 	Can someone helps me?
> 	Thanks.
> 


Mime
View raw message