Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <14657080.post@talk.nabble.com>
Date: Sun, 6 Jan 2008 17:30:38 -0800 (PST)
From: jibjoice <sudarat_jib@hotmail.com>
To: hadoop-user@lucene.apache.org
Subject: Re: Nutch crawl problem
In-Reply-To: <14589912.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <14327978.post@talk.nabble.com> <14410062.post@talk.nabble.com>
 <200712190953.39120.pvvpr@research.iiit.ac.in>
 <14412659.post@talk.nabble.com> <14433510.post@talk.nabble.com>
 <200712201806.52741.pvvpr@research.iiit.ac.in>
 <14450181.post@talk.nabble.com> <14492766.post@talk.nabble.com>
 <38376.220.226.42.133.1198583542.squirrel@research.iiit.ac.in>
 <14589912.post@talk.nabble.com>


why i can crawl http://game.search.com but i can't crawl
http://www.search.com? conf/crawl-urlfilter is

# skip file:, ftp:, & mailto: urls
-^(file|ftp|mailto):

# skip image and other suffixes we can't yet parse
#-\.(png|PNG|ico|ICO|css|sit|eps|wmf|zip|mpg|gz|rpm|tgz|mov|MOV|exe|bmp|BMP)$

# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]

# skip URLs with slash-delimited segment that repeats 3+ times, to break
loops
-.*(/.+?)/.*?\1/.*?\1/

# accept hosts in MY.DOMAIN.NAME
#+^http://([a-z0-9]*\.)*search.com/

# skip everything else
+.
 
and some host i can't crawl because have error "Generator: 0 records
selected for fetching, exiting ..." i set the same config for all host.why?
-- 
View this message in context: http://www.nabble.com/Nutch-crawl-problem-tp14327978p14657080.html
Sent from the Hadoop Users mailing list archive at Nabble.com.