Mailing-List: contact user-help@nutch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@nutch.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4D35797B.10303@unister-gmbh.de>
Date: Tue, 18 Jan 2011 12:28:59 +0100
From: Andrey Sapegin <andrey.sapegin@unister-gmbh.de>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.1.12) Gecko/20100913 Iceowl/1.0b1 Icedove/3.0.7
MIME-Version: 1.0
To: user@nutch.apache.org
Subject: search not working with merged indexes (Total hits: 0)
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Dear all.

I have a problem with nutch Internet crawl/recrawl script (I'm wanted to 
understand how it works, so I wrote it by myself).

After I merge indexes (merging segments seems to be fine), I search 
doesn't work for me:
    $ bin/nutch org.apache.nutch.searcher.NutchBean http
    Total hits: 0

Before recrawling I was able to search (index was placed at crawl/indexes)

My script:
---------------------------------------------
#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-6-sun

#Inject new urls
bin/nutch inject crawl/crawldb dmoz/urls
echo "new URLs injected (dmoz/urls)"

#generate segments
bin/nutch generate crawl/crawldb crawl/segments -topN $3
echo "segments generated"

#generate fetch-list
s1=`ls -d crawl/segments/2* | tail -1`
echo $s1
echo "fetch-list generated"

#fetch
bin/nutch fetch $s1 -threads $2
echo "fetching done"

#update the database with results of fetch
bin/nutch updatedb crawl/crawldb $s1
echo "database updated"

#merge segments
bin/nutch mergesegs crawl/MERGEDsegments crawl/segments/*
rm -r crawl/segments
mv crawl/MERGEDsegments crawl/segments
echo "segments merged"

#inverting links
bin/nutch invertlinks crawl/linkdb -dir crawl/segments
echo "links inverted"

#indexing
bin/nutch index crawl/NEWindexes crawl/crawldb crawl/linkdb crawl/segments/*
echo "indexing done"

#dedup - delete duplicate documents in the index
bin/nutch dedup crawl/NEWindexes
echo "dedup done"

#merging indexes
bin/nutch merge crawl/MERGEDindexes crawl/NEWindexes
echo "indexes merged"

# replace indexes with indexes_merged
mv --verbose crawl/indexes crawl/OLDindexes
mv --verbose crawl/MERGEDindexes crawl/indexes/part-00000

#clean up
rm -rf crawl/NEWindexes
rm -rf crawl/OLDindexes
-------------------------------------------------

What's wrong with the script?

Thank You in advance,
Kind Regards,

-- 

Andrey Sapegin,
Software Developer,

Unister GmbH
andrey.sapegin@unister-gmbh.de
www.unister.de