lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: duplication checking while indexing
Date Mon, 29 Dec 2008 18:41:07 GMT
Chris,

Mark Miller & Co. are working on (Near) Duplicate Detection.  I think the work is in Solr's
JIRA, but some of it might be applicable to Lucene.

 Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: Chris Lu <chris.lu@gmail.com>
> To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>
> Sent: Monday, December 29, 2008 4:55:14 AM
> Subject: duplication checking while indexing
> 
> I am wondering whether there is an easy way to avoid duplication while
> indexing, just using the index being created, without creating other data
> structures.
> In some cases, the incoming document list can have duplicates. For example,
> when creating spell checking indexes for phrases. Each phrase is one
> document. So I want to check whether the phrase is already indexed or not.
> 
> To do so, I can either create a hash map for all the indexed phrases. But
> the hash map would consume a lot of memory.
> A possible alternative is to search existing index. But remember the index
> is being created, and not all contents are flushed to disk yet.
> 
> Is it possible to query the not-yet-closed index?
> 
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes:
> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per request) got
> 2.6 Million Euro funding!


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message