Mailing-List: contact solr-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-dev@lucene.apache.org
Message-ID: <870958535.1223424164551.JavaMail.jira@brutus>
Date: Tue, 7 Oct 2008 17:02:44 -0700 (PDT)
From: "Mark Miller (JIRA)" <jira@apache.org>
To: solr-dev@lucene.apache.org
Subject: [jira] Commented: (SOLR-799) Add support for hash based exact/near
 duplicate document handling
In-Reply-To: <1661911285.1223088224170.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SOLR-799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637719#action_12637719 ] 

Mark Miller commented on SOLR-799:
----------------------------------

Thanks for the review Andrzej. I've made the first two changes (I put at the top of TextProfileSignature that its 'borrowed' from Nutch and grabbed Hadoops MD5Hash class and stripped its Hadoop dependencies) and I'm investigating change 3. I'll put up another patch in a couple days.

- Mark

> Add support for hash based exact/near duplicate document handling
> -----------------------------------------------------------------
>
>                 Key: SOLR-799
>                 URL: https://issues.apache.org/jira/browse/SOLR-799
>             Project: Solr
>          Issue Type: New Feature
>          Components: update
>            Reporter: Mark Miller
>            Priority: Minor
>         Attachments: SOLR-799.patch
>
>
> Hash based duplicate document detection is efficient and allows for blocking as well as field collapsing. Lets put it into solr. 
> http://wiki.apache.org/solr/Deduplication

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.