Mailing-List: contact dev-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mahout.apache.org
Date: Mon, 18 Apr 2011 17:32:05 +0000 (UTC)
From: "Chris Jordan (JIRA)" <jira@apache.org>
To: dev@mahout.apache.org
Message-ID: 
 <818432793.64963.1303147925720.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1205695927.64947.1303147685753.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (MAHOUT-675) LuceneIterator throws an
 IllegalStateException when a null TermFreqVector is encountered for a
 document instead of skipping to the next one
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAHOUT-675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Jordan updated MAHOUT-675:
--------------------------------

    Attachment: MAHOUT-675

I have patch the LuceneIterator. It is a mild change that also adds a Logger instance.

> LuceneIterator throws an IllegalStateException when a null TermFreqVector is encountered for a document instead of skipping to the next one
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAHOUT-675
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-675
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Utils
>            Reporter: Chris Jordan
>         Attachments: MAHOUT-675
>
>
> The org.apache.mahout.utils.vectors.lucene.LuceneIterator currently throws an IllegalStateException if it encounters a document with a null term frequency vector for the target field in the computeNext() method. That is problematic for people who are developing text mining applications on top of lucene as it forces them to check that the documents that they are adding to their lucene indexes actually have terms for the target field. While that check may sound reasonable, it actually is not in practice.
> Lucene in most cases will apply an analyzer to a field in a document as it is added to the index. The StandardAnalyzer is pretty lenient and barely removes any terms. In most cases though, if you want to have better text mining performance, you will create your own custom analyzer. For example, in my current work with document clustering, in order to generate tighter clusters and have more human readable top terms, I am using a stop word list specific to my subject domain and I am filtering out terms that contain numbers. The net result is that some of my documents have no terms for the target field which is a desirable outcome. When I attempt to dump the lucene vectors though, I encounter an IllegalStateException because of those documents.
> Now it is possible for me to check the TokenStream of the target field before I insert into my index however, if we were to follow that approach, it means for each of my applications, I would have to perform this check. That isn't a great practice as someone could be experimenting with custom analyzers to improve text mining performance and then encounter this exception without any real indication that it was due to the custom analyzer.
> I believe a better approach is to log a warning with the field id of the problem document and then skip to the next one. That way, a warning will be in the logs and the lucene vector dump process will not halt.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira