Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
Mime-Version: 1.0 (Apple Message framework v1075.2)
Subject: Re: capturing field length into a stored document field
From: Grant Ingersoll <gsingers@apache.org>
In-Reply-To: <25297690.post@talk.nabble.com>
Date: Fri, 4 Sep 2009 17:27:23 -0500
Content-Transfer-Encoding: 7bit
Message-Id: <F34DF2B8-1986-428F-8D20-DEE577DD9F00@apache.org>
References: <25297690.post@talk.nabble.com>
To: solr-user@lucene.apache.org

The Similarity.lengthNorm() is a callback from Lucene that gives you  
the information you seek.  Of course, the trick still is how to use  
that.  Perhaps you can describe a bit more about why you need that  
length.

On Sep 4, 2009, at 11:34 AM, mike.schultz wrote:

>
> For various statistics I collect from an index it's important for me  
> to know
> the length (measured in tokens) of a document field.  I can get that
> information to some degree from the "norms" for the field but a) the
> resolution isn't that great, and b) more importantly, if boosts are  
> used
> it's almost impossible to get lengths from this.
>
> Here's two ideas I was thinking about that maybe some can comment on.
>
> 1) Use copyto to copy the field in question, fieldA to an addition  
> field,
> fieldALength, which has an extra filter that just counts the tokens  
> and only
> outputs a token representing the length of the field.  This has the
> disadvantage of retokenizing basically the whole document (because  
> the field
> in question is basically the body).  Plus I would think littering  
> the term
> space with these tokens might be bad for performance, I'm not sure.
>
> 2) Add a filter to the field in question which again counts the  
> tokens.
> This filter allows the regular tokens to be indexed as usual but  
> somehow
> manages to get the token-count into a stored field of the document.   
> This
> has the advantage of not having to retokenize the field and instead of
> littering the token space, the count becomes docdata for each doc.   
> Can this
> be done?  Maybe using threadLocal to temporarily store the count?
>
> Thanks.
>
> -- 
> View this message in context: http://www.nabble.com/capturing-field-length-into-a-stored-document-field-tp25297690p25297690.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:
http://www.lucidimagination.com/search