lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom Burton-West (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2393) Utility to output total term frequency and df from a lucene index
Date Mon, 10 May 2010 19:54:30 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865894#action_12865894
] 

Tom Burton-West commented on LUCENE-2393:
-----------------------------------------

Hi Mike,

Thanks for all your help. 

If we replace the current HighFreqTerms with the HighFreqTermsWithTF should there be a command
line switch so that you could ask for the default behavior of the current HighFreqTerms? 
Or perhaps the default should be the current behavior and the switch should turn on the additional
step of gathering and reporting on the totalTF for the terms.   

I haven't bench-marked it but I'm wondering if  getting the totalTF  could  take a significant
 additional amount of time for large indexes.  When I ask for the top 10,000 terms using HighFreqTermsWithTF
for our 500,000 document indexes it takes about 40 minutes to an hour.  I'm guessing that
most of that time is taken in the first step of getting the top 10,000 terms by docFreq, but
still it seems that reading the data and calculating the totalTF for 10,000 terms might be
a significant enough fraction of the total time that the option to skip that step might be
useful.


Tom




> Utility to output total term frequency and df from a lucene index
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2393
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2393
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: contrib/*
>            Reporter: Tom Burton-West
>            Priority: Trivial
>         Attachments: LUCENE-2393, LUCENE-2393.patch, LUCENE-2393.patch, LUCENE-2393.patch,
LUCENE-2393.patch, LUCENE-2393.patch, LUCENE-2393.patch
>
>
> This is a pair of command line utilities that provide information on the total number
of occurrences of a term in a Lucene index.  The first  takes a field name, term, and index
directory and outputs the document frequency for the term and the total number of occurrences
of the term in the index (i.e. the sum of the tf of the term for each document).   The second
reads the index to determine the top N most frequent terms (by document frequency) and then
outputs a list of those terms along with  the document frequency and the total number of occurrences
of the term. Both utilities are useful for estimating the size of the term's entry in the
*prx files and consequent Disk I/O demands. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message