lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eks Dev (JIRA)" <>
Subject [jira] Commented: (LUCENE-1340) Make it posible not to include TF information in index
Date Mon, 21 Jul 2008 18:43:31 GMT


Eks Dev commented on LUCENE-1340:

Great, it is already more than I expected, even indexing is going to be somewhat faster.

I have tried your patch on smallish index with 8Mio documents and it worked on our regression
test without problems. 
it worked fine with and without omitTf(true), no performance drop or bad surprises when we
do not use it. Tomorrow is scheduled real test with production data, around 80Mio very small
documents, with some very extensive tests.... I will report back.

"The one place I know of that will still waste bytes is the term dict
(TermInfo): it stores a long proxPointer on disk (in .tii,.tis) and
also in memory because we load *.tii into RAM.... "

 About this one, it would be nice not to store this as well, but I think the pointers are
already reduced to one byte, as they are 0 for these cases (are they,?) So we have this benefit
without expecting it :)

And yes, more "column stride" is great, if you followed my comments on LUCENE-1278, that would
mean we could easily "inline" very short postings into term dict (here I expect huge performance
benefit, as skip()  on another large file is going to be saved independent from omitTf(true)),
without increase in size (or minimal) of tii (no locality penalty) If we follow Zipfian distribution,
there is *a lot* of terms with postings shorter than e.g. 16 ... 

Thanks again for your support, without you this patch would be just another nice idea :)

> Make it posible not to include TF information in index
> ------------------------------------------------------
>                 Key: LUCENE-1340
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Eks Dev
>            Priority: Minor
>         Attachments: LUCENE-1340.patch, LUCENE-1340.patch, LUCENE-1340.patch, LUCENE-1340.patch,
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> Term Frequency is typically not needed  for all fields, some CPU (reading one VInt less
and one X>>>1...) and IO can be spared by making pure boolen fields possible in Lucene.
This topic has already been discussed and accepted as a part of Flexible Indexing... This
issue tries to push things a bit faster forward as I have some concrete customer demands.
> benefits can be expected for fields that are typical candidates for Filters, enumerations,
user rights, IDs or very short "texts", phone  numbers, zip codes, names...
> Status: just passed standard test (compatibility), commited for early review, I have
not tried new feature, missing some asserts and one two unit tests
> Complexity: simpler than expected
> can be used via omitTf() (who used omitNorms() will know where to find it :)  

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message