lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <>
Subject [jira] Commented: (LUCENE-1035) Optional Buffer Pool to Improve Search Performance
Date Fri, 26 Oct 2007 14:43:50 GMT


Yonik Seeley commented on LUCENE-1035:

A couple of random thoughts
- previous tests showed that vint decoding was often bottleneck, but these tests seem to indicate
otherwise (that the bottleneck is the system call to move data from OS  FS cache to userspace?)...
perhaps this is due to the fact that all queries are "AND" and match a max of 1000 docs? 
The recent addition of multi-level skipping perhaps removes the vint decoding bottleneck for
these types of queries that match few documents.
- most lucene usecases store much more than just the document id... that would really affect
- It seems like a simple LRU cache could really be blown out of the water by certain types
of queries (retrieve a lot of stored fields, or do an expanding term query) that would force
out all previously cached hotspots.  Most OS level caching has protection against this (multi-level
LRU or whatever).  But of our user-level LRU cache fails, we've also messed up the OS level
cache since we've been hiding page hits from it.
- I'd like to see single term queries, "OR" queries, and queries across multiple fields (also
a common usecase) that match more documents tested also.

> Optional Buffer Pool to Improve Search Performance
> --------------------------------------------------
>                 Key: LUCENE-1035
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>            Reporter: Ning Li
>         Attachments: LUCENE-1035.patch
> Index in RAMDirectory provides better performance over that in FSDirectory.
> But many indexes cannot fit in memory or applications cannot afford to
> spend that much memory on index. On the other hand, because of locality,
> a reasonably sized buffer pool may provide good improvement over FSDirectory.
> This issue aims at providing such an optional buffer pool layer. In cases
> where it fits, i.e. a reasonable hit ratio can be achieved, it should provide
> a good improvement over FSDirectory.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message