lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marvin Humphrey (JIRA)" <>
Subject [jira] Commented: (LUCENE-1458) Further steps towards flexible indexing
Date Wed, 19 Nov 2008 00:19:44 GMT


Marvin Humphrey commented on LUCENE-1458:

> I'm not sure I'd trust the OS's IO cache to "make the right decisions" about what to

In KS and Lucy, at least, we're focused on optimizing for the use case of dedicated search
clusters where each box has enough RAM to fit the entire index/shard -- in which case we won't
have to worry about the OS swapping out those pages.

I suspect that in many circumstances the term dictionary would be a hot file even if RAM were
running short, but I don't think it's important to worry about maxing out performance on such
systems -- if the term dictionary isn't hot the posting list files are definitely not hot
and search-time responsiveness is already compromised.

In other words...

* I trust the OS to do a decent enough job on underpowered systems.
* High-powered systems should strive to avoid swapping entirely. To aid in that endeavor,
we minimize per-process RAM consumption by maximizing our use of mmap and treating the system
IO cache backing buffers as interprocess shared memory.

More on designing with modern virtual memory in mind at <>.

> Plus during that binary search the IO system is loading whole pages into
> the IO cache, even though you'll only peak at the first few bytes of each.

I'd originally been thinking of mapping only the term dictionary index files. Those are pretty
small, and the file itself occupies fewer bytes than the decompressed array of term/pointer
pairs. Even better if you have several search app forks and they're all sharing the same memory
mapped system IO buffer.

But hey, we can simplify even further! How about dispensing with the index file? We can just
divide the main dictionary file into blocks and binary search on that.

Killing off the term dictionary index yields a nice improvement in code and file specification
simplicity, and there's no performance penalty for our primary optimization target use case.

> We could also explore something in-between, eg it'd be nice to
> genericize MultiLevelSkipListWriter so that it could index arbitrary
> files, then we could use that to index the terms dict. You could
> choose to spend dedicated process RAM on the higher levels of the skip
> tree, and then tentatively trust IO cache for the lower levels.

That doesn't meet the design goals of bringing the cost of opening/warming an IndexReader
down to near-zero and sharing backing buffers among multiple forks. It's also very complicated,
which of course bothers me more than it bothers you. ;) So I imagine we'll choose different

> I'd like to eventually make the TermsDict index pluggable so one could
> swap in different indexers like this (it's not now).

If we treat the term dictionary as a black box, it has to accept a term and return... a blob,
I guess.  Whatever calls the lookup needs to know how to handle that blob.

> Further steps towards flexible indexing
> ---------------------------------------
>                 Key: LUCENE-1458
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>    Affects Versions: 2.9
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1458.patch, LUCENE-1458.patch, LUCENE-1458.patch
> I attached a very rough checkpoint of my current patch, to get early
> feedback.  All tests pass, though back compat tests don't pass due to
> changes to package-private APIs plus certain bugs in tests that
> happened to work (eg call TermPostions.nextPosition() too many times,
> which the new API asserts against).
> [Aside: I think, when we commit changes to package-private APIs such
> that back-compat tests don't pass, we could go back, make a branch on
> the back-compat tag, commit changes to the tests to use the new
> package private APIs on that branch, then fix nightly build to use the
> tip of that branch?o]
> There's still plenty to do before this is committable! This is a
> rather large change:
>   * Switches to a new more efficient terms dict format.  This still
>     uses tii/tis files, but the tii only stores term & long offset
>     (not a TermInfo).  At seek points, tis encodes term & freq/prox
>     offsets absolutely instead of with deltas delta.  Also, tis/tii
>     are structured by field, so we don't have to record field number
>     in every term.
> .
>     On first 1 M docs of Wikipedia, tii file is 36% smaller (0.99 MB
>     -> 0.64 MB) and tis file is 9% smaller (75.5 MB -> 68.5 MB).
> .
>     RAM usage when loading terms dict index is significantly less
>     since we only load an array of offsets and an array of String (no
>     more TermInfo array).  It should be faster to init too.
> .
>     This part is basically done.
>   * Introduces modular reader codec that strongly decouples terms dict
>     from docs/positions readers.  EG there is no more TermInfo used
>     when reading the new format.
> .
>     There's nice symmetry now between reading & writing in the codec
>     chain -- the current docs/prox format is captured in:
> {code}
> FormatPostingsTermsDictWriter/Reader
> FormatPostingsDocsWriter/Reader (.frq file) and
> FormatPostingsPositionsWriter/Reader (.prx file).
> {code}
>     This part is basically done.
>   * Introduces a new "flex" API for iterating through the fields,
>     terms, docs and positions:
> {code}
> FieldProducer -> TermsEnum -> DocsEnum -> PostingsEnum
> {code}
>     This replaces TermEnum/Docs/Positions.  SegmentReader emulates the
>     old API on top of the new API to keep back-compat.
> Next steps:
>   * Plug in new codecs (pulsing, pfor) to exercise the modularity /
>     fix any hidden assumptions.
>   * Expose new API out of IndexReader, deprecate old API but emulate
>     old API on top of new one, switch all core/contrib users to the
>     new API.
>   * Maybe switch to AttributeSources as the base class for TermsEnum,
>     DocsEnum, PostingsEnum -- this would give readers API flexibility
>     (not just index-file-format flexibility).  EG if someone wanted
>     to store payload at the term-doc level instead of
>     term-doc-position level, you could just add a new attribute.
>   * Test performance & iterate.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message