lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrien Grand <>
Subject Re: Indexing directly from stdin in lucene 3.5
Date Tue, 19 Feb 2013 23:50:24 GMT

On Tue, Feb 19, 2013 at 11:04 AM, A. L. Benhenni <> wrote:
> I am currently writing an indexer class to index texts from stdin. I also
> need the text to be tokenized and stored to access the termvector of the
> document.

Actually, you don't need to store documents to access their term
vectors, these are two different options. The stored fields allow you
to retrieve data as you provided it to the IndexWriter while term
vectors return a single-document inverted index of your document
(mapping every unique term to its frequency, the positions where it
appeared in the original document, etc.).

> 1/ Is there a more appropriate way of handling the indexing of an incoming
> stream ?

Actually, your example is very strange since (if I'm not mistaken)
each iteration of the loop overrides the previous line with the
current one (because path_field_name doesn't change).

If you want your document to be stored (Store.YES), you need to buffer
everything into a String before feeding Lucene with it.

> 2/ Is there an easy way to clean the index ?


> And a subsidiary 3/ Why a field can't store a reader ?

Lucene stores string fields by first writing their length, followed by
their bytes, so it couldn't even start serializing a Reader before
having consumed and buffered it fully (to know its length). Lucene
doesn't allow the creation of stored fields from a Reader because it
would give the impression of being lighweight (no need to load
everything into memory at once) although it wouldn't be. On the
contrary, you can provide a Reader to a field which is indexed and has
term vectors turned on and Lucene will manage to consume it in a truly
streaming fashion.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message