lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Field with reader limitation arbitrary
Date Tue, 15 Sep 2009 17:46:04 GMT

: Someone has made the decision that we will not be interested in
: storing files read using a Reader (at least not with these
: constructors).
: This is rather arbitrary.

No, it was not arbitrary at all.

The javadocs there are not a "decree" of what shall or shan't be 
supported, they are an explanation of how the constructor works so that 
there isn't any confusion.

The *reason* why the code works that way, is because when Lucene 
processes Fields that use a Reader, it doesn't buffer the Reader so it 
can't store the full contents of that Reader. the *reason* it doesnt' 
buffer the reader is because then it would have to make very arbitrary 
memory decisions that could easily result in OOM (Readers can in fact be
infinite streams of characters)

clients that want to store the value, need to be able to provide the 
entire value as a String.

: might want to also store files in the index,  having a queue of 1000
: Documents with 1000 Readers to files is vastly preferable to having
: 1000 documents with 1000 (perhaps very large) Strings with all the
: contents of the files. While this is not the best for all cases (#open

You've just pointed out exactly why it's not feasible for Lucene to store 
the contents of the Reader -- it would have to do the exact same thing you 
describe.  The curerent API leaves it up to the client to decide when to 
do this, and what to do if the Reader produces a String bigger then it 
wants to deal with.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message