lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Glen Newton <glen.new...@gmail.com>
Subject Re: Field with reader limitation arbitrary
Date Tue, 15 Sep 2009 12:32:49 GMT
OK, thanks. :-)

Glen

2009/9/14 Anthony Urso <anthony.urso@gmail.com>:
> It's best to file a feature request on the Lucene issue tracker if you
> are interested in seeing this implemented.
>
> http://issues.apache.org/jira/browse/LUCENE
>
> Just cut and paste your description and attach a patch and/or tests if
> you have them.
>
> Cheers,
> Anthony
>
> On Mon, Sep 14, 2009 at 1:03 PM, Glen Newton <glen.newton@gmail.com> wrote:
>> Hi,
>>
>> In 2.4.1, Field has 2 constructors that involve a Reader:
>> public Field(String name,
>>                  Reader reader)
>> public Field(String name,
>>                  Reader reader,
>>                  Field.TermVector termVector)
>>
>> http://lucene.apache.org/java/2_4_1/api/core/org/apache/lucene/document/Field.html#Field(java.lang.String,%20java.io.Reader,%20org.apache.lucene.document.Field.TermVector)
>>
>> The Reader references a text file on the filesystem. These
>> constructors do the following:
>> "Create a tokenized and indexed field that is not stored, optionally
>> with storing term vectors. The Reader is read only when the Document
>> is added to the index, i.e. you may not close the Reader until
>> IndexWriter.addDocument(Document)  has been called."
>>
>> Someone has made the decision that we will not be interested in
>> storing files read using a Reader (at least not with these
>> constructors).
>> This is rather arbitrary.
>> As someone who has massively parralelized my indexing AND sometimes
>> might want to also store files in the index,  having a queue of 1000
>> Documents with 1000 Readers to files is vastly preferable to having
>> 1000 documents with 1000 (perhaps very large) Strings with all the
>> contents of the files. While this is not the best for all cases (#open
>> file handles, etc), this is a use case which would benefit from being
>> able to do this (i.e. reduced memory footprint, especially for large
>> files or large queues).
>>
>> Suggestion: replace or add a constructor with:
>> public Field(String name,
>>             Reader reader,
>>             Field.Store store,
>>             Field.Index index,
>>             Field.TermVector termVector)
>>
>> Constructively,
>>
>> Glen Newton
>>  http://zzzoot.blogspot.com/
>>  http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>>
>> -
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>



-- 

-

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message