lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2252) stored field retrieve slow
Date Sun, 07 Feb 2010 04:14:27 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830642#action_12830642
] 

Robert Muir commented on LUCENE-2252:
-------------------------------------

bq. In modern machines (at least with the machines we are using, e.g. a macbook pro)

its not really subjective, or based on modern machines. you are talking about 5M documents,
some indexes have a lot more documents and 4bytes/doc in ram adds up to a lot!
for the case of using lucene as a search engine library, this memory could be better spent
on other things.
I dont think this is subjective, because its a search engine library, not a document store.

bq. Furthermore, with the question at hand, even if we do use Directory implementation Uwe
suggested, it is not optimal.

but it is easy, and takes away your disk seek. the "in-memory seek, read/parse" is as you
say, peanuts in comparison.


> stored field retrieve slow
> --------------------------
>
>                 Key: LUCENE-2252
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2252
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>    Affects Versions: 3.0
>            Reporter: John Wang
>
> IndexReader.document() on a stored field is rather slow. Did a simple multi-threaded
test and profiled it:
> 40+% time is spent in getting the offset from the index file
> 30+% time is spent in reading the count (e.g. number of fields to load)
> Although I ran it on my lap top where the disk isn't that great, but still seems to be
much room in improvement, e.g. load field index file into memory (for a 5M doc index, the
extra memory footprint is 20MB, peanuts comparing to other stuff being loaded)
> A related note, are there plans to have custom segments as part of flexible indexing
feature?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message