lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4515) Make MemoryIndex more memory efficient
Date Thu, 01 Nov 2012 13:43:12 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488694#comment-13488694
] 

Simon Willnauer commented on LUCENE-4515:
-----------------------------------------

ok I will try to summarize this again. I have 2 issues. 

1. reduce the GC load on MemIndex if it is used extensively - that is why I moved to ByteBlockPool
/ IntBlockPool
2. control memory consumption / buffering in a very flexible way. When buffers are reused
and how many of them. 

I don't give a fuck about highlighting at this point. I want to solve the issue with mem index
that exists today and if rob wants to solve highlighing ok fine go for it but that is unrelated.
I already removed all the changes to WeightedSpanTermExtractor in the latest patch. I really
really don't see the big problems here to allow expert users to have more control over it.
This patch fixes the 1. problem for everybody and allows folks with problem 2. to make use
of allocators yet with lots of effort (subclass + same package). This makes robs 2. concern
invalid really.
                
> Make MemoryIndex more memory efficient
> --------------------------------------
>
>                 Key: LUCENE-4515
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4515
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: modules/other
>    Affects Versions: 4.0, 4.1, 5.0
>            Reporter: Simon Willnauer
>             Fix For: 4.1, 5.0
>
>         Attachments: LUCENE-4515.patch, LUCENE-4515.patch
>
>
> Currently MemoryIndex uses BytesRef objects to represent terms and holds an int[] per
term per field to represent postings. For highlighting this creates a ton of objects for each
search that 1. need to be GCed and 2. can't be reused.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message