lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC
Date Fri, 04 May 2018 21:05:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464394#comment-16464394
] 

David Smiley commented on SOLR-12312:
-------------------------------------

While I look at this a bit more in my IDE, I wonder if we even need this buf at all.  Notice
that {{fetchPackets}} accepts {{FastInputStream fis}} and that's what we read from.  FIS
contains an accessible {{byte[]}} internally; it's partially why FIS exists in the first place
(vs some generic InputStream).  fetchPackets could be recoded to have a loop within the packet
reading to use FIS's buffer directly.  That would not be an abuse; we're just sending data
along to other places (to a file and some checksum calculator thingy). The net effect is one
less buffer to allocate and copy data to.

That said, I so easily become guilty of scope creep and this is extra work with more intricacies
than the fairly simple change we have on sizing the buf correctly.

> Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC
> ----------------------------------------------------------------------------------
>
>                 Key: SOLR-12312
>                 URL: https://issues.apache.org/jira/browse/SOLR-12312
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: replication (java)
>            Reporter: David Smiley
>            Assignee: David Smiley
>            Priority: Minor
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> IndexFetcher's constructor knows the size of the file it's going to transfer.  As-such,
it ought to initialize the "buf" field to no larger than this size.  This has been shown
to waste Java heap/GC in  an environment with lots of cores of small indexes and thus small
files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message