accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-580) Make size of batch scanner client size buffer configurable
Date Mon, 21 May 2012 22:23:41 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280556#comment-13280556
] 

Keith Turner commented on ACCUMULO-580:
---------------------------------------

The intent of this fixed size buffer is to limit memory usage.  I do not think it does a very
good job at that.  I am thinking this queue should be of unlimited size, however each batch
scanner thread should be limited to two active batches.   So a batch scanner thread would
fetch a batch and throw the entire batch on the queue.  It could then go fetch another batch.
 The thread would then block until at least one of the previous two batches was consumed.
 This would limit memory usage to 2*BatchSizeByte*NumThreads.  The current default batch size
is 1MB, this could be set to 512K to approximate current memory usage.  

Allowing two batches per thread should allow for more parallelism and limit memory usage.

The current fixed size buffer of 1000 entries could lead to obnoxious memory usage if large
key values are returned.  This is the reason I think its not a suitable solution for managing
memory.


                
> Make size of batch scanner client size buffer configurable
> ----------------------------------------------------------
>
>                 Key: ACCUMULO-580
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-580
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client
>            Reporter: Keith Turner
>            Assignee: Billie Rinaldi
>
> The batch scanner has a buffer on the client side where results read from tservers are
stored.  This buffer holds 1000 entries and it not configurable by the client.  When it fills
up, all threads reading from tservers block.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message