drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venki Korukanti (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-2178) Update outgoing record batch size and allocation in PartitionSender
Date Fri, 06 Feb 2015 22:00:36 GMT

     [ https://issues.apache.org/jira/browse/DRILL-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Venki Korukanti updated DRILL-2178:
    Fix Version/s: 0.8.0

> Update outgoing record batch size and allocation in PartitionSender
> -------------------------------------------------------------------
>                 Key: DRILL-2178
>                 URL: https://issues.apache.org/jira/browse/DRILL-2178
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: Venki Korukanti
>            Assignee: Venki Korukanti
>             Fix For: 0.8.0
>         Attachments: 0001-DRILL-2178-Update-outgoing-record-batch-size-and-all.patch
> Currently we allocate memory for vectors in partition sender OutgoingRecordBatch using
allocateNew() which for most ValueVectors allocates space for 4096 record capacity, but we
flush the current record batch as soon as we reach 1000 records causing wasted memory. Automatic
resizing kicks in after flushing few record batches, but auto resize always doubles or halves
the capacity. This cause the buffer record capacity to flip between  512 and 2048.
> This JIRA is to:
> 1. Decide on the outgoing record batch depending upon the number of receivers of partition
sender. Default value is 1024, but when the number of receivers exceeds 1000 change it to
> 2. Allocate value vector space for storing the outgoing record batch size decided in
(1). For this we make use of {{AllocationHelper.allocate(ValueVector v, int valueCount, int
bytesPerValue, int repeatedPerTop)}}. {{bytesPerValue}} and {{repeatedPerTop}} is currently
hard coded to 50 and 10, but this shouldn't matter as these values are applicable for variable
and repeated vectors which have realloc facility if they run out of space.

This message was sent by Atlassian JIRA

View raw message