drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Padma Penumarthy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-6113) Limit batch size for Merge Receiver
Date Fri, 26 Jan 2018 22:21:00 GMT
Padma Penumarthy created DRILL-6113:
---------------------------------------

             Summary: Limit batch size for Merge Receiver
                 Key: DRILL-6113
                 URL: https://issues.apache.org/jira/browse/DRILL-6113
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Flow
    Affects Versions: 1.12.0
            Reporter: Padma Penumarthy
            Assignee: Padma Penumarthy
             Fix For: 1.13.0


Merge receiver has hard coded limit of 32K rows as batch. Since rows can be of varying width,
it is difficult to predict the output batch size (in terms of memory) for this operator. Change
this to derive row count based on actual memory available. We are introducing a new option
called outputBatchSize to limit the batch size of each operator. Use the memory configured
from that. Figure out the average row width of outgoing batch based on averages of batches
coming from incoming streams. Limit the row count based on memory available to use and average
row width.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message