drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4411) HashJoin should not only depend on number of records, but also on size
Date Thu, 18 Feb 2016 17:40:18 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152698#comment-15152698
] 

ASF GitHub Bot commented on DRILL-4411:
---------------------------------------

GitHub user minji-kim opened a pull request:

    https://github.com/apache/drill/pull/381

    DRILL-4411: hash join should limit batch based on size and number of records

    Right now, hash joins can run out of memory if records are large since the batch is limited
only by size (of 4000).  This patch implements a simple heuristic.  If the allocator for the
outputs become larger than 10 MB before outputing 4000 records (say 2000), then set the batch
size limit to 2000 for the future batches.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/minji-kim/drill DRILL-4411

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/381.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #381
    
----
commit 2e3b1c75273e1b87679d79bdc4f3877b72603e3c
Author: Minji Kim <minji@dremio.com>
Date:   2016-02-18T17:05:51Z

    DRILL-4411: hash join should limit batch based on size as well as number of records

----


> HashJoin should not only depend on number of records, but also on size
> ----------------------------------------------------------------------
>
>                 Key: DRILL-4411
>                 URL: https://issues.apache.org/jira/browse/DRILL-4411
>             Project: Apache Drill
>          Issue Type: Bug
>          Components:  Server
>            Reporter: MinJi Kim
>            Assignee: MinJi Kim
>
> In HashJoinProbeTemplate, each batch is limited to TARGET_RECORDS_PER_BATCH (4000). 
But we should not only depend on the number of records, but also size (in case of extremely
large records).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message