drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-5008) Refactor, document and simplify ExternalSortBatch
Date Mon, 07 Nov 2016 18:02:59 GMT

     [ https://issues.apache.org/jira/browse/DRILL-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Paul Rogers updated DRILL-5008:
-------------------------------
    Description: 
ExternalSortBatch provides a spillable sort operator for Drill. The code works fine, but can
be a hard to follow and understand. Make the following changes to improve ease-of-use for
developers:

1. Refactor the large methods into bite-sized chunks to aid understanding.
2. Provide additional explanation of the theory and operation of this operator.

The memory limit cases for the spill-needed test seem inconsistent:

For the test for in-memory sort:
{code}
    long currentlyAvailable =  popConfig.getMaxAllocation() - oAllocator.getAllocatedMemory();
{code}

For reaching the memory limit:
{code}
oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
{code}

That is, one uses `oAllocator.getLimit` ("the current maximum limit this allocator imposes"),
the other uses `popConfig.getMaxAllocation` ("The maximum memory this operator can allocate".)

  was:
ExternalSortBatch provides a spillable sort operator for Drill. The code works fine, but can
be a hard to follow and understand. Make the following changes to improve ease-of-use for
developers:

1. Refactor the large methods into bite-sized chunks to aid understanding.
2. Provide additional explanation of the theory and operation of this operator.

The memory limit cases for the spill-needed test seem inconsistent:

For the test for in-memory sort:
{code}
    long currentlyAvailable =  popConfig.getMaxAllocation() - oAllocator.getAllocatedMemory();
{code}

For reaching the memory limit:
{code}
oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
{code}

That is, one uses `oAllocator.getLimit` ("the current maximum limit this allocator imposes"),
the other uses `popConfig.getMaxAllocation` (not documented, but the memory limit for the
operator as defined by the Foreman.)


> Refactor, document and simplify ExternalSortBatch
> -------------------------------------------------
>
>                 Key: DRILL-5008
>                 URL: https://issues.apache.org/jira/browse/DRILL-5008
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Priority: Minor
>
> ExternalSortBatch provides a spillable sort operator for Drill. The code works fine,
but can be a hard to follow and understand. Make the following changes to improve ease-of-use
for developers:
> 1. Refactor the large methods into bite-sized chunks to aid understanding.
> 2. Provide additional explanation of the theory and operation of this operator.
> The memory limit cases for the spill-needed test seem inconsistent:
> For the test for in-memory sort:
> {code}
>     long currentlyAvailable =  popConfig.getMaxAllocation() - oAllocator.getAllocatedMemory();
> {code}
> For reaching the memory limit:
> {code}
> oAllocator.getAllocatedMemory() > .95 * oAllocator.getLimit()
> {code}
> That is, one uses `oAllocator.getLimit` ("the current maximum limit this allocator imposes"),
the other uses `popConfig.getMaxAllocation` ("The maximum memory this operator can allocate".)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message