drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5080) Create a memory-managed version of the External Sort operator
Date Wed, 01 Feb 2017 04:04:53 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847957#comment-15847957
] 

ASF GitHub Bot commented on DRILL-5080:
---------------------------------------

Github user Ben-Zvi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/717#discussion_r98576202
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
    @@ -63,13 +63,29 @@
       String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
       String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
       String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
    +
    +  // External Sort Boot configuration
    +
       String EXTERNAL_SORT_TARGET_BATCH_SIZE = "drill.exec.sort.external.batch.size";
       String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = "drill.exec.sort.external.spill.batch.size";
       String EXTERNAL_SORT_SPILL_GROUP_SIZE = "drill.exec.sort.external.spill.group.size";
       String EXTERNAL_SORT_SPILL_THRESHOLD = "drill.exec.sort.external.spill.threshold";
       String EXTERNAL_SORT_SPILL_DIRS = "drill.exec.sort.external.spill.directories";
       String EXTERNAL_SORT_SPILL_FILESYSTEM = "drill.exec.sort.external.spill.fs";
    +  String EXTERNAL_SORT_SPILL_FILE_SIZE = "drill.exec.sort.external.spill.file_size";
       String EXTERNAL_SORT_MSORT_MAX_BATCHSIZE = "drill.exec.sort.external.msort.batch.maxsize";
    +  String EXTERNAL_SORT_DISABLE_MANAGED = "drill.exec.sort.external.disable_managed";
    +  String EXTERNAL_SORT_MERGE_LIMIT = "drill.exec.sort.external.merge_limit";
    +  String EXTERNAL_SORT_MIN_SPILL = "drill.exec.sort.external.spill.min_batches";
    +  String EXTERNAL_SORT_MAX_SPILL = "drill.exec.sort.external.spill.max_batches";
    +  String EXTERNAL_SORT_MAX_MEMORY = "drill.exec.sort.external.mem_limit";
    +  String EXTERNAL_SORT_BATCH_LIMIT = "drill.exec.sort.external.batch_limit";
    +
    +  // External Sort Runtime options
    +
    +  BooleanValidator EXTERNAL_SORT_DISABLE_MANAGED_OPTION = new BooleanValidator("exec.sort.disable_managed",
false);
    --- End diff --
    
    "false" means "enabled", but the Jira says that initially the managed feature would be
disabled by default.



> Create a memory-managed version of the External Sort operator
> -------------------------------------------------------------
>
>                 Key: DRILL-5080
>                 URL: https://issues.apache.org/jira/browse/DRILL-5080
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>             Fix For: 1.10
>
>         Attachments: ManagedExternalSortDesign.pdf
>
>
> We propose to create a "managed" version of the external sort operator that works to
a clearly-defined memory limit. Attached is a design specification for the work.
> The project will include fixing a number of bugs related to the external sort, include
as sub-tasks of this umbrella task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message