drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5080) Create a memory-managed version of the External Sort operator
Date Thu, 02 Feb 2017 03:14:52 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849327#comment-15849327
] 

ASF GitHub Bot commented on DRILL-5080:
---------------------------------------

Github user Ben-Zvi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/717#discussion_r99045403
  
    --- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
    @@ -177,13 +177,47 @@ drill.exec: {
       sort: {
         purge.threshold : 1000,
         external: {
    -      batch.size : 4000,
    +      // Drill uses the managed External Sort Batch by default.
    +      // Set this to true to use the legacy, unmanaged version.
    +      // Disabled in the intial commit, to be enabled after
    +      // tests are committed.
    +      disable_managed: true
    +      // Limit on the number of batches buffered in memory.
    +      // Primarily for testing.
    +      // 0 = unlimited
    +      batch_limit: 0
    +      // Limit on the amount of memory used for xsort. Overrides the
    +      // value provided by Foreman. Primarily for testing.
    +      // 0 = unlimited, Supports HOCON memory suffixes.
    +      mem_limit: 0
    +      // Limit on the number of spilled batches that can be merged in
    +      // a single pass. Limits the number of open file handles.
    +      // 0 = unlimited
    +      merge_limit: 0
           spill: {
    -        batch.size : 4000,
    -        group.size : 40000,
    -        threshold : 40000,
    -        directories : [ "/tmp/drill/spill" ],
    -        fs : "file:///"
    +        // Deprecated for managed xsort; used only by legacy xsort
    +        group.size: 40000,
    +        // Deprecated for managed xsort; used only by legacy xsort
    +        threshold: 40000,
    +        // Minimum number of in-memory batches to spill per spill file
    +        // Affects only spilling from memory to disk.
    +        // Primarily for testing.
    +        min_batches: 2,
    +        // Maximum number of in-memory batches to spill per spill file
    +        // Affects only spilling from memory to disk.
    +        // Primarily for testing.
    +        // 0 = unlimited
    +        max_batches: 0,
    +        // File system to use. Local file system by default.
    +        fs: "file:///"
    +        // List of directories to use. Directories are created
    --- End diff --
    
    List of (comma separated) directories....


> Create a memory-managed version of the External Sort operator
> -------------------------------------------------------------
>
>                 Key: DRILL-5080
>                 URL: https://issues.apache.org/jira/browse/DRILL-5080
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>             Fix For: 1.10.0
>
>         Attachments: ManagedExternalSortDesign.pdf
>
>
> We propose to create a "managed" version of the external sort operator that works to
a clearly-defined memory limit. Attached is a design specification for the work.
> The project will include fixing a number of bugs related to the external sort, include
as sub-tasks of this umbrella task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message