drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paul-rogers <...@git.apache.org>
Subject [GitHub] drill pull request #928: DRILL-5716: Queue-driven memory allocation
Date Fri, 06 Oct 2017 03:25:01 GMT
Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/928#discussion_r143088158
  
    --- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
    @@ -392,10 +419,12 @@ drill.exec.options:  {
         exec.query_profile.debug_mode: false,
         exec.query_profile.save: true,
         exec.queue.enable: false,
    -    exec.queue.large: 10,
    -    exec.queue.small: 100,
    +    exec.queue.large: 2,
    --- End diff --
    
    As for the smaller values... Drill ships with a default of 8 GB heap, and allows 2 GB
per query per node by default. Drill only has enough memory to run 3 "normal-sized" queries.
These new numbers are, in fact, wildly optimistic that, in that configuration, we can run
10 small and two large queries. We assume that each will get (8 GB * .8) / 30 = ~210 MB per
small query, about 2 GB per large query.
    
    In fact, in explaining this, I realized that the small queue number is still too large
for the default install and have reduced the numbers to large = 2, small = 4. Memory for small
is now (8 GB * 0.8) / 15 = ~430 MB, with large now ~ 2 GB. This maps a "large" query to the
default memory-per-query-per-node.
    
    Granted, the 20% overhead is a SWAG, we have evidence that it is sometimes larger. Fixing
that is another project...


---

Mime
View raw message