drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arina Ielchiieva (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-4823) Fix OOM while trying to prune partitions with reasonable data size
Date Wed, 03 Aug 2016 09:31:20 GMT
Arina Ielchiieva created DRILL-4823:

             Summary: Fix OOM while trying to prune partitions with reasonable data size
                 Key: DRILL-4823
                 URL: https://issues.apache.org/jira/browse/DRILL-4823
             Project: Apache Drill
          Issue Type: Bug
          Components: Functions - Drill, Query Planning & Optimization
    Affects Versions: 1.6.0
            Reporter: Arina Ielchiieva
            Assignee: Arina Ielchiieva
             Fix For: 1.8.0

_Example query:_
select  '/'||dir0||'/'||dir1||'/'||dir2||'/'||dir3 , count(*)
  FROM dfs.`/path/to/parquet/files`
  WHERE ('/'||dir0||'/'||dir1||'/'||dir2||'/'||dir3)  IN ( 
  group by   '/'||dir0||'/'||dir1||'/'||dir2||'/'||dir3
  order by 1;

OOM while trying to prune partitions:
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 256
due to memory limit. Current allocation: 5242880
	at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) ~[drill-memory-base-1.6.0.jar:1.6.0]
	at org.apache.drill.exec.ops.BufferManagerImpl.getManagedBuffer(BufferManagerImpl.java:60)
	at org.apache.drill.exec.ops.BufferManagerImpl.getManagedBuffer(BufferManagerImpl.java:56)
	at org.apache.drill.exec.ops.QueryContext.getManagedBuffer(QueryContext.java:241) ~[drill-java-exec-1.6.0.jar:1.6.0]
	at org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.getManagedBufferIfAvailable(InterpreterEvaluator.java:158)

Interpreter always asks for a new buffer to hold varchar/varbinary or decimal constant values.
That's why the memory size required would be proportion to # of constant expressions multiplied
by # of input rows (partition). This is different from evaluation from run-time generated
where constant expression will be evaluated once and use only one buffer per value.

To use one buffer for each unique constant value in query.

This message was sent by Atlassian JIRA

View raw message