drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "abdelhakim deneche" <adene...@gmail.com>
Subject Re: Review Request 33829: DRILL-2757: Verify operators correctly handle low memory conditions and cancellations
Date Wed, 06 May 2015 19:10:16 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33829/
-----------------------------------------------------------

(Updated May 6, 2015, 7:10 p.m.)


Review request for drill, Hanifi Gunes, Jacques Nadeau, and Steven Phillips.


Changes
-------

fixed formatting in FixedValueVectors.java


Bugs: DRILL-2757
    https://issues.apache.org/jira/browse/DRILL-2757


Repository: drill-git


Description
-------

includes:
[DRILL-2893](https://issues.apache.org/jira/browse/DRILL-2893): ScanBatch throws a NullPointerException
instead of returning OUT_OF_MEMORY
[DRILL-2894](https://issues.apache.org/jira/browse/DRILL-2894): FixedValueVectors shouldn't
set it's data buffer to null when it fails to allocate it
[DRILL-2895](https://issues.apache.org/jira/browse/DRILL-2895): AbstractRecordBatch.buildSchema()
should properly handle OUT_OF_MEMORY outcome
[DRILL-2905](https://issues.apache.org/jira/browse/DRILL-2905): RootExec implementations should
properly handle IterOutcome.OUT_OF_MEMORY
[DRILL-2920](https://issues.apache.org/jira/browse/DRILL-2920): properly handle OutOfMemoryException
[DRILL-2947](https://issues.apache.org/jira/browse/DRILL-2947): AllocationHelper.allocateNew()
doesn't have a consistent behavior when it can't allocate

also:
- improved how system errors are displayed
- added UserException.memoryError() with a pre assigned error message
- injection site in ScanBatch and unit test that runs various tpch queries and injects
  an exception in the ScanBatch that will cause an OUT_OF_MEMORY outcome to be sent


Diffs (updated)
-----

  common/src/main/java/org/apache/drill/common/exceptions/ErrorHelper.java 4da4ee8 
  common/src/main/java/org/apache/drill/common/exceptions/UserException.java 9283339 
  common/src/test/java/org/apache/drill/common/exceptions/TestUserException.java a145f95 
  exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 6a924b7 
  exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java 8a4b663 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java b108924 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java 4700dbd

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScreenCreator.java 5b4d7bd

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java
67062f3 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/TopNBatch.java 9f6bea9

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/WriterRecordBatch.java
15fb7b5 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
b753574 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
1b90dd8 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java
c1c5cb9 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggTemplate.java
86f3100 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java
d2282c8 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/FilterTemplate2.java
26f2657 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java
dd53477 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java
6466f70 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/NestedLoopJoinBatch.java
d20bfa1 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/limit/LimitRecordBatch.java
eff9e61 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java
c36b0d3 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
cf7ba16 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/producer/ProducerConsumerBatch.java
35bf3cd 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
7b9fffb 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/sort/SortBatch.java 74b7d85

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/RemovingRecordBatch.java
aa9297e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/trace/TraceRecordBatch.java
af45815 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java
d7ea3bb 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java
86d11d5 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
e88bc67 
  exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java 4e348bb

  exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java 7c77ca2

  exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java 2fc5bf3 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java d678cc5 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java ddb828c

  exec/java-exec/src/test/java/org/apache/drill/TestOutOfMemoryOutcome.java PRE-CREATION 

Diff: https://reviews.apache.org/r/33829/diff/


Testing
-------

added new test class "TestOutOfMemory" that runs several tpch queries and injects an OutOfMemoryException
in the ScanBatch. This causes an OUT_OF_MEMORY outcome to be propagated through the execution
stack.

all unit tests are passing along with regression/tpch100


Thanks,

abdelhakim deneche


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message