drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Phillips" <sphill...@maprtech.com>
Subject Re: Review Request 29723: DRILL-1960: Automatically realloc buffers when vector runs out of space
Date Tue, 20 Jan 2015 02:44:29 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29723/
-----------------------------------------------------------

(Updated Jan. 20, 2015, 2:44 a.m.)


Review request for drill, Hanifi Gunes and Jacques Nadeau.


Repository: drill-git


Description
-------

When reaching the end of a buffer, the current way drill handles this is for setSafe() to
return false, and then whatever operator is doing the write will send the batch, and the redo
the last record. This creates a lot of difficulty, because it sometimes requires being able
to "rewind" the input stream to replay the last record.
The proposal is to move the handling of buffer sizing and allocation into the value vectors
themselves, making it transparent to the user of the value vector. The operators will now
no longer have to worry about the possibility that writing into a vector may fail due to lack
of space.


Diffs
-----

  contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
33bf376 
  contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java
82e038c 
  contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveRecordReader.java
625a7b2 
  contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTextRecordReader.java
5406048 
  contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java
79abe60 
  exec/java-exec/src/main/codegen/templates/ComplexReaders.java 027f61d 
  exec/java-exec/src/main/codegen/templates/ComplexWriters.java 5ba1c64 
  exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 58e6ccc 
  exec/java-exec/src/main/codegen/templates/MapWriters.java b8bd73e 
  exec/java-exec/src/main/codegen/templates/NullableValueVectors.java b222024 
  exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java d261050 
  exec/java-exec/src/main/codegen/templates/TypeHelper.java c83c301 
  exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java b8ffe5d 
  exec/java-exec/src/main/java/org/apache/drill/exec/expr/EvaluationVisitor.java 5cf4a35 
  exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillAggFuncHolder.java 6e0b282

  exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillComplexWriterFuncHolder.java
df56174 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
113e883 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
d7cf904 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java
ef85a36 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggTemplate.java
556b260 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/ChainedHashTable.java
0502f7e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java e8ccd62

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java
5b56f8e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java
e82dd29 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java
5deb67f 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinProbe.java
7599f9e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinProbeTemplate.java
c58f9a3 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinTemplate.java
c1dffc1 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java
d0f9d7d 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingReceiverGeneratorBase.java
2885c52 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingReceiverTemplate.java
c29ef75 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java
acbb755 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionProjectorTemplate.java
f5068b4 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
20f6195 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
fa983aa 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectorTemplate.java
49ad390 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/CopierTemplate2.java
5cc308a 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/CopierTemplate4.java
c42332d 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameRecordBatch.java
a3e7940 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameTemplate.java
b4e3fed 
  exec/java-exec/src/main/java/org/apache/drill/exec/record/TransferPair.java 9645be9 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedWidthRepeatedReader.java
0c4437a 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/NullableBitReader.java
b9b808b 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java
83f9bde 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthValuesColumn.java
7c4b33b 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoRecordReader.java 241fa95

  exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoWriter.java 0ffa55c 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/Writers.java fee011a 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java
e0cce8b 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java 51726a3

  exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java d2523c5 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/CopyUtil.java 1ed7f37 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedFixedWidthVector.java
aadc563 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedMutator.java ad2ba1b 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/VariableWidthVector.java ff3ee63

  exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java cc3d24c

  exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java
362d806 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java
e140c8b 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedListReaderImpl.java
c60730c 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedMapReaderImpl.java
15f8a2b 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/SingleMapReaderImpl.java
3ec66ff 
  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/TestWindowFrame.java
7c04477 
  exec/java-exec/src/test/java/org/apache/drill/exec/vector/TestAdaptiveAllocation.java ebc4df7


Diff: https://reviews.apache.org/r/29723/diff/


Testing
-------


Thanks,

Steven Phillips


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message