drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Phillips" <sphill...@maprtech.com>
Subject Re: Review Request 29723: DRILL-1960: Automatically realloc buffers when vector runs out of space
Date Tue, 20 Jan 2015 02:42:47 GMT


> On Jan. 17, 2015, 2:38 a.m., Jason Altekruse wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java,
line 621
> > <https://reviews.apache.org/r/29723/diff/1/?file=813525#file813525line621>
> >
> >     Doesn't this line ensure we have a positive index into the hash table 'array'?

If you look at the code in getBucketIndex(), it is already guaranteed to be positive.


> On Jan. 17, 2015, 2:38 a.m., Jason Altekruse wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/ChainedHashTable.java,
line 164
> > <https://reviews.apache.org/r/29723/diff/1/?file=813523#file813523line164>
> >
> >     This is duplicated frequently, I know that it might be a hard to find a centeralized
place for this to live, but it would be better to encapsulate it so we can make future changes
in a single location.

I will look into this.


- Steven


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29723/#review68495
-----------------------------------------------------------


On Jan. 8, 2015, 8:35 p.m., Steven Phillips wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29723/
> -----------------------------------------------------------
> 
> (Updated Jan. 8, 2015, 8:35 p.m.)
> 
> 
> Review request for drill and Jacques Nadeau.
> 
> 
> Repository: drill-git
> 
> 
> Description
> -------
> 
> When reaching the end of a buffer, the current way drill handles this is for setSafe()
to return false, and then whatever operator is doing the write will send the batch, and the
redo the last record. This creates a lot of difficulty, because it sometimes requires being
able to "rewind" the input stream to replay the last record.
> The proposal is to move the handling of buffer sizing and allocation into the value vectors
themselves, making it transparent to the user of the value vector. The operators will now
no longer have to worry about the possibility that writing into a vector may fail due to lack
of space.
> 
> 
> Diffs
> -----
> 
>   contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
33bf376 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java
82e038c 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveRecordReader.java
625a7b2 
>   contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTextRecordReader.java
5406048 
>   contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java
79abe60 
>   exec/java-exec/src/main/codegen/templates/ComplexReaders.java 027f61d 
>   exec/java-exec/src/main/codegen/templates/ComplexWriters.java 5ba1c64 
>   exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 58e6ccc 
>   exec/java-exec/src/main/codegen/templates/MapWriters.java b8bd73e 
>   exec/java-exec/src/main/codegen/templates/NullableValueVectors.java b222024 
>   exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java d261050 
>   exec/java-exec/src/main/codegen/templates/TypeHelper.java c83c301 
>   exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java b8ffe5d 
>   exec/java-exec/src/main/java/org/apache/drill/exec/expr/EvaluationVisitor.java 5cf4a35

>   exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillAggFuncHolder.java
6e0b282 
>   exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillComplexWriterFuncHolder.java
df56174 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
113e883 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
d7cf904 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java
ef85a36 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggTemplate.java
556b260 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/ChainedHashTable.java
0502f7e 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java
e8ccd62 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java
5b56f8e 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java
e82dd29 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java
5deb67f 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinProbe.java
7599f9e 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinProbeTemplate.java
c58f9a3 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinTemplate.java
c1dffc1 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java
d0f9d7d 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingReceiverGeneratorBase.java
2885c52 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingReceiverTemplate.java
c29ef75 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java
acbb755 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionProjectorTemplate.java
f5068b4 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
20f6195 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java
fa983aa 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectorTemplate.java
49ad390 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/CopierTemplate2.java
5cc308a 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/CopierTemplate4.java
c42332d 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameRecordBatch.java
a3e7940 
>   exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameTemplate.java
b4e3fed 
>   exec/java-exec/src/main/java/org/apache/drill/exec/record/TransferPair.java 9645be9

>   exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedWidthRepeatedReader.java
0c4437a 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/NullableBitReader.java
b9b808b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java
83f9bde 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthValuesColumn.java
7c4b33b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoRecordReader.java
241fa95 
>   exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoWriter.java 0ffa55c

>   exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/Writers.java fee011a

>   exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java
e0cce8b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java 51726a3

>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java d2523c5 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/CopyUtil.java 1ed7f37 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedFixedWidthVector.java
aadc563 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedMutator.java ad2ba1b

>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/VariableWidthVector.java
ff3ee63 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java cc3d24c

>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java
362d806 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java
e140c8b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedListReaderImpl.java
c60730c 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedMapReaderImpl.java
15f8a2b 
>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/SingleMapReaderImpl.java
3ec66ff 
>   exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/TestWindowFrame.java
7c04477 
>   exec/java-exec/src/test/java/org/apache/drill/exec/vector/TestAdaptiveAllocation.java
ebc4df7 
> 
> Diff: https://reviews.apache.org/r/29723/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Steven Phillips
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message