drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Altekruse" <altekruseja...@gmail.com>
Subject Re: Review Request 35484: DRILL-2851: set an upper-bound on # of bytes to re-allocate to prevent overflows
Date Fri, 19 Jun 2015 17:06:18 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35484/#review88551
-----------------------------------------------------------



exec/java-exec/src/main/codegen/templates/FixedValueVectors.java (line 91)
<https://reviews.apache.org/r/35484/#comment141135>

    The reallocation loops will run infinitely if we have it the maximum buffer size. This
can be reproduced by reading a JSON file with a lot of large lists. This could be fixed by
re-intriducing the code we had before re-allocation, but only in the case where we hit one
of these limits. Allocations should be able to fail, and we need to make sure the operators
can handle this case.
    
    We should fail with an OOM error in these cases until we can split lists across batches
or find another way around the fact that the allocator tracks buffer lengths with ints.


- Jason Altekruse


On June 17, 2015, 6:54 p.m., Hanifi Gunes wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35484/
> -----------------------------------------------------------
> 
> (Updated June 17, 2015, 6:54 p.m.)
> 
> 
> Review request for drill, Mehant Baid and Venki Korukanti.
> 
> 
> Repository: drill-git
> 
> 
> Description
> -------
> 
> DRILL-2851: set an upper-bound on # of bytes to re-allocate to prevent overflows
> Vectors
> - set an upper bound on # of bytes to allocate
> - 
> TestValueVector.java	
> - Add unit tests
> 
> 
> Diffs
> -----
> 
>   exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 7103a17108693d47839212c418d11d13fbb8f6f4

>   exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java bd41e10d3f69e13d0f8c426460af5e9a09d93fd9

>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java ec409a3fc59616708226aa500ccab1680cd261f6

>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java 10bdf0752632c7577b9a6eb445c7101ec1a24730

>   exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java
037c8c6d3da94acf5c2ca300ce617338cacb0fb0 
> 
> Diff: https://reviews.apache.org/r/35484/diff/
> 
> 
> Testing
> -------
> 
> all
> 
> 
> Thanks,
> 
> Hanifi Gunes
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message