drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hanifi Gunes" <hanifigu...@gmail.com>
Subject Re: Review Request 35484: DRILL-2851: set an upper-bound on # of bytes to re-allocate to prevent overflows
Date Fri, 19 Jun 2015 19:00:30 GMT


> On June 19, 2015, 5:06 p.m., Jason Altekruse wrote:
> > exec/java-exec/src/main/codegen/templates/FixedValueVectors.java, line 92
> > <https://reviews.apache.org/r/35484/diff/1/?file=985297#file985297line92>
> >
> >     The reallocation loops will run infinitely if we have it the maximum buffer
size. This can be reproduced by reading a JSON file with a lot of large lists. This could
be fixed by re-intriducing the code we had before re-allocation, but only in the case where
we hit one of these limits. Allocations should be able to fail, and we need to make sure the
operators can handle this case.
> >     
> >     We should fail with an OOM error in these cases until we can split lists across
batches or find another way around the fact that the allocator tracks buffer lengths with
ints.
> 
> Jason Altekruse wrote:
>     *hit the maximum buffer size,   *re-introducing the code

I doubt if it is a good idea to fail with OOM. Instead we should consider setting the upper
bound to max allowed by the allocator (Int.MAX) and ensure that this invariant holds across
vector impls.


- Hanifi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35484/#review88551
-----------------------------------------------------------


On June 17, 2015, 6:54 p.m., Hanifi Gunes wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35484/
> -----------------------------------------------------------
> 
> (Updated June 17, 2015, 6:54 p.m.)
> 
> 
> Review request for drill, Mehant Baid and Venki Korukanti.
> 
> 
> Repository: drill-git
> 
> 
> Description
> -------
> 
> DRILL-2851: set an upper-bound on # of bytes to re-allocate to prevent overflows
> Vectors
> - set an upper bound on # of bytes to allocate
> - 
> TestValueVector.java	
> - Add unit tests
> 
> 
> Diffs
> -----
> 
>   exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 7103a17108693d47839212c418d11d13fbb8f6f4

>   exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java bd41e10d3f69e13d0f8c426460af5e9a09d93fd9

>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java ec409a3fc59616708226aa500ccab1680cd261f6

>   exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java 10bdf0752632c7577b9a6eb445c7101ec1a24730

>   exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestValueVector.java
037c8c6d3da94acf5c2ca300ce617338cacb0fb0 
> 
> Diff: https://reviews.apache.org/r/35484/diff/
> 
> 
> Testing
> -------
> 
> all
> 
> 
> Thanks,
> 
> Hanifi Gunes
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message