drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5517) Provide size-aware set operations in value vectors
Date Thu, 29 Jun 2017 15:49:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068504#comment-16068504
] 

ASF GitHub Bot commented on DRILL-5517:
---------------------------------------

Github user amansinha100 commented on a diff in the pull request:

    https://github.com/apache/drill/pull/840#discussion_r124813604
  
    --- Diff: exec/vector/src/main/codegen/templates/NullableValueVectors.java ---
    @@ -674,6 +764,14 @@ public void reset(){
           setCount = 0;
           <#if type.major = "VarLen">lastSet = -1;</#if>
         }
    +
    +    @Override
    +    public void exchange(ValueVector.Mutator other) {
    --- End diff --
    
    How does swapping the value count exchange the buffer ?  The name 'exchange' indicates
more than what this method is doing.  The JIRA description says the 'exchange is for vectors
within a single operator'  but I am not sure how that is enforced considering the ValueVector.Mutator
that is passed in could belong to any ValueVector.  So, essentially we are trusting the caller
to use it only for specific cases. 


> Provide size-aware set operations in value vectors
> --------------------------------------------------
>
>                 Key: DRILL-5517
>                 URL: https://issues.apache.org/jira/browse/DRILL-5517
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.11.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>              Labels: ready-to-commit
>             Fix For: 1.11.0
>
>
> DRILL-5211 describes a memory fragmentation issue in Drill. The resolution is to limit
vector sizes to 16 MB (the size of Netty memory allocation "slabs.") Effort starts by providing
"size-aware" set operations in value vectors which:
> * Operate as {{setSafe()}} while vectors are below 16 MB.
> * Throw a new, specific exception ({{VectorOverflowException}}) if setting the value
(and growing the vector) would exceed the vector limit.
> The methods in value vectors then become the foundation on which we can construct size-aware
record batch "writers."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message