drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5517) Provide size-aware set operations in value vectors
Date Thu, 29 Jun 2017 15:49:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068503#comment-16068503
] 

ASF GitHub Bot commented on DRILL-5517:
---------------------------------------

Github user amansinha100 commented on a diff in the pull request:

    https://github.com/apache/drill/pull/840#discussion_r124833987
  
    --- Diff: exec/vector/src/main/codegen/templates/FixedValueVectors.java ---
    @@ -546,238 +576,400 @@ public DateTime getObject(int index) {
         public ${minor.javaType!type.javaType} getPrimitiveObject(int index) {
           return get(index);
         }
    -    </#if>
     
    +    </#if>
         public void get(int index, ${minor.class}Holder holder){
           <#if minor.class.startsWith("Decimal")>
           holder.scale = getField().getScale();
           holder.precision = getField().getPrecision();
           </#if>
     
    -      holder.value = data.get${(minor.javaType!type.javaType)?cap_first}(index * ${type.width});
    +      holder.value = data.get${(minor.javaType!type.javaType)?cap_first}(index * VALUE_WIDTH);
         }
     
         public void get(int index, Nullable${minor.class}Holder holder){
           holder.isSet = 1;
    -      holder.value = data.get${(minor.javaType!type.javaType)?cap_first}(index * ${type.width});
    +      holder.value = data.get${(minor.javaType!type.javaType)?cap_first}(index * VALUE_WIDTH);
         }
         </#if> <#-- type.width -->
    - }
    -
    - /**
    -  * ${minor.class}.Mutator implements a mutable vector of fixed width values.  Elements
in the
    -  * vector are accessed by position from the logical start of the vector.  Values should
be pushed
    -  * onto the vector sequentially, but may be randomly accessed.
    -  *   The width of each element is ${type.width} byte(s)
    -  *   The equivalent Java primitive is '${minor.javaType!type.javaType}'
    -  *
    -  * NB: this class is automatically generated from ValueVectorTypes.tdd using FreeMarker.
    -  */
    -  public final class Mutator extends BaseDataValueVector.BaseMutator {
    -
    -    private Mutator(){};
    -   /**
    -    * Set the element at the given index to the given value.  Note that widths smaller
than
    -    * 32 bits are handled by the DrillBuf interface.
    -    *
    -    * @param index   position of the bit to set
    -    * @param value   value to set
    -    */
    +  }
    +
    +  /**
    +   * ${minor.class}.Mutator implements a mutable vector of fixed width values.  Elements
in the
    +   * vector are accessed by position from the logical start of the vector.  Values should
be pushed
    +   * onto the vector sequentially, but may be randomly accessed.
    +   * <ul>
    +   * <li>The width of each element is {@link #VALUE_WIDTH} (= ${type.width}) byte(s).</li>
    +   * <li>The equivalent Java primitive is '${minor.javaType!type.javaType}'</li>
    +   * </ul>
    +   *
    +   * NB: this class is automatically generated from ValueVectorTypes.tdd using FreeMarker.
    +   */
    +   public final class Mutator extends BaseDataValueVector.BaseMutator {
    +
    +    private Mutator() {};
    +
    +    /**
    +     * Set the element at the given index to the given value.  Note that widths smaller
than
    +     * 32 bits are handled by the DrillBuf interface.
    +     *
    +     * @param index   position of the bit to set
    +     * @param value   value to set
    +     */
    +
       <#if (type.width > 8)>
         public void set(int index, <#if (type.width > 4)>${minor.javaType!type.javaType}<#else>int</#if>
value) {
    -      data.setBytes(index * ${type.width}, value, 0, ${type.width});
    +      data.setBytes(index * VALUE_WIDTH, value, 0, VALUE_WIDTH);
         }
     
         public void setSafe(int index, <#if (type.width > 4)>${minor.javaType!type.javaType}<#else>int</#if>
value) {
           while(index >= getValueCapacity()) {
             reAlloc();
           }
    -      data.setBytes(index * ${type.width}, value, 0, ${type.width});
    +      data.setBytes(index * VALUE_WIDTH, value, 0, VALUE_WIDTH);
         }
     
    -  <#if (minor.class == "Interval")>
    -    public void set(int index, int months, int days, int milliseconds){
    -      final int offsetIndex = index * ${type.width};
    -      data.setInt(offsetIndex, months);
    -      data.setInt((offsetIndex + ${minor.daysOffset}), days);
    -      data.setInt((offsetIndex + ${minor.millisecondsOffset}), milliseconds);
    +    /**
    +     * Set the value of a required or nullable vector. Enforces the value
    +     * and size limits.
    +     * @param index item to write
    +     * @return true if the item was written, false if the index would
    --- End diff --
    
    Method returns a void but comment says the method returns a boolean..


> Provide size-aware set operations in value vectors
> --------------------------------------------------
>
>                 Key: DRILL-5517
>                 URL: https://issues.apache.org/jira/browse/DRILL-5517
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.11.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>              Labels: ready-to-commit
>             Fix For: 1.11.0
>
>
> DRILL-5211 describes a memory fragmentation issue in Drill. The resolution is to limit
vector sizes to 16 MB (the size of Netty memory allocation "slabs.") Effort starts by providing
"size-aware" set operations in value vectors which:
> * Operate as {{setSafe()}} while vectors are below 16 MB.
> * Throw a new, specific exception ({{VectorOverflowException}}) if setting the value
(and growing the vector) would exceed the vector limit.
> The methods in value vectors then become the foundation on which we can construct size-aware
record batch "writers."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message