drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5517) Provide size-aware set operations in value vectors
Date Thu, 29 Jun 2017 15:49:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068505#comment-16068505
] 

ASF GitHub Bot commented on DRILL-5517:
---------------------------------------

Github user amansinha100 commented on a diff in the pull request:

    https://github.com/apache/drill/pull/840#discussion_r124706419
  
    --- Diff: exec/vector/src/main/codegen/templates/FixedValueVectors.java ---
    @@ -31,68 +31,98 @@
     import org.apache.drill.exec.util.DecimalUtility;
     
     /**
    - * ${minor.class} implements a vector of fixed width values.  Elements in the vector
are accessed
    - * by position, starting from the logical start of the vector.  Values should be pushed
onto the
    - * vector sequentially, but may be randomly accessed.
    - *   The width of each element is ${type.width} byte(s)
    - *   The equivalent Java primitive is '${minor.javaType!type.javaType}'
    + * ${minor.class} implements a vector of fixed width values. Elements in the vector are
accessed
    + * by position, starting from the logical start of the vector. Values should be pushed
onto the
    + * vector sequentially, but may be accessed randomly.
    + * <ul>
    + * <li>The width of each element is {@link #VALUE_WIDTH} (= ${type.width}) byte<#if
type.width != 1>s</#if>.</li>
    + * <li>The equivalent Java primitive is '${minor.javaType!type.javaType}'.</li>
    + * </ul>
      *
      * NB: this class is automatically generated from ${.template_name} and ValueVectorTypes.tdd
using FreeMarker.
      */
    -public final class ${minor.class}Vector extends BaseDataValueVector implements FixedWidthVector{
    +public final class ${minor.class}Vector extends BaseDataValueVector implements FixedWidthVector
{
       private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(${minor.class}Vector.class);
     
    +  /**
    +   * Width of each fixed-width value.
    +   */
    +
    +  public static final int VALUE_WIDTH = ${type.width};
    +
    +  /**
    +   * Maximum number of values that this fixed-width vector can hold
    +   * and stay below the maximum vector size limit. This is the limit
    +   * enforced when the vector is used to hold values in a repeated
    +   * vector.
    +   */
    +
    +  public static final int MAX_VALUE_COUNT = MAX_BUFFER_SIZE / VALUE_WIDTH;
    +
    +  /**
    +   * Maximum number of values that this fixed-width vector can hold
    +   * and stay below the maximum vector size limit and/or stay below
    +   * the maximum row count. This is the limit enforced when the
    +   * vector is used to hold scalar (required or nullable) values.
    +   */
    +
    +  public static final int MAX_SCALAR_COUNT = Math.min(MAX_ROW_COUNT, MAX_VALUE_COUNT);
    --- End diff --
    
    Where is the constant MAX_ROW_COUNT defined ?   (later I saw it defined in the ValueVector
interface...perhaps add a comment here)


> Provide size-aware set operations in value vectors
> --------------------------------------------------
>
>                 Key: DRILL-5517
>                 URL: https://issues.apache.org/jira/browse/DRILL-5517
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.11.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>              Labels: ready-to-commit
>             Fix For: 1.11.0
>
>
> DRILL-5211 describes a memory fragmentation issue in Drill. The resolution is to limit
vector sizes to 16 MB (the size of Netty memory allocation "slabs.") Effort starts by providing
"size-aware" set operations in value vectors which:
> * Operate as {{setSafe()}} while vectors are below 16 MB.
> * Throw a new, specific exception ({{VectorOverflowException}}) if setting the value
(and growing the vector) would exceed the vector limit.
> The methods in value vectors then become the foundation on which we can construct size-aware
record batch "writers."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message