drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Boaz Ben-Zvi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5446) Offset Vector in VariableLengthVectors may waste up to 256KB per value vector
Date Wed, 26 Apr 2017 21:52:04 GMT
Boaz Ben-Zvi created DRILL-5446:

             Summary: Offset Vector in VariableLengthVectors may waste up to 256KB per value
                 Key: DRILL-5446
                 URL: https://issues.apache.org/jira/browse/DRILL-5446
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Relational Operators
    Affects Versions: 1.10.0
            Reporter: Boaz Ben-Zvi
            Assignee: Boaz Ben-Zvi
             Fix For: 1.11.0

In exec/vector/src/main/codegen/templates/VariableLengthVectors.java -- the implementation
uses an "offset vector" to note the BEGINNING of each variable length element. In order to
find the length (i.e. the END of the element), need to look at the FOLLOWING element. 
  This requires the "offset vector" to have ONE MORE entry than the total number of elements
-- in order to find the END of the LAST element.
  Some places in the code (e.g., the hash table) use the maximum number of elements - 64K
( = 65536 ).  And each entry in the "offset vector" is 4-byte UInt4, hence looks like needing
  However because of that "ONE MORE", the code in this case allocates for 65537, thus (rounding
to next power of 2) allocating 512KB, where half is not used !!!! 
 (And this is per each varchar value vector, per each batch; e.g., in the qa test Functional/aggregates/tpcds_variants/text/aggregate25.q
where there are 10 key columns, each hash-table batch is wasting 2.5MB !).

Possible fix: change the logic in VariableLengthVectors.java to keep the END point of each
variable length element - the first element's beginning is always ZERO, so it need not be


This message was sent by Atlassian JIRA

View raw message