drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-6230) Extend row set readers to handle hyper vectors
Date Sun, 01 Apr 2018 01:05:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421541#comment-16421541
] 

ASF GitHub Bot commented on DRILL-6230:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1161#discussion_r178445424
  
    --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/rowSet/HyperRowSetImpl.java
---
    @@ -45,8 +50,67 @@ public RowSetReader buildReader(HyperRowSet rowSet, SelectionVector4
sv4) {
           TupleMetadata schema = rowSet.schema();
           HyperRowIndex rowIndex = new HyperRowIndex(sv4);
           return new RowSetReaderImpl(schema, rowIndex,
    -          buildContainerChildren(rowSet.container(),
    -              new MetadataRetrieval(schema)));
    +          buildContainerChildren(rowSet.container(), schema));
    +    }
    +  }
    +
    +  public static class HyperRowSetBuilderImpl implements HyperRowSetBuilder {
    +
    +    private final BufferAllocator allocator;
    +    private final List<VectorContainer> batches = new ArrayList<>();
    +    private int totalRowCount;
    +
    +    public HyperRowSetBuilderImpl(BufferAllocator allocator) {
    +      this.allocator = allocator;
    +    }
    +
    +    @Override
    +    public void addBatch(SingleRowSet rowSet) {
    +      if (rowSet.rowCount() == 0) {
    +        return;
    +      }
    +      if (rowSet.indirectionType() != SelectionVectorMode.NONE) {
    +        throw new IllegalArgumentException("Batches must not have a selection vector.");
    +      }
    +      batches.add(rowSet.container());
    +      totalRowCount += rowSet.rowCount();
    +    }
    +
    +    @Override
    +    public void addBatch(VectorContainer container) {
    +      if (container.getRecordCount() == 0) {
    +        return;
    +      }
    +      if (container.getSchema().getSelectionVectorMode() != SelectionVectorMode.NONE)
{
    +        throw new IllegalArgumentException("Batches must not have a selection vector.");
    +      }
    +      batches.add(container);
    +      totalRowCount += container.getRecordCount();
    +    }
    +
    +    @SuppressWarnings("resource")
    +    @Override
    +    public HyperRowSet build() throws SchemaChangeException {
    +      SelectionVector4 sv4 = new SelectionVector4(allocator, totalRowCount);
    +      ExpandableHyperContainer hyperContainer = new ExpandableHyperContainer();
    +      for (VectorContainer container : batches) {
    +        hyperContainer.addBatch(container);
    +      }
    +
    +      // TODO: This has a bug. If the hyperset has two batches with unions,
    +      // and the first union contains only VARCHAR, while the second contains
    --- End diff --
    
    [DRILL-6304](https://issues.apache.org/jira/browse/DRILL-6304)


> Extend row set readers to handle hyper vectors
> ----------------------------------------------
>
>                 Key: DRILL-6230
>                 URL: https://issues.apache.org/jira/browse/DRILL-6230
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Major
>             Fix For: 1.14.0
>
>
> The current row set readers have incomplete support for hyper-vectors. To add full support,
we need an interface that supports either single batches or hyper batches. Accessing vectors
in hyper batches differs depending on whether the vector is at the top level or is nested.
SeeĀ [this post|https://github.com/paul-rogers/drill/wiki/BH-Column-Readers] for details.
Also includes a simpler reader template: replaces the original three classes with one, in
parallel with the writers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message