drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5344) External sort priority queue copier fails with an empty batch
Date Sat, 11 Mar 2017 01:38:04 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905988#comment-15905988
] 

ASF GitHub Bot commented on DRILL-5344:
---------------------------------------

Github user Ben-Zvi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/778#discussion_r105520793
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/PriorityQueueCopierTemplate.java
---
    @@ -57,9 +57,12 @@ public void setup(FragmentContext context, BufferAllocator allocator,
VectorAcce
     
         queueSize = 0;
         for (int i = 0; i < size; i++) {
    -      vector4.set(i, i, batchGroups.get(i).getNextIndex());
    -      siftUp();
    -      queueSize++;
    +      int index = batchGroups.get(i).getNextIndex();
    +      vector4.set(i, i, index);
    --- End diff --
    
    If the record batch is empty, why set this (65535) value in the vector ?
    Probably usually the empty batch is last, so queueSize would not be incremented; otherwise
if the next iteration brings a non empty batch, then siftUp() may run into that 65535 ....
(not sure ...)



> External sort priority queue copier fails with an empty batch
> -------------------------------------------------------------
>
>                 Key: DRILL-5344
>                 URL: https://issues.apache.org/jira/browse/DRILL-5344
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>             Fix For: 1.11.0
>
>
> The external sort uses a "priority queue copier" to merge batches when spilling or when
merging spilled batches.
> The code will fail with an {{IndexOutOfBoundsException}} if any record batch is empty.
The reason is a faulty assumption in generated code:
> {code}
>   public void setup(...) {
>     ...
>       vector4.set(i, i, batchGroups.get(i).getNextIndex());
>     ...
>   }
>   public int getNextIndex() {
>     if (pointer == getRecordCount()) {
>       return -1;
>     }
>     ...
>   }
> {code}
> The code to get the next index returns -1 when the "position" in a record batch is zero.
The -1 position translates (when truncated) into 65535 which produces the index exception.
> The workaround has been to special case empty batches elsewhere in the code, apparently
to avoid hitting this error.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message