tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Travis Woodruff (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3757) Integer overflow in PipelinedSorter
Date Fri, 09 Jun 2017 14:53:18 GMT
Travis Woodruff created TEZ-3757:

             Summary: Integer overflow in PipelinedSorter
                 Key: TEZ-3757
                 URL: https://issues.apache.org/jira/browse/TEZ-3757
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.8.4, 0.8.5
            Reporter: Travis Woodruff

This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to the {{SortSpan}}

      //TODO: fix per item being passed.
      span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(), (1024*1024),
          perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));

{{SortSpan}}'s constructor then calculates {{dataSize}} as follows:

      int dataSize = maxItems * perItem;

This means that if {{perItem}} is >= 2040, {{dataSize}} overflows, which (usually?) ends
up causing the capacity check to not work correctly, which causes subsequent buffer operations
to fail.

This message was sent by Atlassian JIRA

View raw message