Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Wed, 2 Aug 2017 20:29:00 +0000 (UTC)
From: "Eugene Kirpichov (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13091996.1501705695000.77736.1501705740301@Atlassian.JIRA>
In-Reply-To: <JIRA.13091996.1501705695000@Atlassian.JIRA>
References: <JIRA.13091996.1501705695000@Atlassian.JIRA> <JIRA.13091996.1501705695882@jira-lw-us.apache.org>
Subject: [jira] [Created] (BEAM-2716) AvroReader should refuse dynamic
 splits while in the last block
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 02 Aug 2017 20:29:06 -0000

Eugene Kirpichov created BEAM-2716:
--------------------------------------

             Summary: AvroReader should refuse dynamic splits while in the last block
                 Key: BEAM-2716
                 URL: https://issues.apache.org/jira/browse/BEAM-2716
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-core
            Reporter: Eugene Kirpichov
            Assignee: Eugene Kirpichov
            Priority: Minor


AvroReader is able to detect when it's in the last block:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L728

It could also use this information to avoid wastefully producing dynamic splits starting in the range of the current block.

One way to do this would be to have OffsetRangeTracker have a "claim range" operation: claim range of [a, b) is, in terms of correctness, equivalent to claiming "a" (it checks whether "a" is within the range), but sets the last claimed position to "b" rather than "a", thus protecting more positions from being split away.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)