From issues-return-194678-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Oct 16 06:51:31 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2246E180674 for ; Tue, 16 Oct 2018 06:51:30 +0200 (CEST) Received: (qmail 3282 invoked by uid 500); 16 Oct 2018 04:51:29 -0000 Mailing-List: contact issues-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list issues@flink.apache.org Received: (qmail 3260 invoked by uid 99); 16 Oct 2018 04:51:29 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Oct 2018 04:51:29 +0000 From: GitBox To: issues@flink.apache.org Subject: [GitHub] zhijiangW commented on a change in pull request #6705: [FLINK-10356][network] add sanity checks to SpillingAdaptiveSpanningRecordDeserializer Message-ID: <153966548902.4244.14689037526967166442.gitbox@gitbox.apache.org> Date: Tue, 16 Oct 2018 04:51:29 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit zhijiangW commented on a change in pull request #6705: [FLINK-10356][network] add sanity checks to SpillingAdaptiveSpanningRecordDeserializer URL: https://github.com/apache/flink/pull/6705#discussion_r225396776 ########## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpillingAdaptiveSpanningRecordDeserializer.java ########## @@ -549,21 +584,53 @@ private void addNextChunkFromMemorySegment(MemorySegment segment, int offset, in } else { spillingChannel.close(); + spillingChannel = null; - BufferedInputStream inStream = new BufferedInputStream(new FileInputStream(spillFile), 2 * 1024 * 1024); + BufferedInputStream inStream = + new BufferedInputStream( + new FileInputStream(checkNotNull(spillFile)), + 2 * 1024 * 1024); this.spillFileReader = new DataInputViewStreamWrapper(inStream); } } } - private void moveRemainderToNonSpanningDeserializer(NonSpanningWrapper deserializer) { + private void moveRemainderToNonSpanningDeserializer(NonSpanningWrapper deserializer) throws IOException { + Optional deserializationError = getDeserializationError(0); + if (deserializationError.isPresent()) { + throw new IOException(deserializationError.get()); + } + deserializer.clear(); if (leftOverData != null) { deserializer.initializeFromMemorySegment(leftOverData, leftOverStart, leftOverLimit); } } + private Optional getDeserializationError(int addToReadBytes) { + Optional deserializationError = Optional.empty(); + int remainingSpanningBytes = 0, leftOverDataStart = 0, leftOverDataLimit = 0; + if (this.spillFileReader == null) { + remainingSpanningBytes = this.serializationReadBuffer.available() - addToReadBytes; + } else { + try { + remainingSpanningBytes = this.spillFileReader.available() - addToReadBytes; + } catch (IOException ignored) { + } + } + if (this.leftOverData != null) { Review comment: Because the following condition `remainingSpanningBytes != 0` is not always true, this condition is not always necessary. I am not sure whether it is worth putting this condition inside the below one. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services