From notifications-return-612-archive-asf-public=cust-asf.ponee.io@nemo.apache.org Fri Jun 21 04:11:14 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5A3CA180670 for ; Fri, 21 Jun 2019 06:11:14 +0200 (CEST) Received: (qmail 17757 invoked by uid 500); 21 Jun 2019 04:10:34 -0000 Mailing-List: contact notifications-help@nemo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nemo.apache.org Delivered-To: mailing list notifications@nemo.apache.org Received: (qmail 17599 invoked by uid 99); 21 Jun 2019 04:09:56 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jun 2019 04:09:56 +0000 From: GitBox To: notifications@nemo.apache.org Subject: [GitHub] [incubator-nemo] hy00nc commented on a change in pull request #222: [NEMO-350] Implement Off-heap SerializedMemoryStore & [NEMO-384] Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore Message-ID: <156109019614.3788.9377591688608007255.gitbox@gitbox.apache.org> Date: Fri, 21 Jun 2019 04:09:56 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit hy00nc commented on a change in pull request #222: [NEMO-350] Implement Off-heap SerializedMemoryStore & [NEMO-384] Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore URL: https://github.com/apache/incubator-nemo/pull/222#discussion_r296089729 ########## File path: runtime/executor/src/main/java/org/apache/nemo/runtime/executor/data/block/FileBlock.java ########## @@ -83,11 +96,13 @@ public FileBlock(final String blockId, */ private void writeToFile(final Iterable> serializedPartitions) throws IOException { - try (FileOutputStream fileOutputStream = new FileOutputStream(filePath, true)) { + try (FileChannel fileOutputChannel = new FileOutputStream(filePath, true).getChannel()) { for (final SerializedPartition serializedPartition : serializedPartitions) { // Reserve a partition write and get the metadata. metadata.writePartitionMetadata(serializedPartition.getKey(), serializedPartition.getLength()); - fileOutputStream.write(serializedPartition.getData(), 0, serializedPartition.getLength()); + for (final ByteBuffer buffer: serializedPartition.getBuffer()) { + fileOutputChannel.write(buffer); Review comment: The previous implementation using the `DirectByteArrayOutputStream` included invalid bytes in the `serializedPartition` so we needed to specify the valid length in the data to only fetch the valid data. Data that is written with `ByteBuffer`s using `DirectByteArrayOutputStream` has a `limit` which automatically stops reading the data when it is not valid even without specifying the length, so we no longer need one. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services