From dev-return-59316-archive-asf-public=cust-asf.ponee.io@thrift.apache.org Tue Oct 6 19:34:02 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id ABD27180636 for ; Tue, 6 Oct 2020 21:34:02 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 260BD6496D for ; Tue, 6 Oct 2020 19:34:02 +0000 (UTC) Received: (qmail 78816 invoked by uid 500); 6 Oct 2020 19:34:01 -0000 Mailing-List: contact dev-help@thrift.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@thrift.apache.org Delivered-To: mailing list dev@thrift.apache.org Received: (qmail 78804 invoked by uid 99); 6 Oct 2020 19:34:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2020 19:34:01 +0000 Received: from jira2-he-de.apache.org (static.54.33.119.168.clients.your-server.de [168.119.33.54]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8368E40C96 for ; Tue, 6 Oct 2020 19:34:00 +0000 (UTC) Received: from jira2-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira2-he-de.apache.org (ASF Mail Server at jira2-he-de.apache.org) with ESMTP id 06D3AC800AA for ; Tue, 6 Oct 2020 19:34:00 +0000 (UTC) Date: Tue, 6 Oct 2020 19:34:00 +0000 (UTC) From: "David Mollitor (Jira)" To: dev@thrift.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (THRIFT-5288) Move Support for ByteBuffer into TTransport MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/THRIFT-5288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated THRIFT-5288: ----------------------------------- Summary: Move Support for ByteBuffer into TTransport (was: Move Support for ByteBuffer in TTransport) > Move Support for ByteBuffer into TTransport > ------------------------------------------- > > Key: THRIFT-5288 > URL: https://issues.apache.org/jira/browse/THRIFT-5288 > Project: Thrift > Issue Type: Improvement > Reporter: David Mollitor > Assignee: David Mollitor > Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > {code:java|title=TCompactProtocol.java} > /** > * Write a byte array, using a varint for the size. > */ > public void writeBinary(ByteBuffer bin) throws TException { > int length = bin.limit() - bin.position(); > writeBinary(bin.array(), bin.position() + bin.arrayOffset(), length); > } > {code} > I was working on something with Parquet and this code was causing some issues: > {code:java} > java.lang.Exception: java.nio.ReadOnlyBufferException > at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) > at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) > Caused by: java.nio.ReadOnlyBufferException > at java.nio.ByteBuffer.array(ByteBuffer.java:996) > at shaded.parquet.org.apache.thrift.protocol.TCompactProtocol.writeBinary(TCompactProtocol.java:375) > at org.apache.parquet.format.InterningProtocol.writeBinary(InterningProtocol.java:135) > at org.apache.parquet.format.ColumnIndex$ColumnIndexStandardScheme.write(ColumnIndex.java:945) > at org.apache.parquet.format.ColumnIndex$ColumnIndexStandardScheme.write(ColumnIndex.java:820) > at org.apache.parquet.format.ColumnIndex.write(ColumnIndex.java:728) > at org.apache.parquet.format.Util.write(Util.java:372) > at org.apache.parquet.format.Util.writeColumnIndex(Util.java:69) > at org.apache.parquet.hadoop.ParquetFileWriter.serializeColumnIndexes(ParquetFileWriter.java:1087) > at org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:1050) > {code} > This happens, because not all {{Buffer}} allow for direct access to the backing Array,... for example a ByteBuffer tied to a file does not have an Array. Read-only (immutable) {{ByteBuffer}} do not allow for this kind of access to the array since it could then be modified. > There are two approaches here: > # Assert and throw Exception if the backing array must be allowed for access > # Deal natively with the ByteBuffer > I propose the latter. However, the initial naive I approach I propose is to "deal natively" with the ByteBuffer by making a copy of the contents. -- This message was sent by Atlassian Jira (v8.3.4#803005)