parquet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ziva...@apache.org
Subject parquet-mr git commit: PARQUET-1152: Parquet-thrift doesn't compile with Thrift 0.9.3
Date Thu, 09 Nov 2017 10:25:28 GMT
Repository: parquet-mr
Updated Branches:
  refs/heads/master 328c5deb0 -> 170cfa758


PARQUET-1152: Parquet-thrift doesn't compile with Thrift 0.9.3

Author: Nandor Kollar <nkollar@cloudera.com>

Closes #432 from nandorKollar/PARQUET-1152 and squashes the following commits:

fd578ec [Zoltan Ivanfi] Undo unrelated whitespace changes.
8bbcfad [Nandor Kollar] PARQUET-1152: Parquet-thrift doesn't compile with Thrift 0.9.3


Project: http://git-wip-us.apache.org/repos/asf/parquet-mr/repo
Commit: http://git-wip-us.apache.org/repos/asf/parquet-mr/commit/170cfa75
Tree: http://git-wip-us.apache.org/repos/asf/parquet-mr/tree/170cfa75
Diff: http://git-wip-us.apache.org/repos/asf/parquet-mr/diff/170cfa75

Branch: refs/heads/master
Commit: 170cfa758547b4d9be50058ad93cf60ce0da5564
Parents: 328c5de
Author: Nandor Kollar <nkollar@cloudera.com>
Authored: Thu Nov 9 11:16:09 2017 +0100
Committer: Zoltan Ivanfi <zi@cloudera.com>
Committed: Thu Nov 9 11:23:01 2017 +0100

----------------------------------------------------------------------
 .../hadoop/thrift/ThriftBytesWriteSupport.java  | 22 +-------------------
 1 file changed, 1 insertion(+), 21 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/parquet-mr/blob/170cfa75/parquet-thrift/src/main/java/org/apache/parquet/hadoop/thrift/ThriftBytesWriteSupport.java
----------------------------------------------------------------------
diff --git a/parquet-thrift/src/main/java/org/apache/parquet/hadoop/thrift/ThriftBytesWriteSupport.java
b/parquet-thrift/src/main/java/org/apache/parquet/hadoop/thrift/ThriftBytesWriteSupport.java
index f6f511b..ba46c84 100644
--- a/parquet-thrift/src/main/java/org/apache/parquet/hadoop/thrift/ThriftBytesWriteSupport.java
+++ b/parquet-thrift/src/main/java/org/apache/parquet/hadoop/thrift/ThriftBytesWriteSupport.java
@@ -123,28 +123,8 @@ public class ThriftBytesWriteSupport extends WriteSupport<BytesWritable>
{
     return thriftWriteSupport.init(configuration);
   }
 
-  private static boolean IS_READ_LENGTH_SETABLE = false;
-  static {
-    try {
-      TBinaryProtocol.class.getMethod("setReadLength", int.class);
-      IS_READ_LENGTH_SETABLE = true;
-    } catch (NoSuchMethodException e) {
-      IS_READ_LENGTH_SETABLE = false;
-    }
-  }
-
   private TProtocol protocol(BytesWritable record) {
-    TProtocol protocol = protocolFactory.getProtocol(new TIOStreamTransport(new ByteArrayInputStream(record.getBytes())));
-
-    /* Reduce the chance of OOM when data is corrupted. When readBinary is called on TBinaryProtocol,
it reads the length of the binary first,
-     so if the data is corrupted, it could read a big integer as the length of the binary
and therefore causes OOM to happen.
-     Currently this fix only applies to TBinaryProtocol which has the setReadLength defined.
-      */
-    if (IS_READ_LENGTH_SETABLE && protocol instanceof TBinaryProtocol) {
-      ((TBinaryProtocol)protocol).setReadLength(record.getLength());
-    }
-
-    return protocol;
+    return protocolFactory.getProtocol(new TIOStreamTransport(new ByteArrayInputStream(record.getBytes())));
   }
 
   @Override


Mime
View raw message