spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joshro...@apache.org
Subject spark git commit: [SPARK-4901] [SQL] Hot fix for ByteWritables.copyBytes
Date Fri, 19 Dec 2014 16:04:58 GMT
Repository: spark
Updated Branches:
  refs/heads/master 283263ffa -> 5479450c4


[SPARK-4901] [SQL] Hot fix for ByteWritables.copyBytes

HiveInspectors.scala failed in compiling with Hadoop 1, as the BytesWritable.copyBytes is
not available in Hadoop 1.

Author: Cheng Hao <hao.cheng@intel.com>

Closes #3742 from chenghao-intel/settable_oi_hotfix and squashes the following commits:

bb04d1f [Cheng Hao] hot fix for ByteWritables.copyBytes


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5479450c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5479450c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5479450c

Branch: refs/heads/master
Commit: 5479450c45038ddb4485cd676f945a961f2420d8
Parents: 283263f
Author: Cheng Hao <hao.cheng@intel.com>
Authored: Fri Dec 19 08:04:41 2014 -0800
Committer: Josh Rosen <joshrosen@databricks.com>
Committed: Fri Dec 19 08:04:41 2014 -0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/sql/hive/HiveInspectors.scala     | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/5479450c/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
index 0618934..e114747 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
@@ -297,7 +297,13 @@ private[hive] trait HiveInspectors {
       case x: ByteObjectInspector if x.preferWritable() => x.get(data)
       case x: HiveDecimalObjectInspector => HiveShim.toCatalystDecimal(x, data)
       case x: BinaryObjectInspector if x.preferWritable() =>
-        x.getPrimitiveWritableObject(data).copyBytes()
+        // BytesWritable.copyBytes() only available since Hadoop2
+        // In order to keep backward-compatible, we have to copy the
+        // bytes with old apis
+        val bw = x.getPrimitiveWritableObject(data)
+        val result = new Array[Byte](bw.getLength()) 
+        System.arraycopy(bw.getBytes(), 0, result, 0, bw.getLength())
+        result
       case x: DateObjectInspector if x.preferWritable() =>
         x.getPrimitiveWritableObject(data).get()
       // org.apache.hadoop.hive.serde2.io.TimestampWritable.set will reset current time object


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message