hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin" <vanzin+...@cloudera.com>
Subject Re: Review Request 28779: [spark-client] Netty-based RPC implementation.
Date Mon, 08 Dec 2014 21:54:29 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28779/
-----------------------------------------------------------

(Updated Dec. 8, 2014, 9:54 p.m.)


Review request for hive, Brock Noland, chengxiang li, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-9036
    https://issues.apache.org/jira/browse/HIVE-9036


Repository: hive-git


Description
-------

This patch replaces akka with a simple netty-based RPC layer. It doesn't add any features
on top of the existing spark-client API, which is unchanged (except for the need to add empty
constructors in some places).

With the new backend we can think about adding some nice features such as future listeners
(which were awkward with akka because of Scala), but those are left for a different time.

The full change set, with more detailed descriptions, can be seen here:
https://github.com/vanzin/hive/commits/spark-client-netty


Diffs (updated)
-----

  pom.xml 630b10ce35032e4b2dee50ef3dfe5feb58223b78 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/RemoteSparkJobStatus.java PRE-CREATION

  spark-client/pom.xml PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/ClientUtils.java PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/JobHandleImpl.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/Protocol.java PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/SparkClientFactory.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/rpc/KryoMessageCodec.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/rpc/README.md PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java PRE-CREATION 
  spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcConfiguration.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcException.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcServer.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/counter/SparkCounter.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/counter/SparkCounterGroup.java PRE-CREATION

  spark-client/src/main/java/org/apache/hive/spark/counter/SparkCounters.java PRE-CREATION

  spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java PRE-CREATION

  spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestKryoMessageCodec.java PRE-CREATION

  spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java PRE-CREATION 

Diff: https://reviews.apache.org/r/28779/diff/


Testing
-------

spark-client unit tests, plus some qtests.


Thanks,

Marcelo Vanzin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message