flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From m..@apache.org
Subject flink git commit: [FLINK-1635] remove Apache Thrift and Google Protobuf dependencies
Date Mon, 08 Jun 2015 09:21:02 GMT
Repository: flink
Updated Branches:
  refs/heads/master 3e9af8202 -> 804430bdf


[FLINK-1635] remove Apache Thrift and Google Protobuf dependencies

- remove Maven dependencies
- add docs to show how to register your own serializer with Kryo

This closes #794.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/804430bd
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/804430bd
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/804430bd

Branch: refs/heads/master
Commit: 804430bdf48a6593be278ac54707735119f9b43c
Parents: 3e9af82
Author: Maximilian Michels <mxm@apache.org>
Authored: Fri Jun 5 15:36:28 2015 +0200
Committer: Maximilian Michels <mxm@apache.org>
Committed: Mon Jun 8 11:20:13 2015 +0200

----------------------------------------------------------------------
 docs/apis/best_practices.md                     | 91 +++++++++++++++++++-
 flink-java/pom.xml                              | 34 --------
 .../typeutils/runtime/kryo/Serializers.java     | 28 ------
 .../main/resources/archetype-resources/pom.xml  |  4 -
 .../main/resources/archetype-resources/pom.xml  |  4 -
 pom.xml                                         |  1 -
 6 files changed, 89 insertions(+), 73 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/804430bd/docs/apis/best_practices.md
----------------------------------------------------------------------
diff --git a/docs/apis/best_practices.md b/docs/apis/best_practices.md
index 985737f..d8619d4 100644
--- a/docs/apis/best_practices.md
+++ b/docs/apis/best_practices.md
@@ -37,7 +37,7 @@ For example for specifying input and output sources (like paths or addresses),
a
 
 Since version 0.9 we are providing a simple utility called `ParameterTool` to provide at
least some basic tooling for solving these problems.
 
-Please note that you don't have to use the `ParameterTool` explained here. Other frameworks
such as [Commons CLI](https://commons.apache.org/proper/commons-cli/), 
+Please note that you don't have to use the `ParameterTool` explained here. Other frameworks
such as [Commons CLI](https://commons.apache.org/proper/commons-cli/),
 [argparse4j](http://argparse4j.sourceforge.net/) and others work well with Flink as well.
 
 
@@ -125,7 +125,7 @@ public static final class Tokenizer extends RichFlatMapFunction<String,
Tuple2<S
 	@Override
 	public void open(Configuration parameters) throws Exception {
 		parameters.getInteger("myInt", -1);
-		// .. do 
+		// .. do
 {% endhighlight %}
 
 
@@ -182,3 +182,90 @@ public static class CustomType extends Tuple11<String, String, ...,
String> {
 ~~~
 
 
+## Register a custom serializer for your Flink program
+
+If you use a custom type in your Flink program which cannot be serialized by the
+Flink type serializer, Flink falls back to using the generic Kryo
+serializer. You may register your own serializer or a serialization system like
+Google Protobuf or Apache Thrift with Kryo. To do that, simply register the type
+class and the serializer in the `ExecutionConfig` of your Flink program.
+
+
+{% highlight java %}
+final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+
+// register the class of the serializer as serializer for a type
+env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, MyCustomSerializer.class);
+
+// register an instance as serializer for a type
+MySerializer mySerializer = new MySerializer();
+env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, mySerializer);
+{% endhighlight %}
+
+Note that your custom serializer has to extend Kryo's Serializer class. In the
+case of Google Protobuf or Apache Thrift, this has already been done for
+you:
+
+{% highlight java %}
+
+final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+
+// register the Google Protobuf serializer with Kryo
+env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, ProtobufSerializer.class);
+
+// register the serializer included with Apache Thrift as the standard serializer
+// TBaseSerializer states it should be initalized as a default Kryo serializer
+env.getConfig().addDefaultKryoSerializer(MyCustomType.class, TBaseSerializer.class);
+
+{% endhighlight %}
+
+For the above example to work, you need to include the necessary dependencies in
+your Maven project file (pom.xml). In the dependency section, add the following
+for Apache Thrift:
+
+{% highlight xml %}
+
+<dependency>
+	<groupId>com.twitter</groupId>
+	<artifactId>chill-thrift</artifactId>
+	<version>0.5.2</version>
+</dependency>
+<!-- libthrift is required by chill-thrift -->
+<dependency>
+	<groupId>org.apache.thrift</groupId>
+	<artifactId>libthrift</artifactId>
+	<version>0.6.1</version>
+	<exclusions>
+		<exclusion>
+			<groupId>javax.servlet</groupId>
+			<artifactId>servlet-api</artifactId>
+		</exclusion>
+		<exclusion>
+			<groupId>org.apache.httpcomponents</groupId>
+			<artifactId>httpclient</artifactId>
+		</exclusion>
+	</exclusions>
+</dependency>
+
+{% endhighlight %}
+
+For Google Protobuf you need the following Maven dependency:
+
+{% highlight xml %}
+
+<dependency>
+	<groupId>com.twitter</groupId>
+	<artifactId>chill-protobuf</artifactId>
+	<version>0.5.2</version>
+</dependency>
+<!-- We need protobuf for chill-protobuf -->
+<dependency>
+	<groupId>com.google.protobuf</groupId>
+	<artifactId>protobuf-java</artifactId>
+	<version>2.5.0</version>
+</dependency>
+
+{% endhighlight %}
+
+
+Please adjust the versions of both libraries as needed.

http://git-wip-us.apache.org/repos/asf/flink/blob/804430bd/flink-java/pom.xml
----------------------------------------------------------------------
diff --git a/flink-java/pom.xml b/flink-java/pom.xml
index 8879803..fcedfa3 100644
--- a/flink-java/pom.xml
+++ b/flink-java/pom.xml
@@ -77,40 +77,6 @@ under the License.
 		</dependency>
 
 		<dependency>
-			<groupId>com.twitter</groupId>
-			<artifactId>chill-protobuf</artifactId>
-			<version>${chill.version}</version>
-		</dependency>
-		<!-- We need protobuf for chill-protobuf -->
-		<dependency>
-			<groupId>com.google.protobuf</groupId>
-			<artifactId>protobuf-java</artifactId>
-			<version>${protobuf.version}</version>
-		</dependency>
-
-		<dependency>
-			<groupId>com.twitter</groupId>
-			<artifactId>chill-thrift</artifactId>
-			<version>${chill.version}</version>
-		</dependency>
-		<!-- libthrift is required by chill-thrift -->
-		<dependency>
-			<groupId>org.apache.thrift</groupId>
-			<artifactId>libthrift</artifactId>
-			<version>0.6.1</version>
-			<exclusions>
-				<exclusion>
-					<groupId>javax.servlet</groupId>
-					<artifactId>servlet-api</artifactId>
-				</exclusion>
-				<exclusion>
-					<groupId>org.apache.httpcomponents</groupId>
-					<artifactId>httpclient</artifactId>
-				</exclusion>
-			</exclusions>
-		</dependency>
-
-		<dependency>
 			<groupId>de.javakaffee</groupId>
 			<artifactId>kryo-serializers</artifactId>
 			<version>0.27</version>

http://git-wip-us.apache.org/repos/asf/flink/blob/804430bd/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/Serializers.java
----------------------------------------------------------------------
diff --git a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/Serializers.java
b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/Serializers.java
index 36c2375..9e0da25 100644
--- a/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/Serializers.java
+++ b/flink-java/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/Serializers.java
@@ -22,9 +22,6 @@ import com.esotericsoftware.kryo.Serializer;
 import com.esotericsoftware.kryo.io.Input;
 import com.esotericsoftware.kryo.io.Output;
 import com.esotericsoftware.kryo.serializers.CollectionSerializer;
-import com.google.protobuf.Message;
-import com.twitter.chill.protobuf.ProtobufSerializer;
-import com.twitter.chill.thrift.TBaseSerializer;
 import de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer;
 import de.javakaffee.kryoserializers.jodatime.JodaIntervalSerializer;
 import org.apache.avro.Schema;
@@ -32,7 +29,6 @@ import org.apache.avro.generic.GenericData;
 import org.apache.avro.specific.SpecificRecordBase;
 import org.apache.flink.api.common.ExecutionConfig;
 import org.apache.flink.api.java.typeutils.TypeExtractor;
-import org.apache.thrift.protocol.TMessage;
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 
@@ -99,12 +95,6 @@ public class Serializers {
 	}
 
 	public static void addSerializerForType(ExecutionConfig reg, Class<?> type) {
-		if(Message.class.isAssignableFrom(type)) {
-			registerProtoBuf(reg);
-		}
-		if(TMessage.class.isAssignableFrom(type)) {
-			registerThrift(reg);
-		}
 		if(GenericData.Record.class.isAssignableFrom(type)) {
 			registerGenericAvro(reg);
 		}
@@ -117,24 +107,6 @@ public class Serializers {
 	}
 
 	/**
-	 * Register serializers required for Google Protocol Buffers
-	 * with Flink runtime.
-	 */
-	public static void registerProtoBuf(ExecutionConfig reg) {
-		// Google Protobuf (FLINK-1392)
-		reg.registerTypeWithKryoSerializer(Message.class, ProtobufSerializer.class);
-	}
-
-	/**
-	 * Register Apache Thrift messages
-	 */
-	public static void registerThrift(ExecutionConfig reg) {
-		// TBaseSerializer states it should be initalized as a default Kryo serializer
-		reg.addDefaultKryoSerializer(TMessage.class, TBaseSerializer.class);
-		reg.registerKryoType(TMessage.class);
-	}
-
-	/**
 	 * Register these serializers for using Avro's {@see GenericData.Record} and classes
 	 * implementing {@see org.apache.avro.specific.SpecificRecordBase}
 	 */

http://git-wip-us.apache.org/repos/asf/flink/blob/804430bd/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/pom.xml
----------------------------------------------------------------------
diff --git a/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/pom.xml
b/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/pom.xml
index 30f2315..3e26ece 100644
--- a/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/pom.xml
+++ b/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/pom.xml
@@ -164,10 +164,6 @@ under the License.
 									<exclude>com.twitter:chill-bijection_*</exclude>
 									<exclude>com.twitter:bijection-core_*</exclude>
 									<exclude>com.twitter:bijection-avro_*</exclude>
-									<exclude>com.twitter:chill-protobuf</exclude>
-									<exclude>com.google.protobuf:protobuf-java</exclude>
-									<exclude>com.twitter:chill-thrift</exclude>
-									<exclude>org.apache.thrift:libthrift</exclude>
 									<exclude>commons-lang:commons-lang</exclude>
 									<exclude>junit:junit</exclude>
 									<exclude>de.javakaffee:kryo-serializers</exclude>

http://git-wip-us.apache.org/repos/asf/flink/blob/804430bd/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml
----------------------------------------------------------------------
diff --git a/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml
b/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml
index e940b90..6fbb745 100644
--- a/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml
+++ b/flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/pom.xml
@@ -169,10 +169,6 @@ under the License.
 									<exclude>com.twitter:chill-bijection_*</exclude>
 									<exclude>com.twitter:bijection-core_*</exclude>
 									<exclude>com.twitter:bijection-avro_*</exclude>
-									<exclude>com.twitter:chill-protobuf</exclude>
-									<exclude>com.google.protobuf:protobuf-java</exclude>
-									<exclude>com.twitter:chill-thrift</exclude>
-									<exclude>org.apache.thrift:libthrift</exclude>
 									<exclude>commons-lang:commons-lang</exclude>
 									<exclude>junit:junit</exclude>
 									<exclude>de.javakaffee:kryo-serializers</exclude>

http://git-wip-us.apache.org/repos/asf/flink/blob/804430bd/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 83dddc8..4eaa766 100644
--- a/pom.xml
+++ b/pom.xml
@@ -83,7 +83,6 @@ under the License.
 		<akka.version>2.3.7</akka.version>
 		<scala.macros.version>2.0.1</scala.macros.version>
 		<kryoserialization.version>0.3.2</kryoserialization.version>
-		<protobuf.version>2.5.0</protobuf.version>
 		<chill.version>0.5.2</chill.version>
 		<asm.version>5.0.4</asm.version>
 		<tez.version>0.6.1</tez.version>


Mime
View raw message