Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 057A9200BB3 for ; Wed, 2 Nov 2016 22:39:46 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 04357160B0F; Wed, 2 Nov 2016 21:39:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 097B3160AFB for ; Wed, 2 Nov 2016 22:39:44 +0100 (CET) Received: (qmail 89341 invoked by uid 500); 2 Nov 2016 21:39:44 -0000 Mailing-List: contact commits-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kudu.apache.org Delivered-To: mailing list commits@kudu.apache.org Received: (qmail 89311 invoked by uid 99); 2 Nov 2016 21:39:44 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Nov 2016 21:39:44 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E6E10E0FC4; Wed, 2 Nov 2016 21:39:43 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: todd@apache.org To: commits@kudu.apache.org Date: Wed, 02 Nov 2016 21:39:44 -0000 Message-Id: In-Reply-To: <86bd5cd21af645cab89e0320c1a46194@git.apache.org> References: <86bd5cd21af645cab89e0320c1a46194@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [2/6] kudu git commit: [kudu-spark] Allow build against Spark 2.0 with Scala 2.11 archived-at: Wed, 02 Nov 2016 21:39:46 -0000 [kudu-spark] Allow build against Spark 2.0 with Scala 2.11 This patch adds build profiles to build an artifact for Spark 2.0 with Scala 2.11. There was one tiny code change to package.scala required to support Spark 2.0. To handle this, the different versions of the package.scala reside in different directories and the build profile includes the proper one. The two artifactIds are - kudu-spark_2.10 - kudu-spark2_2.11 The Jenkins job builds and tests both the default and spark2_2.11 profiles. Change-Id: Ie055ac1e8f7be7cb992124afa0b601376db14f91 Reviewed-on: http://gerrit.cloudera.org:8080/4886 Tested-by: Kudu Jenkins Reviewed-by: Jean-Daniel Cryans Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/e46e9518 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/e46e9518 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/e46e9518 Branch: refs/heads/master Commit: e46e9518b12471efdd65e3a989e05d4ffbbc96c9 Parents: 3a3f714 Author: Will Berkeley Authored: Sun Oct 30 18:48:38 2016 -0400 Committer: Will Berkeley Committed: Wed Nov 2 18:15:54 2016 +0000 ---------------------------------------------------------------------- build-support/jenkins/build-and-test.sh | 7 ++++ java/README.md | 14 ++++++++ java/kudu-spark/pom.xml | 33 ++++++++++++++++- .../org/apache/kudu/spark/kudu/package.scala | 38 -------------------- .../org/apache/kudu/spark/kudu/package.scala | 38 ++++++++++++++++++++ .../org/apache/kudu/spark/kudu/package.scala | 38 ++++++++++++++++++++ 6 files changed, 129 insertions(+), 39 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kudu/blob/e46e9518/build-support/jenkins/build-and-test.sh ---------------------------------------------------------------------- diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh index ca27217..6448e17 100755 --- a/build-support/jenkins/build-and-test.sh +++ b/build-support/jenkins/build-and-test.sh @@ -350,6 +350,13 @@ if [ "$BUILD_JAVA" == "1" ]; then EXIT_STATUS=1 FAILURES="$FAILURES"$'Java build/test failed\n' fi + # Test kudu-spark with Spark 2.x + Scala 2.11 profile + # This won't work if there are ever Spark integration tests! + rm -rf kudu-spark/target/ + if ! mvn $MVN_FLAGS -Pspark2_2.11 -Dtest="org.apache.kudu.spark.kudu.*" test; then + EXIT_STATUS=1 + FAILURES="$FAILURES"$'Java build/test failed\n' + fi set +x popd fi http://git-wip-us.apache.org/repos/asf/kudu/blob/e46e9518/java/README.md ---------------------------------------------------------------------- diff --git a/java/README.md b/java/README.md index 345e030..ac45186 100644 --- a/java/README.md +++ b/java/README.md @@ -51,6 +51,20 @@ may be run with: $ mvn verify +Building the Kudu-Spark integration for Spark 2.x with Scala 2.11 +------------------------------------------------------------ + +The Spark integration builds for Spark 1.x and Scala 2.10 by default. +Additionally, there is a build profile available for Spark 2.x with +Scala 2.11: from the kudu-spark directory, run + +$ mvn clean package -P spark2_2.11 + +The two artifactIds are + +1. kudu-spark_2.10 for Spark 1.x with Scala 2.10 +2. kudu-spark2_2.11 for Spark 2.x with Scala 2.11 + State of Eclipse integration ------------------------------------------------------------ http://git-wip-us.apache.org/repos/asf/kudu/blob/e46e9518/java/kudu-spark/pom.xml ---------------------------------------------------------------------- diff --git a/java/kudu-spark/pom.xml b/java/kudu-spark/pom.xml index 4c4cda9..f9a0fcc 100644 --- a/java/kudu-spark/pom.xml +++ b/java/kudu-spark/pom.xml @@ -20,14 +20,16 @@ 1.1.0-SNAPSHOT - kudu-spark_${scala.binary.version} + kudu-${spark.version.label}_${scala.binary.version} Kudu Spark Bindings 1.6.1 + spark 2.10.4 2.10 ${project.basedir}/.. + src/main/spark1 @@ -161,6 +163,35 @@ + + + org.codehaus.mojo + build-helper-maven-plugin + + + generate-sources + add-source + + + ${compat.src} + + + + + + + + + spark2_2.11 + + 2.0.1 + src/main/spark2 + spark2 + 2.11.6 + 2.11 + + + http://git-wip-us.apache.org/repos/asf/kudu/blob/e46e9518/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/package.scala ---------------------------------------------------------------------- diff --git a/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/package.scala b/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/package.scala deleted file mode 100755 index 6f49023..0000000 --- a/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/package.scala +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.kudu.spark - -import org.apache.spark.sql.{DataFrame, DataFrameReader, DataFrameWriter} - -package object kudu { - - /** - * Adds a method, `kudu`, to DataFrameReader that allows you to read Kudu tables using - * the DataFrameReader. - */ - implicit class KuduDataFrameReader(reader: DataFrameReader) { - def kudu: DataFrame = reader.format("org.apache.kudu.spark.kudu").load - } - - /** - * Adds a method, `kudu`, to DataFrameWriter that allows writes to Kudu using - * the DataFileWriter - */ - implicit class KuduDataFrameWriter(writer: DataFrameWriter) { - def kudu = writer.format("org.apache.kudu.spark.kudu").save - } -} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/kudu/blob/e46e9518/java/kudu-spark/src/main/spark1/org/apache/kudu/spark/kudu/package.scala ---------------------------------------------------------------------- diff --git a/java/kudu-spark/src/main/spark1/org/apache/kudu/spark/kudu/package.scala b/java/kudu-spark/src/main/spark1/org/apache/kudu/spark/kudu/package.scala new file mode 100755 index 0000000..6f49023 --- /dev/null +++ b/java/kudu-spark/src/main/spark1/org/apache/kudu/spark/kudu/package.scala @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.kudu.spark + +import org.apache.spark.sql.{DataFrame, DataFrameReader, DataFrameWriter} + +package object kudu { + + /** + * Adds a method, `kudu`, to DataFrameReader that allows you to read Kudu tables using + * the DataFrameReader. + */ + implicit class KuduDataFrameReader(reader: DataFrameReader) { + def kudu: DataFrame = reader.format("org.apache.kudu.spark.kudu").load + } + + /** + * Adds a method, `kudu`, to DataFrameWriter that allows writes to Kudu using + * the DataFileWriter + */ + implicit class KuduDataFrameWriter(writer: DataFrameWriter) { + def kudu = writer.format("org.apache.kudu.spark.kudu").save + } +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/kudu/blob/e46e9518/java/kudu-spark/src/main/spark2/org/apache/kudu/spark/kudu/package.scala ---------------------------------------------------------------------- diff --git a/java/kudu-spark/src/main/spark2/org/apache/kudu/spark/kudu/package.scala b/java/kudu-spark/src/main/spark2/org/apache/kudu/spark/kudu/package.scala new file mode 100755 index 0000000..19ae9ef --- /dev/null +++ b/java/kudu-spark/src/main/spark2/org/apache/kudu/spark/kudu/package.scala @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.kudu.spark + +import org.apache.spark.sql.{DataFrame, DataFrameReader, DataFrameWriter} + +package object kudu { + + /** + * Adds a method, `kudu`, to DataFrameReader that allows you to read Kudu tables using + * the DataFrameReader. + */ + implicit class KuduDataFrameReader(reader: DataFrameReader) { + def kudu: DataFrame = reader.format("org.apache.kudu.spark.kudu").load + } + + /** + * Adds a method, `kudu`, to DataFrameWriter that allows writes to Kudu using + * the DataFileWriter + */ + implicit class KuduDataFrameWriter[T](writer: DataFrameWriter[T]) { + def kudu = writer.format("org.apache.kudu.spark.kudu").save + } +} \ No newline at end of file