Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8675D200B96 for ; Thu, 22 Sep 2016 08:14:23 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8508A160ADE; Thu, 22 Sep 2016 06:14:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 797A5160ADB for ; Thu, 22 Sep 2016 08:14:22 +0200 (CEST) Received: (qmail 12298 invoked by uid 500); 22 Sep 2016 06:14:21 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 12282 invoked by uid 99); 22 Sep 2016 06:14:21 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Sep 2016 06:14:21 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 67F882C0050 for ; Thu, 22 Sep 2016 06:14:21 +0000 (UTC) Date: Thu, 22 Sep 2016 06:14:21 +0000 (UTC) From: "Ferdinand Xu (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 22 Sep 2016 06:14:23 -0000 [ https://issues.apache.org/jira/browse/HIVE-14029?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1551= 2315#comment-15512315 ]=20 Ferdinand Xu commented on HIVE-14029: ------------------------------------- Hi [~lirui],=20 bq. I don't think we need all the spark jars. Agree. It's not required for all Spark jars. From Hive pom.xml file, we see= that it depends only on Spark_core. You can find the dependency for Spark_= core. All of them should be included into HIVE_CLASSPATH. Do we really need= to filter those jars? It isn't very user friendly because users have to fi= nd them one by one in Spark jars and add it to HIVE classpath. I think we c= an simple add the whole folder to HIVE classpath when running Hive on Spark= . Any thoughts? {code} core]# mvn dependency:tree [INFO] Scanning for projects... [INFO] = =20 [INFO] --------------------------------------------------------------------= ---- [INFO] Building Spark Project Core 2.0.0 [INFO] --------------------------------------------------------------------= ---- [INFO]=20 [INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ spark-core_2.1= 1 --- [INFO] org.apache.spark:spark-core_2.11:jar:2.0.0 [INFO] +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile [INFO] | +- org.apache.avro:avro-ipc:jar:1.7.7:compile [INFO] | | \- org.apache.avro:avro:jar:1.7.7:compile [INFO] | +- org.apache.avro:avro-ipc:jar:tests:1.7.7:test [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile [INFO] | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile [INFO] +- com.google.guava:guava:jar:14.0.1:provided [INFO] +- com.twitter:chill_2.11:jar:0.8.0:compile [INFO] | \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile [INFO] | \- com.esotericsoftware:minlog:jar:1.3.0:compile [INFO] +- com.twitter:chill-java:jar:0.8.0:compile [INFO] +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile [INFO] +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | | +- commons-io:commons-io:jar:2.4:compile [INFO] | | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compi= le [INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | | +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile [INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] | | \- org.tukaani:xz:jar:1.0:compile [INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile [INFO] | | \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compil= e [INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:= compile [INFO] | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile [INFO] | | | | \- com.google.inject:guice:jar:3.0:compile [INFO] | | | | +- javax.inject:javax.inject:jar:1:compile [INFO] | | | | \- aopalliance:aopalliance:jar:1.0:compile [INFO] | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:co= mpile [INFO] | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0= :compile [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compi= le [INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:= compile [INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile [INFO] +- org.apache.spark:spark-launcher_2.11:jar:2.0.0:compile [INFO] +- org.apache.spark:spark-network-common_2.11:jar:2.0.0:compile [INFO] +- org.apache.spark:spark-network-shuffle_2.11:jar:2.0.0:compile [INFO] | +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile [INFO] | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compi= le [INFO] +- org.apache.spark:spark-unsafe_2.11:jar:2.0.0:compile [INFO] +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile [INFO] | +- commons-codec:commons-codec:jar:1.10:compile [INFO] | \- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] +- org.apache.curator:curator-recipes:jar:2.4.0:compile [INFO] | +- org.apache.curator:curator-framework:jar:2.4.0:compile [INFO] | | \- org.apache.curator:curator-client:jar:2.4.0:compile [INFO] | \- org.apache.zookeeper:zookeeper:jar:3.4.5:compile [INFO] +- org.eclipse.jetty:jetty-plus:jar:9.2.16.v20160414:compile [INFO] | +- org.eclipse.jetty:jetty-webapp:jar:9.2.16.v20160414:compile [INFO] | | \- org.eclipse.jetty:jetty-xml:jar:9.2.16.v20160414:compile [INFO] | \- org.eclipse.jetty:jetty-jndi:jar:9.2.16.v20160414:compile [INFO] +- org.eclipse.jetty:jetty-security:jar:9.2.16.v20160414:compile [INFO] +- org.eclipse.jetty:jetty-util:jar:9.2.16.v20160414:compile [INFO] +- org.eclipse.jetty:jetty-server:jar:9.2.16.v20160414:compile [INFO] | \- org.eclipse.jetty:jetty-io:jar:9.2.16.v20160414:compile [INFO] +- org.eclipse.jetty:jetty-http:jar:9.2.16.v20160414:compile [INFO] +- org.eclipse.jetty:jetty-continuation:jar:9.2.16.v20160414:compile [INFO] +- org.eclipse.jetty:jetty-servlet:jar:9.2.16.v20160414:compile [INFO] +- org.eclipse.jetty:jetty-servlets:jar:9.2.16.v20160414:compile [INFO] +- javax.servlet:javax.servlet-api:jar:3.1.0:compile [INFO] +- org.apache.commons:commons-lang3:jar:3.3.2:compile [INFO] +- org.apache.commons:commons-math3:jar:3.4.1:compile [INFO] +- com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] +- org.slf4j:slf4j-api:jar:1.7.16:compile [INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.16:compile [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile [INFO] +- log4j:log4j:jar:1.2.17:compile [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile [INFO] +- com.ning:compress-lzf:jar:1.0.3:compile [INFO] +- org.xerial.snappy:snappy-java:jar:1.1.2.4:compile [INFO] +- net.jpountz.lz4:lz4:jar:1.3.0:compile [INFO] +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile [INFO] +- commons-net:commons-net:jar:2.2:compile [INFO] +- org.scala-lang:scala-library:jar:2.11.8:compile [INFO] +- org.json4s:json4s-jackson_2.11:jar:3.2.11:compile [INFO] | \- org.json4s:json4s-core_2.11:jar:3.2.11:compile [INFO] | +- org.json4s:json4s-ast_2.11:jar:3.2.11:compile [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile [INFO] | \- org.scala-lang:scalap:jar:2.11.8:compile [INFO] | \- org.scala-lang:scala-compiler:jar:2.11.8:compile [INFO] | \- org.scala-lang.modules:scala-parser-combinators_2.11:= jar:1.0.4:compile [INFO] +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile [INFO] | +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile [INFO] | +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile [INFO] | | +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile [INFO] | | \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0= -b34:compile [INFO] | +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:compile [INFO] | \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile [INFO] +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile [INFO] | +- javax.annotation:javax.annotation-api:jar:1.2:compile [INFO] | +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.= 2:compile [INFO] | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile [INFO] +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:compile [INFO] | +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:compil= e [INFO] | \- javax.validation:validation-api:jar:1.1.0.Final:compile [INFO] +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22= .2:compile [INFO] +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar= :2.22.2:compile [INFO] +- org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1:compile [INFO] +- io.netty:netty-all:jar:4.0.29.Final:compile [INFO] +- io.netty:netty:jar:3.8.0.Final:compile [INFO] +- com.clearspring.analytics:stream:jar:2.7.0:compile [INFO] +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile [INFO] +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile [INFO] +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile [INFO] +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:compile [INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:compile [INFO] +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:= compile [INFO] | +- org.scala-lang:scala-reflect:jar:2.11.8:compile [INFO] | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.= 5:compile [INFO] +- org.apache.derby:derby:jar:10.11.1.1:test [INFO] +- org.apache.ivy:ivy:jar:2.4.0:compile [INFO] +- oro:oro:jar:2.0.8:compile [INFO] +- org.seleniumhq.selenium:selenium-java:jar:2.52.0:test [INFO] | +- org.seleniumhq.selenium:selenium-chrome-driver:jar:2.52.0:test [INFO] | | \- org.seleniumhq.selenium:selenium-remote-driver:jar:2.52.0:t= est [INFO] | | +- cglib:cglib-nodep:jar:2.1_3:test [INFO] | | +- com.google.code.gson:gson:jar:2.3.1:test [INFO] | | \- org.seleniumhq.selenium:selenium-api:jar:2.52.0:test [INFO] | +- org.seleniumhq.selenium:selenium-edge-driver:jar:2.52.0:test [INFO] | | \- org.apache.commons:commons-exec:jar:1.3:test [INFO] | +- org.seleniumhq.selenium:selenium-firefox-driver:jar:2.52.0:tes= t [INFO] | +- org.seleniumhq.selenium:selenium-ie-driver:jar:2.52.0:test [INFO] | | +- net.java.dev.jna:jna:jar:4.1.0:test [INFO] | | \- net.java.dev.jna:jna-platform:jar:4.1.0:test [INFO] | +- org.seleniumhq.selenium:selenium-safari-driver:jar:2.52.0:test [INFO] | +- org.seleniumhq.selenium:selenium-support:jar:2.52.0:test [INFO] | +- org.webbitserver:webbit:jar:0.4.14:test [INFO] | \- org.seleniumhq.selenium:selenium-leg-rc:jar:2.52.0:test [INFO] +- org.seleniumhq.selenium:selenium-htmlunit-driver:jar:2.52.0:test [INFO] | +- net.sourceforge.htmlunit:htmlunit:jar:2.18:test [INFO] | | +- xalan:xalan:jar:2.7.2:test [INFO] | | | \- xalan:serializer:jar:2.7.2:test [INFO] | | +- org.apache.httpcomponents:httpmime:jar:4.5.2:test [INFO] | | +- net.sourceforge.htmlunit:htmlunit-core-js:jar:2.17:test [INFO] | | +- xerces:xercesImpl:jar:2.11.0:test [INFO] | | +- net.sourceforge.nekohtml:nekohtml:jar:1.9.22:test [INFO] | | +- net.sourceforge.cssparser:cssparser:jar:0.9.16:test [INFO] | | | \- org.w3c.css:sac:jar:1.3:test [INFO] | | +- commons-logging:commons-logging:jar:1.2:test [INFO] | | \- org.eclipse.jetty.websocket:websocket-client:jar:9.2.12.v20= 150709:test [INFO] | | \- org.eclipse.jetty.websocket:websocket-common:jar:9.2.12.= v20150709:test [INFO] | | \- org.eclipse.jetty.websocket:websocket-api:jar:9.2.12.= v20150709:test [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile [INFO] | \- org.apache.httpcomponents:httpclient:jar:4.5.2:test [INFO] | \- org.apache.httpcomponents:httpcore:jar:4.4.4:test [INFO] +- xml-apis:xml-apis:jar:1.4.01:test [INFO] +- org.hamcrest:hamcrest-core:jar:1.3:test [INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test [INFO] +- org.mockito:mockito-core:jar:1.10.19:test [INFO] | \- org.objenesis:objenesis:jar:2.1:compile [INFO] +- org.scalacheck:scalacheck_2.11:jar:1.12.5:test [INFO] | \- org.scala-sbt:test-interface:jar:1.0:test [INFO] +- org.apache.curator:curator-test:jar:2.4.0:test [INFO] | +- org.javassist:javassist:jar:3.15.0-GA:compile [INFO] | \- org.apache.commons:commons-math:jar:2.2:compile [INFO] +- net.razorvine:pyrolite:jar:4.9:compile [INFO] +- net.sf.py4j:py4j:jar:0.10.1:compile [INFO] +- org.apache.spark:spark-tags_2.11:jar:2.0.0:compile [INFO] +- org.apache.commons:commons-crypto:jar:1.0.0:compile [INFO] +- org.spark-project.spark:unused:jar:1.0.0:compile [INFO] +- org.scalatest:scalatest_2.11:jar:2.2.6:test [INFO] | \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile [INFO] +- junit:junit:jar:4.12:test [INFO] \- com.novocode:junit-interface:jar:0.11:test [INFO] --------------------------------------------------------------------= ---- [INFO] BUILD SUCCESS [INFO] --------------------------------------------------------------------= ---- [INFO] Total time: 2.475 s [INFO] Finished at: 2016-09-22T06:34:12+08:00 [INFO] Final Memory: 23M/963M [INFO] --------------------------------------------------------------------= ---- {code} > Update Spark version to 2.0.0 > ----------------------------- > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug > Reporter: Ferdinand Xu > Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, HIVE-14029.3= .patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up= Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn=E2=80=99t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetric= s return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)