phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shekhar reddy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3729) Phoenix 4.7 integration issue with Spark 2.1.0
Date Wed, 17 May 2017 07:21:04 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013662#comment-16013662
] 

shekhar reddy commented on PHOENIX-3729:
----------------------------------------

Hello Josh Mahonin,

I am integrating Phoenix with spark as blow.

step1. coping below files into /usr/lib/spark/lib
              phoenix-core-4.7.0-HBase-1.0.jar
              phoenix-server-4.7.0-HBase-1.0.jar
              phoenix-spark-4.7.0-HBase-1.0.jar
              phoenix-HBase-client.jar

Step2. creating soft links in /usr/lib/spark/lib
 phoenix-HBase-client.jar -> /usr/lib/spark/lib/phoenix-server-client-4.7.0-HBase-1.0.jar
 phoenix-spark-HBase.jar -> /usr/lib/spark/lib/phoenix-spark-4.7.0-HBase-1.0.jar

I had integration issue with jackson-core-XXX.jar files. As Spark 2.1 using 2.6.5 version
of Jackson apis but Phonenix-Core using 1.9.13. phoneix-core module pom.xml having  <addClasspath>true</addClasspath>
so the output of MANIFEST file has below content and Class-Path configured with older version
of jackson api and start getting exception  like some of the methods not available. I solved
the issue by removing <addClasspath>true</addClasspath> and  building Phonenix-Core
module then Class-path attribute not generated in Manifest file so it's started using Spark
version of jars in run time.

Manifest file with <addClasspath>true</addClasspath>:
	Manifest-Version: 1.0
	Implementation-Title: Phoenix Core
	Implementation-Version: 4.7.0-HBase-1.0
	Archiver-Version: Plexus Archiver
	Built-By: mas_user
	Specification-Vendor: Apache Software Foundation
	Specification-Title: Phoenix Core
	Implementation-Vendor-Id: org.apache.phoenix
	Class-Path: tephra-api-0.7.0.jar tephra-core-0.7.0.jar gson-2.2.4.jar 
	 guice-3.0.jar javax.inject-1.jar aopalliance-1.0.jar guice-assistedin
	 ject-3.0.jar libthrift-0.9.0.jar fastutil-6.5.6.jar twill-common-0.6.
	 0-incubating.jar twill-core-0.6.0-incubating.jar twill-api-0.6.0-incu
	 bating.jar asm-all-5.0.2.jar twill-discovery-api-0.6.0-incubating.jar
	  twill-discovery-core-0.6.0-incubating.jar twill-zookeeper-0.6.0-incu
	 bating.jar metrics-core-3.1.0.jar tephra-hbase-compat-1.0-0.7.0.jar a
	 ntlr-3.5.jar ST4-4.0.7.jar antlr-runtime-3.5.jar stringtemplate-3.2.1
	 .jar antlr-2.7.7.jar jline-2.11.jar sqlline-1.1.8.jar guava-13.0.1.ja
	 r joda-time-1.6.jar annotations-1.3.2.jar jackson-core-asl-1.9.13.jar
	  jackson-mapper-asl-1.9.13.jar junit-4.12.jar hamcrest-core-1.3.jar p
	 rotobuf-java-2.5.0.jar httpclient-4.0.1.jar httpcore-4.0.1.jar common
	 s-logging-1.2.jar log4j-1.2.17.jar slf4j-api-1.6.4.jar snappy-0.3.jar
	  htrace-core-3.1.0-incubating.jar netty-all-4.0.23.Final.jar commons-
	 codec-1.7.jar commons-collections-3.2.1.jar commons-csv-1.0.jar jsr30
	 5-2.0.1.jar slf4j-log4j12-1.7.5.jar hadoop-auth-2.6.0-cdh5.4.8.jar ap
	 acheds-kerberos-codec-2.0.0-M15.jar apacheds-i18n-2.0.0-M15.jar api-a
	 sn1-api-1.0.0-M20.jar api-util-1.0.0-M20.jar curator-framework-2.7.1.
	 jar hadoop-core-2.6.0-mr1-cdh5.4.8.jar hsqldb-1.8.0.10.jar core-3.1.1
	 .jar hadoop-hdfs-2.6.0-cdh5.4.8.jar commons-daemon-1.0.13.jar leveldb
	 jni-all-1.8.jar hadoop-hdfs-2.6.0-cdh5.4.8-tests.jar findbugs-annotat
	 ions-1.3.9-1.jar jersey-client-1.9.jar commons-cli-1.2.jar commons-ma
	 th-2.1.jar commons-lang-2.6.jar hbase-common-1.0.0-cdh5.4.8.jar hbase
	 -annotations-1.0.0-cdh5.4.8.jar commons-io-2.4.jar jetty-util-6.1.26.
	 cloudera.4.jar hbase-protocol-1.0.0-cdh5.4.8.jar hbase-client-1.0.0-c
	 dh5.4.8.jar netty-3.6.6.Final.jar zookeeper-3.4.5-cdh5.4.8.jar jcodin
	 gs-1.0.8.jar hbase-server-1.0.0-cdh5.4.8.jar commons-httpclient-3.1.j
	 ar jersey-core-1.9.jar jersey-server-1.9.jar asm-3.1.jar jetty-6.1.26
	 .cloudera.4.jar jetty-sslengine-6.1.26.cloudera.4.jar jsp-2.1-6.1.14.
	 jar jsp-api-2.1-6.1.14.jar servlet-api-2.5-6.1.14.jar jasper-compiler
	 -5.5.23.jar jasper-runtime-5.5.23.jar jamon-runtime-2.3.1.jar jersey-
	 json-1.9.jar jettison-1.1.jar jaxb-impl-2.2.3-1.jar jaxb-api-2.1.jar 
	 activation-1.1.jar hbase-server-1.0.0-cdh5.4.8-tests.jar hadoop-commo
	 n-2.6.0-cdh5.4.8.jar commons-math3-3.1.1.jar xmlenc-0.52.jar commons-
	 net-3.1.jar servlet-api-2.5.jar jsp-api-2.1.jar commons-el-1.0.jar je
	 ts3t-0.9.0.jar java-xmlbuilder-0.4.jar commons-configuration-1.6.jar 
	 commons-digester-1.8.jar commons-beanutils-1.7.0.jar commons-beanutil
	 s-core-1.8.0.jar avro-1.7.6-cdh5.4.8.jar paranamer-2.3.jar jsch-0.1.4
	 2.jar curator-client-2.7.1.jar curator-recipes-2.7.1.jar htrace-core-
	 3.0.4.jar commons-compress-1.4.1.jar xz-1.0.jar hadoop-annotations-2.
	 6.0-cdh5.4.8.jar hadoop-mapreduce-client-core-2.6.0-cdh5.4.8.jar hado
	 op-yarn-common-2.6.0-cdh5.4.8.jar jersey-guice-1.9.jar guice-servlet-
	 3.0.jar hadoop-yarn-api-2.6.0-cdh5.4.8.jar joni-2.1.2.jar
	Implementation-Vendor: Apache Software Foundation
	Created-By: Apache Maven 3.3.9
	Build-Jdk: 1.8.0_65
	Specification-Version: 4.7.0-HBase-1.0



Manifest file without <addClasspath>true</addClasspath>:
	Manifest-Version: 1.0
	Implementation-Title: Phoenix Core
	Implementation-Version: 4.7.0-HBase-1.0
	Archiver-Version: Plexus Archiver
	Built-By: mas_user
	Specification-Vendor: Apache Software Foundation
	Specification-Title: Phoenix Core
	Implementation-Vendor-Id: org.apache.phoenix
	Implementation-Vendor: Apache Software Foundation
	Created-By: Apache Maven 3.3.9
	Build-Jdk: 1.8.0_65
	Specification-Version: 4.7.0-HBase-1.0

thanks,
Shekhar Reddy

> Phoenix 4.7 integration issue with Spark 2.1.0
> ----------------------------------------------
>
>                 Key: PHOENIX-3729
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3729
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.7.0, 4.9.0, 4.8.1, 4.10.0, 5.0
>            Reporter: shekhar reddy
>            Priority: Blocker
>
> I was using phoenix 4.7 with spark 1.6 after I upgraded spark 1.6 to 2.1.0 my spark stopped
working. I build Phoenix 4.7.0 with spark version 2.1.0  and copied latest jars in /usr/lib/spark/jars
then started getting Jackson jars conflicts as spark 2.1.0 also having different versions
of Jackson jars in /usr/lb/spark/jars folder.
> I fixed this issue by updating phoenix-core/pom.xml by commenting below line and Spark
started working!!!!!!!!
> <addClasspath>true</addClasspath>
> Please let me how to push this changes in Phoenix code if you guys feel this approach
OK.
> I am happy to contribute!!!!!!!!
> Thanks,
> Shekhar Reddy



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message