apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Gupta <gau...@datatorrent.com>
Subject Re: Running CassandraOutputOperator with newer library
Date Fri, 18 Dec 2015 19:24:26 GMT
Priyanka,

Guava is coming from Hadoop. Execute "mvn dependency:tree" under apex/api and you should see
something like.

 org.apache.apex:apex-api:jar:3.3.0-incubating-SNAPSHOT
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO] |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO] |  |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile

Thanks
- Gaurav

> On Dec 18, 2015, at 7:20 AM, Thomas Weise <thomas@datatorrent.com> wrote:
> 
> It wasn't clear where the conflict is, so shading may or may not be a
> solution.
> 
> If your app with the Cassandra operator needs a newer version of Guava,
> include that newer version into the app package. The container's class path
> will have the dependencies packaged with the app before anything from Hadoop
> .
> 
> If the newer version of Guava isn't backward compatible, then things could
> get complicated. If the conflict is between the Hadoop stuff and your app
> (including Cassandra), shading won't help because those are dependencies
> that we use as is. In this case, it may be necessary to run the Cassandra
> code in a fully isolated class loader. If the conflict is between Apex code
> and the app, then shading may help.
> 
> Thomas
> 
> 
> 
> 
> 
> 
> On Fri, Dec 18, 2015 at 5:43 AM, Timothy Farkas <tim@datatorrent.com> wrote:
> 
>> Hi Priyanka,
>> 
>> I haven't seen this issue yet, but I am not using the newest version of
>> cassandra driver. This is the version I am using:
>> 
>> 
>>        <dependency>
>>          <groupId>com.datastax.cassandra</groupId>
>>          <artifactId>cassandra-driver-core</artifactId>
>>          <version>2.1.8</version>
>>        </dependency>
>> 
>> I did run into an issue where I was getting strange cassandra exceptions,
>> so I set the protocol to version 2.0 in the CassandraStore class like this
>> and the issue was resolved:
>> 
>> 
>>      cluster = Cluster.builder()
>> 
>> 
>> .addContactPoint(node).withProtocolVersion(ProtocolVersion.V2).withCredentials(userName,
>> password).build();
>> 
>> If you are still experiencing missing method issues due to different guava
>> versions it may be necessary to shade the version of guava used by apex.
>> Shading changes the package names of all the packages in a library by
>> appending a prefix, so that
>> all the classes in the guava version used by apex would look like this:
>> org.apache.apex.com.google.guavapackage.GuavaClass. This prevents version
>> clashes, and this technique is used by Flink for that reason. Perhaps we
>> can start that discussion again in a separate thread.
>> 
>> A temporary workaround could be to exclude guava from the apex dependencies
>> in your pom, that way the newest version of guava will be used.
>> 
>> Thanks,
>> Tim
>> 
>> On Fri, Dec 18, 2015 at 4:33 AM, Priyanka Gugale <priyag@apache.org>
>> wrote:
>> 
>>> Hi,
>>> 
>>> I am facing classpath issues when I try to use newer version (e.g. 2.1.x
>> or
>>> 3.0.x) of cassandra library for CassandraOutputOperator.
>>> 
>>> From datastax documents looks like we need to include few more libraries
>> to
>>> setup java environment. (reference:
>>> 
>>> 
>> http://docs.datastax.com/en/developer/java-driver/2.1/java-driver/reference/settingUpJavaProgEnv_r.html
>>> ).
>>> When I included the required libraries my ClassNotFound issue was
>> resolved
>>> but then I was getting some methods missing error.
>>> Looks like, apex platform also uses guava library and it's version is
>> lower
>>> than 14.0.1, I couldn't find the source though (Note: I was just trying
>> to
>>> run unit tests here, not an application).
>>> 
>>> Has anyone faced such issue earlier?
>>> 
>>> -Priyanka
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message