flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nico Kruber <n...@data-artisans.com>
Subject Re: Flink 1.4 with cassandra-connector: Shading error
Date Tue, 19 Dec 2017 10:14:31 GMT
Hi Dominik,
nice assessment of the issue: in the version of the cassandra-driver we
use there is even a comment about why:

----
try {
 // prevent this string from being shaded
 Class.forName(String.format("%s.%s.channel.Channel", "io", "netty"));
 shaded = false;
} catch (ClassNotFoundException e) {
 try {
   Class.forName("com.datastax.shaded.netty.channel.Channel");
   shaded = true;
 } catch (ClassNotFoundException e1) {
   throw new AssertionError("Cannot locate Netty classes in the
classpath:" + e1);
 }
}
----

@Chesnay: Should we instead shade into datastax' namespace as shown?
This would also make sure to follow the shaded path in that class which,
for example, deactivates epoll.


Nico


On 18/12/17 15:43, Timo Walther wrote:
> Hi Dominik,
> 
> thanks for reporting your issue. I will loop in Chesnay that might know
> more about your problem.
> 
> There were a lot of dependency changes in 1.4 to make the future more
> dependency friendly. Maybe this has not been tested properly.
> 
> Regards,
> Timo
> 
> 
> Am 12/18/17 um 3:07 PM schrieb dominik@dbruhn.de:
>> Hey everyone,
>> I'm trying to migrate one of my jobs to Flink 1.4 and I'm running into
>> a classpath/shading error.
>>
>> What happens is that when Flink calls into Cluster.connect(),
>> somewhere down in the stream, the cassandra library tries to
>> initialize Netty, and I'm getting the following exception:
>>
>> Caused by: java.lang.AssertionError: Cannot locate Netty classes in
>> the classpath:java.lang.ClassNotFoundException:
>> com.datastax.shaded.netty.channel.Channel.
>> Full exception here:
>> https://gist.github.com/anonymous/16a44eabb45ad7f20d551dd29b83d2fb
>>
>> I think this can be explained very easily:
>> 1. The flink-cassandra-connector-1.4 jar contains the Datastax
>> Cassandra driver inside
>> 2. The flink-cassandra-connector-1.4 jar contains a shaded (to
>> org.apache.flink.cassandra.shaded.io.netty) netty
>> 3. The DataStax driver executes the following code when it initializes
>> netty (in NettyUtils starting line 58):
>>
>> ----
>>  try {
>>             Class.forName(String.format("%s.%s.channel.Channel", "io",
>> "netty"));
>>             shaded = false;
>>         } catch (ClassNotFoundException var9) {
>>             try {
>> Class.forName("com.datastax.shaded.netty.channel.Channel");
>>                 shaded = true;
>>             } catch (ClassNotFoundException var8) {
>>                 throw new AssertionError("Cannot locate Netty classes
>> in the classpath:" + var8);
>>             }
>>         }
>> ----
>>
>>
>> Neither of the two packages (io.netty.channel and
>> com.datastax.shaded.netty.channel.Channel) are available and the
>> shaded one from Flink is not in the list and is never used here.
>>
>> My question: Did anyone ever use the cassandra connector on 1.4. As it
>> looks to me it is completely broken and can never work. But maybe I'm
>> failing to see something. I don't include netty in my dependencies,
>> that could of course fix it, but I'm suspecting I will run into more
>> dependency problems then.
>>
>> Here are my dependencies:
>> https://gist.github.com/anonymous/565a0ad017976502a62b2919115b31fd
>>
>> What additional information can I provide?
>>
>> Thanks,
>> Dominik
> 
> 


Mime
View raw message