flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Flink 1.2 and Cassandra Connector
Date Thu, 16 Mar 2017 17:13:03 GMT
I've created a pull request for the fix:
https://github.com/apache/flink/pull/3556

It would be nice if one of the issue reporters could validate that the
cassandra connector works after the fix.
If it is a valid fix, I would like to include it into the upcoming 1.2.1
release.

On Thu, Mar 16, 2017 at 6:08 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Yep, this is definitively a bug / misconfiguration in the build system.
>
> The cassandra client defines metrics-core as a dependency, but the shading
> is dropping the dependency when building the dependency reduced pom.
> To resolve the issue, we need to add the following line into the shading
> config of the cassandra module:
>
> <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
>
> This makes the metrics dependency appear again in the dep red pom.
>
>
> I've filed a JIRA: https://issues.apache.org/jira/browse/FLINK-6084 and
> will open a PR.
>
> On Thu, Mar 16, 2017 at 1:08 PM, Stephan Ewen <sewen@apache.org> wrote:
>
>> Can we improve the Flink experience here by adding this dependency
>> directly to the cassandra connector pom.xml (so that user jars always pull
>> it in via transitivity)?
>>
>> On Wed, Mar 15, 2017 at 4:09 PM, Nico <nico.franzeck@gmail.com> wrote:
>>
>>> Hi @all,
>>>
>>> I came back to this issue today...
>>>
>>> @Robert:
>>> "com/codahale/metrics/Metric" class was not available in the user code
>>> jar
>>>
>>> Even after adding the metric class into the build-jar profile of the pom
>>> file, more "class not found" errors occur. So the only solution was to add
>>> the whole dependency:
>>>
>>> <dependency>
>>>    <groupId>com.codahale.metrics</groupId>
>>>    <artifactId>metrics-core</artifactId>
>>>    <version>3.0.2</version>
>>> </dependency>
>>>
>>>
>>> This worked for me.
>>>
>>> Best,
>>> Nico
>>>
>>>
>>>
>>> 2017-03-06 11:46 GMT+01:00 Chesnay Schepler <chesnay@apache.org>:
>>>
>>>> Hello,
>>>>
>>>> i believe the cassandra connector is not shading it's dependencies
>>>> properly. This didn't cause issues in the
>>>> past since flink used to have a dependency on codahale metrics as well.
>>>>
>>>> Please open a JIRA for this issue.
>>>>
>>>> Regards,
>>>> Chesnay
>>>>
>>>>
>>>> On 06.03.2017 11:32, Tarandeep Singh wrote:
>>>>
>>>> Hi Robert & Nico,
>>>>
>>>> I am facing the same problem (java.lang.NoClassDefFoundError:
>>>> com/codahale/metrics/Metric)
>>>> Can you help me identify shading issue in pom.xml file.
>>>>
>>>> My pom.xml content-
>>>> ---------------------------------------------
>>>>
>>>> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>      <groupId>rfk-dataplatform</groupId>
  <artifactId>stream-processing</artifactId>   <version>0.1.0</version>
  <packaging>jar</packaging>   <name>Stream processing</name>   <properties>
     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>      <flink.version>1.2.0</flink.version>
     <slf4j.version>1.7.7</slf4j.version>      <log4j.version>1.2.17</log4j.version>
  </properties>   <dependencies>      <dependency>         <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_2.10</artifactId>         <version>${flink.version}</version>
     </dependency>      <dependency>         <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients_2.10</artifactId>         <version>${flink.version}</version>
     </dependency>      <dependency>         <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-cassandra_2.10</artifactId>         <version>1.2.0</version>
     </dependency>        <dependency>            <groupId>org.apache.flink</groupId>
           <artifactId>flink-statebackend-rocksdb_2.10</artifactId>          
 <version>1.2.0</version>        </dependency>      <dependency> 
       <groupId>org.slf4j</groupId>         <artifactId>slf4j-log4j12</artifactId>
        <version>${slf4j.version}</version>      </dependency>      <dependency>
        <groupId>log4j</groupId>         <artifactId>log4j</artifactId>
        <version>${log4j.version}</version>      </dependency>        <dependency>
           <groupId>org.apache.avro</groupId>            <artifactId>avro</artifactId>
           <version>1.8.1</version>        </dependency>        <dependency>
           <groupId>org.testng</groupId>            <artifactId>testng</artifactId>
           <version>6.8</version>            <scope>test</scope> 
      </dependency>        <!-- flink kafka connector -->        <dependency>
           <groupId>org.apache.flink</groupId>            <artifactId>flink-connector-kafka-0.8_2.10</artifactId>
           <version>1.2.0</version>        </dependency>        <!--
java influxdb client -->        <dependency>            <groupId>org.influxdb</groupId>
           <artifactId>influxdb-java</artifactId>            <version>2.5</version>
       </dependency>    </dependencies>   <profiles>      <profile>
        <!-- Profile for packaging correct JAR files -->         <id>build-jar</id>
        <activation>            <activeByDefault>false</activeByDefault>
        </activation>         <dependencies>            <dependency>   
           <groupId>org.apache.flink</groupId>               <artifactId>flink-java</artifactId>
              <version>${flink.version}</version>               <scope>provided</scope>
           </dependency>            <dependency>               <groupId>org.apache.flink</groupId>
              <artifactId>flink-streaming-java_2.10</artifactId>             
 <version>${flink.version}</version>               <scope>provided</scope>
           </dependency>            <dependency>               <groupId>org.apache.flink</groupId>
              <artifactId>flink-clients_2.10</artifactId>               <version>${flink.version}</version>
              <scope>provided</scope>            </dependency>         
  <dependency>               <groupId>org.slf4j</groupId>              
<artifactId>slf4j-log4j12</artifactId>               <version>${slf4j.version}</version>
              <scope>provided</scope>            </dependency>         
  <dependency>               <groupId>log4j</groupId>               <artifactId>log4j</artifactId>
              <version>${log4j.version}</version>               <scope>provided</scope>
           </dependency>         </dependencies>            <build>    
       <plugins>               <!-- disable the exclusion rules -->          
    <plugin>                  <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-shade-plugin</artifactId>                 
<version>2.4.1</version>                  <executions>                 
   <execution>                        <phase>package</phase>           
            <goals>                           <goal>shade</goal>       
                </goals>                        <configuration>              
            <artifactSet>                              <excludes combine.self="override"></excludes>
                          </artifactSet>                        </configuration>
                    </execution>                  </executions>              
</plugin>            </plugins>         </build>      </profile> 
 </profiles>   <build>      <plugins>         <plugin>           
<groupId>org.apache.maven.plugins</groupId>            <artifactId>maven-shade-plugin</artifactId>
           <version>2.4.1</version>            <executions>            
  <!-- Run shade goal on package phase -->               <execution>         
        <phase>package</phase>                  <goals>                
    <goal>shade</goal>                  </goals>                  <configuration>
                    <artifactSet>                        <excludes>          
                <!-- This list contains all dependencies of flink-dist                
          Everything else will be packaged into the fat-jar                           -->
                          <exclude>org.apache.flink:flink-annotations</exclude>
                          <exclude>org.apache.flink:flink-shaded-hadoop2</exclude>
                          <exclude>org.apache.flink:flink-shaded-curator-recipes</exclude>
                          <exclude>org.apache.flink:flink-core</exclude>     
                     <exclude>org.apache.flink:flink-java</exclude>          
                <exclude>org.apache.flink:flink-scala_2.10</exclude>         
                 <exclude>org.apache.flink:flink-runtime_2.10</exclude>      
                    <exclude>org.apache.flink:flink-optimizer_2.10</exclude> 
                         <exclude>org.apache.flink:flink-clients_2.10</exclude>
                          <exclude>org.apache.flink:flink-avro_2.10</exclude>
                          <exclude>org.apache.flink:flink-examples-batch_2.10</exclude>
                          <exclude>org.apache.flink:flink-examples-streaming_2.10</exclude>
                          <exclude>org.apache.flink:flink-streaming-java_2.10</exclude>
                          <exclude>org.apache.flink:flink-streaming-scala_2.10</exclude>
                          <exclude>org.apache.flink:flink-scala-shell_2.10</exclude>
                          <exclude>org.apache.flink:flink-python</exclude>   
                       <exclude>org.apache.flink:flink-metrics-core</exclude>
                          <exclude>org.apache.flink:flink-metrics-jmx</exclude>
                          <exclude>org.apache.flink:flink-statebackend-rocksdb_2.10</exclude>
                          <!-- Also exclude very big transitive dependencies of Flink 
                         WARNING: You have to remove these excludes if your code relies on
other                           versions of these dependencies.                          
-->                           <exclude>log4j:log4j</exclude>              
            <exclude>org.scala-lang:scala-library</exclude>                  
        <exclude>org.scala-lang:scala-compiler</exclude>                     
     <exclude>org.scala-lang:scala-reflect</exclude>                         
 <exclude>com.data-artisans:flakka-actor_*</exclude>                         
 <exclude>com.data-artisans:flakka-remote_*</exclude>                        
  <exclude>com.data-artisans:flakka-slf4j_*</exclude>                        
  <exclude>io.netty:netty-all</exclude>                           <exclude>io.netty:netty</exclude>
                          <exclude>commons-fileupload:commons-fileupload</exclude>
                          <exclude>org.apache.avro:avro</exclude>            
              <exclude>commons-collections:commons-collections</exclude>     
                     <exclude>org.codehaus.jackson:jackson-core-asl</exclude>
                          <exclude>org.codehaus.jackson:jackson-mapper-asl</exclude>
                          <exclude>com.thoughtworks.paranamer:paranamer</exclude>
                          <exclude>org.xerial.snappy:snappy-java</exclude>   
                       <exclude>org.apache.commons:commons-compress</exclude>
                          <exclude>org.tukaani:xz</exclude>                  
        <exclude>com.esotericsoftware.kryo:kryo</exclude>                    
      <exclude>com.esotericsoftware.minlog:minlog</exclude>                  
        <exclude>org.objenesis:objenesis</exclude>                           <exclude>com.twitter:chill_*</exclude>
                          <exclude>com.twitter:chill-java</exclude>          
                <exclude>commons-lang:commons-lang</exclude>                 
         <exclude>junit:junit</exclude>                           <exclude>org.apache.commons:commons-lang3</exclude>
                          <exclude>org.slf4j:slf4j-api</exclude>             
             <exclude>org.slf4j:slf4j-log4j12</exclude>                      
    <exclude>log4j:log4j</exclude>                           <exclude>org.apache.commons:commons-math</exclude>
                          <exclude>org.apache.sling:org.apache.sling.commons.json</exclude>
                          <exclude>commons-logging:commons-logging</exclude> 
                         <exclude>commons-codec:commons-codec</exclude>      
                    <exclude>com.fasterxml.jackson.core:jackson-core</exclude>
                          <exclude>com.fasterxml.jackson.core:jackson-databind</exclude>
                          <exclude>com.fasterxml.jackson.core:jackson-annotations</exclude>
                          <exclude>stax:stax-api</exclude>                   
       <exclude>com.typesafe:config</exclude>                           <exclude>org.uncommons.maths:uncommons-maths</exclude>
                          <exclude>com.github.scopt:scopt_*</exclude>        
                  <exclude>commons-io:commons-io</exclude>                   
       <exclude>commons-cli:commons-cli</exclude>                        </excludes>
                    </artifactSet>                     <filters>             
          <filter>                           <artifact>org.apache.flink:*</artifact>
                          <excludes>                              <!-- exclude shaded
google but include shaded curator -->                              <exclude>org/apache/flink/shaded/com/**</exclude>
                             <exclude>web-docs/**</exclude>                  
        </excludes>                        </filter>                        <filter>
                          <artifact>*:*</artifact>                           <excludes>
                             <exclude>META-INF/*.SF</exclude>                
             <exclude>META-INF/*.DSA</exclude>                              <exclude>META-INF/*.RSA</exclude>
                          </excludes>                        </filter>       
             </filters>                     <createDependencyReducedPom>false</createDependencyReducedPom>
                 </configuration>               </execution>            </executions>
        </plugin>         <plugin>            <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>            <version>3.1</version>
           <configuration>               <source>1.8</source>          
    <target>1.8</target>            </configuration>         </plugin>
           <plugin>                <groupId>org.apache.avro</groupId>  
             <artifactId>avro-maven-plugin</artifactId>                <version>1.8.1</version>
               <executions>                    <execution>                   
    <phase>generate-sources</phase>                        <goals>     
                      <goal>schema</goal>                        </goals>
                       <configuration>                            <stringType>String</stringType>
                           <fieldVisibility>PRIVATE</fieldVisibility>        
                   <sourceDirectory>${project.basedir}/src/main/schema/</sourceDirectory>
                           <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                       </configuration>                    </execution>      
         </executions>            </plugin>            <plugin>        
       <groupId>org.apache.maven.plugins</groupId>                <artifactId>maven-compiler-plugin</artifactId>
               <configuration>                    <source>1.8</source> 
                  <target>1.8</target>                </configuration> 
          </plugin>      </plugins>         </build></project>
>>>>
>>>>
>>>> ---------------------------------------------
>>>>
>>>>
>>>> On Sun, Feb 12, 2017 at 1:56 AM, Robert Metzger <rmetzger@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Nico,
>>>>> The cassandra connector should be available on Maven central:
>>>>> http://search.maven.org/#artifactdetails%7Corg.apac
>>>>> he.flink%7Cflink-connector-cassandra_2.10%7C1.2.0%7Cjar
>>>>>
>>>>> Potentially, the issue you've mentioned is due to some shading issue.
>>>>> Is the "com/codahale/metrics/Metric" class in your user code jar?
>>>>>
>>>>> On Thu, Feb 9, 2017 at 2:56 PM, Nico < <nico.franzeck@gmail.com>
>>>>> nico.franzeck@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I would like to upgrade to the new stable version 1.2 - but i get
an
>>>>>> ClassNotFound exception when i start the application.
>>>>>>
>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>> com/codahale/metrics/Metric
>>>>>>     at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1
>>>>>> 367)
>>>>>>     at com.datastax.driver.core.Cluster.init(Cluster.java:162)
>>>>>>     at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:3
>>>>>> 33)
>>>>>>     at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:3
>>>>>> 08)
>>>>>>     at com.datastax.driver.core.Cluster.connect(Cluster.java:250)
>>>>>>     at org.apache.flink.streaming.connectors.cassandra.CassandraSin
>>>>>> kBase.open(CassandraSinkBase.java:67)
>>>>>>     at org.apache.flink.streaming.connectors.cassandra.CassandraTup
>>>>>> leSink.open(CassandraTupleSink.java:42)
>>>>>>     at org.apache.flink.api.common.functions.util.FunctionUtils.ope
>>>>>> nFunction(FunctionUtils.java:36)
>>>>>>     at org.apache.flink.streaming.api.operators.AbstractUdfStreamOp
>>>>>> erator.open(AbstractUdfStreamOperator.java:112)
>>>>>>     at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllO
>>>>>> perators(StreamTask.java:386)
>>>>>>     at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(S
>>>>>> treamTask.java:262)
>>>>>>     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
>>>>>>     at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>> So I think the cassandra connector is the reason for it. Moreover,
i
>>>>>> don't see a version 1.2 in the maven repository for the connector
as
>>>>>> mentioned in the doc.
>>>>>>
>>>>>> <dependency>
>>>>>>   <groupId>org.apache.flink</groupId>
>>>>>>   <artifactId>flink-connector-cassandra_2.10</artifactId>
>>>>>>   <version>1.2.0</version>
>>>>>> </dependency>
>>>>>>
>>>>>> Is there a plan to release a new version?
>>>>>>
>>>>>> Best,
>>>>>> Nico
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message