flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Srinivasaraghavan <vijikar...@yahoo.com.INVALID>
Subject Re: Shading Plugin + Hadoop Dependency
Date Tue, 21 Jun 2016 15:47:52 GMT
Thanks Rob for trying. In my machine, I don't see any of minikdc related files in the target
jar.
How did you run the build?
RegardsVijay 

    On Tuesday, June 21, 2016 8:39 AM, Robert Metzger <rmetzger@apache.org> wrote:
 

 Cool, thank you.
After adding the line you've mentioned, my jar contains the missing MiniKdc class:
robert@robert-da ...adoop/flink-shaded-include-yarn-tests (git)-[master] % jar tf target/flink-shaded-include-yarn-tests-1.1-SNAPSHOT.jar
| grep minikdcminikdc-krb5.confminikdc.ldifforg/apache/hadoop/minikdc/org/apache/hadoop/minikdc/KerberosSecurityTestcase.classorg/apache/hadoop/minikdc/MiniKdc$1.classorg/apache/hadoop/minikdc/MiniKdc.classMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/META-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.xmlMETA-INF/maven/org.apache.hadoop/hadoop-minikdc/pom.propertiesrobert@robert-da
...adoop/flink-shaded-include-yarn-tests (git)-[master] %     
Is the same true on your machine?



On Tue, Jun 21, 2016 at 5:36 PM, Vijay Srinivasaraghavan <vijikarthi@yahoo.com.invalid>
wrote:

Hi Rob,
You need to include below lines to the pom.xml to resolve the chain dependency error.
 <!--    https://issues.apache.org/jira/browse/DIRSHARED-134    Mini-KDC dependency
error    --> <build> <plugins> <plugin> <groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId> <inherited>true</inherited>
<extensions>true</extensions> </plugin> </plugins> </build>
RegardsVijay

    On Tuesday, June 21, 2016 8:27 AM, Robert Metzger <rmetzger@apache.org> wrote:


 Hi,
I didn't even get the shading module to build:
[INFO] ------------------------------------------------------------------------[INFO] Building
flink-shaded-include-yarn-tests 1.1-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO]
------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO]
------------------------------------------------------------------------[INFO] Total time:
0.756 s[INFO] Finished at: 2016-06-21T17:25:00+02:00[INFO] Final Memory: 16M/303M[INFO] ------------------------------------------------------------------------[ERROR]
Failed to execute goal on project flink-shaded-include-yarn-tests: Could not resolve dependencies
for project org.apache.flink:flink-shaded-include-yarn-tests:jar:1.1-SNAPSHOT: Failure to
find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in https://repo.maven.apache.org/maven2
was cached in the local repository, resolution will not be reattempted until the update interval
of central has elapsed or updates are forced -> [Help 1][ERROR] 

How did you do it?
Regards,Robert

On Tue, Jun 21, 2016 at 5:06 PM, Vijay Srinivasaraghavan <vijikarthi@yahoo.com.invalid>
wrote:

Hi Rob,
Yes I checked but the result jar does not contain the classes from hadoop-minikdc package.
RegardsVijay

    On Tuesday, June 21, 2016 7:59 AM, Robert Metzger <rmetzger@apache.org> wrote:


 Hi Vijay,
did you check if the artifact produced by the "flink-shaded-include-yarn-tests" module contains
the required classes?
Regards,Robert

On Mon, Jun 20, 2016 at 6:18 PM, Vijay Srinivasaraghavan <vijikarthi@yahoo.com.invalid>
wrote:

Hello,
I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency
in "flink-shaded-include-yarn-tests" pom.xml file.
<dependency>    <groupId>org.apache.hadoop</groupId>    <artifactId>hadoop-minikdc</artifactId>    <version>${hadoop.version}</version></dependency>
The dependency inclusion seems to be working from IDE. IntelliJ picked up the jar and did
not report any compilation issue when I create a new test program in "flink-yarn-tests" module
and try to use the minikdc class.
However, when I try to run the test using maven command, I am getting compilation error for
the MiniKdc package.
Could someone please share some insights on how the dependency inclusion works with the shaded
hadoop components?
mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests -Dtest=YARNSessionFIFOITCase

[ERROR] COMPILATION ERROR : [INFO] -------------------------------------------------------------[ERROR]
/workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[24,33]
package org.apache.hadoop.minikdc does not exist[ERROR] /workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[46,24]
cannot find symbol  symbol:   class MiniKdc  location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
/workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[58,38]
cannot find symbol  symbol:   variable MiniKdc  location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase[ERROR]
/workspace/git-projects/junk/flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionSecuredFIFOITCase.java:[59,27]
cannot find symbol  symbol:   class MiniKdc  location: class org.apache.flink.yarn.YARNSessionSecuredFIFOITCase
RegardsVijay



 



  



  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message