hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Curtis Ullerich <curtuller...@gmail.com>
Subject Re: Compile Just a Subproject
Date Sat, 22 Jun 2013 00:38:40 GMT
Executing this doesn't seem to work either.
C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell>hadoop
jar
target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
org.apache.hadoop.yarn.applications.distributedshell.Client -jar
target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
-shell_command whoami

Thank you for pointing out the classpath issue, though. I checked in
hadoop-dist/.../share and found that hadoop had been using the jar there.
The command can't find the Client class unless it's in that directory, so I
suppose a suitable workaround would be to add the distributedshell/target
directory to the classpath myself, or just copy the jar to /share. If
anyone has suggestions for the best way to get around that issue, I would
welcome them. Otherwise, this relieves my confusion--thanks for all the
help!

Curtis



On Fri, Jun 21, 2013 at 5:10 PM, Hitesh Shah <hitesh@apache.org> wrote:

> Hello Curtis
>
> Try the following:
>
> hadoop jar ./target/distributed-shell.jar.rebuilt-one
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar ...
>
> If you are running hadoop without the jar command, it will find the first
> instance of Client.class in its classpath which I am guessing is from the
> jars used to start the cluster and not the re-built jar.
>
> -- Hitesh
>
>
> On Jun 21, 2013, at 4:09 PM, Curtis Ullerich wrote:
>
> > I should clarify that the build doesn't actually fail (so that bug
> doesn't seem to be the issue). It reports success. But then the output from
> the just-build jar doesn't change.
> >
> > Here's my output from building:
> >
> >
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell>mvn
> clean package -DskipTests
> > [INFO] Scanning for projects...
> > [INFO]
> > [INFO]
> ------------------------------------------------------------------------
> > [INFO] Building hadoop-yarn-applications-distributedshell 3.0.0-SNAPSHOT
> > [INFO]
> ------------------------------------------------------------------------
> > [INFO]
> > [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO] Deleting
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target
> > [INFO]
> > [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO] Executing tasks
> >
> > main:
> >     [mkdir] Created dir:
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\test-dir
> > [INFO] Executed tasks
> > [INFO]
> > [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO] Using default encoding to copy filtered resources.
> > [INFO]
> > [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO] Compiling 3 source files to
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\classes
> > [INFO]
> > [INFO] --- maven-resources-plugin:2.2:testResources
> (default-testResources) @ hadoop-yarn-applications-distributedshell ---
> > [INFO] Using default encoding to copy filtered resources.
> > [INFO]
> > [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile)
> @ hadoop-yarn-applications-distributedshell ---
> > [INFO] Compiling 1 source file to
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\test-classes
> > [INFO]
> > [INFO] --- maven-jar-plugin:2.3.1:jar (default) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO] Building jar:
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
> > [INFO]
> > [INFO] --- maven-surefire-plugin:2.12.3:test (default-test) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO] Tests are skipped.
> > [INFO]
> > [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO]
> > [INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @
> hadoop-yarn-applications-distributedshell ---
> > [INFO]
> ------------------------------------------------------------------------
> > [INFO] BUILD SUCCESS
> > [INFO]
> ------------------------------------------------------------------------
> > [INFO] Total time: 3.487s
> > [INFO] Finished at: Fri Jun 21 16:08:02 PDT 2013
> > [INFO] Final Memory: 19M/367M
> > [INFO]
> ------------------------------------------------------------------------
> >
> >
> >
> > On Fri, Jun 21, 2013 at 3:57 PM, Arpit Agarwal <aagarwal@hortonworks.com>
> wrote:
> > Curtis, if you are running 'mvn clean' in a subproject you may be
> hitting this bug:
> > https://issues.apache.org/jira/browse/HADOOP-9383
> >
> > You might have better luck if you include install along with package in
> the same invocation.
> >
> > -Arpit
> >
> >
> > On Fri, Jun 21, 2013 at 3:12 PM, Curtis Ullerich <curtullerich@gmail.com>
> wrote:
> > I've executed the commands as you've said, and the jar that is run is
> unchanged. Here's exactly what I did.
> >
> > C:\l\> start-dfs
> > C:\l\> start-yarn
> > C:\l\> cd
> hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell
> >
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell>
> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
> -shell_command whoami
> > ...output...
> > ...modify logging code in
> C:/l/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
> >
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell>
> mvn clean install -DskipTests
> >
> C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell>
> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
> -shell_command whoami
> > ...identical output to before...
> >
> > Do you see anything wrong with what I've done?
> >
> > Thanks,
> > Curtis
> >
> >
> >
> >
> > On Thu, Jun 20, 2013 at 7:17 PM, Omkar Joshi <ojoshi@hortonworks.com>
> wrote:
> > Hi Curtis,
> >
> > where are you picking your jar file from? once you run above command you
> will see the updated jar file in
> >
> "..../hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar"
> >
> > I hope you are not using below jar file
> >
> "..../hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar"
> >
> >
> > What chris has said is right.
> >
> > once you have taken latest code, you should follow below steps
> > 1) mvn clean install -DskipTests (clean to remove previously generated
> code)
> > 2) now say you are updating distributed shell client code. then go to
> ""..../hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/"
> and then run "mvn clean install" and use jar from target sub folder.
> >
> > Thanks,
> > Omkar Joshi
> > Hortonworks Inc.
> >
> >
> > On Thu, Jun 20, 2013 at 11:47 AM, Curtis Ullerich <
> curtullerich@gmail.com> wrote:
> > Hi Chris,
> >
> > I really appreciate the response. What you described is what I initially
> tried. The changes never seem to take effect though. Here's what I've done
> (this is Windows):
> >
> > cd %hadoop_install_dir%
> > mvn clean package -DskipTests
> > mvn install -DskipTests
> >
> > --- modify the code in distributed shell's Client.java ---
> >
> > cd
> hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell
> > mvn clean
> > mvn package -DskipTests
> > mvn install -DskipTests
> >
> > Then I run the jar, just as before. I've just been changing log
> statements to see if the changes worked. They don't--the output doesn't
> change after doing this rebuild. I've also tried clearing the files put on
> HDFS in my user directory in case that was the issue. I've taken a more
> thorough look at BUILDING.txt and I seemed to be consistent with the
> procedures described there. Am I missing anything else? I've tried
> restarting yarn and dfs, though I didn't think that would matter.
> >
> > Thanks,
> > Curtis
> >
> >
> >
> >
> > On Thu, Jun 20, 2013 at 11:17 AM, Chris Nauroth <
> cnauroth@hortonworks.com> wrote:
> > Hi Curtis,
> >
> > I handle this by running "mvn install -DskipTests" once from the root of
> the whole hadoop project to install the sub-modules in my local Maven
> repository.  Then, you can change your working directory to any sub-module
> (like distributed shell) and run a successful Maven build for just that
> sub-module.  Full details are in the BUILDING.txt file in the root of the
> hadoop project.  Look for the section titled "Building components
> separately".  I hope this helps.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, Jun 19, 2013 at 4:53 PM, Curtis Ullerich <curtullerich@gmail.com>
> wrote:
> > What is the way to compile just a subproject of Hadoop with Maven
> (without rebuilding parent modules)? In my case I'm wondering about
> DistributedShell. I thought that this would have worked but no changes seem
> to take effect after building this way.
> >
> > Thanks,
> > Curtis
> >
> >
> >
> >
> >
> >
>
>

Mime
View raw message