Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of xeonmailinglist@gmail.com
 designates 209.85.212.180 as permitted sender)
Message-ID: <54D0E208.1040702@gmail.com>
Date: Tue, 03 Feb 2015 14:58:16 +0000
From: xeonmailinglist <xeonmailinglist@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.4.0
MIME-Version: 1.0
To: user@hadoop.apache.org
Subject: Pass lib jars when invoking an hadoop program
Content-Type: multipart/alternative;
 boundary="------------040604070207000300090409"

This is a multi-part message in MIME format.
--------------040604070207000300090409
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

I am trying to run |distcp| using a java class, but I get the error of 
class not found |DistCpOptions|. I have used the argument |-libjars 
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar| to pass the jar file, 
but it seems that is not right. How I pass the lib properly?

Output:

|vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop jar wordcount.jar  -libjars ./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar /input1 /outputmp /output1
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml
-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar
/input1
/outputmp
/output1
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
     at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
     at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCpOptions
     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
     at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
|

My class:

|public static void main(String[] args) throws Exception {
         Configuration conf = new Configuration();

         String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
         if (otherArgs.length < 2) {
             System.err.println("Usage: wordcount <in> [<in>...] <out>");
             System.exit(2);
         }

         System.out.println(conf.toString());
         for (int i = 0; i < args.length; i++) {
             System.out.println(args[i]);
         }

         // distcp
         String proto = "webhdfs://";
         String src = "hadoop-coc-1/input1";
         String dest = "hadoop-coc-2/input1";
         List<Path> lsrc = new ArrayList<Path>();
         lsrc.add(new Path(src));
         DistCpOptions options = new DistCpOptions(lsrc, new Path(dest));
         DistCp distcp = new DistCp(new Configuration(), options);
         distcp.execute();
     }
|

​

--------------040604070207000300090409
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div style="" class="markdown-here-wrapper"
      data-md-url="Thunderbird">
      <p style="margin: 1.2em 0px ! important;">Hi,</p>
      <p style="margin: 1.2em 0px ! important;">I am trying to run <code
          style="font-size: 0.85em; font-family:
          Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em;
          padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid
          rgb(234, 234, 234); background-color: rgb(248, 248, 248);
          border-radius: 3px; display: inline;">distcp</code> using a
        java class, but I get the error of class not found <code
          style="font-size: 0.85em; font-family:
          Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em;
          padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid
          rgb(234, 234, 234); background-color: rgb(248, 248, 248);
          border-radius: 3px; display: inline;">DistCpOptions</code>. I
        have used the argument <code style="font-size: 0.85em;
          font-family: Consolas,Inconsolata,Courier,monospace;margin:
          0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border:
          1px solid rgb(234, 234, 234); background-color: rgb(248, 248,
          248); border-radius: 3px; display: inline;">-libjars
          ./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar</code> to
        pass the jar file, but it seems that is not right. How I pass
        the lib properly?</p>
      <p style="margin: 1.2em 0px ! important;">Output: </p>
      <pre style="font-size: 0.85em; font-family: Consolas,Inconsolata,Courier,monospace;font-size: 1em; line-height: 1.2em;margin: 1.2em 0px;"><code style="font-size: 0.85em; font-family: Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid rgb(234, 234, 234); background-color: rgb(248, 248, 248); border-radius: 3px; display: inline;white-space: pre; overflow: auto; border-radius: 3px; border: 1px solid rgb(204, 204, 204); padding: 0.5em 0.7em; display: block ! important;">vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop jar wordcount.jar  -libjars ./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar /input1 /outputmp /output1
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml
-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar
/input1
/outputmp
/output1
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
    at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCpOptions
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
</code></pre>
      <p style="margin: 1.2em 0px ! important;">My class:</p>
      <pre style="font-size: 0.85em; font-family: Consolas,Inconsolata,Courier,monospace;font-size: 1em; line-height: 1.2em;margin: 1.2em 0px;"><code style="font-size: 0.85em; font-family: Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid rgb(234, 234, 234); background-color: rgb(248, 248, 248); border-radius: 3px; display: inline;white-space: pre; overflow: auto; border-radius: 3px; border: 1px solid rgb(204, 204, 204); padding: 0.5em 0.7em; display: block ! important;">public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();

        String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
        if (otherArgs.length &lt; 2) {
            System.err.println("Usage: wordcount &lt;in&gt; [&lt;in&gt;...] &lt;out&gt;");
            System.exit(2);
        }

        System.out.println(conf.toString());
        for (int i = 0; i &lt; args.length; i++) {
            System.out.println(args[i]);
        }

        // distcp
        String proto = "webhdfs://";
        String src = "hadoop-coc-1/input1";
        String dest = "hadoop-coc-2/input1";
        List&lt;Path&gt; lsrc = new ArrayList&lt;Path&gt;();
        lsrc.add(new Path(src));
        DistCpOptions options = new DistCpOptions(lsrc, new Path(dest));
        DistCp distcp = new DistCp(new Configuration(), options); 
        distcp.execute();
    }
</code></pre>
      <div
title="MDH:SGksPGJyPjxicj5JIGFtIHRyeWluZyB0byBydW4gYGRpc3RjcGAgdXNpbmcgYSBqYXZhIGNsYXNzLCBidXQgSSBnZXQgdGhlIGVycm9yIG9mIGNsYXNzIG5vdCBmb3VuZCBgRGlzdENwT3B0aW9uc2Au
IEkgaGF2ZSB1c2VkIHRoZSBhcmd1bWVudCBgLWxpYmphcnMgLi9zaGFyZS9oYWRvb3AvdG9vbHMv
bGliL2hhZG9vcC1kaXN0Y3AtMi42LjAuamFyYCB0byBwYXNzIHRoZSBqYXIgZmlsZSwgYnV0IGl0
IHNlZW1zIHRoYXQgaXMgbm90IHJpZ2h0LiBIb3cgSSBwYXNzIHRoZSBsaWIgcHJvcGVybHk/PGJy
Pjxicj5PdXRwdXQ6IDxicj48YnI+YGBgPGJyPnZhZ3JhbnRAaGFkb29wLWNvYy0xOn4vUHJvZ3Jh
bXMvaGFkb29wJCBoYWRvb3AgamFyIHdvcmRjb3VudC5qYXLCoCAtbGliamFycyAuL3NoYXJlL2hh
ZG9vcC90b29scy9saWIvaGFkb29wLWRpc3RjcC0yLjYuMC5qYXIgL2lucHV0MSAvb3V0cHV0bXAg
L291dHB1dDE8YnI+Q29uZmlndXJhdGlvbjogY29yZS1kZWZhdWx0LnhtbCwgY29yZS1zaXRlLnht
bCwgbWFwcmVkLWRlZmF1bHQueG1sLCBtYXByZWQtc2l0ZS54bWwsIHlhcm4tZGVmYXVsdC54bWws
IHlhcm4tc2l0ZS54bWwsIGhkZnMtZGVmYXVsdC54bWwsIGhkZnMtc2l0ZS54bWw8YnI+LWxpYmph
cnM8YnI+Li9zaGFyZS9oYWRvb3AvdG9vbHMvbGliL2hhZG9vcC1kaXN0Y3AtMi42LjAuamFyPGJy
Pi9pbnB1dDE8YnI+L291dHB1dG1wPGJyPi9vdXRwdXQxPGJyPkV4Y2VwdGlvbiBpbiB0aHJlYWQg
Im1haW4iIGphdmEubGFuZy5Ob0NsYXNzRGVmRm91bmRFcnJvcjogb3JnL2FwYWNoZS9oYWRvb3Av
dG9vbHMvRGlzdENwT3B0aW9uczxicj7CoMKgwqAgYXQgb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVk
LmV4YW1wbGVzLldvcmRDb3VudC5tYWluKFdvcmRDb3VudC5qYXZhOjEwMSk8YnI+wqDCoMKgIGF0
IHN1bi5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5pbnZva2UwKE5hdGl2ZSBNZXRo
b2QpPGJyPsKgwqDCoCBhdCBzdW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuaW52
b2tlKE5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5qYXZhOjU3KTxicj7CoMKgwqAgYXQgc3VuLnJl
ZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5pbnZva2UoRGVsZWdhdGluZ01ldGhv
ZEFjY2Vzc29ySW1wbC5qYXZhOjQzKTxicj7CoMKgwqAgYXQgamF2YS5sYW5nLnJlZmxlY3QuTWV0
aG9kLmludm9rZShNZXRob2QuamF2YTo2MDYpPGJyPsKgwqDCoCBhdCBvcmcuYXBhY2hlLmhhZG9v
cC51dGlsLlJ1bkphci5ydW4oUnVuSmFyLmphdmE6MjIxKTxicj7CoMKgwqAgYXQgb3JnLmFwYWNo
ZS5oYWRvb3AudXRpbC5SdW5KYXIubWFpbihSdW5KYXIuamF2YToxMzYpPGJyPkNhdXNlZCBieTog
amF2YS5sYW5nLkNsYXNzTm90Rm91bmRFeGNlcHRpb246IG9yZy5hcGFjaGUuaGFkb29wLnRvb2xz
LkRpc3RDcE9wdGlvbnM8YnI+wqDCoMKgIGF0IGphdmEubmV0LlVSTENsYXNzTG9hZGVyJDEucnVu
KFVSTENsYXNzTG9hZGVyLmphdmE6MzY2KTxicj7CoMKgwqAgYXQgamF2YS5uZXQuVVJMQ2xhc3NM
b2FkZXIkMS5ydW4oVVJMQ2xhc3NMb2FkZXIuamF2YTozNTUpPGJyPsKgwqDCoCBhdCBqYXZhLnNl
Y3VyaXR5LkFjY2Vzc0NvbnRyb2xsZXIuZG9Qcml2aWxlZ2VkKE5hdGl2ZSBNZXRob2QpPGJyPsKg
wqDCoCBhdCBqYXZhLm5ldC5VUkxDbGFzc0xvYWRlci5maW5kQ2xhc3MoVVJMQ2xhc3NMb2FkZXIu
amF2YTozNTQpPGJyPsKgwqDCoCBhdCBqYXZhLmxhbmcuQ2xhc3NMb2FkZXIubG9hZENsYXNzKENs
YXNzTG9hZGVyLmphdmE6NDI1KTxicj7CoMKgwqAgYXQgamF2YS5sYW5nLkNsYXNzTG9hZGVyLmxv
YWRDbGFzcyhDbGFzc0xvYWRlci5qYXZhOjM1OCk8YnI+YGBgPGJyPjxicj48YnI+TXkgY2xhc3M6
PGJyPjxicj5gYGA8YnI+cHVibGljIHN0YXRpYyB2b2lkIG1haW4oU3RyaW5nW10gYXJncykgdGhy
b3dzIEV4Y2VwdGlvbiB7PGJyPsKgwqAgwqDCoMKgIMKgQ29uZmlndXJhdGlvbiBjb25mID0gbmV3
IENvbmZpZ3VyYXRpb24oKTs8YnI+PGJyPsKgwqAgwqDCoMKgIMKgU3RyaW5nW10gb3RoZXJBcmdz
ID0gbmV3IEdlbmVyaWNPcHRpb25zUGFyc2VyKGNvbmYsIGFyZ3MpLmdldFJlbWFpbmluZ0FyZ3Mo
KTs8YnI+wqDCoCDCoMKgwqAgwqBpZiAob3RoZXJBcmdzLmxlbmd0aCAmbHQ7IDIpIHs8YnI+wqDC
oCDCoMKgwqAgwqDCoMKgIMKgU3lzdGVtLmVyci5wcmludGxuKCJVc2FnZTogd29yZGNvdW50ICZs
dDtpbiZndDsgWyZsdDtpbiZndDsuLi5dICZsdDtvdXQmZ3Q7Iik7PGJyPsKgwqAgwqDCoMKgIMKg
wqDCoCDCoFN5c3RlbS5leGl0KDIpOzxicj7CoMKgIMKgwqDCoCDCoH08YnI+PGJyPsKgwqAgwqDC
oMKgIMKgU3lzdGVtLm91dC5wcmludGxuKGNvbmYudG9TdHJpbmcoKSk7PGJyPsKgwqAgwqDCoMKg
IMKgZm9yIChpbnQgaSA9IDA7IGkgJmx0OyBhcmdzLmxlbmd0aDsgaSsrKSB7PGJyPsKgwqAgwqDC
oMKgIMKgwqDCoCDCoFN5c3RlbS5vdXQucHJpbnRsbihhcmdzW2ldKTs8YnI+wqDCoCDCoMKgwqAg
wqB9PGJyPjxicj7CoMKgIMKgwqDCoCDCoC8vIGRpc3RjcDxicj7CoMKgIMKgwqDCoCDCoFN0cmlu
ZyBwcm90byA9ICJ3ZWJoZGZzOi8vIjs8YnI+wqDCoCDCoMKgwqAgwqBTdHJpbmcgc3JjID0gImhh
ZG9vcC1jb2MtMS9pbnB1dDEiOzxicj7CoMKgIMKgwqDCoCDCoFN0cmluZyBkZXN0ID0gImhhZG9v
cC1jb2MtMi9pbnB1dDEiOzxicj7CoMKgIMKgwqDCoCDCoExpc3QmbHQ7UGF0aCZndDsgbHNyYyA9
IG5ldyBBcnJheUxpc3QmbHQ7UGF0aCZndDsoKTs8YnI+wqDCoCDCoMKgwqAgwqBsc3JjLmFkZChu
ZXcgUGF0aChzcmMpKTs8YnI+wqDCoCDCoMKgwqAgwqBEaXN0Q3BPcHRpb25zIG9wdGlvbnMgPSBu
ZXcgRGlzdENwT3B0aW9ucyhsc3JjLCBuZXcgUGF0aChkZXN0KSk7PGJyPsKgwqAgwqDCoMKgIMKg
RGlzdENwIGRpc3RjcCA9IG5ldyBEaXN0Q3AobmV3IENvbmZpZ3VyYXRpb24oKSwgb3B0aW9ucyk7
IDxicj7CoMKgIMKgwqDCoCDCoGRpc3RjcC5leGVjdXRlKCk7PGJyPsKgwqAgwqB9PGJyPmBgYDxi
        cj4="
style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0;">​</div>
    </div>
  </body>
</html>

--------------040604070207000300090409--