hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Valentin Logvinskiy (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13939) hadoop jar -libjars not working with mainClass argument
Date Tue, 27 Dec 2016 22:55:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15781458#comment-15781458
] 

Valentin Logvinskiy edited comment on HADOOP-13939 at 12/27/16 10:55 PM:
-------------------------------------------------------------------------

[~templedf] Yes, just rechecked.

{code}
public class MainClass extends Configured implements Tool {
    private static final Logger log = LoggerFactory.getLogger(MainClass.class);
    public static void main(String[] args) throws Exception {
        int res = ToolRunner.run(HBaseConfiguration.create(), new MainClass(), args);
        System.exit(res);
    }
    @Override
    public int run(String[] args) throws Exception {
        Configuration conf = this.getConf();
        log.info(conf.get("tmpjars"));
        return 0;
    }
}
{code}

{noformat}
hadoop jar file.jar MainClass -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:21,252  INFO EventsToJourneysJobMain:69 -
{noformat}

{noformat}
hadoop jar file.jar -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:42,677  INFO EventsToJourneysJobMain:69 - file:/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
{noformat}

More information with debug file GenericOptionsParser on line:
{code}
this.commandLine = parser.parse(opts, this.preProcessForWindows(args), true);   //version
2.7.2 from maven repo
{code}

With MainClass:
{noformat}
commandLine = {CommandLine@1337} 
 args = {LinkedList@1343}  size = 3
  0 = "MainClass"
  1 = "-libjars"
  2 = "/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar"
 options = {ArrayList@1344}  size = 0
{noformat}

Without MainClass argument:
{noformat}
commandLine = {CommandLine@1337} 
 args = {LinkedList@1342}  size = 0
 options = {ArrayList@1343}  size = 1
  0 = {Option@1345} "[ option: libjars  [ARG] :: comma separated jar files to include in the
classpath. ]"
   opt = "libjars"
   longOpt = null
   argName = "paths"
   description = "comma separated jar files to include in the classpath."
   required = false
   optionalArg = false
   numberOfArgs = 1
   type = null
   values = {ArrayList@1350}  size = 1
   valuesep = '\u0000' 0
{noformat}


was (Author: gron1987):
[~templedf] Yes, just recheck.

{code}
public class MainClass extends Configured implements Tool {
    private static final Logger log = LoggerFactory.getLogger(MainClass.class);
    public static void main(String[] args) throws Exception {
        int res = ToolRunner.run(HBaseConfiguration.create(), new MainClass(), args);
        System.exit(res);
    }
    @Override
    public int run(String[] args) throws Exception {
        Configuration conf = this.getConf();
        log.info(conf.get("tmpjars"));
        return 0;
    }
}
{code}

{noformat}
hadoop jar file.jar MainClass -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:21,252  INFO EventsToJourneysJobMain:69 -
{noformat}

{noformat}
hadoop jar file.jar -libjars /Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
23:48:42,677  INFO EventsToJourneysJobMain:69 - file:/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar
{noformat}

More information with debug file GenericOptionsParser on line:
{code}
this.commandLine = parser.parse(opts, this.preProcessForWindows(args), true);   //version
2.7.2 from maven repo
{code}

With MainClass:
{noformat}
commandLine = {CommandLine@1337} 
 args = {LinkedList@1343}  size = 3
  0 = "MainClass"
  1 = "-libjars"
  2 = "/Users/vlogvinskiy/.m2/repository/com/test/0.0.1/test-0.0.1.jar"
 options = {ArrayList@1344}  size = 0
{noformat}

Without MainClass argument:
{noformat}
commandLine = {CommandLine@1337} 
 args = {LinkedList@1342}  size = 0
 options = {ArrayList@1343}  size = 1
  0 = {Option@1345} "[ option: libjars  [ARG] :: comma separated jar files to include in the
classpath. ]"
   opt = "libjars"
   longOpt = null
   argName = "paths"
   description = "comma separated jar files to include in the classpath."
   required = false
   optionalArg = false
   numberOfArgs = 1
   type = null
   values = {ArrayList@1350}  size = 1
   valuesep = '\u0000' 0
{noformat}

> hadoop jar -libjars not working with mainClass argument
> -------------------------------------------------------
>
>                 Key: HADOOP-13939
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13939
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.7.2
>            Reporter: Valentin Logvinskiy
>
> If we have 
> {code:java}
> public class MainClass extends Configured implements Tool {
>     public static void main(String[] args) throws Exception {
>         ToolRunner.run(conf, new MainClass(), args);
>     }
>     
>     ...
>     
>    @Override
>     public int run(String[] args) throws Exception {
>         ...
>     }
> }
> {code}
> Running this code with -libjars argument in distributed mode will not work.
> {noformat}
> hadoop jar file.jar MainClass -libjars /path/to/file
> {noformat}
> This not working due to error in GenericOptionsParser
> GenericOptionsParser.java:
> {code}
> commandLine = parser.parse(opts, args, true);
> {code}
> As far as 3rd argument is "true" this mean:
> {noformat}
> * @param stopAtNonOption specifies whether to continue parsing the
>      * arguments if a non option is encountered.
> {noformat}
> Our run arguments are:
> {noformat}
> MainClass -libjars /path/to/file
> {noformat}
> MainClass is nonOption and after this argument everything else don't start parsing.
> If we have only arguments then processGeneralOptions method will return nothing and will
not add tmpjars config.
> Changing to 
> {noformat}
> commandLine = parser.parse(opts, args, false); 
> {noformat}
> works for me, but could affect a lot of developers (in theory)
> Fast fix - do not use mainClass argument at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message