hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10820) Empty entry in libjars results in working directory being recursively localized
Date Tue, 12 Aug 2014 00:32:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093560#comment-14093560
] 

Andrew Wang commented on HADOOP-10820:
--------------------------------------

Hi Zhihai, only comment I have is that it would be nice to validate for empty string before
the file list length, e.g.:

{noformat}
-> % hadoop fs -files ,, 
Exception in thread "main" java.lang.IllegalArgumentException: File list length can't be zero
{noformat}

I think if we just need to do this check on the finalArr instead at the end. +1 pending this.

> Empty entry in libjars results in working directory being recursively localized
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-10820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10820
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Alex Holmes
>            Priority: Minor
>         Attachments: HADOOP-10820-1.patch, HADOOP-10820-2.patch, HADOOP-10820.patch
>
>
> An empty token (e.g. "a.jar,,b.jar") in the -libjars option causes the current working
directory to be recursively localized.
> Here's an example of this in action (using Hadoop 2.2.0):
> {code}
> # create a temp directory and touch three JAR files
> mkdir -p tmp/path && cd tmp && touch a.jar b.jar c.jar path/d.jar
> # Run an example job only specifying two of the JARs.
> # Include an empty entry in libjars.
> hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
pi -libjars a.jar,,c.jar 2 1000000000
> # As the job is running examine the localized directory in HDFS.
> # Notice that not only are the two JAR's specified in libjars copied,
> # but in addition the contents of the working directory are also recursively copied.
> $ hadoop fs -lsr /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/a.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/c.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/a.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/b.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/c.jar
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/path
> /tmp/hadoop-yarn/staging/aholmes/.staging/job_1404752711144_0018/libjars/tmp/path/d.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message