hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitesh Shah (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
Date Fri, 07 Oct 2011 01:42:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122483#comment-13122483
] 

Hitesh Shah commented on MAPREDUCE-3123:
----------------------------------------

Vinod/Thomas/Arun - a query on task.sh - the final command i.e. "exec setsid /bin/bash -c
..." encapsulates the user provided command with "". The assumption here seems to be they
can use environment variables with the command for expansion and that the onus of correctly
escaping the command i.e escaping any special chars is on the users themselves. Would that
be correct?

For example, creating the symlink above is pretty simply by just quoting the file name but
if the symlink is something that will be executed then special chars like "&" or even
basic things like spaces can create problems if not quoted and escaped properly.  
                
> Symbolic links with special chars causing container/task.sh to fail
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3123
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Thomas Graves
>            Assignee: Hitesh Shah
>            Priority: Blocker
>             Fix For: 0.23.0
>
>
> the following job throws an exception when you have the special characters in it.
> hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -Dmapreduce.job.queuename=queue1
-files file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+=
-input Streaming/streaming-980/input.txt  -mapper 'xargs cat' -reducer cat -output Streaming/streaming-980/Output
-jobconf mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=*
> Exception:
> 2011-09-27 20:58:48,903 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor:
launchContainer:
> [container-executor, hadoopuser, 1, application_1317077272567_0239,
> container_1317077272567 0239_01_000001,
> tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001,
> tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01
000001/task.sh,
> tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_000001/container_1317077272567_0239_01_000001.tokens]1109221111-tests.jar:hadoop-mapreduce-p2011-09-27
> 20:58:48,944 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit
code from container is : 2    
>                                                                                     
                      2011-09-27
> 20:58:48,946 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
from container-launch :  
>                                                                                     
                     
> org.apache.hadoop.util.Shell$ExitCodeException:
> /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001/task.sh:
> line 26: syntax error near unexpected token `-_+='       
> /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_000001/task.sh:
> line 26: `ln -sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir
test
> ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar:
>      at org.apache.hadoop.util.Shell.runCommand(Shell.java:261)                     
                                  
>                                                                                     
                                  
>    at org.apache.hadoop.util.Shell.run(Shell.java:188)                              
                                  
>                                                                                     
                                  
>  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)       
                                  
>                                                                                     
                                 
> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174)
  
>                                                                                     
                                
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197)
 
>                                                                                     
                                
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> 2011-09-27 20:58:48,951 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
> 2011-09-27 20:58:48,951 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Processing
> container_1317077272567_0239_01_000001 of type UPDATE_DIAGNOSTICS_MSG
> 2011-09-27 20:58:48,951 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Container exited with a non-zero exit code 2

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message