hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-477) Streaming should execute Unix commands and scripts in well known languages without user specifying the path
Date Tue, 29 Aug 2006 04:23:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-477?page=comments#action_12431134 ] 
Owen O'Malley commented on HADOOP-477:

The simplest way to fix this is to make sure that the environment PATH, etc. of the account
that starts the TaskTrackers is reasonable on each of the nodes.

A slightly more involved fix would be to create a config variable in the TaskTracker's config
that specifies the PATH for the Tasks and have the TaskTracker set it on the child process.
But that leads to a fairly slippery slope of throwing more and more process environment into
the config.

> Streaming should execute Unix commands and scripts in well known languages without user
specifying the path
> -----------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-477
>                 URL: http://issues.apache.org/jira/browse/HADOOP-477
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/streaming
>            Reporter: arkady borkovsky
> If the executables for -mapper or -reducer are well-known (grep, cat, awk), Streaming
should make sure that the executable is found.
> If a script  for -mapper or -reducer are in a well-known language (.pl, .py), Streaming
should  execute it  with the correct language processor.
> Reason:
> many jobs get started from machines with a different environment from that on the cluster.
> On another hand, different clusters may have different environments.  
> Also, a user may have no access to the cluster machines.
> Because of this, a user may be unable to specify correct paths for standard commands,
and correct language processors for scripts.
> Implementation:
> Stream may tailr the commands by prepending the path, or the name of language processor.
> Another solution is to make sure that the commands are executed in a "meaningful" environment
(with good $PATH, and other variables Unix users are accustomed to count upon).
> Once again, Streaming is user facing tool -- it is not a library or a hackable example
that the users are to modify for their needs.  So it should work out of the box.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message