incubator-oozie-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Islam <misla...@yahoo.com>
Subject Re: Oozie streaming script not found on nodes
Date Sun, 02 Oct 2011 08:40:42 GMT
Hi Sridhar,
Please excuse my limited knowledge of python.

How the import in python works? Does it look into current directory by default?

You could double check the current directory from where the actual streaming job was launched
by accessing the launcher mapper log. You might already knew that Launcher mapper is started
by oozie for most of the actions. You could find it in your JT job list with name something
oozie launcher. The first few lines of the task attempt log show the list of the files in
current dir. If possible, please cut-paste the relevant log.

If you find the contents of the current directory are as expected, please create an exact
test environment in your local setup and run the command. 

Regards,
Mohammad


 


________________________________
From: Sridhar Narayanan <srinar@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oozie-users@incubator.apache.org>
Sent: Friday, September 30, 2011 4:39 AM
Subject: Oozie streaming script not found on nodes

I am running a workflow- currently only single actions at a time for
diagnostics.
                                    Workflow looks like(relevant section):
<action name='preclean'>
            <streaming>
                <mapper>python mapper1 </mapper>
                <reducer>python reducer1 </reducer>
            </streaming>
                     ...
                     ...
<file>$path/mapper1#mapper1</file>
            <file>$path/reducer1#reducer1</file>
            <file>$path/tools.py#tools.py</file>
            <file>$path/streaming.py#streaming.py</file>
Note: $path and all other variables are defined in the config file.
         Files exist at these locations
         All action stages work executed through a script
On running the workflow(OOZIE URL set, correct config file,
updated(3.0.2) version of oozie, examples run fine), ALL map tasks
fail and the following is the error :
                                                          Error:
File "reducer1", line 10, in ?
    import streaming
ImportError: No module named streaming
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
failed with code 1
                                                        Questions:
1. Does the path of the file point to a HDFS location or home?
(currently points to the HDFS location, which is a mirror image of
home, for this purpose)
2. Is this a permissions problem?
3. Does forcing sim-links create this problem?(I have had earlier
problems because the mapper cannot follow sim-links, but either way
doesn't change the error here)
(All paths are absolute, and correct. All scripts work. All
archives(where used) are uncorrupted)
Advice is much appreciated. Please feel free to ask for additional
information.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message