hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jitendra Yadav <jeetuyadav200...@gmail.com>
Subject Re: Error while using the Hadoop Streaming
Date Fri, 24 May 2013 18:10:02 GMT
Hi,

In your first mail you were using "/usr/bin/python" binary file just after
"- mapper", I don't think we need python executable to run this example.

Make sure that you are using correct path of you files "mapper.py and
reducer.py"  while executing.


~Thanks



On Fri, May 24, 2013 at 11:31 PM, Adamantios Corais <
adamantios.corais@gmail.com> wrote:

> Hi,
>
> Thanks a lot for your response.
>
> Unfortunately, I run into the same problem though.
>
> What do you mean by "python binary"? This is what I have in the very first
> line of both scripts: #!/usr/bin/python
>
> Any ideas?
>
>
> On Fri, May 24, 2013 at 7:41 PM, Jitendra Yadav <
> jeetuyadav200890@gmail.com> wrote:
>
>> Hi,
>>
>> I have run Michael's python map reduce example several times without any
>> issue.
>>
>> I think this issue is related to your file path 'mapper.py'.  you are
>> using python binary?
>>
>> try this,
>>
>> hadoop jar
>> /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/hadoop-streaming-1.1.2.jar
>> \
>>  -input /user/yyy/20417-8.txt \
>> -output /user/yyy/output \
>>  -file /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \
>> -mapper /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \
>>  -file /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py \
>> -reducer /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py
>>
>>
>> Thanks~
>>
>> On Fri, May 24, 2013 at 10:12 PM, Adamantios Corais <
>> adamantios.corais@gmail.com> wrote:
>>
>>> I tried this nice example:
>>> http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
>>>
>>> The python scripts work pretty fine from my laptop (through terminal),
>>> but they don't when I execute them on the CDH3 (Pseudo-Distributed Mode).
>>>
>>> Any ideas?
>>>
>>> hadoop jar
>>> /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/hadoop-streaming-1.1.2.jar
>>> \
>>> -input /user/yyy/20417-8.txt \
>>> -output /user/yyy/output \
>>> -file /usr/bin/python
>>> /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \
>>> -mapper /usr/bin/python
>>> /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \
>>> -file /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py \
>>> -reducer /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py
>>>
>>> -----------------------
>>>
>>> 2013-05-24 18:21:12,232 INFO org.apache.hadoop.util.NativeCodeLoader:
>>> Loaded the native-hadoop library
>>> 2013-05-24 18:21:12,569 INFO
>>> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
>>> symlink:
>>> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/jars/job.jar
>>> <-
>>> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/attempt_201305160832_0020_m_000000_0/work/job.jar
>>> 2013-05-24 18:21:12,586 INFO
>>> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
>>> symlink:
>>> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/jars/.job.jar.crc
>>> <-
>>> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/attempt_201305160832_0020_m_000000_0/work/.job.jar.crc
>>> 2013-05-24 18:21:12,717 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>>> Initializing JVM Metrics with processName=MAP, sessionId=
>>> 2013-05-24 18:21:13,062 INFO org.apache.hadoop.util.ProcessTree: setsid
>>> exited with exit code 0
>>> 2013-05-24 18:21:13,087 INFO org.apache.hadoop.mapred.Task:  Using
>>> ResourceCalculatorPlugin :
>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1358f03
>>> 2013-05-24 18:21:13,452 WARN
>>> org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is
>>> available
>>> 2013-05-24 18:21:13,452 INFO
>>> org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library
>>> loaded
>>> 2013-05-24 18:21:13,464 INFO org.apache.hadoop.mapred.MapTask:
>>> numReduceTasks: 1
>>> 2013-05-24 18:21:13,477 INFO org.apache.hadoop.mapred.MapTask:
>>> io.sort.mb = 100
>>> 2013-05-24 18:21:13,635 INFO org.apache.hadoop.mapred.MapTask: data
>>> buffer = 79691776/99614720
>>> 2013-05-24 18:21:13,635 INFO org.apache.hadoop.mapred.MapTask: record
>>> buffer = 262144/327680
>>> 2013-05-24 18:21:13,724 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2013-05-24 18:21:13,733 INFO org.apache.hadoop.streaming.PipeMapRed:
>>> PipeMapRed exec [mapper.py]
>>> 2013-05-24 18:21:13,783 ERROR org.apache.hadoop.streaming.PipeMapRed:
>>> configuration exception
>>> java.io.IOException: Cannot run program "mapper.py":
>>> java.io.IOException: error=2, No such file or directory
>>>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>>>     at
>>> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
>>>     at
>>> org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>>     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>>>     at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>>     at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
>>>     at org.apache.hadoop.mapred.Child.main(Child.java:260)
>>> Caused by: java.io.IOException: java.io.IOException: error=2, No such
>>> file or directory
>>>     at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>>     at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>>>     ... 24 more
>>> 2013-05-24 18:21:13,869 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2013-05-24 18:21:13,871 WARN org.apache.hadoop.mapred.Child: Error
>>> running child
>>> java.lang.RuntimeException: Error in configuring object
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>>>     at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>>     at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
>>>     at org.apache.hadoop.mapred.Child.main(Child.java:260)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>>>     ... 9 more
>>> Caused by: java.lang.RuntimeException: Error in configuring object
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>>     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>>>     ... 14 more
>>> Caused by: java.lang.reflect.InvocationTargetException
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>     at
>>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>>>     ... 17 more
>>> Caused by: java.lang.RuntimeException: configuration exception
>>>     at
>>> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230)
>>>     at
>>> org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
>>>     ... 22 more
>>> Caused by: java.io.IOException: Cannot run program "mapper.py":
>>> java.io.IOException: error=2, No such file or directory
>>>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>>>     at
>>> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
>>>     ... 23 more
>>> Caused by: java.io.IOException: java.io.IOException: error=2, No such
>>> file or directory
>>>     at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>>     at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>>>     ... 24 more
>>> 2013-05-24 18:21:13,879 INFO org.apache.hadoop.mapred.Task: Runnning
>>> cleanup for the task
>>>
>>
>>
>

Mime
View raw message