hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simone Leo <simone....@crs4.it>
Subject Re: Creating files through the hadoop streaming interface
Date Thu, 07 Feb 2013 15:39:41 GMT

the lack of an HDFS API is just one of the drawbacks that motivated us 
to abandon Streaming and develop Pydoop.  Unfortunately, in the blog 
post cited by Harsh J, Pydoop is just briefly mentioned because the 
author failed to build and install it.

Here is how you solve your problem in Pydoop (for details on how to run 
programs, see the docs at http://pydoop.sourceforge.net/docs):

import pydoop.pipes as pp
import pydoop.hdfs as hdfs

class Mapper(pp.Mapper):

   def __init__(self, context):
     super(Mapper, self).__init__(context)
     jc = context.getJobConf()
     fname = "%s/%s" % (jc.get("mapred.output.dir"), 
     self.fo = hdfs.open(fname, "w", user="simleo")
     self.fo = hdfs.open(fname, "a", user="simleo")

   def map(self, context):
     l = len(context.getInputValue())
     self.fo.write("%d\n" % l)

   def close(self):

class Reducer(pp.Reducer):

if __name__ == "__main__":
   pp.runTask(pp.Factory(Mapper, Reducer))

Note that I'm embedding the task attempt info into the file name, to 
avoid clashes due to different mappers trying to access the same file at 
the same time.


On 02/07/2013 06:18 AM, Harsh J wrote:
> The raw streaming interface has much issues of this manner. The python
> open(…, 'w') calls won't open files on HDFS, further. Perhaps, since
> you wish to use Python for its various advantages, check out this
> detailed comparison guide of various Python-based Hadoop frameworks
> (including the raw streaming we offer as part of Apache Hadoop) at
> http://blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/
> by Uri? Many of these provide python extensions to HDFS/etc., letting
> you do much more than plain streaming.
> On Thu, Feb 7, 2013 at 6:43 AM, Julian Bui <julianbui@gmail.com> wrote:
>> Hi hadoop users,
>> I am trying to use the streaming interface to use my python script mapper to
>> create some files but am running into difficulties actually creating files
>> on the hdfs.
>> I have a python script mapper with no reducers.  Currently, it doesn't even
>> read the input and instead reads in the env variable for the output dir
>> (outdir = os.environ['mapred_output_dir']) and attempts to create an empty
>> file at that location.  However, that appears to fail with the [vague] error
>> message appended to this email.
>> I am using the streaming interface because the python file examples seem so
>> much cleaner and abstract a lot of the details away for me but if I instead
>> need to use the java bindings (and create a mapper and reducer class) then
>> please let me know.  I'm still learning hadoop.  As I understand it, I
>> should be able to create files in hadoop but perhaps there is limited
>> ability while using the streaming i/o interface.
>> Further questions: If my mapper absolutely must send my output to stdout, is
>> there a way to rename the file after it has been created?
>> Please help.
>> Thanks,
>> -Julian
>> Python mapper code:
>> outdir = os.environ['mapred_output_dir']
>> f = open(outdir + "/testfile.txt", "wb")
>> f.close()
>> 13/02/06 17:07:55 INFO streaming.StreamJob:  map 100%  reduce 100%
>> 13/02/06 17:07:55 INFO streaming.StreamJob: To kill this job, run:
>> 13/02/06 17:07:55 INFO streaming.StreamJob:
>> /opt/hadoop/libexec/../bin/hadoop job
>> -Dmapred.job.tracker=gcn-13-88.ibnet0:54311 -kill job_201302061706_0001
>> 13/02/06 17:07:55 INFO streaming.StreamJob: Tracking URL:
>> http://gcn-13-88.ibnet0:50030/jobdetails.jsp?jobid=job_201302061706_0001
>> 13/02/06 17:07:55 ERROR streaming.StreamJob: Job not successful. Error: # of
>> failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask:
>> task_201302061706_0001_m_000000
>> 13/02/06 17:07:55 INFO streaming.StreamJob: killJob...
>> Streaming Command Failed!
> --
> Harsh J

Simone Leo
Data Fusion - Distributed Computing
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simone.leo@crs4.it

View raw message