hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From smriti singh <smritis...@gmail.com>
Subject Re: How to access contents of a Map Reduce job's working directory
Date Tue, 02 Aug 2011 10:18:19 GMT
Hi Harsh, let me explain this in detail:

this is what I am trying to do in my mapper:

File setupFolder = new File(setupFolderName);


MARD mard = new MARD(setupFolder);

Text valuz = new Text();

IntWritable intval = new IntWritable();

File original = new File("Vca1652.txt");

File mardedxml = new File("Vca1652-mardedxml.txt");

File marded = new File("Vca1652-marded.txt");



NormalisationStats stats;

try {

stats = mard.normaliseFile(original,mardedxml,marded,50.0);

//This meathod requires access to the myMardfolder


} catch (MARDException e) {

// TODO Auto-generated catch block



1. This mard.normalise() creates files in the "setup" folder.
2. I have no control on this method as I just a got a jar mard.jar to call
this meathod.
3. Mard.normalise() searches for folder called "foul" in the working
directory. If not found throws on Exception. It is this folder's data that
mard.normalise() method process on to generate files in the "setup"
folder. I passed this folder to the working directory through the
-archives option (**by first compressing it)
3. I am not using the "input path" data in anyway in the mapper.
4. Hence not using the key and values generated.
5. I am using and Identity reducer as there is no need of any reduction .
6. Hence also the output is of no use for me.
7. I need to get the content of "setup folder", but I dont know the method
to do so.

* I might be wrong in the way I am doing it because I had no formal hadoop
training I have just learned it by reading articles on the net.

Thanking you in anticipation



On Tue, Aug 2, 2011 at 11:33 AM, Harsh J <harsh@cloudera.com> wrote:

> Smriti,
> By working directory, do you mean the task attempt's working directory
> or the global job staging directory?
> On Tue, Aug 2, 2011 at 6:22 AM, smriti singh <smritisoft@gmail.com> wrote:
> > I want to run a MapReduce job in hadoop which needs to create a "setup"
> > folder in working directory. During the execution the job will generate
> > some additional text files within this "setup" folder. The problem is I
> > dont know how to access or move this setup folder content to my local
> file
> > system as at end of the job, the job directory will be cleaned up.
> >
> > It would be great if you can help.
> > Regards
> > Smriti
> --
> Harsh J

View raw message