Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 738F867A5 for ; Tue, 2 Aug 2011 10:19:19 +0000 (UTC) Received: (qmail 37507 invoked by uid 500); 2 Aug 2011 10:19:16 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 36820 invoked by uid 500); 2 Aug 2011 10:18:59 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 36784 invoked by uid 99); 2 Aug 2011 10:18:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2011 10:18:48 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of smritisoft@gmail.com designates 209.85.210.176 as permitted sender) Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2011 10:18:40 +0000 Received: by iyi20 with SMTP id 20so11672609iyi.35 for ; Tue, 02 Aug 2011 03:18:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=DtohNmRqZL8/pbMnPNC/A9wlI2rgtoipKXMztherzoA=; b=ct4wAq6PM9smxNtaT29TmMRVAQXZyMqZlQXwLBx/NGYFriiHxldE5bFW65rq/wXKoS hjUAJETST4Xd1twg/JsUq5pyvxoPVhHnIrb40vAw7xiSqU/nhGWqlxWef0BCwVWox6+j ZjOqtObi/MGopKDJtaE6ZZjyBz7sL24iobS4Y= MIME-Version: 1.0 Received: by 10.231.190.204 with SMTP id dj12mr736882ibb.195.1312280299667; Tue, 02 Aug 2011 03:18:19 -0700 (PDT) Received: by 10.231.33.11 with HTTP; Tue, 2 Aug 2011 03:18:19 -0700 (PDT) In-Reply-To: References: Date: Tue, 2 Aug 2011 15:48:19 +0530 Message-ID: Subject: Re: How to access contents of a Map Reduce job's working directory From: smriti singh To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016363b888c1b817804a983114b X-Virus-Checked: Checked by ClamAV on apache.org --0016363b888c1b817804a983114b Content-Type: text/plain; charset=ISO-8859-1 Hi Harsh, let me explain this in detail: this is what I am trying to do in my mapper: File setupFolder = new File(setupFolderName); setupFolder.mkdirs(); MARD mard = new MARD(setupFolder); Text valuz = new Text(); IntWritable intval = new IntWritable(); File original = new File("Vca1652.txt"); File mardedxml = new File("Vca1652-mardedxml.txt"); File marded = new File("Vca1652-marded.txt"); mardedxml.createNewFile(); marded.createNewFile(); NormalisationStats stats; try { stats = mard.normaliseFile(original,mardedxml,marded,50.0); //This meathod requires access to the myMardfolder System.out.println(stats); } catch (MARDException e) { // TODO Auto-generated catch block e.printStackTrace(); } Now, 1. This mard.normalise() creates files in the "setup" folder. 2. I have no control on this method as I just a got a jar mard.jar to call this meathod. 3. Mard.normalise() searches for folder called "foul" in the working directory. If not found throws on Exception. It is this folder's data that mard.normalise() method process on to generate files in the "setup" folder. I passed this folder to the working directory through the -archives option (**by first compressing it) 3. I am not using the "input path" data in anyway in the mapper. 4. Hence not using the key and values generated. 5. I am using and Identity reducer as there is no need of any reduction . 6. Hence also the output is of no use for me. 7. I need to get the content of "setup folder", but I dont know the method to do so. * I might be wrong in the way I am doing it because I had no formal hadoop training I have just learned it by reading articles on the net. Thanking you in anticipation Regards Smriti On Tue, Aug 2, 2011 at 11:33 AM, Harsh J wrote: > Smriti, > > By working directory, do you mean the task attempt's working directory > or the global job staging directory? > > On Tue, Aug 2, 2011 at 6:22 AM, smriti singh wrote: > > I want to run a MapReduce job in hadoop which needs to create a "setup" > > folder in working directory. During the execution the job will generate > > some additional text files within this "setup" folder. The problem is I > > dont know how to access or move this setup folder content to my local > file > > system as at end of the job, the job directory will be cleaned up. > > > > It would be great if you can help. > > Regards > > Smriti > > > > -- > Harsh J > --0016363b888c1b817804a983114b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Harsh, let me explain this in detail:

this is what I am trying to do in my mapper:<= br>
File setupFolder =3D new File(setupFolderName);

setupFolder.mkdi= rs();



MARD mard =3D new MARD(setupFolder);

Text valuz= =3D new Text();

IntWritable intval =3D new IntWritable();

Fi= le original =3D new File("Vca1652.txt");

File mardedxml =3D new File("Vca1652-mardedxml.txt");

= File marded =3D new File("Vca1652-marded.txt");



ma= rdedxml.createNewFile();

marded.createNewFile();

Normalisatio= nStats stats;

try {

stats =3D mard.normaliseFile(original,mardedxml,marded,50.= 0);

//This meathod requires access to the myMardfolder


Sy= stem.out.println(stats);

} catch (MARDException e) {

// TODO = Auto-generated catch block

e.printStackTrace();

}


Now,
1. This mard.normalise() creates files in the "setup" folder.
= 2. I have no control on this method as I just a got a jar mard.jar to call<= br>this meathod.
3. Mard.normalise() searches for folder called "fo= ul" in the working
directory. If not found throws on Exception. It is this folder's data t= hat
mard.normalise() method process on to generate files in the "se= tup"
folder. I passed this folder to the working directory through = the
-archives option (**by first compressing it)
3. I am not using the "= ;input path" data in anyway in the mapper.
4. Hence not using the k= ey and values generated.
5. I am using and Identity reducer as there is = no need of any reduction .
6. Hence also the output is of no use for me.
7. I need to get the conte= nt of "setup folder", but I dont know the method
to do so.
=

* I might be wrong in the way I am doing it because I had no formal= hadoop
training I have just learned it by reading articles on the net.

Thanking you in anticipation

Regards

Smriti

<= span class=3D"Apple-style-span" style=3D"border-collapse: collapse;">
On Tue, Aug 2, 2011 at 11:33 AM, H= arsh J <harsh@cl= oudera.com> wrote:
Smriti,

By working directory, do you mean the task attempt's working directory<= br> or the global job staging directory?

On Tue, Aug 2, 2011 at 6:22 AM, smriti singh <smritisoft@gmail.com> wrote:
> I want to run a MapReduce job in hadoop which needs to create a "= setup"
> folder in working directory. During the execution the job will generat= e
> some additional text files within this "setup" folder. The p= roblem is I
> dont know how to access or move this setup folder content to my local = file
> system as at end of the job, the job directory will be cleaned up.
>
> It would be great if you can help.
> Regards
> Smriti



--
Harsh J

--0016363b888c1b817804a983114b--