Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 72144D446 for ; Fri, 24 May 2013 17:44:34 +0000 (UTC) Received: (qmail 48987 invoked by uid 500); 24 May 2013 17:44:29 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 48874 invoked by uid 500); 24 May 2013 17:44:29 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 48854 invoked by uid 99); 24 May 2013 17:44:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 May 2013 17:44:28 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Sanjay.Subramanian@wizecommerce.com designates 216.32.181.181 as permitted sender) Received: from [216.32.181.181] (HELO ch1outboundpool.messaging.microsoft.com) (216.32.181.181) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 May 2013 17:44:20 +0000 Received: from mail47-ch1-R.bigfish.com (10.43.68.229) by CH1EHSOBE002.bigfish.com (10.43.70.52) with Microsoft SMTP Server id 14.1.225.23; Fri, 24 May 2013 17:43:58 +0000 Received: from mail47-ch1 (localhost [127.0.0.1]) by mail47-ch1-R.bigfish.com (Postfix) with ESMTP id B2F79160354 for ; Fri, 24 May 2013 17:43:58 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.232.197;KIP:(null);UIP:(null);IPV:NLI;H:BLUPRD0411HT002.namprd04.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: -3 X-BigFish: PS-3(zz936eI181fMc85ehdd85kzz1f42h1ee6h1de0h1fdah1202h1e76h1d1ah1d2ah1fc6hzz18c673h8275bhz2fh2a8h668h839hbe3he5bhf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d0ch1d2eh1d3fh1dc1h1155h) Received-SPF: pass (mail47-ch1: domain of wizecommerce.com designates 157.56.232.197 as permitted sender) client-ip=157.56.232.197; envelope-from=Sanjay.Subramanian@wizecommerce.com; helo=BLUPRD0411HT002.namprd04.prod.outlook.com ;.outlook.com ; Received: from mail47-ch1 (localhost.localdomain [127.0.0.1]) by mail47-ch1 (MessageSwitch) id 1369417436626291_6101; Fri, 24 May 2013 17:43:56 +0000 (UTC) Received: from CH1EHSMHS019.bigfish.com (snatpool1.int.messaging.microsoft.com [10.43.68.241]) by mail47-ch1.bigfish.com (Postfix) with ESMTP id 8B9791400C5 for ; Fri, 24 May 2013 17:43:56 +0000 (UTC) Received: from BLUPRD0411HT002.namprd04.prod.outlook.com (157.56.232.197) by CH1EHSMHS019.bigfish.com (10.43.70.19) with Microsoft SMTP Server (TLS) id 14.1.225.23; Fri, 24 May 2013 17:43:51 +0000 Received: from BLUPRD0411MB426.namprd04.prod.outlook.com ([169.254.10.213]) by BLUPRD0411HT002.namprd04.prod.outlook.com ([10.255.127.37]) with mapi id 14.16.0311.000; Fri, 24 May 2013 17:43:49 +0000 From: Sanjay Subramanian To: "user@hadoop.apache.org" Subject: Re: Where to begin from?? Thread-Topic: Where to begin from?? Thread-Index: AQHOWCgR0Rz2xWYiakGHlW3WtVXI25kToa+A//+PT4CAAIaLAIAACWMAgAAFngCAAGBJgA== Date: Fri, 24 May 2013 17:43:48 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.255.127.4] Content-Type: multipart/alternative; boundary="_000_CDC4F0B77C5Bsanjaysubramanianwizecommercecom_" MIME-Version: 1.0 X-OriginatorOrg: wizecommerce.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_CDC4F0B77C5Bsanjaysubramanianwizecommercecom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hey guys Is there a way to dynamically change the input dir and outputdir I have the following CONSTANT directories in HDFS * /path/to/input/9999-99-99 (empty directory ) * /path/to/output/9999-99-99 (empty directory) A new directory with yesterdays date like /path/to/input/2013-05-23 gets cr= eated every day I set the job params mapreduce.input.fileinputformat.inputdir=3D/path/to/input/9999-99-99 mapreduce.output.fileoutputformat.outputdir=3D/path/to/output/9999-99-99 But in my Mapper I thought I can sneak in this code=85.But it does not work= ? protected void setup(Context context) throws IOException, InterruptedException{ org.apache.hadoop.conf.Configuration hadoopConf =3D ((JobContext)conte= xt).getConfiguration(); String inputDir =3D hadoopConf.get("mapreduce.input.fileinputformat.inp= utdir"); String outputDir =3D hadoopConf.get("mapreduce.output.fileoutputformat.= outputdir"); String yesterdaysDate =3DDateUtils.getDayYYYYMMDD("-", -1); String inputDirUseThis =3D inputDir.replaceAll("9999-99-99", yesterdays= Date); String outputDirUseThis =3D inputDir.replaceAll("9999-99-99", yesterday= sDate); hadoopConf.set("mapreduce.input.fileinputformat.inputdir",inputDirUseTh= is); hadoopConf.set("mapreduce.output.fileoutputformat.outputdir",outputDirU= seThis); } Thanks sanjay CONFIDENTIALITY NOTICE =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply ema= il and destroy all copies of the original message along with any attachment= s, from your computer system. If you are the intended recipient, please be = advised that the content of this message is subject to access, review and d= isclosure by the sender's Email System Administrator. --_000_CDC4F0B77C5Bsanjaysubramanianwizecommercecom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <5C1E3D3C245D9F4ABA68D8FBF16163DD@namprd04.prod.outlook.com> Content-Transfer-Encoding: quoted-printable
Hey guys 

Is there a way to dynamically change the in= put dir and outputdir

I have the following CONSTANT directories i= n HDFS 
  • /path/to/input/9999-99-99 (empty directory )
  • /path/to/output/99= 99-99-99 (empty directory)
A new directory with yesterdays date like&n= bsp;/path/to/input/2013-05-23 gets created every day 

I set the job params
mapreduce.input.fileinputformat.inputdir=3D= /path/to/input/9999-99-99
mapreduce.output.fileoutputformat.outputdir= =3D/path/to/output/9999-99-99

But in my Mapper I = thought I can sneak in this code=85.But it does not work ?

  &n= bsp; protected void setup(Context context)

  &n= bsp;         throws IOException,

 &nb= sp;                 InterruptedExce= ption{

  &n= bsp;  org.apache.hadoop.conf.Configuration hadoopConf =3D ((JobContext)con= text).getConfiguration();

  &n= bsp; String inputDir =3D hadoopConf.get("mapreduce.in= put.fileinputformat.inputdir");

  &n= bsp; String outputDir = =3D hadoopConf.get("mapreduce.output.fileoutputformat.o= utputdir");

  &n= bsp; String yesterdaysDate =3DDateUtils.getDayYYYYMMDD("-&quo= t;, -1);

  &n= bsp; String inputDirUseThis =3D inputDir.replaceAll("9999-99-99"= ;, yesterdaysDate);

  &n= bsp; String outputDirUseThis =3D inputDir.replaceAll("9999-99-99&quo= t;, yesterdaysDate);

  &n= bsp; hadoopConf.set("mapreduce.input.fileinputformat.= inputdir",inputDirUseThis);

  &n= bsp; hadoopConf.set("mapreduce.output.fileoutputforma= t.outputdir",outputDirUseThis);

  &n= bsp; }


Thanks

sanjay
  

CONFIDENTIALITY NOTICE
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply email and destroy all copies of the ori= ginal message along with any attachments, from your computer system. If you= are the intended recipient, please be advised that the content of this mes= sage is subject to access, review and disclosure by the sender's Email System Administrator.
--_000_CDC4F0B77C5Bsanjaysubramanianwizecommercecom_--