Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of sshi@pivotal.io designates
 74.125.82.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <FEE0CE3FD27B5B4B8D29843789038BA88C72D0246B@MX17A.corp.emc.com>
References: <FEE0CE3FD27B5B4B8D29843789038BA88C72D0246B@MX17A.corp.emc.com>
Date: Thu, 28 Aug 2014 14:15:23 +0800
Message-ID: 
 <CAKjCjS9L+e7yp-ivegPH1tine1Yd2WukAcqx7gxtqkkcJe-_pg@mail.gmail.com>
Subject: Re: What happens when .....?
From: Stanley Shi <sshi@pivotal.io>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7bb0398c454f330501aa760b

--047d7bb0398c454f330501aa760b
Content-Type: text/plain; charset=UTF-8

Normally MR job is used for batch processing. So I don't think this is a
good use case here for MR.
Since you need to run the program periodically, you cannot submit a single
mapreduce job for this.
An possible way is to create a cron job to scan the folder size and submit
a MR job if necessary;


On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Nikhil.Kandoi@emc.com>
wrote:

> Hi All,
>
>
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
>
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
>
>
> Regards,
> Nikhil Kandoi
>
>
>


-- 
Regards,
*Stanley Shi,*

--047d7bb0398c454f330501aa760b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Normally MR job is used for batch processing. So I don&#39=
;t think this is a good use case here for MR.<div>Since you need to run the=
 program periodically, you cannot submit a single mapreduce job for this.=
=C2=A0</div>
<div>An possible way is to create a cron job to scan the folder size and su=
bmit a MR job if necessary;</div><div><br></div></div><div class=3D"gmail_e=
xtra"><br><br><div class=3D"gmail_quote">On Wed, Aug 27, 2014 at 7:38 PM, K=
andoi, Nikhil <span dir=3D"ltr">&lt;<a href=3D"mailto:Nikhil.Kandoi@emc.com=
" target=3D"_blank">Nikhil.Kandoi@emc.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div lang=3D"EN-US" link=3D"blue" vlink=3D"p=
urple"><div><p class=3D"MsoNormal">Hi All,<u></u><u></u></p><p class=3D"Mso=
Normal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">I have a system where files are coming in hdfs at re=
gular intervals and I perform an operation everytime the directory size goe=
s above a particular point.<u></u><u></u></p><p class=3D"MsoNormal">My Ques=
tion is that when I submit a map reduce job, would it only work on the file=
s present at that point ??<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p class=3D"MsoNormal" style=
=3D"margin-right:0in;margin-bottom:5.0pt;margin-left:0in;text-autospace:non=
e"><span style=3D"color:#1f497d">Regards,<br>Nikhil Kandoi<br><br></span><s=
pan style=3D"color:gray"><u></u><u></u></span></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div></div></blockquote></d=
iv><br><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"><div>Regard=
s,</div><div><b>Stanley Shi,</b></div><img src=3D"http://www.gopivotal.com/=
files/media/logos/pivotal-logo-email-signature.png"><br>
</div>
</div>

--047d7bb0398c454f330501aa760b--