hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanley Shi <s...@pivotal.io>
Subject Re: What happens when .....?
Date Thu, 28 Aug 2014 06:15:23 GMT
Normally MR job is used for batch processing. So I don't think this is a
good use case here for MR.
Since you need to run the program periodically, you cannot submit a single
mapreduce job for this.
An possible way is to create a cron job to scan the folder size and submit
a MR job if necessary;



On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Nikhil.Kandoi@emc.com>
wrote:

> Hi All,
>
>
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
>
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
>
>
> Regards,
> Nikhil Kandoi
>
>
>



-- 
Regards,
*Stanley Shi,*

Mime
View raw message