hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanley Shi <s...@pivotal.io>
Subject Re: What happens when .....?
Date Thu, 28 Aug 2014 06:15:23 GMT
Normally MR job is used for batch processing. So I don't think this is a
good use case here for MR.
Since you need to run the program periodically, you cannot submit a single
mapreduce job for this.
An possible way is to create a cron job to scan the folder size and submit
a MR job if necessary;

On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Nikhil.Kandoi@emc.com>

> Hi All,
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
> Regards,
> Nikhil Kandoi

*Stanley Shi,*

View raw message