hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahesh Khandewal <mahesh.k....@gmail.com>
Subject Re: What happens when .....?
Date Thu, 28 Aug 2014 13:50:06 GMT
unsubscribe


On Thu, Aug 28, 2014 at 6:42 PM, Eric Payne <eric.payne1000@yahoo.com>
wrote:

> Or, maybe have a look at Apache Falcon:
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
>
>
>
>
>
>
> Falcon - Apache Falcon - Data management and processing platform
> <http://falcon.incubator.apache.org/>
> Apache Falcon - Data management and processing platform
> View on falcon.incubator.apache.org <http://falcon.incubator.apache.org/>
> Preview by Yahoo
>
>
>    *From:* Stanley Shi <sshi@pivotal.io>
> *To:* "user@hadoop.apache.org" <user@hadoop.apache.org>
> *Sent:* Thursday, August 28, 2014 1:15 AM
> *Subject:* Re: What happens when .....?
>
> Normally MR job is used for batch processing. So I don't think this is a
> good use case here for MR.
> Since you need to run the program periodically, you cannot submit a single
> mapreduce job for this.
> An possible way is to create a cron job to scan the folder size and submit
> a MR job if necessary;
>
>
>
> On Wed, Aug 27, 2014 at 7:38 PM, Kandoi, Nikhil <Nikhil.Kandoi@emc.com>
> wrote:
>
> Hi All,
>
> I have a system where files are coming in hdfs at regular intervals and I
> perform an operation everytime the directory size goes above a particular
> point.
> My Question is that when I submit a map reduce job, would it only work on
> the files present at that point ??
>
> Regards,
> Nikhil Kandoi
>
>
>
>
>
>
> --
> Regards,
> *Stanley Shi,*
>
>
>
>

Mime
View raw message