Return-Path: Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: (qmail 52894 invoked from network); 25 Mar 2011 14:26:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 25 Mar 2011 14:26:09 -0000 Received: (qmail 79455 invoked by uid 500); 25 Mar 2011 14:26:09 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 79418 invoked by uid 500); 25 Mar 2011 14:26:09 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 79410 invoked by uid 99); 25 Mar 2011 14:26:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 14:26:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 14:26:01 +0000 Received: by vws7 with SMTP id 7so1234780vws.35 for ; Fri, 25 Mar 2011 07:25:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.0.131 with SMTP id 3mr1131648vde.260.1301063140316; Fri, 25 Mar 2011 07:25:40 -0700 (PDT) Received: by 10.220.4.148 with HTTP; Fri, 25 Mar 2011 07:25:40 -0700 (PDT) X-Originating-IP: [113.31.32.3] In-Reply-To: References: Date: Fri, 25 Mar 2011 22:25:40 +0800 Message-ID: Subject: Re: A way to monitor HDFS for a file to come live, and then kick off a job? From: "Bai, Gang" To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf304346e64f17fc049f4f5eb1 --20cf304346e64f17fc049f4f5eb1 Content-Type: text/plain; charset=ISO-8859-1 Hi Jon, Oozie could handle this nicely. You may just specify a Oozie coordinator jobs. But if you don't have a Oozie server handy, cron jobs could also meet your needs. Regards, -BaiGang On Fri, Mar 25, 2011 at 1:09 AM, Jonathan Coveney wrote: > I am not sure if this is the right listserv, forgive me if it is not. My > goal is this: monitor HDFS until a file is create, and then kick off a job. > Ideally I'd want to do this continuously, but the file would be create > hourly (with some sort of variance). I guess I could make a script that > would ping the server every 5 minutes or something, but I was wondering if > there might be a more elegant way? > > Thanks > Jon > --20cf304346e64f17fc049f4f5eb1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Hi Jon,

Oozie could handle this nicely. You may ju= st specify a Oozie coordinator jobs. But if you don't have a Oozie serv= er handy, cron jobs could also meet your needs.

Regards,=

-BaiGang

On Fri, Mar 25, 2011 at 1:0= 9 AM, Jonathan Coveney <jcoveney@gmail.com> wrote:

I am not sure if this is the right listserv, forgive me if it is not. My go= al is this: monitor HDFS until a file is create, and then kick off a job. I= deally I'd want to do this continuously, but the file would be create h= ourly (with some sort of variance). I guess I could make a script that woul= d ping the server every 5 minutes or something, but I was wondering if ther= e might be a more elegant way?

Thanks
Jon

--20cf304346e64f17fc049f4f5eb1--