hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Kent" <matthew.e.k...@gmail.com>
Subject Re: FW: Removing files after processing
Date Tue, 28 Aug 2007 20:34:30 GMT
I would find it useful to have some sort of listener mechanism, where
you could register an object to be notified of a job completion event
and then respond to it accordingly.

Matt

On 8/28/07, Stu Hood <stuhood@webmail.us> wrote:
>
>
> Does anyone have any ideas on this issue?
>
> Otherwise, if I were to write a patch to add this option for jobs to Hadoop, would it
be useful for anyone else?
>
> Thanks
> Stu
>
> -----Original Message-----
> From: Stu Hood <stuhood@webmail.us>
> Sent: Fri, August 24, 2007 9:43 am
> To: hadoop-user@lucene.apache.org
> Subject: Removing files after processing
>
>
> Hello,
>
> Whats the best way to go about doing cleanup after MapReduce jobs? I'd like to have the
job delete its input files when it has finished successfully (but preferably before it is
marked as having finished: so I don't have to deal with a race condition).
>
> Obviously, I don't want to have to track which files are being processed for each job,
since that data is stored anyway. Also, I'm using JobClient.submitJob(), so I can't sit around
and wait to do the cleanup manually.
>
> Any suggestions?
>
> Thanks!
>
> Stu Hood
> Webmail.us
> "You manage your business. We'll manage your email."(r)

Mime
View raw message