hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: DFS temporary files?
Date Mon, 08 Sep 2008 11:21:06 GMT
Owen O'Malley wrote:
> Currently there isn't a way to do that. In Hadoop 0.19, there will be a way
> to have a clean up method that runs at the end of the job. See
> HADOOP-3150<https://issues.apache.org/jira/browse/HADOOP-3150>
> .

another bit of feature creep would be an expires: attribute on files, 
and something to purge expired files every so often. Which ensures that 
even if a job dies or the entire cluster is reset, stuff gets cleaned up

Before someone rushes to implement this, I've been burned in the past by 
differences in a clusters machines and clocks. Even if everything really 
is in sync with NTP, and not configured to talk to a NTP server that the 
production site can't see, you still need to be 100% that all your boxes 
are in the same time zone.


View raw message