pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com>
Subject Re: Clear temp files
Date Thu, 09 Jul 2009 15:04:58 GMT
Hi Pallavi,

Yes, it looks like we may be going down the same path. Regarding the hadoop
dfs -rmr, that DID work. What did NOT work was issuing RMR /tmp/tmp* from
pig itself.

Thanks!
Chris


On 7/8/09 8:23 PM, "Palleti, Pallavi" <pallavi.palleti@corp.aol.com> wrote:

> Hi Chris,
> 
> We too faced similar issue and finally we ended up writing a cron job which
> deletes temporary files which are one day older from /tmp directory of HDFS. I
> am not sure why "hadoop dfs -rmr /tmp/tmp*" didn't work for you as it worked
> for me when I tried manually.
> 
> Thanks
> Pallavi
> 
> -----Original Message-----
> From: Chris Riccomini [mailto:criccomini@linkedin.com]
> Sent: Wednesday, July 08, 2009 11:57 PM
> To: pig-user@hadoop.apache.org
> Subject: Re: Clear temp files
> 
> Understood. I believe that the temp files are remaining when the script
> fails or is killed. This is a bit of a bummer since some of the temp files
> are 1+ TB, although I can't think of an easy way to fix the problem.
> 
> Thanks for the information.
> 
> Chris
> 
> 
> On 7/8/09 11:23 AM, "Olga Natkovich" <olgan@yahoo-inc.com> wrote:
> 
>> Pradeep is absolutely right. As for your command, Pig does not support globs
>> for DFS commands - only in the load statement. The reason you don't see an
>> error is because rmf command does not error out if file is not found. If you
>> run rm, you would see an error.
>> 
>> Olga
>> 
>> -----Original Message-----
>> From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
>> Sent: Wednesday, July 08, 2009 11:05 AM
>> To: pig-user@hadoop.apache.org
>> Subject: RE: Clear temp files
>> 
>> Temp files created on DFS by pig during execution are to store intermediate
>> results used by later statements in the script and should not be deleted
>> while
>> the script is executing.
>> 
>> Pig cleans up these intermediate files once the script execution completes
>> -do
>> you see temp files even after a run? If so, can you attach a pig script with
>> sample data which can show this behavior?
>> 
>> Pradeep
>> 
>> -----Original Message-----
>> From: Chris Riccomini [mailto:criccomini@linkedin.com]
>> Sent: Wednesday, July 08, 2009 10:53 AM
>> To: pig-user@hadoop.apache.org
>> Subject: Clear temp files
>> 
>> Hi All,
>> 
>> Is there an easy way to clear temp files that Pig creates when a script
>> runs?
>> 
>> I tried adding RMF /tmp/tmp* and /tmp/temp*, but it doesn¹t seem to work
>> (although it doesn¹t fail either ....).
>> 
>> Thanks!
>> Chris
> 


Mime
View raw message