spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <michael.engl...@nomura.com>
Subject RE: Cleaning up spark.local.dir automatically
Date Tue, 13 Jan 2015 16:01:30 GMT
That’s really useful, thanks.


From: Andrew Ash [mailto:andrew@andrewash.com]
Sent: 09 January 2015 22:42
To: England, Michael (IT/UK)
Cc: raghavendra.pandey@gmail.com; user
Subject: Re: Cleaning up spark.local.dir automatically

That's a worker setting which cleans up the files left behind by executors, so spark.cleaner.ttl
isn't at the RDD level.  After https://issues.apache.org/jira/browse/SPARK-1860 the cleaner
won't clean up directories left by running executors.


On Fri, Jan 9, 2015 at 7:38 AM, <michael.england@nomura.com<mailto:michael.england@nomura.com>>
wrote:
Thanks, I imagine this will kill any cached RDDs if their files are beyond the ttl?

Thanks


From: Raghavendra Pandey [mailto:raghavendra.pandey@gmail.com<mailto:raghavendra.pandey@gmail.com>]
Sent: 09 January 2015 15:29
To: England, Michael (IT/UK); user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Cleaning up spark.local.dir automatically

You may like to look at spark.cleaner.ttl configuration which is infinite by default. Spark
has that configuration to delete temp files time to time.
On Fri Jan 09 2015 at 8:34:10 PM <michael.england@nomura.com<mailto:michael.england@nomura.com>>
wrote:
Hi,

Is there a way of automatically cleaning up the spark.local.dir after a job has been run?
I have noticed a large number of temporary files have been stored here and are not cleaned
up. The only solution I can think of is to run some sort of cron job to delete files older
than a few days. I am currently using a mixture of standalone and YARN spark builds.

Thanks,
Michael


This e-mail (including any attachments) is private and confidential, may contain proprietary
or privileged information and is intended for the named recipient(s) only. Unintended recipients
are strictly prohibited from taking action on the basis of information in this e-mail and
must contact the sender immediately, delete this e-mail (and all attachments) and destroy
any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness
of, or the presence of any virus or disabling code in, this e-mail. If verification is sought
please request a hard copy. Any reference to the terms of executed transactions should be
treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves
the right to retain, monitor and intercept e-mail communications through its networks (subject
to and in accordance with applicable laws). No confidentiality or privilege is waived or lost
by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference
to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications
Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm

This e-mail (including any attachments) is private and confidential, may contain proprietary
or privileged information and is intended for the named recipient(s) only. Unintended recipients
are strictly prohibited from taking action on the basis of information in this e-mail and
must contact the sender immediately, delete this e-mail (and all attachments) and destroy
any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness
of, or the presence of any virus or disabling code in, this e-mail. If verification is sought
please request a hard copy. Any reference to the terms of executed transactions should be
treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves
the right to retain, monitor and intercept e-mail communications through its networks (subject
to and in accordance with applicable laws). No confidentiality or privilege is waived or lost
by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference
to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications
Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm



This e-mail (including any attachments) is private and confidential, may contain proprietary
or privileged information and is intended for the named recipient(s) only. Unintended recipients
are strictly prohibited from taking action on the basis of information in this e-mail and
must contact the sender immediately, delete this e-mail (and all attachments) and destroy
any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness
of, or the presence of any virus or disabling code in, this e-mail. If verification is sought
please request a hard copy. Any reference to the terms of executed transactions should be
treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves
the right to retain, monitor and intercept e-mail communications through its networks (subject
to and in accordance with applicable laws). No confidentiality or privilege is waived or lost
by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference
to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications
Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm

Mime
View raw message