Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Jens Rantil <jens.rantil@tink.se>
Date: Thu, 8 Oct 2015 12:43:06 +0200
Message-ID: 
 <CAL+ArfXMw+fUV2y2B8eJtWDdDXHT8ei7hPd74qJGoZWGKYpEHA@mail.gmail.com>
Subject: Best practises to clean up RDDs for old applications
To: user@spark.apache.org
Content-Type: multipart/alternative; boundary=001a114dab9860f7940521958837

--001a114dab9860f7940521958837
Content-Type: text/plain; charset=UTF-8

Hi,

I have a couple of old application RDDs under /var/lib/spark/rdd that
haven't been properly cleaned up after themselves. Example:

# du -shx /var/lib/spark/rdd/*
44K /var/lib/spark/rdd/liblz4-java1011984124691611873.so
48K /var/lib/spark/rdd/snappy-1.0.5-libsnappyjava.so
2.3G /var/lib/spark/rdd/spark-local-20150903112858-a72d
23M /var/lib/spark/rdd/spark-local-20150929141201-143f

The applications (such as "20150903112858") aren't running anymore. What
are best practises to clean these up? A cron job? Enabling some kind of
cleaner in Spark? I'm currently running Spark 1.1, but will eventually move
to 1.2 and then 1.4.

Thanks,
Jens

-- 
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
 Twitter <https://twitter.com/tink>

--001a114dab9860f7940521958837
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I have a couple of old application =
RDDs under /var/lib/spark/rdd that haven&#39;t been properly cleaned up aft=
er themselves. Example:</div><div><br></div><div><div># du -shx /var/lib/sp=
ark/rdd/*</div><div>44K<span class=3D"" style=3D"white-space:pre">	</span>/=
var/lib/spark/rdd/liblz4-java1011984124691611873.so</div><div>48K<span clas=
s=3D"" style=3D"white-space:pre">	</span>/var/lib/spark/rdd/<a href=3D"http=
://snappy-1.0.5-libsnappyjava.so">snappy-1.0.5-libsnappyjava.so</a></div><d=
iv>2.3G<span class=3D"" style=3D"white-space:pre">	</span>/var/lib/spark/rd=
d/spark-local-20150903112858-a72d</div><div>23M<span class=3D"" style=3D"wh=
ite-space:pre">	</span>/var/lib/spark/rdd/spark-local-20150929141201-143f</=
div><div><br></div><div>The applications (such as &quot;20150903112858&quot=
;) aren&#39;t running anymore. What are best practises to clean these up? A=
 cron job? Enabling some kind of cleaner in Spark? I&#39;m currently runnin=
g Spark 1.1, but will eventually move to 1.2 and then 1.4.</div><div><br></=
div><div>Thanks,</div><div>Jens=C2=A0</div><div><br></div>-- <br><div class=
=3D"gmail_signature"><div dir=3D"ltr"><div>Jens Rantil</div><div>Backend en=
gineer</div><div>Tink AB</div><div><br></div><div>Email:=C2=A0<a href=3D"ma=
ilto:jens.rantil@tink.se" style=3D"color:rgb(17,85,204)" target=3D"_blank">=
jens.rantil@tink.se</a></div><div>Phone: +46 708 84 18 32</div><div>Web:=C2=
=A0<a href=3D"http://www.tink.se/" style=3D"color:rgb(17,85,204)" target=3D=
"_blank">www.tink.se</a></div><div><br></div><div><a href=3D"https://www.fa=
cebook.com/#!/tink.se" style=3D"color:rgb(17,85,204);font-family:arial;font=
-size:small" target=3D"_blank">Facebook</a><span style=3D"font-family:arial=
;font-size:small">=C2=A0</span><a href=3D"http://www.linkedin.com/company/2=
735919?trk=3Dvsrp_companies_res_photo&amp;trkInfo=3DVSRPsearchId%3A10570233=
81369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary" style=3D"colo=
r:rgb(17,85,204);font-family:arial;font-size:small" target=3D"_blank">Linke=
din</a><span style=3D"font-family:arial;font-size:small">=C2=A0</span><a hr=
ef=3D"https://twitter.com/tink" style=3D"color:rgb(17,85,204);font-family:a=
rial;font-size:small" target=3D"_blank">Twitter</a></div></div></div>
</div></div>

--001a114dab9860f7940521958837--