Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <1452307828595-25926.post@n3.nabble.com>
References: <1452307828595-25926.post@n3.nabble.com>
Date: Fri, 8 Jan 2016 19:34:39 -0800
Message-ID: 
 <CAFzSYuy7wdeS66cxv6ukPU-ACVMUc3R9RFSEw1UJx2QwiGR4+g@mail.gmail.com>
Subject: Re: how garbage collection works on parallelize
From: Josh Rosen <joshrosen@databricks.com>
To: jluan <jayluan7@gmail.com>
Cc: user <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c08851e3803fa0528de623f

--94eb2c08851e3803fa0528de623f
Content-Type: text/plain; charset=UTF-8

It won't be GC'd as long as the RDD which results from `parallelize()` is
kept around; that RDD keeps strong references to the parallelized
collection's elements in order to enable fault-tolerance.

On Fri, Jan 8, 2016 at 6:50 PM, jluan <jayluan7@gmail.com> wrote:

> Hi,
>
> I am curious about garbage collect on an object which gets parallelized.
> Say
> if we have a really large array (say 40GB in ram) that we want to
> parallelize across our machines.
>
> I have the following function:
>
> def doSomething(): RDD[Double] = {
> val reallyBigArray = Array[Double[(some really big value)
> sc.parallelize(reallyBigArray)
> }
>
> Theoretically, will reallyBigArray be marked for GC? Or will reallyBigArray
> not be GC'd because parallelize somehow has a reference on reallyBigArray?
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-garbage-collection-works-on-parallelize-tp25926.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

--94eb2c08851e3803fa0528de623f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">It won&#39;t be GC&#39;d as long as the RDD which results =
from `parallelize()` is kept around; that RDD keeps strong references to th=
e parallelized collection&#39;s elements in order to enable fault-tolerance=
.</div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Fri, Ja=
n 8, 2016 at 6:50 PM, jluan <span dir=3D"ltr">&lt;<a href=3D"mailto:jayluan=
7@gmail.com" target=3D"_blank">jayluan7@gmail.com</a>&gt;</span> wrote:<br>=
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi,<br>
<br>
I am curious about garbage collect on an object which gets parallelized. Sa=
y<br>
if we have a really large array (say 40GB in ram) that we want to<br>
parallelize across our machines.<br>
<br>
I have the following function:<br>
<br>
def doSomething(): RDD[Double] =3D {<br>
val reallyBigArray =3D Array[Double[(some really big value)<br>
sc.parallelize(reallyBigArray)<br>
}<br>
<br>
Theoretically, will reallyBigArray be marked for GC? Or will reallyBigArray=
<br>
not be GC&#39;d because parallelize somehow has a reference on reallyBigArr=
ay?<br>
<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href=3D"http://apache-spark-user-list.1001=
560.n3.nabble.com/how-garbage-collection-works-on-parallelize-tp25926.html"=
 rel=3D"noreferrer" target=3D"_blank">http://apache-spark-user-list.1001560=
.n3.nabble.com/how-garbage-collection-works-on-parallelize-tp25926.html</a>=
<br>
Sent from the Apache Spark User List mailing list archive at Nabble.com.<br=
>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href=3D"mailto:user-unsubscribe@spark.apache.org=
">user-unsubscribe@spark.apache.org</a><br>
For additional commands, e-mail: <a href=3D"mailto:user-help@spark.apache.o=
rg">user-help@spark.apache.org</a><br>
<br>
</blockquote></div><br></div>

--94eb2c08851e3803fa0528de623f--