Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
Sender: holden.karau@gmail.com
In-Reply-To: <1483936217357-20515.post@n3.nabble.com>
References: <1483731016869-20496.post@n3.nabble.com> <1483936217357-20515.post@n3.nabble.com>
From: Holden Karau <holden@pigscanfly.ca>
Date: Sun, 8 Jan 2017 21:40:25 -0800
Message-ID: <CAJLcJd-LCYRETeYaqhaqgg+DfTRL6JWqKa0o9ZK82=p2rwHo8w@mail.gmail.com>
Subject: Re: handling of empty partitions
To: Liang-Chi Hsieh <viirya@gmail.com>, "user@spark.apache.org" <user@spark.apache.org>
Cc: "dev@spark.apache.org" <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a113de47ef15c380545a2cd13
archived-at: Mon, 09 Jan 2017 05:40:39 -0000

--001a113de47ef15c380545a2cd13
Content-Type: text/plain; charset=UTF-8

Hi Georg,

Thanks for the question along with the code (as well as posting to stack
overflow). In general if a question is well suited for stackoverflow its
probably better suited to the user@ list instead of the dev@ list so I've
cc'd the user@ list for you.

As far as handling empty partitions when working mapPartitions (and
similar), the general approach is to return an empty iterator of the
correct type when you have an empty input iterator.

It looks like your code is doing this, however it seems like you likely
have a bug in your application logic (namely it assumes that if a partition
has a record missing a value it will either have had a previous row in the
same partition which is good OR that the previous partition is not empty
and has a good row - which need not necessarily be the case). You've
partially fixed this problem by going through and for each partition
collecting the last previous good value, and then if you don't have a good
value at the start of a partition look up the value in the collected array.

However, if this also happens at the same time the previous partition is
empty, you will need to go and lookup the previous previous partition value
until you find the one you are looking for. (Note this assumes that the
first record in your dataset is valid, if it isn't your code will still
fail).

Your solution is really close to working but just has some minor
assumptions which don't always necessarily hold.

Cheers,

Holden :)

On Sun, Jan 8, 2017 at 8:30 PM, Liang-Chi Hsieh <viirya@gmail.com> wrote:

>
> Hi Georg,
>
> Can you describe your question more clear?
>
> Actually, the example codes you posted in stackoverflow doesn't crash as
> you
> said in the post.
>
>
> geoHeil wrote
> > I am working on building a custom ML pipeline-model / estimator to impute
> > missing values, e.g. I want to fill with last good known value.
> > Using a window function is slow / will put the data into a single
> > partition.
> > I built some sample code to use the RDD API however, it some None / null
> > problems with empty partitions.
> >
> > How should this be implemented properly to handle such empty partitions?
> > http://stackoverflow.com/questions/41474175/spark-
> mappartitionswithindex-handling-empty-partitions
> >
> > Kind regards,
> > Georg
>
>
>
>
>
> -----
> Liang-Chi Hsieh | @viirya
> Spark Technology Center
> http://www.spark.tc/
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/handling-of-empty-
> partitions-tp20496p20515.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>


-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

--001a113de47ef15c380545a2cd13
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Georg,<div><br></div><div>Thanks for the question along=
 with the code (as well as posting to stack overflow). In general if a ques=
tion is well suited for stackoverflow its probably better suited to the use=
r@ list instead of the dev@ list so I&#39;ve cc&#39;d the user@ list for yo=
u.</div><div><br></div><div>As far as handling empty partitions when workin=
g mapPartitions (and similar), the general approach is to return an empty i=
terator of the correct type when you have an empty input iterator.</div><di=
v><br></div><div>It looks like your code is doing this, however it seems li=
ke you likely have a bug in your application logic (namely it assumes that =
if a partition has a record missing a value it will either have had a previ=
ous row in the same partition which is good OR that the previous partition =
is not empty and has a good row - which need not necessarily be the case). =
You&#39;ve partially fixed this problem by going through and for each parti=
tion collecting the last previous good value, and then if you don&#39;t hav=
e a good value at the start of a partition look up the value in the collect=
ed array.</div><div><br></div><div>However, if this also happens at the sam=
e time the previous partition is empty, you will need to go and lookup the =
previous previous partition value until you find the one you are looking fo=
r. (Note this assumes that the first record in your dataset is valid, if it=
 isn&#39;t your code will still fail).<br></div><div><br></div><div>Your so=
lution is really close to working but just has some minor assumptions which=
 don&#39;t always necessarily hold.</div><div><br></div><div>Cheers,</div><=
div><br></div><div>Holden :)</div></div><div class=3D"gmail_extra"><br><div=
 class=3D"gmail_quote">On Sun, Jan 8, 2017 at 8:30 PM, Liang-Chi Hsieh <spa=
n dir=3D"ltr">&lt;<a href=3D"mailto:viirya@gmail.com" target=3D"_blank">vii=
rya@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Georg,<br>
<br>
Can you describe your question more clear?<br>
<br>
Actually, the example codes you posted in stackoverflow doesn&#39;t crash a=
s you<br>
said in the post.<br>
<br>
<br>
geoHeil wrote<br>
<span class=3D"">&gt; I am working on building a custom ML pipeline-model /=
 estimator to impute<br>
&gt; missing values, e.g. I want to fill with last good known value.<br>
&gt; Using a window function is slow / will put the data into a single<br>
&gt; partition.<br>
&gt; I built some sample code to use the RDD API however, it some None / nu=
ll<br>
&gt; problems with empty partitions.<br>
&gt;<br>
&gt; How should this be implemented properly to handle such empty partition=
s?<br>
&gt; <a href=3D"http://stackoverflow.com/questions/41474175/spark-mappartit=
ionswithindex-handling-empty-partitions" rel=3D"noreferrer" target=3D"_blan=
k">http://stackoverflow.com/<wbr>questions/41474175/spark-<wbr>mappartition=
swithindex-<wbr>handling-empty-partitions</a><br>
&gt;<br>
&gt; Kind regards,<br>
&gt; Georg<br>
<br>
<br>
<br>
<br>
<br>
</span>-----<br>
Liang-Chi Hsieh | @viirya<br>
Spark Technology Center<br>
<a href=3D"http://www.spark.tc/" rel=3D"noreferrer" target=3D"_blank">http:=
//www.spark.tc/</a><br>
--<br>
View this message in context: <a href=3D"http://apache-spark-developers-lis=
t.1001551.n3.nabble.com/handling-of-empty-partitions-tp20496p20515.html" re=
l=3D"noreferrer" target=3D"_blank">http://apache-spark-<wbr>developers-list=
.1001551.n3.<wbr>nabble.com/handling-of-empty-<wbr>partitions-tp20496p20515=
.html</a><br>
<div class=3D"HOEnZb"><div class=3D"h5">Sent from the Apache Spark Develope=
rs List mailing list archive at Nabble.com.<br>
<br>
------------------------------<wbr>------------------------------<wbr>-----=
----<br>
To unsubscribe e-mail: <a href=3D"mailto:dev-unsubscribe@spark.apache.org">=
dev-unsubscribe@spark.apache.<wbr>org</a><br>
<br>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=
=3D"ltr"><div><div dir=3D"ltr"><div>Cell : 425-233-8271</div><div>Twitter:=
=C2=A0<a href=3D"https://twitter.com/holdenkarau" target=3D"_blank">https:/=
/twitter.com/holdenkarau</a></div></div></div></div></div>
</div>

--001a113de47ef15c380545a2cd13--