spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-21057) Do not use a PascalDistribution in countApprox
Date Sun, 11 Jun 2017 22:58:19 GMT


Sean Owen commented on SPARK-21057:

I don't know if that's relevant to the analysis. Nothing is literally picking elements with
or without replacement. Neither does the Poisson model literally model what's actually taking
place, either. The argument isn't solely that the expect value is correct, of course. Still
I don't see the reason NB was introduced here, and it looks no longer useful or even valid.
I have a PR ready anyway as I needed to test it.

> Do not use a PascalDistribution in countApprox
> ----------------------------------------------
>                 Key: SPARK-21057
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.1
>            Reporter: Lovasoa
> I was reading the source of Spark, and found this:
> This is the function that estimates the probability distribution of the total count of
elements in an RDD given the count of only some partitions.
> This function does a strange thing: when the number of elements counted so far is less
than 10 000, it models the total count with a negative binomial (Pascal) law, else, it models
it with a Poisson law.
> Modeling our number of uncounted elements with a negative binomial law is like saying
that we ran over elements, counting only some, and stopping after having counted a given number
of elements.
> But this does not model what really happened.  Our counting was limited in time, not
in number of counted elements, and we can't count only some of the elements in a partition.
> I propose to use the Poisson distribution in every case, as it can be justified under
the hypothesis that the number of elements in each partition is independent and follows a
Poisson law.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message