Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 15 Jul 2016 11:32:20 +0000 (UTC)
From: "Sean Owen (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.12987590.1467920887000.40717.1468582340502@Atlassian.JIRA>
In-Reply-To: <JIRA.12987590.1467920887000@Atlassian.JIRA>
References: <JIRA.12987590.1467920887000@Atlassian.JIRA> <JIRA.12987590.1467920887826@arcas>
Subject: [jira] [Updated] (SPARK-16426) IsotonicRegression produces NaNs
 with certain data
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 15 Jul 2016 11:32:22 -0000


     [ https://issues.apache.org/jira/browse/SPARK-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-16426:
------------------------------
    Assignee: Nic Eggert
    Priority: Minor  (was: Major)

> IsotonicRegression produces NaNs with certain data
> --------------------------------------------------
>
>                 Key: SPARK-16426
>                 URL: https://issues.apache.org/jira/browse/SPARK-16426
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.3.1, 1.4.1, 1.5.2, 1.6.2
>            Reporter: Nic Eggert
>            Assignee: Nic Eggert
>            Priority: Minor
>             Fix For: 2.1.0
>
>
> {code}
> val r = sc.parallelize(Seq[(Double, Double, Double)]((2, 1, 1), (1, 1, 1), (0, 2, 1), (1, 2, 1), (0.5, 3, 1), (0, 3, 1)), 2)
> val i = new IsotonicRegression().run(r)
> scala> i.predict(3.0)
> res12: Double = NaN
> scala> i.predictions
> res13: Array[Double] = Array(0.75, 0.75, NaN, NaN)
> {code}
> I believe I understand the problem so I'll submit a PR shortly.
> The problem happens when rows with the same feature value but different labels end up on different partitions. The merge function in poolAdjacentViolators introduces 0-weight points to be used for linear interpolation. This works fine, as long as they are always next to a non-0-weight point, but in the above case, you can end up with two 0-weight points  with the same feature value, which end up next to each other in the final PAV step. If these points are pooled, it creates a NaN.
> One solution to this is to ensure that the all points with identical feature values end up on the same partition. This is the solution I intend to submit a PR for. Another option would be to try to get rid of the 0-weight points, but that seems trickier to me.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org