spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <>
Subject [jira] [Resolved] (SPARK-1552) GraphX performs type comparison incorrectly
Date Fri, 06 Jun 2014 06:34:02 GMT


Reynold Xin resolved SPARK-1552.

       Resolution: Fixed
    Fix Version/s: 1.1.0

> GraphX performs type comparison incorrectly
> -------------------------------------------
>                 Key: SPARK-1552
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: GraphX
>            Reporter: Ankur Dave
>            Assignee: Ankur Dave
>             Fix For: 1.1.0
> In GraphImpl, mapVertices and outerJoinVertices use a more efficient implementation when
the map function preserves vertex attribute types. This is implemented by comparing the ClassTags
of the old and new vertex attribute types. However, ClassTags store _erased_ types, so the
comparison will return a false positive for types with different type parameters, such as
Option[Int] and Option[Double].
> Thanks to Pierre-Alexandre Fonta for reporting this bug on the [mailing list|].
> Demo in the Scala shell:
> scala> import scala.reflect.{classTag, ClassTag}
> scala> def typesEqual[A: ClassTag, B: ClassTag](a: A, b: B): Boolean = classTag[A]
equals classTag[B]
> scala> typesEqual(Some(1), Some(2.0)) // should return false
> res2: Boolean = true
> We can require richer TypeTags for these methods, or just take a flag from the caller
specifying whether the types are equal.

This message was sent by Atlassian JIRA

View raw message