spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph K. Bradley (JIRA)" <>
Subject [jira] [Created] (SPARK-4791) Create SchemaRDD from case classes with multiple constructors
Date Tue, 09 Dec 2014 01:23:13 GMT
Joseph K. Bradley created SPARK-4791:

             Summary: Create SchemaRDD from case classes with multiple constructors
                 Key: SPARK-4791
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 1.3.0
            Reporter: Joseph K. Bradley
            Priority: Minor

Issue: One can usually take an RDD of case classes and create a SchemaRDD, where Spark SQL
infers the schema from the case class metadata.  However, if the case class has multiple constructors,
then ScalaReflection.schemaFor gets confused.

Motivation: In, I would like to create a class with the following signature:
case class LabeledPoint(label: Double, features: Vector, weight: Double) {
  def this(label: Double, features: Vector) = this(label, features, 1.0)

Proposed fix: Change ScalaReflection.schemaFor so it checks for whether there are multiple
constructors.  If there are multiple ones, it should take the primary constructor.  This will
not change the behavior of existing code since it currently only supports case classes with
1 constructor.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message