spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Glenn Weidner (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-7425) spark.ml Predictor should support other numeric types for label
Date Wed, 13 May 2015 00:41:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541099#comment-14541099
] 

Glenn Weidner edited comment on SPARK-7425 at 5/13/15 12:40 AM:
----------------------------------------------------------------

Thank you for the tip!
In PredictorParams.validateAndTransformSchema, I've replaced SchemaUtils.checkColumnType with
isInstanceOf[NumericType] as shown below
(where NumericType imported from org.apache.spark.sql.types.NumericType).

    if (fitting) 
      // TODO: Allow other numeric types
      //SchemaUtils.checkColumnType(schema, $(labelCol), DoubleType)

      val actualDataType = schema($(labelCol)).dataType
      require(actualDataType.isInstanceOf[NumericType],
        s"Column $labelCol must be of type NumericType but was actually $actualDataType.")
    

A quick debug run of existing LinearRegressionSuite stepped into modified PredictorParams.validateAndTransformSchema
and verified test still passed with DoubleType.  Next step is to try test with other numeric
types.



was (Author: gweidner):
Thank you for the tip!
In PredictorParams.validateAndTransformSchema, I've replaced SchemaUtils.checkColumnType with
isInstanceOf[NumericType] as shown below
(where NumericType imported from org.apache.spark.sql.types.NumericType).

    if (fitting) {
      // TODO: Allow other numeric types
      //SchemaUtils.checkColumnType(schema, $(labelCol), DoubleType)

      val actualDataType = schema($(labelCol)).dataType
      require(actualDataType.isInstanceOf[NumericType],
        s"Column $labelCol must be of type NumericType but was actually $actualDataType.")
    }

A quick debug run of existing LinearRegressionSuite stepped into modified PredictorParams.validateAndTransformSchema
and verified test still passed with DoubleType.  Next step is to try test with other numeric
types.


> spark.ml Predictor should support other numeric types for label
> ---------------------------------------------------------------
>
>                 Key: SPARK-7425
>                 URL: https://issues.apache.org/jira/browse/SPARK-7425
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML
>            Reporter: Joseph K. Bradley
>            Priority: Minor
>              Labels: starter
>
> Currently, the Predictor abstraction expects the input labelCol type to be DoubleType,
but we should support other numeric types.  This will involve updating the PredictorParams.validateAndTransformSchema
method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message