spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wayne Zhang (JIRA)" <>
Subject [jira] [Created] (SPARK-20574) Allow Bucketizer to handle non-Double column
Date Wed, 03 May 2017 06:00:12 GMT
Wayne Zhang created SPARK-20574:

             Summary: Allow Bucketizer to handle non-Double column
                 Key: SPARK-20574
             Project: Spark
          Issue Type: Bug
          Components: ML
    Affects Versions: 2.1.0
            Reporter: Wayne Zhang

Bucketizer currently requires input column to be Double, but the logic should work on any
numeric data types. Many practical problems have integer/float data types, and it could get
very tedious to manually cast them into Double before calling bucketizer. This transformer
could be extended to handle all numeric types.  

The example below shows failure of Bucketizer on integer data. 
val splits = Array(-3.0, 0.0, 3.0)
val data: Array[Int] = Array(-2, -1, 0, 1, 2)
val expectedBuckets = Array(0.0, 0.0, 1.0, 1.0, 1.0)
val dataFrame ="feature", "expected")
val bucketizer = new Bucketizer()

java.lang.IllegalArgumentException: requirement failed: Column feature must be of type DoubleType
but was actually IntegerType.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message