spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Takeshi Yamamuro (JIRA)" <>
Subject [jira] [Commented] (SPARK-19217) Offer easy cast from vector to array
Date Sun, 21 Jan 2018 05:32:00 GMT


Takeshi Yamamuro commented on SPARK-19217:

If it still makes some sense and nobody takes this, I'll do. How about adding a new method
for supported cast types in UserDefinedType ( []
 In this example we could cast from VectorUDT to Array and _vice versa_.
scala> val df1 = Seq((1, Vectors.dense(Array(1.0, 2.0, 3.0)))).toDF("a", "b")
scala> val df2 = df1.selectExpr("CAST(b AS ARRAY<DOUBLE>)")
scala> df2.printSchema
 |-- b: array (nullable = true)
 |    |-- element: double (containsNull = true)

|              b|
|[1.0, 2.0, 3.0]|

scala> val df3 = Seq((1, Seq(1.0, 2.0, 3.0))).toDF("a", "b")
scala> val df4 ="b").cast(new VectorUDT()))
scala> df4.printSchema
 |-- b: vector (nullable = true)

|            b|

WDYT cc: [~cloud_fan]

> Offer easy cast from vector to array
> ------------------------------------
>                 Key: SPARK-19217
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML, PySpark, SQL
>    Affects Versions: 2.1.0
>            Reporter: Nicholas Chammas
>            Priority: Minor
> Working with ML often means working with DataFrames with vector columns. You can't save
these DataFrames to storage (edit: at least as ORC) without converting the vector columns
to array columns, and there doesn't appear to an easy way to make that conversion.
> This is a common enough problem that it is [documented on Stack Overflow|].
The current solutions to making the conversion from a vector column to an array column are:
> # Convert the DataFrame to an RDD and back
> # Use a UDF
> Both approaches work fine, but it really seems like you should be able to do something
like this instead:
> {code}
> (le_data
>     .select(
>         col('features').cast('array').alias('features')
>     ))
> {code}
> We already have an {{ArrayType}} in {{pyspark.sql.types}}, but it appears that {{cast()}}
doesn't support this conversion.
> Would this be an appropriate thing to add?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message