spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kurt Fehlhauer <kfehl...@gmail.com>
Subject Re: how to create a DataType Object using the String representation in Java using Spark 2.2.0?
Date Fri, 26 Jan 2018 07:11:56 GMT
Can you share your code and a sample of your data? WIthout seeing it, I
can't give a definitive answer. I can offer some hints. If you have a
column of strings you should either be able to create a new column casted
to Integer. This can be accomplished two ways:

df.withColumn("newColumn", df.currentColumn.cast(IntegerType))

or

val df = df.select("cast(CurretColumn as int) newColum")


Without seeing your json, I really can't offer assistance.


On Thu, Jan 25, 2018 at 11:39 PM, kant kodali <kanth909@gmail.com> wrote:

> It seems like its hard to construct a DataType given its String literal
> representation.
>
> dataframe.types() return column names and its corresponding Types. for
> example say I have an integer column named "sum" doing dataframe.dtypes()
> would return "sum" and "IntegerType" but this string  representation
> "IntegerType" doesnt seem to be very useful because I cannot do
> DataType.fromJson("IntegerType") This will throw an error. so I am not
> quite sure how to construct a DataType given its String representation ?
>
> On Thu, Jan 25, 2018 at 4:22 PM, kant kodali <kanth909@gmail.com> wrote:
>
>> Hi All,
>>
>> I have a datatype "IntegerType" represented as a String and now I want to
>> create DataType object out of that. I couldn't find in the DataType or
>> DataTypes api on how to do that?
>>
>> Thanks!
>>
>
>

Mime
View raw message