spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darshan Pandya <darshanpan...@gmail.com>
Subject Re: pivot over non numerical data
Date Thu, 02 Feb 2017 20:10:16 GMT
Thanks Kevin,
Worked like a charm.
FYI for readers,
val temp1 =
temp.groupBy("reference_id").pivot("char_name").agg(max($"char_value"))

I didn't know I can use 'agg' with a string max. I was using it incorrectly
as below
temp.groupBy("reference_id").pivot("char_name").max("char_value")

On Wed, Feb 1, 2017 at 11:56 PM, Kevin Mellott <kevin.r.mellott@gmail.com>
wrote:

> This should work for non-numerical data as well - can you please elaborate
> on the error you are getting and provide a code sample? As a preliminary
> hint, you can "aggregate" text values using *max*.
>
> df.groupBy("someCol")
>   .pivot("anotherCol")
>   .agg(max($"textCol"))
>
> Thanks,
> Kevin
>
> On Wed, Feb 1, 2017 at 2:02 PM, Darshan Pandya <darshanpandya@gmail.com>
> wrote:
>
>> Hello,
>>
>> I am trying to transpose some data using groupBy pivot aggr as mentioned
>> in this blog
>> https://databricks.com/blog/2016/02/09/reshaping-data-with-
>> pivot-in-apache-spark.html
>>
>> But this works only for numerical data.
>> Any hints for doing the same thing for non numerical data ?
>>
>>
>> --
>> Sincerely,
>> Darshan
>>
>>
>


-- 
Sincerely,
Darshan

Mime
View raw message