spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Magnus Nilsson <ma...@kth.se>
Subject Re: Windowing LAG function Usage in Spark2.2 Dataset scala
Date Thu, 14 Mar 2019 08:09:51 GMT
import org.apache.spark.sql.expressions.Window

val partitionBy = Window.partitionBy("name", "sit").orderBy("data_date")

val newDf = df.withColumn("PreviousDate", lag("uniq_im",
1).over(partitionBy))

Cheers...

On Thu, Mar 14, 2019 at 4:55 AM anbu <anbutech17@gmail.com> wrote:

> Hi,
>
> To calculate LAG functions difference for the two data_date(current date
> and
> previous date) on the same column
> Could you please help me to implement the below scenario using scala spark
> Dataset.
> uniq_im - LAG(uniq_im,1,0) OVER PARTITION BY(name,sit,plc,country,state)
> order by (data_date) as calc_value.
>
> could you please help me how to implement using spark scala Dataset.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message