Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 43B29200BA3 for ; Thu, 20 Oct 2016 11:18:45 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 42338160AE0; Thu, 20 Oct 2016 09:18:45 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 63264160ADB for ; Thu, 20 Oct 2016 11:18:44 +0200 (CEST) Received: (qmail 91738 invoked by uid 500); 20 Oct 2016 09:18:42 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 91728 invoked by uid 99); 20 Oct 2016 09:18:42 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2016 09:18:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A237E1A7A0C for ; Thu, 20 Oct 2016 09:18:41 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=rsa.com header.b=Cf7mNKqf; dkim=pass (1024-bit key) header.d=rsa.com header.b=hSFYXyPV Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id rj3DHjAfhh-c for ; Thu, 20 Oct 2016 09:18:39 +0000 (UTC) Received: from esa7.dell-outbound.iphmx.com (esa7.dell-outbound.iphmx.com [68.232.153.96]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id CC7025FBD6 for ; Thu, 20 Oct 2016 09:18:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=rsa.com; i=@rsa.com; q=dns/txt; s=jan2013; t=1476955118; x=1508491118; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; bh=rTi1RhMCRKBK93vil9siNaJE00H0PYUq0VDqHo6icpo=; b=Cf7mNKqffblHtixmB2VyEYH8BQzFEeu01c/88zfS8O+yFqHQgtfp9h6v OXn1yySkylTvC/FR+NPqDPRsaQ4gET9TwMfpCAe5SS6klGLeXWowlojK8 XDg5tmAGmucN33e6fQX6UX2FryUdMOGgbKo1Eh5rhUIe93M1/v4wngHAE U=; Received: from esa2.dell-outbound2.iphmx.com ([68.232.153.202]) by esa7.dell-outbound.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Oct 2016 04:18:37 -0500 Received: from mailuogwdur.emc.com ([128.221.224.79]) by esa2.dell-outbound2.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Oct 2016 15:18:37 +0600 Received: from maildlpprd53.lss.emc.com (maildlpprd53.lss.emc.com [10.106.48.157]) by mailuogwprd51.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id u9K9Ia9r030438 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 20 Oct 2016 05:18:36 -0400 X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd51.lss.emc.com u9K9Ia9r030438 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=rsa.com; s=jan2013; t=1476955116; bh=+cVPCDZ8Z1goeFKMpjUGmg/OBOQ=; h=From:To:Subject:Date:Message-ID:References:In-Reply-To: Content-Type:MIME-Version; b=hSFYXyPVMgx1YNqoeyBumd/W3+VGhYCcjiumhDk74n5cADQBmocQZM8fTs4au3wWo EM3SvjImQj6azzrwp09/GnK23W7/O75p9HG94LvucaLCGhCE/veKkpp0hQZ2VMzsJG 7Ntdyk6p+q7raXVL1abjw4N6ars5KM34Ol7/Xm6A= X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd51.lss.emc.com u9K9Ia9r030438 Received: from mailusrhubprd52.lss.emc.com (mailusrhubprd52.lss.emc.com [10.106.48.25]) by maildlpprd53.lss.emc.com (RSA Interceptor); Thu, 20 Oct 2016 05:18:23 -0400 Received: from MXHUB315.corp.emc.com (MXHUB315.corp.emc.com [10.146.3.93]) by mailusrhubprd52.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id u9K9IOQT005841 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=FAIL); Thu, 20 Oct 2016 05:18:24 -0400 Received: from MX302CL04.corp.emc.com ([fe80::18ad:6300:21eb:2b39]) by MXHUB315.corp.emc.com ([10.146.3.93]) with mapi id 14.03.0266.001; Thu, 20 Oct 2016 05:18:23 -0400 From: "Mendelson, Assaf" To: =?iso-8859-1?Q?muhammet_paky=FCrek?= , "user@spark.apache.org" Subject: RE: pyspark dataframe codes for lead lag to column Thread-Topic: pyspark dataframe codes for lead lag to column Thread-Index: AQHSKqxzGqr2uGFAEEWIkFPexzG7BKCxEERg Date: Thu, 20 Oct 2016 09:18:23 +0000 Message-ID: <4E2D62B698F0814BB1E367C9F243AB8005CFB5@MX302CL04.corp.emc.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.76.70.64] Content-Type: multipart/alternative; boundary="_000_4E2D62B698F0814BB1E367C9F243AB8005CFB5MX302CL04corpemcc_" MIME-Version: 1.0 X-Sentrion-Hostname: mailusrhubprd52.lss.emc.com X-RSA-Classifications: public archived-at: Thu, 20 Oct 2016 09:18:45 -0000 --_000_4E2D62B698F0814BB1E367C9F243AB8005CFB5MX302CL04corpemcc_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Depending on your usecase, you may want to take a look at window functions From: muhammet paky=FCrek [mailto:mpak85@hotmail.com] Sent: Thursday, October 20, 2016 11:36 AM To: user@spark.apache.org Subject: pyspark dataframe codes for lead lag to column is there pyspark dataframe codes for lead lag to column? lead/lag column is something 1 lag -1 lead 2 2 1 3 3 2 4 4 3 5 5 4 -1 --_000_4E2D62B698F0814BB1E367C9F243AB8005CFB5MX302CL04corpemcc_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Depending on your usecase= , you may want to take a look at window functions

 <= /p>

From: muhammet= paky=FCrek [mailto:mpak85@hotmail.com]
Sent: Thursday, October 20, 2016 11:36 AM
To: user@spark.apache.org
Subject: pyspark dataframe codes for lead lag to column

 

 

 

is there pyspark dataframe codes for le= ad lag to column?

 

lead/lag column is something

 

1      lag   -1   &= nbsp;    lead     2

2             =    1                 &nbs= p;  3

3             =    2                 &nbs= p;  4

4             =    3                 &nbs= p;  5

5             =    4                 &nbs= p; -1

--_000_4E2D62B698F0814BB1E367C9F243AB8005CFB5MX302CL04corpemcc_--