spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Dubey <saurabh.s.du...@oracle.com>
Subject RE: Window Functions with SQLContext
Date Thu, 01 Sep 2016 05:40:19 GMT
Hi Adline,

rowNumber and row_number are same functions:
@scala.deprecated("Use row_number. This will be removed in Spark 2.0.")
def rowNumber() : org.apache.spark.sql.Column = { /* compiled code */ }
def row_number() : org.apache.spark.sql.Column = { /* compiled code */ }

but the issue here is about using SQLContext instead of HiveContext.

Thanks,
Saurabh

-----Original Message-----
From: Adline Dsilva [mailto:adline.dsilva@mimos.my] 
Sent: 01 September 2016 11:05
To: saurabh3d; user@spark.apache.org
Subject: RE: Window Functions with SQLContext

Hi,
  Use function rowNumber instead of row_number

df1.withColumn("row_number", rowNumber.over(w));

Regards,
Adline
________________________________________
From: saurabh3d [saurabh.s.dubey@oracle.com]
Sent: 01 September 2016 13:16
To: user@spark.apache.org
Subject: Window Functions with SQLContext

Hi All,

As per  SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001>   ,
Window functions should be supported by SQLContext. But when i try to run

SQLContext sqlContext = new SQLContext(jsc); WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w)); df_2.show(false);

it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not resolve window
function 'row_number'. Note that, using window functions currently requires a HiveContext;

This code runs fine with HiveContext.
Any idea what's going on?  Is this a known issue and is there a workaround to make Window
function work without HiveContext.

Thanks,
Saurabh




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


________________________________
DISCLAIMER:


This e-mail (including any attachments) is for the addressee(s) only and may be confidential,
especially as regards personal data. If you are not the intended recipient, please note that
any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited.
If you have received this email in error, please notify the sender immediately and delete
the original message (including any attachments).

MIMOS Berhad is a research and development institution under the purview of the Malaysian
Ministry of Science, Technology and Innovation. Opinions, conclusions and other information
in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries
shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries
and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability
arising from or in connection with computer viruses and/or corrupted e-mails is excluded to
the fullest extent permitted by law.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message