spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From java8964 <java8...@hotmail.com>
Subject RE: Calculating Min and Max Values using Spark Transformations?
Date Fri, 28 Aug 2015 17:09:57 GMT
Or RDD.max() and RDD.min() won't work for you?
Yong

Subject: Re: Calculating Min and Max Values using Spark Transformations?
To: ashen@wso2.com
CC: user@spark.apache.org
From: jfchen@us.ibm.com
Date: Fri, 28 Aug 2015 09:28:43 -0700


If you already loaded csv data into a dataframe, why not register it as a table, and use Spark
SQL

to find max/min or any other aggregates? SELECT MAX(column_name) FROM dftable_name ... seems
natural.












JESSE CHEN

Big Data Performance | IBM Analytics



Office:  408 463 2296

Mobile: 408 828 9068

Email:   jfchen@us.ibm.com








ashensw ---08/28/2015 05:40:07 AM---Hi all, I have a dataset which consist of large number
of features(columns). It is



From:	ashensw <ashen@wso2.com>

To:	user@spark.apache.org

Date:	08/28/2015 05:40 AM

Subject:	Calculating Min and Max Values using Spark Transformations?








Hi all,



I have a dataset which consist of large number of features(columns). It is

in csv format. So I loaded it into a spark dataframe. Then I converted it

into a JavaRDD<Row> Then using a spark transformation I converted that into

JavaRDD<String[]>. Then again converted it into a JavaRDD<double[]>. So now

I have a JavaRDD<double[]>. So is there any method to calculate max and min

values of each columns in this JavaRDD<double[]> ?  



Or Is there any way to access the array if I store max and min values to a

array inside the spark transformation class?



Thanks.







--

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Calculating-Min-and-Max-Values-using-Spark-Transformations-tp24491.html

Sent from the Apache Spark User List mailing list archive at Nabble.com.



---------------------------------------------------------------------

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org

For additional commands, e-mail: user-help@spark.apache.org




 		 	   		  
Mime
View raw message