spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Nguyen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-13804) Spark SQL's DataFrame.count() Major Divergent (Non-Linear) Performance Slowdown going from 4million rows to 16+ million rows
Date Tue, 15 Mar 2016 16:52:33 GMT

    [ https://issues.apache.org/jira/browse/SPARK-13804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195625#comment-15195625
] 

Michael Nguyen commented on SPARK-13804:
----------------------------------------

I posted this issue to user@ at 

http://apache-spark-user-list.1001560.n3.nabble.com/DataFrame-count-Major-Divergent-Non-Linear-Performance-Slowdown-when-data-set-increases-from-4-millis-td26493.html

However, it has not been accepted by the mailing list yet. What needs to be done for it to
be accepted ? And what is the typical turn-around for postings to be accepted ?

> Spark SQL's DataFrame.count()  Major Divergent (Non-Linear) Performance Slowdown going
from 4million rows to 16+ million rows
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-13804
>                 URL: https://issues.apache.org/jira/browse/SPARK-13804
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0
>         Environment: - 3 nodes Spark cluster: 1 master node and 2 slave nodes
> - Each node is an EC2 with c3.4xlarge
> - Each node has 16 cores and 30GB of RAM
>            Reporter: Michael Nguyen
>
> Spark SQL is used to load cvs files via com.databricks.spark.csv and then run dataFrame.count()

> In the same environment with plenty of CPU and RAM, Spark SQL takes 
> - 18.25 seconds to load  a table with 4 millions vs
> - 346.624 seconds (5.77 minutes) to load a table with 16 million rows.
> Even though the number of rows increases by 4 times, the time it takes Spark SQL to run
dataframe.count () increases by 19.22 times. So the performance of dataframe.count () diverges
so drastically.
> 1. Why does Spark SQL's performance not proportional to the number of rows while there
is plenty of CPU and RAM (it uses only 10GB out of 30GB RAM) ?
> 2. What can be done to fix  this performance issue ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message