spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kazuaki Ishizaki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24486) Slow performance reading ArrayType columns
Date Thu, 07 Jun 2018 15:56:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504845#comment-16504845
] 

Kazuaki Ishizaki commented on SPARK-24486:
------------------------------------------

Thank you for reporting a problem.
Could you please let us know which value is shown for each of three results in `sum(...)`?

> Slow performance reading ArrayType columns
> ------------------------------------------
>
>                 Key: SPARK-24486
>                 URL: https://issues.apache.org/jira/browse/SPARK-24486
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, SQL
>    Affects Versions: 2.3.0
>            Reporter: Luca Canali
>            Priority: Minor
>
> We have found an issue of slow performance in one of our applications when running on
Spark 2.3.0 (the same workload does not have a performance issue on Spark 2.2.1). We suspect
a regression in the area of handling columns of ArrayType. I have built a simplified test
case showing a manifestation of the issue to help with troubleshooting:
>  
>  
> {code:java}
> // prepare test data
> val stringListValues=Range(1,30000).mkString(",")
> sql(s"select 1 as myid, Array($stringListValues) as myarray from range(20000)").repartition(1).write.parquet("file:///tmp/deleteme1")
> // run test
> spark.read.parquet("file:///tmp/deleteme1").limit(1).show(){code}
> Performance measurements:
>  
> On a desktop-size test system, the test runs in about 2 sec using Spark 2.2.1 (runtime
goes down to subsecond in subsequent runs) and takes close to 20 sec on Spark 2.3.0
>  
> Additional drill-down using Spark task metrics data, show that in Spark 2.2.1 only 2
records are read by this workload, while on Spark 2.3.0 all rows in the file are read, which
appears anomalous.
> Example:
> {code:java}
> bin/spark-shell --master local[*] --driver-memory 2g --packages ch.cern.sparkmeasure:spark-measure_2.11:0.11
> val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) 
> stageMetrics.runAndMeasure(spark.read.parquet("file:///tmp/deleteme1").limit(1).show())
> {code}
>  
>  
> Selected metrics from Spark 2.3.0 run:
>  
> {noformat}
> elapsedTime => 17849 (18 s)
> sum(numTasks) => 11
> sum(recordsRead) => 20000
> sum(bytesRead) => 1136448171 (1083.0 MB){noformat}
>  
>  
> From Spark 2.2.1 run:
>  
> {noformat}
> elapsedTime => 1329 (1 s)
> sum(numTasks) => 2
> sum(recordsRead) => 2
> sum(bytesRead) => 269162610 (256.0 MB)
> {noformat}
>  
> Note: Using Spark built from master (as I write this, June 7th 2018) shows the same behavior
as found in Spark 2.3.0
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message