spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liang-Chi Hsieh <vii...@gmail.com>
Subject Re: [SQL][ML] Pipeline performance regression between 1.6 and 2.x
Date Fri, 03 Feb 2017 07:34:20 GMT

Hi Maciej,

FYI, this fix is submitted at https://github.com/apache/spark/pull/16785.


Liang-Chi Hsieh wrote
> Hi Maciej,
> 
> After looking into the details of the time spent on preparing the executed
> plan, the cause of the significant difference between 1.6 and current
> codebase when running the example, is the optimization process to generate
> constraints.
> 
> There seems few operations in generating constraints which are not
> optimized. Plus the fact the query plan grows continuously, the time spent
> on generating constraints increases more and more.
> 
> I am trying to reduce the time cost. Although not as low as 1.6 because we
> can't remove the process of generating constraints, it is significantly
> lower than current codebase (74294 ms -> 2573 ms).
> 
> 385 ms
> 107 ms
> 46 ms
> 58 ms
> 64 ms
> 105 ms
> 86 ms
> 122 ms
> 115 ms
> 114 ms
> 100 ms
> 109 ms
> 169 ms
> 196 ms
> 174 ms
> 212 ms
> 290 ms
> 254 ms
> 318 ms
> 405 ms
> 347 ms
> 443 ms
> 432 ms
> 500 ms
> 544 ms
> 619 ms
> 697 ms
> 683 ms
> 807 ms
> 802 ms
> 960 ms
> 1010 ms
> 1155 ms
> 1251 ms
> 1298 ms
> 1388 ms
> 1503 ms
> 1613 ms
> 2279 ms
> 2349 ms
> 2573 ms
> 
> Liang-Chi Hsieh wrote
>> Hi Maciej,
>> 
>> Thanks for the info you provided.
>> 
>> I tried to run the same example with 1.6 and current branch and record
>> the difference between the time cost on preparing the executed plan.
>> 
>> Current branch:
>> 
>> 292 ms                                                                          
  
>> 95 ms                             
>> 57 ms
>> 34 ms
>> 128 ms
>> 120 ms
>> 63 ms
>> 106 ms
>> 179 ms
>> 159 ms
>> 235 ms
>> 260 ms
>> 334 ms
>> 464 ms
>> 547 ms                             
>> 719 ms
>> 942 ms
>> 1130 ms
>> 1928 ms
>> 1751 ms
>> 2159 ms                            
>> 2767 ms
>> 3333 ms
>> 4175 ms
>> 5106 ms
>> 6269 ms
>> 7683 ms
>> 9210 ms
>> 10931 ms
>> 13237 ms
>> 15651 ms
>> 19222 ms
>> 23841 ms
>> 26135 ms
>> 31299 ms
>> 38437 ms
>> 47392 ms
>> 51420 ms
>> 60285 ms
>> 69840 ms
>> 74294 ms
>> 
>> 1.6:
>> 
>> 3 ms
>> 4 ms
>> 10 ms
>> 4 ms
>> 17 ms
>> 8 ms
>> 12 ms
>> 21 ms
>> 15 ms
>> 15 ms
>> 19 ms
>> 23 ms
>> 28 ms
>> 28 ms
>> 58 ms
>> 39 ms
>> 43 ms
>> 61 ms
>> 56 ms
>> 60 ms
>> 81 ms
>> 73 ms
>> 100 ms
>> 91 ms
>> 96 ms
>> 116 ms
>> 111 ms
>> 140 ms
>> 127 ms
>> 142 ms
>> 148 ms
>> 165 ms
>> 171 ms
>> 198 ms
>> 200 ms
>> 233 ms
>> 237 ms
>> 253 ms
>> 256 ms
>> 271 ms
>> 292 ms
>> 452 ms
>> 
>> Although they both take more time after each iteration due to the grown
>> query plan, it is obvious that current branch takes much more time than
>> 1.6 branch. The optimizer and query planning in current branch is much
>> more complicated than 1.6.
>> zero323 wrote
>>> Hi Liang-Chi,
>>> 
>>> Thank you for your answer and PR but what I think I wasn't specific
>>> enough. In hindsight I should have illustrate this better. What really
>>> troubles me here is a pattern of growing delays. Difference between
>>> 1.6.3 (roughly 20s runtime since the first job):
>>> 
>>> 
>>> 1.6 timeline
>>> 
>>> vs 2.1.0 (45 minutes or so in a bad case):
>>> 
>>> 2.1.0 timeline
>>> 
>>> The code is just an example and it is intentionally dumb. You easily
>>> mask this with caching, or using significantly larger data sets. So I
>>> guess the question I am really interested in is - what changed between
>>> 1.6.3 and 2.x (this is more or less consistent across 2.0, 2.1 and
>>> current master) to cause this and more important, is it a feature or is
>>> it a bug? I admit, I choose a lazy path here, and didn't spend much time
>>> (yet) trying to dig deeper.
>>> 
>>> I can see a bit higher memory usage, a bit more intensive GC activity,
>>> but nothing I would really blame for this behavior, and duration of
>>> individual jobs is comparable with some favor of 2.x. Neither
>>> StringIndexer nor OneHotEncoder changed much in 2.x. They used RDDs for
>>> fitting in 1.6 and, as far as I can tell, they still do that in 2.x. And
>>> the problem doesn't look that related to the data processing part in the
>>> first place.
>>> 
>>> 
>>> On 02/02/2017 07:22 AM, Liang-Chi Hsieh wrote:
>>>> Hi Maciej,
>>>>
>>>> FYI, the PR is at https://github.com/apache/spark/pull/16775.
>>>>
>>>>
>>>> Liang-Chi Hsieh wrote
>>>>> Hi Maciej,
>>>>>
>>>>> Basically the fitting algorithm in Pipeline is an iterative operation.
>>>>> Running iterative algorithm on Dataset would have RDD lineages and
>>>>> query
>>>>> plans that grow fast. Without cache and checkpoint, it gets slower
>>>>> when
>>>>> the iteration number increases.
>>>>>
>>>>> I think it is why when you run a Pipeline with long stages, it gets
>>>>> much
>>>>> longer time to finish. As I think it is not uncommon to have long
>>>>> stages
>>>>> in a Pipeline, we should improve this. I will submit a PR for this.
>>>>> zero323 wrote
>>>>>> Hi everyone,
>>>>>>
>>>>>> While experimenting with ML pipelines I experience a significant
>>>>>> performance regression when switching from 1.6.x to 2.x.
>>>>>>
>>>>>> import org.apache.spark.ml.{Pipeline, PipelineStage}
>>>>>> import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer,
>>>>>> VectorAssembler}
>>>>>>
>>>>>> val df = (1 to 40).foldLeft(Seq((1, "foo"), (2, "bar"), (3,
>>>>>> "baz")).toDF("id", "x0"))((df, i) => df.withColumn(s"x$i", $"x0"))
>>>>>> val indexers = df.columns.tail.map(c => new StringIndexer()
>>>>>>   .setInputCol(c)
>>>>>>   .setOutputCol(s"${c}_indexed")
>>>>>>   .setHandleInvalid("skip"))
>>>>>>
>>>>>> val encoders = indexers.map(indexer => new OneHotEncoder()
>>>>>>   .setInputCol(indexer.getOutputCol)
>>>>>>   .setOutputCol(s"${indexer.getOutputCol}_encoded")
>>>>>>   .setDropLast(true))
>>>>>>
>>>>>> val assembler = new
>>>>>> VectorAssembler().setInputCols(encoders.map(_.getOutputCol))
>>>>>> val stages: Array[PipelineStage] = indexers ++ encoders :+ assembler
>>>>>>
>>>>>> new Pipeline().setStages(stages).fit(df).transform(df).show
>>>>>>
>>>>>> Task execution time is comparable and executors are most of the time
>>>>>> idle so it looks like it is a problem with the optimizer. Is it a
>>>>>> known
>>>>>> issue? Are there any changes I've missed, that could lead to this
>>>>>> behavior?
>>>>>>
>>>>>> -- 
>>>>>> Best,
>>>>>> Maciej
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: 
>>>>>> dev-unsubscribe@.apache
>>>>
>>>>
>>>>
>>>>
>>>> -----
>>>> Liang-Chi Hsieh | @viirya 
>>>> Spark Technology Center 
>>>> http://www.spark.tc/ 
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/SQL-ML-Pipeline-performance-regression-between-1-6-and-2-x-tp20803p20822.html
>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>> Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: 

>>> dev-unsubscribe@.apache

>>>>
>>> 
>>> -- 
>>> Maciej Szymkiewicz
>>> 
>>> 
>>> 
>>> nM15AWH.png (19K)
>>> &lt;http://apache-spark-developers-list.1001551.n3.nabble.com/attachment/20827/0/nM15AWH.png&gt;
>>> KHZa7hL.png (26K)
>>> &lt;http://apache-spark-developers-list.1001551.n3.nabble.com/attachment/20827/1/KHZa7hL.png&gt;





-----
Liang-Chi Hsieh | @viirya 
Spark Technology Center 
http://www.spark.tc/ 
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SQL-ML-Pipeline-performance-regression-between-1-6-and-2-x-tp20803p20837.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message