spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-14966) SizeEstimator should ignore classes in the scala.reflect package
Date Wed, 27 Apr 2016 20:33:12 GMT
Josh Rosen created SPARK-14966:
----------------------------------

             Summary: SizeEstimator should ignore classes in the scala.reflect package
                 Key: SPARK-14966
                 URL: https://issues.apache.org/jira/browse/SPARK-14966
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
            Reporter: Josh Rosen
            Assignee: Josh Rosen


In local benchmarks, I noticed SizeEstimator spending tons of time estimating the size of
objects which contain TypeTag or ClassTag fields. The problem with these tags is that they
reference global Scala reflection objects, which, in turn, reference many singletons, such
as TestHive. This throws off the size estimation and wastes tons of time traversing the huge
object graph.

As a result, I think that SizeEstimator should ignore any classes in the {{scala.reflect}}
package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message