spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-23880) table cache should be lazy and don't trigger any jobs.
Date Fri, 06 Apr 2018 06:26:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-23880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428011#comment-16428011
] 

Wenchen Fan commented on SPARK-23880:
-------------------------------------

cc [~ZenWzh] [~maropu] is any of you interested in doing this?

> table cache should be lazy and don't trigger any jobs.
> ------------------------------------------------------
>
>                 Key: SPARK-23880
>                 URL: https://issues.apache.org/jira/browse/SPARK-23880
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Wenchen Fan
>            Priority: Major
>
> {code}
> val df = spark.range(10000000000L)
>   .filter('id > 1000)
>   .orderBy('id.desc)
>   .cache()
> {code}
> This triggers a job while the cache should be lazy. The problem is that, when creating
`InMemoryRelation`, we build the RDD, which calls `SparkPlan.execute` and may trigger jobs,
like sampling job for range partitioner, or broadcast job.
> We should create the RDD at physical phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message