spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenchen Fan <cloud0...@gmail.com>
Subject Re: Why Dataset.hint uses logicalPlan (= analyzed not planWithBarrier)?
Date Fri, 26 Jan 2018 08:08:38 GMT
Looks like we missed this one, feel free to submit a patch, thanks for your
finding!

On Fri, Jan 26, 2018 at 3:39 PM, Jacek Laskowski <jacek@japila.pl> wrote:

> Hi,
>
> I've just noticed that every time Dataset.hint is used it triggers
> execution of logical commands, their unions and hint resolution (among
> other things that analyzer does).
>
> Why?
>
> Why does hint trigger hint resolution (through QueryExecution.analyzed)?
> [1]
>
> And moreover why not to use planWithBarrier instead? [2] Looks like an
> oversight, doesn't it?
>
> [1] https://github.com/apache/spark/blob/master/sql/core/
> src/main/scala/org/apache/spark/sql/Dataset.scala#L1219
>
> [2] https://github.com/apache/spark/blob/master/sql/core/
> src/main/scala/org/apache/spark/sql/Dataset.scala#L195
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> Mastering Spark SQL https://bit.ly/mastering-spark-sql
> Spark Structured Streaming https://bit.ly/spark-structured-streaming
> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
> Follow me at https://twitter.com/jaceklaskowski
>

Mime
View raw message