spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: [SQL] Purpose of RuntimeReplaceable unevaluable unary expressions?
Date Thu, 31 May 2018 09:38:54 GMT
Yay! That's right!!! Thanks Reynold. Such a short answer with so much
information. Thanks.

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Mastering Spark SQL https://bit.ly/mastering-spark-sql
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
Follow me at https://twitter.com/jaceklaskowski

On Wed, May 30, 2018 at 8:10 PM, Reynold Xin <rxin@databricks.com> wrote:

> SQL expressions?
>
> On Wed, May 30, 2018 at 11:09 AM Jacek Laskowski <jacek@japila.pl> wrote:
>
>> Hi,
>>
>> I've been exploring RuntimeReplaceable expressions [1] and have been
>> wondering what their purpose is.
>>
>> Quoting the scaladoc [2]:
>>
>> > An expression that gets replaced at runtime (currently by the
>> optimizer) into a different expression for evaluation. This is mainly used
>> to provide compatibility with other databases.
>>
>> For example, ParseToTimestamp expression is a RuntimeReplaceable
>> expression and it is replaced by Cast(left, TimestampType)
>> or Cast(UnixTimestamp(left, format), TimestampType) per to_timestamp
>> function (there are two variants).
>>
>> My question is why is this RuntimeReplaceable better than simply using
>> the Casts as the implementation of to_timestamp functions?
>>
>> def to_timestamp(s: Column, fmt: String): Column = withExpr {
>>   // pseudocode
>>   Cast(UnixTimestamp(left, format), TimestampType)
>> }
>>
>> What's wrong with the above implementation compared to the current one?
>>
>> [1] https://github.com/apache/spark/blob/master/sql/
>> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
>> expressions/Expression.scala#L275
>>
>> [2] https://github.com/apache/spark/blob/master/sql/
>> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
>> expressions/Expression.scala#L266-L267
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://about.me/JacekLaskowski
>> Mastering Spark SQL https://bit.ly/mastering-spark-sql
>> Spark Structured Streaming https://bit.ly/spark-structured-streaming
>> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
>> Follow me at https://twitter.com/jaceklaskowski
>>
>

Mime
View raw message