calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remus Rusanu <rrus...@hortonworks.com>
Subject Re: Towards Calcite release 1.12
Date Tue, 14 Mar 2017 13:13:59 GMT
Hello Julian,

I added an extra check and prevent constant reduction in the HiveExecutor for TIMESTAMPs that
have precision higher than 3. Not based on type, since Hive has always TIMESTAMP(9), but on
actual values that contain non-zero sub-millisecond nanos. This almost works, the problem
is that the expression still gets reduced at a later stage:

	at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1239)
	at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:1236)
	at org.apache.calcite.rex.RexExecutable.reduce(RexExecutable.java:86)
	at org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:128)
	at org.apache.calcite.rex.RexUtil.simplifyCast(RexUtil.java:2450)
	at org.apache.calcite.rex.RexUtil.simplify(RexUtil.java:1633)
	at org.apache.calcite.rex.RexUtil.simplify(RexUtil.java:1587)
	at org.apache.calcite.rex.RexUtil.simplifyList(RexUtil.java:1747)
	at org.apache.calcite.rex.RexUtil.simplifyComparison(RexUtil.java:1658)
	at org.apache.calcite.rex.RexUtil.simplify(RexUtil.java:1648)
	at org.apache.calcite.rex.RexUtil$ExprSimplifier.visitCall(RexUtil.java:3051)
	at org.apache.calcite.rex.RexUtil$ExprSimplifier.visitCall(RexUtil.java:3016)
	at org.apache.calcite.rex.RexCall.accept(RexCall.java:104)
	at org.apache.calcite.rex.RexShuttle.apply(RexShuttle.java:279)

Here the issue is that the custom HiveExecutor gets lost on the stack and the default EXECUTOR
is used instead, resulting in incorrect behavior. I created CALCITE-1695 Not all RexUtil.simplifyXxx
code paths carry the provided executor. This new CALCITE-1695 issue is fairly trivial to fix.
I know Julian talked about making the executor a property of the RexBuilder and that would
be better long term, but right now I tried just extending the signature of simplifyComparison/SimplifyList
to accept and pass on a RexExecutor and this seems to solve the issues I was seeing in Hive.

 
Thanks,
~Remus


On 3/13/17, 4:32 PM, "Julian Hyde" <jhyde@apache.org> wrote:

    1690 is a big one. For now, can you just disable constant reduction for timestamp literals
that have high precision?
    
    > On Mar 13, 2017, at 4:30 PM, Remus Rusanu <rrusanu@hortonworks.com> wrote:
    > 
    > CALCITE-1681 is not needed anymore
    > The only pending issue for Hive is CALCITE-1690: Calcite timestamp literals cannot
express precision above millisecond.
    > 
    > Thanks,
    > ~Remus
    > 
    > ________________________________________
    > From: Julian Hyde <jhyde@apache.org>
    > Sent: Tuesday, March 14, 2017 1:23 AM
    > To: dev@calcite.apache.org
    > Subject: Re: Towards Calcite release 1.12
    > 
    > OK, what’s left before RC0?
    > 
    > I am working on committing 1561, 1684, 1691, 1688. Also, I will review 1566 shortly.
I may also get to 1689.
    > 
    > Remus, Are all of your Hive issues resolved? In particular do you need 1681?
    > 
    > Jesus, Where are you with 1421, 1456, 1413? Can you also commit Nishant’s 1683
since you +1’d it?
    > 
    > Julian
    > 
    > 
    >> On Mar 8, 2017, at 12:25 PM, Julian Hyde <jhyde@apache.org> wrote:
    >> 
    >> We have that dependency already, albeit transitively: CALCITE-1681 blocks HIVE-15708,
which is depended upon by CALCITE-1657 “Release Calcite 1.12.0”.
    >> 
    >>> On Mar 8, 2017, at 12:12 PM, Remus Rusanu <rrusanu@hortonworks.com>
wrote:
    >>> 
    >>> Looks like CALCITE-1681 ‘Provide a way to copy RelNode trees between clusters’
is also needed. I hope I’ll have it ready this week.
    >>> 
    >>> On 3/8/17, 10:49 AM, "Julian Hyde" <jhyde@apache.org> wrote:
    >>> 
    >>>  Same comments as for CALCITE-1615. The list contains commits up to 2/24.
I will fix before the release.
    >>> 
    >>>> On Mar 8, 2017, at 10:43 AM, Eli Levine <elilevine@gmail.com> wrote:
    >>>> 
    >>>> https://issues.apache.org/jira/browse/CALCITE-1598 (Pig adapter) does
not
    >>>> seem to be on the list.
    >>>> 
    >>>> Thanks,
    >>>> 
    >>>> Eli
    >>>> 
    >>>> 
    >>>> On Wed, Mar 8, 2017 at 10:40 AM, Julian Hyde <jhyde@apache.org>
wrote:
    >>>> 
    >>>>> Commit 49888a6c581f6243a5b3f1a48ea2c9edd6aac9d5, which fixed
    >>>>> CALCITE-1615, had not made it to https://github.com/julianhyde/
    >>>>> calcite/tree/branch-1.12 <https://github.com/julianhyde/calcite/tree/
    >>>>> branch-1.12> at the time I wrote that draft of the release notes.
There
    >>>>> are about a dozen other cases fixed since then. I’ll be sure to
include it.
    >>>>> 
    >>>>> Julian
    >>>>> 
    >>>>> 
    >>>>>> On Mar 8, 2017, at 10:29 AM, Haohui Mai <ricetons@gmail.com>
wrote:
    >>>>>> 
    >>>>>> Hi Julian,
    >>>>>> 
    >>>>>> I think CALCITE-1615 is not on the list. Can you please double
check?
    >>>>>> 
    >>>>>> Haohui
    >>>>>> On Wed, Mar 8, 2017 at 9:49 AM Julian Hyde <jhyde@apache.org>
wrote:
    >>>>>> 
    >>>>>>> First draft of release notes:
    >>>>>>> https://github.com/julianhyde/calcite/blob/branch-1.12/site/
    >>>>> _docs/history.md
    >>>>>>> <
    >>>>>>> https://github.com/julianhyde/calcite/blob/branch-1.12/site/
    >>>>> _docs/history.md>
    >>>>>>> (contains commits up to 12 days ago).
    >>>>>>> 
    >>>>>>> 
    >>>>>>>> On Mar 8, 2017, at 9:16 AM, Julian Hyde <jhyde@apache.org>
wrote:
    >>>>>>>> 
    >>>>>>>> Thanks. That last change should be fairly low-risk, so
we should be
    >>>>> able
    >>>>>>> to move to an RC and vote soon afterwards.
    >>>>>>>> 
    >>>>>>>> Committers, Let’s try to stabilize. I propose we get
any “risky”
    >>>>> changes
    >>>>>>> in by the end of the weekend.
    >>>>>>>> 
    >>>>>>>> By the way, I volunteer to be release manager.
    >>>>>>>> 
    >>>>>>>> Julian
    >>>>>>>> 
    >>>>>>>>> On Mar 8, 2017, at 2:53 AM, Remus Rusanu <rrusanu@hortonworks.com>
    >>>>>>> wrote:
    >>>>>>>>> 
    >>>>>>>>> We have only one issue left to resolve, the materialized
views RelNode
    >>>>>>> cloning into a new planner. I expect to have a resolution
by end of this
    >>>>>>> week, so the all-done ETA from me would be mid-next week,
let say March
    >>>>> 15.
    >>>>>>>>> 
    >>>>>>>>> Thanks,
    >>>>>>>>> ~Remus
    >>>>>>>>> 
    >>>>>>>>> On 3/7/17, 5:08 PM, "Julian Hyde" <jhyde@apache.org>
wrote:
    >>>>>>>>> 
    >>>>>>>>> Remus,
    >>>>>>>>> 
    >>>>>>>>> Do you have an ETA for HIVE-15708? Just a guess is
fine, but it helps
    >>>>>>> the rest of us know when this release is likely to drop.
    >>>>>>>>> 
    >>>>>>>>> Julian
    >>>>>>>>> 
    >>>>>>>>>> On Feb 28, 2017, at 10:57 AM, Julian Hyde <jhyde@apache.org>
wrote:
    >>>>>>>>>> 
    >>>>>>>>>> On Tue, Feb 28, 2017 at 1:40 AM, Remus Rusanu
<
    >>>>> rrusanu@hortonworks.com>
    >>>>>>> wrote:
    >>>>>>>>>>> Put on hold, for how long?
    >>>>>>>>>> 
    >>>>>>>>>> Until someone chimed in with their time constraints,
as you just did.
    >>>>>>> :)
    >>>>>>>>>> 
    >>>>>>>>>> We make decisions on-list, but for that we need
people to provide
    >>>>>>>>>> information on-list.
    >>>>>>>>>> 
    >>>>>>>>>>> [We] did not yet have a 100% successful run
for HIVE-15708,
    >>>>>>>>>>> but we're close and once we have the changes
ready, we
    >>>>>>>>>>> cannot do the move unless Calcite does the
release.
    >>>>>>>>>> 
    >>>>>>>>>> What's your best guess for when HIVE-15708 will
be fixed? We will
    >>>>>>>>>> start the RC vote after that.
    >>>>>>>>>> 
    >>>>>>>>>> Julian
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>>> 
    >>>>>>>> 
    >>>>>>> 
    >>>>>>> 
    >>>>> 
    >>>>> 
    >>> 
    >>> 
    >>> 
    >>> 
    >> 
    > 
    > 
    
    
    

Mime
View raw message