From dev-return-47715-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Sep 27 15:08:19 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 304D3180638 for ; Fri, 27 Sep 2019 17:08:19 +0200 (CEST) Received: (qmail 70867 invoked by uid 500); 27 Sep 2019 15:08:18 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 70853 invoked by uid 99); 27 Sep 2019 15:08:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Sep 2019 15:08:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B04B4C2E83 for ; Fri, 27 Sep 2019 15:08:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.201 X-Spam-Level: X-Spam-Status: No, score=-0.201 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id h7ImhyXb2qSU for ; Fri, 27 Sep 2019 15:08:15 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.50; helo=mail-lf1-f50.google.com; envelope-from=gvvinblade@gmail.com; receiver= Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 5B170BC572 for ; Fri, 27 Sep 2019 15:08:15 +0000 (UTC) Received: by mail-lf1-f50.google.com with SMTP id r134so2161821lff.12 for ; Fri, 27 Sep 2019 08:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=5EeNfz2VAd8k8lb/Q14Zx2+wArwX1/R0n/fVPYQPKyY=; b=iEaKRJJoHpxSjZuJCUCZdKcHJQE5rPmrDBVRxGTGWLRzRkwSJIXYYV2pBGg6lA8qsI 9bpjYA5elNL2XryGbMGqr2oVAP50jL2/LXn8WnNYxTRMISHrhUQueMEz49Gp167VFkoe EDGkqatdkWiIzmGKqjoFOX7D6JYXb5TIRE60CCX9gFLG/ER3fz+jIlXjOymNs6lqhhdG L4tDMSQJTuMs2BQZVRbW/p5SupJf1L5SOjOxVfMg1Qs7NvsBAx1u8AoVZ03bU9clOdT/ LxZDFtMpl3QaJ06WIbDsLf1HzTbFqfyJph/2B5TRXPVVaANbM9/t98uQpf8n8nH0l3Gq cGtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=5EeNfz2VAd8k8lb/Q14Zx2+wArwX1/R0n/fVPYQPKyY=; b=PJOgxzHqkipBj8K+ShKIbuf4vCDPxkbngyg1nCwFn2vDnigdPpZM/hfUQ3Qzs+D4pW QHL2WmJFVObLaVDhmwi5OYA51fAxnIHW+4Lyn+JmVwPrZ6/+C3925bG8K/8I7sqAU/h1 DZx0xDeg8tK094jI5x3s1koIoVik/khv+0kB2Gjg6t/OSoBPlXS1WdXIhHrTUSg6Fh2I DaB65h33VfF/syb9ibEqye2mJIsup+6pD2mbTXXq1mjDZzVPMVBmXeJIJHwJex23pixb wFk8WvagECMt1SLQVRthbslO+tpR0RcFifXVHYYG979IAPoAlBekNUMn0F2uXtIQpcbd cEVw== X-Gm-Message-State: APjAAAVmzybHDdk99x2EqAtrXskt6AYkILNHq72FbI/VJ9SgxtiS0amk G2IW9dpNey3n8+sFS1JSIrkIz9Xn X-Google-Smtp-Source: APXvYqyVucOB6HBvBMDSJClUZYWCv4COq0hpOuM4c0wRZr1ggcHkv/ZjZSW5589uwIs0tNvBNYV41Q== X-Received: by 2002:a19:488f:: with SMTP id v137mr3100181lfa.26.1569596893643; Fri, 27 Sep 2019 08:08:13 -0700 (PDT) Received: from iseliverstov-pc.gridgain.local ([195.239.208.174]) by smtp.gmail.com with ESMTPSA id f21sm610065lfm.90.2019.09.27.08.08.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Sep 2019 08:08:12 -0700 (PDT) From: Seliverstov Igor Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: New SQL execution engine Date: Fri, 27 Sep 2019 18:08:11 +0300 References: <2051c3982ae9a6c474a19f47cde1e17c08053f64.camel@gmail.com> <0a9bae410574e5d6c245c674ef277128e58e80cd.camel@gmail.com> <7167ba80-39b7-092a-5e3c-bcec73db42a5@mail.ru> <2fb4868da7ced75107c01c3bdfc6379287c32a4c.camel@gmail.com> <45FA4A1B-D5A0-4165-9A7A-FD2F205485BD@gmail.com> <4bebc4b049212c7a5bd3c397340c64cfcede0f33.camel@gmail.com> To: dev@ignite.apache.org In-Reply-To: <4bebc4b049212c7a5bd3c397340c64cfcede0f33.camel@gmail.com> Message-Id: <040F78F5-6323-4E83-8819-DCCCBE382DB0@gmail.com> X-Mailer: Apple Mail (2.3445.104.11) Nikolay, At last we have better questions. There is no decision, here we should decide. Doing nothing isn=E2=80=99t a decision, it=E2=80=99s just doing nothing Spark Catalyst is a good example, but under the hood it has absolutely = the same idea, but adopted to Spark. Calcite is the same, but general. = That=E2=80=99s why it=E2=80=99s better start point. Implementing an engine from scratch is really cool, but looks like = inventing a bicycle, don=E2=80=99t think it makes sense. At least I = against this option. I added requirements to IEP (as you asked), you may see it=E2=80=99s in = DRAFT state and will be complemented by details. We have some thoughts on how to make smooth replacement, but at first we = should decide what to replace and what with. At now Calcite based engine is placed in different module, we checked it = can build execution graph for both local and distributed cases, it has = good expandability.=20 We talked to Calcite community to identify possible future issues and = everything points to the fact it=E2=80=99s the best option.=20 It=E2=80=99s possible to develop it as an experimental extension at = first (not a replacement) until we make sure that it works as expected. = This way there are no risks for anybody who uses Ignite on production = environment. Regards, Igor > 27 =D1=81=D0=B5=D0=BD=D1=82. 2019 =D0=B3., =D0=B2 17:25, Nikolay = Izhikov =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0= =B0): >=20 > Igor. >=20 >> The main issue - there is no *selection*. >=20 > 1. I don't remember community decision about this. >=20 > 2. We should avoid to make such long-term decision so quickly. > We done this kind of decision with H2 and come to the point when we = should review it. >=20 >> 1) Implementing white papers from scratch >> 2) Adopting Calcite to our needs. >=20 > The third option don't fix issues we have with H2. > The fourth option I know is using spark-catalyst. >=20 > What is wrong with writing engine from scratch? >=20 > I ask you to start with engine requirements. > Can we, please, discuss it? >=20 >> If you have an alternative - you're welcome, I'll gratefully listen = to you. >=20 > We have alternative for now - H2 based engine. >=20 >> The main question isn't "WHAT" but "HOW" - that's the discussion = topic from my point of view. >=20 > When we make a decision about engine we can discuss roadmap for = replacement. > One more time - replacement of SQL engine to some more customizable = make sense for me. > But, this kind of decisions need carefull discussion. >=20 > =D0=92 =D0=9F=D1=82, 27/09/2019 =D0=B2 17:08 +0300, Seliverstov Igor = =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> Nikolay, >>=20 >> The main issue - there is no *selection*. >>=20 >> There is a field of knowledge - relational algebra, which describes = how to transform relational expressions saving their semantics, and a = couple of implementations (Calcite is only one written in Java). >>=20 >> There are only two alternatives: >>=20 >> 1) Implementing white papers from scratch >> 2) Adopting Calcite to our needs. >>=20 >> The second way was chosen by several other projects, there is = experience, there is a list of known issues (like using indexes) so, = almost everything is already done for us. >>=20 >> Implementing a planner is a big deal, I think anybody understands it = there. That's why our proposal to reuse others experience is obvious. >>=20 >> If you have an alternative - you're welcome, I'll gratefully listen = to you. >>=20 >> The main question isn't "WHAT" but "HOW" - that's the discussion = topic from my point of view. >>=20 >> Regards, >> Igor >>=20 >>> 27 =D1=81=D0=B5=D0=BD=D1=82. 2019 =D0=B3., =D0=B2 16:37, Nikolay = Izhikov =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0= =B0): >>>=20 >>> Roman. >>>=20 >>>> Nikolay, Maxim, I understand that our arguments may not be as = obvious=20 >>>> for you as it obvious for SQL team. So, please arrange your = questions in=20 >>>> a more constructive way. >>>=20 >>> What is SQL team? >>> I only know Ignite community :) >>>=20 >>> Please, share you knowledge in IEP. >>> I want to join to the process of engine *selection*. >>> It should start with the requirements to such engine. >>> Can you write it in IEP, please? >>>=20 >>> My point is very simple: >>>=20 >>> 1. We made the wrong decision with H2 >>> 2. We should make a well-thought decision about the new engine. >>>=20 >>>> How many tickets would satisfy you? >>>=20 >>> You write about "issueS" with the H2. >>> All I see is one open ticket. >>> IEP doesn't provide enough information. >>> So it's not about the number of tickets, it's about >>>=20 >>>> These two points (single map-reduce execution and inflexible = optimizer)=20 >>>> are the main problems with the current engine. >>>=20 >>> We may come to the point when Calcite(or any other engine) brings us = third and other "main problems". >>> This is how it happens with H2. >>>=20 >>> Let's start from what we want to get with the engine and move = forward from this base. >>> What do you think? >>>=20 >>>=20 >>>=20 >>> =D0=92 =D0=9F=D1=82, 27/09/2019 =D0=B2 16:15 +0300, Roman Kondakov = =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >>>> Maxim, Nikolay, >>>>=20 >>>> I've listed two issues which show the ideological flaws of the = current=20 >>>> engine. >>>>=20 >>>> 1. IGNITE-11448 - Open. This ticket describes the impossibility of=20= >>>> executing queries which can not be fit in the hardcoded one pass=20 >>>> map-reduce paradigm. >>>>=20 >>>> 2. IGNITE-6085 - Closed (won't fix) - This ticket describes the = second=20 >>>> major problem with the current engine: H2 query optimizer is very=20= >>>> primitive and can not perform many useful optimizations. >>>>=20 >>>> These two points (single map-reduce execution and inflexible = optimizer)=20 >>>> are the main problems with the current engine. It means that our = engine=20 >>>> is currently suitable for execution only a very limited subset of = the=20 >>>> typical SQL queries. For example it can not even run most of the = TPC-H=20 >>>> benchmark queries because they don't fit to the simple map-reduce = paradigm. >>>>=20 >>>>> All I see is links to two tickets: >>>>=20 >>>> How many tickets would satisfy you? I named two. And it looks like = it is=20 >>>> not enough from your point of view. Ok, so how many is enough? The = set=20 >>>> of problems caused by listed above tickets is infinite, therefore I = can=20 >>>> not create a ticket for each of them. >>>>> Tech details also should be added. >>>>=20 >>>> Tech details are in the tickets. >>>>=20 >>>>> We can't discuss such a huge change as an execution engine = replacement with descrition like: >>>>> "No data co-location control, i.e. arbitrary data can be returned = silently" or >>>>> "Low control on how query executes internally, as a result we have = limited possibility to implement improvements/fixes." >>>>=20 >>>> Why not? Don't you understand these problems? Or you don't think = this is=20 >>>> a problem? >>>>=20 >>>>> Let's make these descriptions more specific. >>>>=20 >>>> What do you mean by "more specific"? What is the criteria of the=20 >>>> specific description? >>>>=20 >>>>=20 >>>>=20 >>>> Nikolay, Maxim, I understand that our arguments may not be as = obvious=20 >>>> for you as it obvious for SQL team. So, please arrange your = questions in=20 >>>> a more constructive way. >>>>=20 >>>> Thank you! >>=20 >>=20