From dev-return-47711-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Sep 27 14:22:20 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 65B9F180638 for ; Fri, 27 Sep 2019 16:22:20 +0200 (CEST) Received: (qmail 57886 invoked by uid 500); 27 Sep 2019 14:22:19 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 57864 invoked by uid 99); 27 Sep 2019 14:22:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Sep 2019 14:22:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D14EC1A4663 for ; Fri, 27 Sep 2019 14:22:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.148 X-Spam-Level: X-Spam-Status: No, score=0.148 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Aq8U4vAyXQ-u for ; Fri, 27 Sep 2019 14:22:14 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.194; helo=mail-lj1-f194.google.com; envelope-from=nizhikov.dev@gmail.com; receiver= Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id D9B10BC563 for ; Fri, 27 Sep 2019 14:22:13 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id d1so2655488ljl.13 for ; Fri, 27 Sep 2019 07:22:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:message-id:subject:to:date:in-reply-to:references :mime-version; bh=0o8vY6hOgXU8Q0qL372aAOiO0Q3Xhtt6+2QZDHvcezc=; b=AsZAqnCD8mHHiDP3uVZ4cTfD533PgEiSIZJ7tNHg2s12hWE2e3SMHehrrNS8VsoSIg wDOIsZQUtkeFCGRhvTDympKBgBneOuAK0wdvgULRsNWm0awiditQsJvyp9uGnMJ6MFZ5 HAnf1cNbNmPZldmKzk+JwXjEkrQMcVMTv5jdAFxzIgRNn8uy1bcaX7FvYK26JRc8rW6u MeyV7Q2QCn8Zx9X3VfvPgY5jEsgFw2Y64/BhOYwQ6ULYqkGg6yh8+zMCi8S7iqchR/Cd FK/inwdtv3mBMx1MJP1CZrsYEZDpeGZSUzex8h2uZ2kachkgWB/jV0qPzzkH7qnhJxN2 7hWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:message-id:subject:to:date :in-reply-to:references:mime-version; bh=0o8vY6hOgXU8Q0qL372aAOiO0Q3Xhtt6+2QZDHvcezc=; b=B/vmy2rRf610MQAYlpmijM60ZqGjHMMD4Tfebw42UU13tA8xlb4S9POZyjYZcbB1ub tX37rOulDMTKNoCg/hYNnB9mT0BGSPbVp64TqjgXsi1usb0WhL/rlCxw4aaO7lbHeOkN U2/MSCh7K1FIQgDOeCFw9q9eY7Ko/QSA7nRjvbyeHdqxkwqHF0oQapGzSVvLSKc1sekj Nd2XR8fXhCKp+Qzd/PVqBgBpv+xcAd/isosQhAYCJqGjFH2UebK7w9kYRNsJY6zKUNQi nnNOW7WcIS8fWW5EkUoDcaLZZeRPh31cL21dB17FoivrXZNnmNP7FJAPLZ+PZDlJU/Ty qg7Q== X-Gm-Message-State: APjAAAUZw72u2tt3EAFBiuU+A213GWzEEjzzryXZgO0NWW+lh69zzjDk 5CbTit8r6Rx7MOXIm0UoIOwa3aVI X-Google-Smtp-Source: APXvYqxqm0vSexyWSNxg4Egn+AO26F5vISFEDpzj8IyANGmeGdLCGM2hFJvGezWb+7ZJ86h305Ktdw== X-Received: by 2002:a2e:89c9:: with SMTP id c9mr3232090ljk.64.1569594132618; Fri, 27 Sep 2019 07:22:12 -0700 (PDT) Received: from newDragon ([194.186.207.214]) by smtp.googlemail.com with ESMTPSA id h5sm500520ljf.83.2019.09.27.07.22.11 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 27 Sep 2019 07:22:11 -0700 (PDT) Sender: =?UTF-8?B?0J3QuNC60L7Qu9Cw0Lkg0JjQttC40LrQvtCy?= From: Nikolay Izhikov X-Google-Original-From: Nikolay Izhikov Message-ID: <4bebc4b049212c7a5bd3c397340c64cfcede0f33.camel@gmail.com> Subject: Re: New SQL execution engine To: dev@ignite.apache.org Date: Fri, 27 Sep 2019 17:25:37 +0300 In-Reply-To: <45FA4A1B-D5A0-4165-9A7A-FD2F205485BD@gmail.com> References: <2051c3982ae9a6c474a19f47cde1e17c08053f64.camel@gmail.com> <0a9bae410574e5d6c245c674ef277128e58e80cd.camel@gmail.com> <7167ba80-39b7-092a-5e3c-bcec73db42a5@mail.ru> <2fb4868da7ced75107c01c3bdfc6379287c32a4c.camel@gmail.com> <45FA4A1B-D5A0-4165-9A7A-FD2F205485BD@gmail.com> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-zDFnPi7g8bLICfKdAnix" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.1 Mime-Version: 1.0 --=-zDFnPi7g8bLICfKdAnix Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Igor. > The main issue - there is no *selection*. 1. I don't remember community decision about this. 2. We should avoid to make such long-term decision so quickly. We done this kind of decision with H2 and come to the point when we should = review it. > 1) Implementing white papers from scratch > 2) Adopting Calcite to our needs. The third option don't fix issues we have with H2. The fourth option I know is using spark-catalyst. What is wrong with writing engine from scratch? I ask you to start with engine requirements. Can we, please, discuss it? > If you have an alternative - you're welcome, I'll gratefully listen to yo= u. We have alternative for now - H2 based engine. > The main question isn't "WHAT" but "HOW" - that's the discussion topic fr= om my point of view. When we make a decision about engine we can discuss roadmap for replacement= . One more time - replacement of SQL engine to some more customizable make se= nse for me. But, this kind of decisions need carefull discussion. =D0=92 =D0=9F=D1=82, 27/09/2019 =D0=B2 17:08 +0300, Seliverstov Igor =D0=BF= =D0=B8=D1=88=D0=B5=D1=82: > Nikolay, >=20 > The main issue - there is no *selection*. >=20 > There is a field of knowledge - relational algebra, which describes how t= o transform relational expressions saving their semantics, and a couple of = implementations (Calcite is only one written in Java). >=20 > There are only two alternatives: >=20 > 1) Implementing white papers from scratch > 2) Adopting Calcite to our needs. >=20 > The second way was chosen by several other projects, there is experience,= there is a list of known issues (like using indexes) so, almost everything= is already done for us. >=20 > Implementing a planner is a big deal, I think anybody understands it ther= e. That's why our proposal to reuse others experience is obvious. >=20 > If you have an alternative - you're welcome, I'll gratefully listen to yo= u. >=20 > The main question isn't "WHAT" but "HOW" - that's the discussion topic fr= om my point of view. >=20 > Regards, > Igor >=20 > > 27 =D1=81=D0=B5=D0=BD=D1=82. 2019 =D0=B3., =D0=B2 16:37, Nikolay Izhiko= v =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0): > >=20 > > Roman. > >=20 > > > Nikolay, Maxim, I understand that our arguments may not be as obvious= =20 > > > for you as it obvious for SQL team. So, please arrange your questions= in=20 > > > a more constructive way. > >=20 > > What is SQL team? > > I only know Ignite community :) > >=20 > > Please, share you knowledge in IEP. > > I want to join to the process of engine *selection*. > > It should start with the requirements to such engine. > > Can you write it in IEP, please? > >=20 > > My point is very simple: > >=20 > > 1. We made the wrong decision with H2 > > 2. We should make a well-thought decision about the new engine. > >=20 > > > How many tickets would satisfy you? > >=20 > > You write about "issueS" with the H2. > > All I see is one open ticket. > > IEP doesn't provide enough information. > > So it's not about the number of tickets, it's about > >=20 > > > These two points (single map-reduce execution and inflexible optimize= r)=20 > > > are the main problems with the current engine. > >=20 > > We may come to the point when Calcite(or any other engine) brings us th= ird and other "main problems". > > This is how it happens with H2. > >=20 > > Let's start from what we want to get with the engine and move forward f= rom this base. > > What do you think? > >=20 > >=20 > >=20 > > =D0=92 =D0=9F=D1=82, 27/09/2019 =D0=B2 16:15 +0300, Roman Kondakov =D0= =BF=D0=B8=D1=88=D0=B5=D1=82: > > > Maxim, Nikolay, > > >=20 > > > I've listed two issues which show the ideological flaws of the curren= t=20 > > > engine. > > >=20 > > > 1. IGNITE-11448 - Open. This ticket describes the impossibility of= =20 > > > executing queries which can not be fit in the hardcoded one pass=20 > > > map-reduce paradigm. > > >=20 > > > 2. IGNITE-6085 - Closed (won't fix) - This ticket describes the secon= d=20 > > > major problem with the current engine: H2 query optimizer is very=20 > > > primitive and can not perform many useful optimizations. > > >=20 > > > These two points (single map-reduce execution and inflexible optimize= r)=20 > > > are the main problems with the current engine. It means that our engi= ne=20 > > > is currently suitable for execution only a very limited subset of th= e=20 > > > typical SQL queries. For example it can not even run most of the TPC-= H=20 > > > benchmark queries because they don't fit to the simple map-reduce par= adigm. > > >=20 > > > > All I see is links to two tickets: > > >=20 > > > How many tickets would satisfy you? I named two. And it looks like it= is=20 > > > not enough from your point of view. Ok, so how many is enough? The se= t=20 > > > of problems caused by listed above tickets is infinite, therefore I c= an=20 > > > not create a ticket for each of them. > > > > Tech details also should be added. > > >=20 > > > Tech details are in the tickets. > > >=20 > > > > We can't discuss such a huge change as an execution engine replacem= ent with descrition like: > > > > "No data co-location control, i.e. arbitrary data can be returned s= ilently" or > > > > "Low control on how query executes internally, as a result we have = limited possibility to implement improvements/fixes." > > >=20 > > > Why not? Don't you understand these problems? Or you don't think this= is=20 > > > a problem? > > >=20 > > > > Let's make these descriptions more specific. > > >=20 > > > What do you mean by "more specific"? What is the criteria of the=20 > > > specific description? > > >=20 > > >=20 > > >=20 > > > Nikolay, Maxim, I understand that our arguments may not be as obvious= =20 > > > for you as it obvious for SQL team. So, please arrange your questions= in=20 > > > a more constructive way. > > >=20 > > > Thank you! >=20 >=20 --=-zDFnPi7g8bLICfKdAnix Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQEzBAABCgAdFiEEOiTcLcdgyP2exB5ZbiaPbjg91GUFAl2OG+EACgkQbiaPbjg9 1GUgeAf+LCovllpLlXqMVZKEqF1aMdci87f4eBRxOxZNNqwqfYfvrjAPL5TO1aW3 jBuYQY7EwxUm8h3s+ycoADg2fOu/vcXDaNcYeWWuDUEJYY9CZ3lAEBNUaFn2veee s5BFJq2vHvv0a4wMPjLG3BpmaLrF9cRgGYXEaZj4IaJL+h1lKE7OHbegQqXJDodH 3aSylq4PDGmosst/AXFz9iSpFyrNo/0PLsTr269y2k8H/1Ych4m5CWJM/68g72cX KU8JFewf2uCRvGY3P26l7uS5Wpj6v2GQN1d7U/mBRQ1fw9EOSCdy1VO0e0sk8UHs Ab64nFrL9O3y6yTeMbiyuBTGaXMOrA== =nTW4 -----END PGP SIGNATURE----- --=-zDFnPi7g8bLICfKdAnix--