From dev-return-40776-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Sat Oct 20 23:09:13 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 30D92180668 for ; Sat, 20 Oct 2018 23:09:13 +0200 (CEST) Received: (qmail 41722 invoked by uid 500); 20 Oct 2018 21:09:12 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 41710 invoked by uid 99); 20 Oct 2018 21:09:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Oct 2018 21:09:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0FEB4C0368 for ; Sat, 20 Oct 2018 21:09:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.01 X-Spam-Level: X-Spam-Status: No, score=-0.01 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id G_5glVZMtI2i for ; Sat, 20 Oct 2018 21:09:10 +0000 (UTC) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 771C55F10F for ; Sat, 20 Oct 2018 21:09:09 +0000 (UTC) Received: by mail-lj1-f178.google.com with SMTP id v6-v6so33668147ljc.11 for ; Sat, 20 Oct 2018 14:09:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:message-id:subject:to:date:mime-version; bh=ktYtk/JYHQnpbAqVpNmj+GN1c002jIITmB6htYi0/Ik=; b=bvZ4stjxw1pvyEuJj5bYde1msBH9NCVOE8j7LCYI/o09RXbqcg15YrroHsFjhwjve+ dLpR2UrVkho173g1oo0mTuF6fP3WGV1vUUtBNQCmIzrZPa2TEBZNf5nCd1m/GPRv2MIE Xm0CSEwTppFA5XnPmgvau4XJftMmAzAH8u8iN6DzJdGx8isF8ZUoPenUyb/qazBlI1jM N9HVdVz7PRgu77KracyKcjB13S4NAkjsk2lesaW0zE/hGW37u8213d9UM2pfT7IXjD6W y/yZgKudrYoVwnHoeEMv+qRd7G0cE7e/Y3ck+BAiEdVjqMixMV97UgJ+fNa800d3FmOI HYtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:message-id:subject:to:date :mime-version; bh=ktYtk/JYHQnpbAqVpNmj+GN1c002jIITmB6htYi0/Ik=; b=gsP2vrTixUATrbsoFfjvrbUhmB/vW/eSnt1JxQVfRkhEsNuXzLI+OJXxHtHTaZ/z5F r3jnXrwXBx6KPICBWPqpgIoEvfqvjDW/iDV8B1FJ761rxDh6h5j5ClPOureh+QJAJH8W /hK6gbixCcg+wq60Upstps7kzk2+TRyXdabHB0xHeUWLBvz6ITYGUkZx/5PwjcjfQaqz 8XBFGfhD+y4Xk59aPCIQx6UX6Uav6XPLTLVmlbYYiruG8uz1kxoC3Y+ddoBOnKEF1mxV /sNTCyFN3a8Qza0rKCJF4wAc4hfPp3AWw/dx7qtko2QYXbvYdpnJ7yVVhYGGy51yyz5M s3Nw== X-Gm-Message-State: ABuFfogw3DSGt2w6Rj5cOoIyYKd5IavgpscRsNu9MYz7yj9OoAhSeCkU wJJ0Cq16yJ56PedZ+tNeCMOyI1ia X-Google-Smtp-Source: ACcGV63gwHpQsbkbA22I4YtrxyBsc+5s5CLMvHGzoJasTvd6ZGxPY7FjMKOTbRXisEYHJObpooFS6g== X-Received: by 2002:a2e:3e0b:: with SMTP id l11-v6mr28224636lja.16.1540069748643; Sat, 20 Oct 2018 14:09:08 -0700 (PDT) Received: from newDragon ([213.87.152.196]) by smtp.googlemail.com with ESMTPSA id h21-v6sm1033537lfh.38.2018.10.20.14.09.06 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 20 Oct 2018 14:09:07 -0700 (PDT) Sender: =?UTF-8?B?0J3QuNC60L7Qu9Cw0Lkg0JjQttC40LrQvtCy?= From: Nikolay Izhikov X-Google-Original-From: Nikolay Izhikov Message-ID: <20cc20cec397bf94918644310df198df8dcebdbc.camel@gmail.com> Subject: [DISCUSSION] Spark Data Frame through Thin Client To: dev@ignite.apache.org Date: Sun, 21 Oct 2018 00:09:13 +0300 Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-oPBNKRi3PLAybKuF8+pN" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.1 Mime-Version: 1.0 --=-oPBNKRi3PLAybKuF8+pN Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, Igniters. Currently, Spark Data Frame integration implemented via client node connect= ion. Whenever we need to retrieve some data into Spark worker(or master) from Ig= nite we start a client node. It has several major disadvantages: 1. We should copy whole Ignite distribution on to each Spark worker [1] 2. We should copy whole Ignite distribution on to Spark master to get cata= logue works. 3. We should have the same absolute path to Ignite configuration file on e= very worker and provide it during data frame construction [2] 4. We should additionally configure Spark workerks classpath to include Ig= nite libraries. For now, almost all operation we need to do in Spark Data Frame integration= is supported by Java Thin Client. * obtain the list of caches. * get cache configuration. * execute SQL query. * stream data to the table - don't support by the thin client for now, but= can be implemented using simple SQL INSERT statements. Advantages of usage Java Thin Client in Spark integration(they all known fr= om Java Thin Client advantages): 1. Easy to configure: only IP addresses of server nodes are required. 2. Easy to deploy: only 1 additional jar required. No server side(Ignite w= orker) configuration required. I propose to implement Spark Data Frame integration through Java Thin Clien= t. Thoughts? [1] https://apacheignite-fs.readme.io/docs/installation-deployment [2] https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-ignite= -dataframe-options --=-oPBNKRi3PLAybKuF8+pN Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQEzBAABCgAdFiEEOiTcLcdgyP2exB5ZbiaPbjg91GUFAlvLmXkACgkQbiaPbjg9 1GVPIggAoeBdOHMDjWZM/SnIKYybpc6JOSjMzb7Iy0PMLVfEUqMGnYn4pOriYQTz DsRv16ZbGRE9lWi4V4Fu9CMxQ7z+8XdGTcMQe73V6mmY81rkykTWASFfbu4hCQs/ D0gg9PhYcO2yHgRAVToj+wk3v2fU4def6UJJ6jI9lMHk3elYiOBxjHq8d/eqc+G6 7Q7nZldp2aNlElt9hdB9fVzT10po9iVYNB6HnckI4MKwm4z2BGEKH4LFPkV9PtRj dcWVii5x75ScBYowKg/paVy/KkcdT1VHsLJT8GIWfwcLY4tbOw887D6+ctEP/mWm AYUoZS/ZpMlkcASPMjO63IZankkl5A== =9ANs -----END PGP SIGNATURE----- --=-oPBNKRi3PLAybKuF8+pN--