From dev-return-29844-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Mon Jan 8 17:11:24 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id E5863180607 for ; Mon, 8 Jan 2018 17:11:24 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D4FCA160C2C; Mon, 8 Jan 2018 16:11:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 25FF0160C1E for ; Mon, 8 Jan 2018 17:11:23 +0100 (CET) Received: (qmail 95636 invoked by uid 500); 8 Jan 2018 16:11:23 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 95616 invoked by uid 99); 8 Jan 2018 16:11:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Jan 2018 16:11:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 7F4D8C0903 for ; Mon, 8 Jan 2018 16:11:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.121 X-Spam-Level: X-Spam-Status: No, score=-0.121 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id iw_B8-zan0s5 for ; Mon, 8 Jan 2018 16:11:21 +0000 (UTC) Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com [209.85.215.42]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 2FA4B5F306 for ; Mon, 8 Jan 2018 16:11:21 +0000 (UTC) Received: by mail-lf0-f42.google.com with SMTP id u84so12665398lff.7 for ; Mon, 08 Jan 2018 08:11:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:subject:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=ARFMYpCuw2jAxjwb5X9eA25eNsOXGJstV6Wb9zzBV3w=; b=mYnslxuW01A2kpUmmi+Utap0q3cBamGclKqI2NY3RJHvyYB3SC3vlHTiRNcG1hgOjg yg5isVZJ2bpF9FYMTZIw5vu+6qtfmS3PJj65uKUlRVaoSAU/zC8Qj7Udoalyp8FGcM5p Z5zPsxR0yC4a1nDgyZWfd4a+B5WugcKGqToKwSNWCtuMp7nohyQTuBEqe+vTtXZJpVJU PwAsiJHn5hOyTFrKVWfDyGE+FeSE1PxeUs0LM/OurSMg+50JU3fCwU9EYNJVuz0CzhKk FuzhW2SqLmsZO0n8uSOnd0rsLGldapNRvgtIjWqSS+Wf0SZk3UT4bAJJPQRNJvS3BG/n 4pbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:subject:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=ARFMYpCuw2jAxjwb5X9eA25eNsOXGJstV6Wb9zzBV3w=; b=JPV/ap6MgI3t+ciDx6qQS3DWYbSII2YH+mlX2ytDSp1S3cD5xd3JSMySmHiZ20x8ze egEqLf0yS1pxA0PM8MShIW8MLOFudB1xMm3SwU6l6hY8jYE7elDy2wf7rYZH7p/047tu 4eB9awnmH6iCtx3qbgcjKUpSYg3eJZrOIO9yfen3Fbj65724HeCkWLYdW7gXJ8JOFsjl OUj+tjYWGyoC2yed5w26vxnwclBeJdvaLJo5zic91qOYAsVCN7JysDg0xOychNnv9Pdq IZbowSZTQEZhiucffwIKSJmbExCPL3Mxa0uvMKhU1Z5yUMSzMOTpOWqLjm912kw8QaZn 1XNw== X-Gm-Message-State: AKGB3mKzfTYpqFoVAkLuxkoyn3+5ndt7yeTh6y5rBG934a2P9qUbQF7w r7T3gsztQW+y1E7qmCTp4wQ= X-Google-Smtp-Source: ACJfBotS2iBR5Zp8paIRmhmzNxi72fASv+yvTVYS9/YOdScBGb9YvO87VeJaiQmZgFS20PXbN7qwfw== X-Received: by 10.25.1.139 with SMTP id 133mr6410390lfb.131.1515427874531; Mon, 08 Jan 2018 08:11:14 -0800 (PST) Received: from newDragon ([91.210.95.4]) by smtp.googlemail.com with ESMTPSA id r65sm2188283lfr.52.2018.01.08.08.11.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 08 Jan 2018 08:11:13 -0800 (PST) From: Nikolay Izhikov X-Google-Original-From: Nikolay Izhikov Message-ID: <1515427872.2076.1.camel@gmail.com> Subject: Re: Spark data frames integration merged To: Denis Magda Cc: dev@ignite.apache.org Date: Mon, 08 Jan 2018 19:11:12 +0300 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.26.1-1 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Hello, Denis. > Nikolay, could you document the feature before the release [1]? Yes, I can. I will document these feature in a next few days. В Пт, 29/12/2017 в 13:37 -0800, Denis Magda пишет: > Great news, > > Thanks Nikolay and Val! > > Nikolay, could you document the feature before the release [1]? I’ve > granted you required permission. > > More on the doc process can be found here [2]. > > [1] https://issues.apache.org/jira/browse/IGNITE-7345 > [2] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Documen > t > > — > Denis > > > On Dec 29, 2017, at 1:22 PM, Valentin Kulichenko > ko@gmail.com> wrote: > > > > Igniters, > > > > Great news! We completed and merged first part of integration with > > Spark data frames [1]. It contains implementation of Spark data > > source which allows to use DataFrame API to query Ignite data, as > > well as join it with other data frames originated from different > > sources. > > > > Next planned steps are the following: > > - Implement custom execution strategy to avoid transferring data > > from Ignite to Spark when possible [2]. This should give serious > > performance improvement in cases when only Ignite tables > > participate in a query. > > - Implement ability to save a data frame into Ignite via > > DataFrameWrite API [3]. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-3084 > > [2] https://issues.apache.org/jira/browse/IGNITE-7077 > > [3] https://issues.apache.org/jira/browse/IGNITE-7337 > > > > Nikolay Izhikov, thanks for the contribution and for all the hard > > work! > > > > -Val > >