From dev-return-10827-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Mon Mar 4 17:45:24 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8C768180627 for ; Mon, 4 Mar 2019 18:45:23 +0100 (CET) Received: (qmail 95881 invoked by uid 500); 4 Mar 2019 17:45:17 -0000 Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@arrow.apache.org Delivered-To: mailing list dev@arrow.apache.org Received: (qmail 95869 invoked by uid 99); 4 Mar 2019 17:45:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Mar 2019 17:45:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 71C3FCA952 for ; Mon, 4 Mar 2019 17:45:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.8 X-Spam-Level: * X-Spam-Status: No, score=1.8 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Ysxy_oWxky-m for ; Mon, 4 Mar 2019 17:45:15 +0000 (UTC) Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 857B85F562 for ; Mon, 4 Mar 2019 17:45:14 +0000 (UTC) Received: by mail-lj1-f170.google.com with SMTP id a17so5077922ljd.4 for ; Mon, 04 Mar 2019 09:45:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=wHQTQNMWgDNUgc8ieL6gxUDEyVrFiUNHKMOEsj24qQU=; b=iBmO64hV3+jUlBE0XF2sMPQHudXnsRO8R8WVFFpYoxFnvXgDj6HV2+uswsxtz8LeXY RbxmKlnMT+pH9Jo12viH+grwT9LUTFbksC4DltLSQTVmk5Wkomym+Q8OIz028Nh0kOmT bA6m8Lh4gvIXlbuVJC6ZsXTnWaB4oXpD5IB97+iikXXrmUHAuk+PcVT96rmstilFdoOe QIsrpjQ8XFWX1kBH//fg34wSLosn0B20BoYt1LVBs33FE2jm4iVmvSqWq3g4fjWfM8JS t6ZltHGbAnpIVcJ9Y/vATgCSzCkazytO/pqVQ7TA9CBGDE4Eyl/Krog9ocVOISjY2QrS Csew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=wHQTQNMWgDNUgc8ieL6gxUDEyVrFiUNHKMOEsj24qQU=; b=OcYP7lB5/LxPxqT30+hQq1IGCbBXd+x8eYw6iguPo6iTB0sXL3GfCNeH5c9OpP6+u4 hUH+txxhOAfMP3zu4JWeCTa3h1gXnrSOBjjRlvueksrg1TWfElh2dwfFytg3Md1MGIe0 SvymRaZa636B63KUfETdMREjm8564z7QODNYJKMt8Fjyy0Ki2gLk4SHkH6I9xGbDCuzs qt1mwy+XCFyp6oEn/L8SCXzR/ASmVpznAZ3P4uqTlaOJDUitrqk9n8bWdYEdzGBJTjTE TQDLWFqo68QLtN/qRBf4PYAA7sc2//U7uDzVFVkqH6YmHMvFTlpYiTfZ81+z5Wceqivn kDFQ== X-Gm-Message-State: APjAAAWHn9UW1MMXL5fLrjugUhH4KGei/hqM63XMRsJKsd8yN6Ee+Rbs 7YiOR5K9jkLiQA6lxVG43+N0voIo+lgwKmv/RfyJag== X-Google-Smtp-Source: APXvYqzzfOz1+yOqOI18PLyxr/s2eK4bhihIttwGT5TNaf86t0MDI15QFouNRjQcH33hg06V/UV8ivCoK4lMe3c6lb4= X-Received: by 2002:a2e:9ac8:: with SMTP id p8mr12148647ljj.142.1551721513005; Mon, 04 Mar 2019 09:45:13 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Neville Dipale Date: Mon, 4 Mar 2019 19:44:59 +0200 Message-ID: Subject: Re: [Rust] [DataFusion] Preferences on futures / threading crates? To: dev@arrow.apache.org Content-Type: multipart/alternative; boundary="00000000000092b71905834851fd" --00000000000092b71905834851fd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'm a fan of using Rayon. Perhaps if it's not too much work, we could compare the two On Mon, 4 Mar 2019 at 15:04, Kriszti=C3=A1n Sz=C5=B1cs wrote: > On Mon, Mar 4, 2019 at 5:55 AM Andy Grove wrote: > > > I have been working on a PoC of parallel query execution and it is > working > > well, and I am now starting to create PRs for the various refactors > > necessary for this in DataFusion. > > > > I haven't been following the async/await and futures/tokio developments > > lately but for the PoC I used tokio-threadpool which seems simple to us= e. > > > > I just wanted to give everyone a chance to give their thoughts on this > > before I get too far with my batch of PRs. Is anyone opposed to using > > tokio-threadpool? > > > DataFusion's tasks should be CPU bound and according to tokio-threadpool'= s > documentation [1], it is more suitable for event loops: > "It is optimized for the primary Tokio use case of many independent tasks > with limited computation and with most tasks waiting on I/O." > > Rayon seems to follow different semantics, but depending on futures-rs is > considerable, especially because it is maintained by the rust lang nurser= y. > > [1] https://docs.rs/tokio-threadpool/0.1.12/tokio_threadpool/ > > Cheers, Krisztian > --00000000000092b71905834851fd--